Wikipedia’s Ways of Knowing – Part 2

[social_share/]



In the first part of this two-part piece, I discussed arborescent (vertical, discrete, hierarchical) and rhizomatic (horizontal, overlapping, interconnected) ways of acquiring and classifying knowledge, as well as the convergence of the arbor and the rhizome in modern knowledge systems. In this part, I discuss how this applies to Wikipedia.

Most of us use the Web rhizomatically: we enter a search term in Google or Wikipedia, look at the search results, and follow the links, whether to other Wikipedia pages or other online or offline resources. As I said in the previous post, this lets us explore pathways that interest us most, and may also lead to more engaged learning. But this approach can be both over-inclusive, requiring us to sift through a lot of information before getting to what we need the most, and under-inclusive, in that we can miss relevant material that requires different search terms than the ones we used for searching. Fortunately, Wikipedia provides a number of ways for accessing knowledge that makes it a more powerful encyclopedia than it would be if you had to rely on the search engine alone.

The first of these is Portals, which in turn direct you to Categories. A teaser is found on Wikipedia’s home page, which has links to eight named portals as well as a link to all of them (see linked screen shot below).

Top bar on Wikipedia's home page with link to major categories.

The Biography portal reveals that biographies are categorized in various ways (e.g., by association, ethnicity, gender, nationality or occupation). Thus, using this method, you could find, for example, a linked list of all Wikipedia biographies on signatories of important documents (an association category), which in turn leads to a subcategory of Signatories of declarations of independence. The most famous of these is the United States Declaration of Independence, and a further subcategory links to the Wikipedia articles on all 56 signatories of that document. These articles are also listed in the Wikipedia page Signing of the United States Declaration of Independence, showing that there is often more than one path to an information source on Wikipedia.

There are currently 1491 portals on English Wikipedia, with one of these portals being an alphabetical index to all 5.4 million Wikipedia articles from Aa (you might be surprised to know how many rivers are named “Aa”) to ZZ (a scale used in model railroading). When it comes to categories, there is also an alphabetical index to all categories, though given the many thousands of categories, it may be more useful to access the topical category index, which organizes categories under 12 broad headings and provides a separate search engine just for categories (see linked screen shot below).

Wikipedia's Contents - Categories

I tested this by searching for a highly topical subject—cybercrime. The results show that this is a named category (https://en.wikipedia.org/wiki/Category:Cybercrime, the standard syntax for categories in Wikipedia). The results list a number of other related categories, such as Cyberwarfare and Cybercrime by country, several of which may be relevant to writing or editing a Wikipedia article. The Cybercrime category in turn currently has 20 subcategories and 123 pages, and the subcategories also have subcategories and pages of their own. Using this approach, I quickly found, for example, the recently created article on the WannaCry cyberattack, reported to have affected more than 230,000 computers in over 150 countries in May 2017.

The above approach proceeds from the general to the particular in a branching or arborescent fashion (though I could have started at an even higher level of generality—Crime by type). The point I want to make, though, is that you can also discover this structure by going from the particular to the general. For example, if I use Wikipedia’s basic search engine and type in “WannaCry,” I get the article on the WannaCry cyberattack; then, scrolling to the bottom of the article, I find lots of related information, including links to six portals; a template for the category Hacking in the 2010s, showing a timeline with major incidents, groups and vulnerabilities; and links to several other related categories, from “2017 in computer science” to “Ransomware.” From here, I can switch modes and proceed rhizomatically to other topics, while still being able to see the overall structures within which they are organized. In other words, Wikipedia lends itself to a high degree of convergence of both arbor and rhizome.

Image of the links to the major incidence of hacking in the 2010s

Wikipedia provides other methods of organization besides portals and categories, including third-party systems such as the Library of Congress Classification (LCC) system that I mused about previously (see below).

Letter Subject area
A General Works
B Philosophy, Psychology, and Religion
C Auxiliary Sciences of History
D General and Old World History
E History of America
F History of the United States and British, Dutch, French, and Latin America
G Geography, Anthropology, and Recreation
H Social Sciences
J Political Science
K Law
L Education
M Music
N Fine Arts
P Language and Literature
Q Science
R Medicine
S Agriculture
T Technology
U Military Science
V Naval Science
Z Bibliography, Library Science, and General Information Resources

While the LCC system has been criticized as being more a guide to the books in a library’s collection than a classification of the world’s knowledge, it is well developed and in use by many academic and research libraries. Its 21 classes are further subdivided into a large number of two- or three-letter subclasses that are listed on the LCC page, with, in many cases, links to their own pages. For example, subclass AE – Encyclopedias, mentioned in my last post, has its own Wikipedia entry. Many of these are important subject articles in their own right. For example, RB – Pathology, a subclass of class R – Medicine, links to the Wikipedia article on Pathology, considered of high importance to WikiProject Medicine and currently rated B-class on the project’s quality scale. (On these rating systems, see my blog posts from February 14 and February 28.) Thus for a researcher already familiar with the LCC system (or a component of it such as, in my case, subclass K – Law), Wikipedia offers a way to translate that familiarity and move easily from browsing a library shelf to browsing an online encyclopedia.

In my next post, I’ll bring together a number of concepts that I’ve been writing about. Specifically, I’ll tell you how you can work with Portals, WikiProjects, and the Wiki Ed program to create a course that incorporates Wikipedia assignments and that uses the Wiki Ed dashboard to keep track of student work.


John Kleefeld is an associate professor at the College of Law and a 2017 teaching fellow at the Gwenna Moss Centre for Teaching Effectiveness, where he is coordinating a campus-wide project on integrating Wikipedia assignments into course materials. Portions of this blog series are from an article that he and a former law student wrote about using a Wikipedia assignment for class credit. See J. Kleefeld and K. Rattray, 2016. “Write a Wikipedia Article for Law School Credit—Really?” Journal of Legal Education, 65:3, 597-621.

Wikipedia’s Ways of Knowing – Part 1

[social_share/]



In my previous post, I characterized the subject categories in the Requested articles page as idiosyncratic and mused that they might be better based on the Library of Congress Classification system. As it happens, Wikipedia does map some of its articles (pages) into the LCC system, and also provides several other methods of organizing knowledge. Some of these are well known, some less so. I want to discuss them because I think that instructors and students alike should be familiar with ways of finding knowledge beyond today’s default method of keyword searching. First, though, I want to talk about two approaches to knowing or learning, which philosophers Gilles Deleuze and Félix Guattari termed arborescent and rhizomatic in their 1980 book, A Thousand Plateaus. I will then consider how these approaches are converging in web-based retrieval systems, including Wikipedia.

The Arbor and the Rhizome

The word arborescent is Latinate for treelike. Tree diagrams, also called dendograms (Greek: dendro, tree + gramma, drawing) have depicted genealogical relationships (“family trees”) since medieval times; the “tree of life” or variants on it have been used at least since Carl Linnaeus classified relationships among organisms; and the ubiquitous “org chart” is a kind of inverted tree. Arborescent ways of knowing are said to emphasize totalizing principles (the notion that universal facts can be discovered and classified), binary opposition (e.g., male-female; predator-prey), and relationships characterized by discrete branching hierarchies rather than horizontal interconnections.

The rhizome is also a botanical metaphor. A rhizome (Greek: “mass of roots”) is a stemlike root lying on or just below the soil surface and having the ability to send roots and shoots from its nodes. Think of ginger, ginseng, and many grasses. The concept allows for multiple, non-hierarchical entry and exit points, with one thing potentially networked with many other things, sometimes in unpredictable ways. Ecosystems, power grids, and Web 2.0 are quintessentially rhizomatic systems. And Wikipedia is a quintessentially a Web 2.0 phenomenon, in allowing users to collaboratively create knowledge with multiple jumping-in and jumping-off points based on hyperlinked text and images.

Having identified the differences, I don’t want to convey the impression that rhizomatic ways of knowing are either entirely new or necessarily superior to arborescent ways.

As to newness, consider the dictionary, which, even before the 1755 publication of Samuel Johnston’s opus, came to be organized alphabetically rather than topically. This form of organization, which must at first have seemed arbitrary compared to its forebears, turns out to be surprisingly rhizomatic: I can open a dictionary at any page, read a word’s definition, and be led to another word and its definition through an italicized or bolded word in the definition. The idea is so powerful that it stuck: with today’s dictionary apps, you jump to the new word by clicking on it instead of turning pages. Consider also the polymath—a distinctly rhizomatous type of scholar. An early example was Aristotle, whose knowledge spanned physics, metaphysics, poetry, theatre, music, logic, rhetoric, politics, ethics, and biology. In a similar vein, Hildegard of Bingen was an abbess, writer, composer, philosopher, musician, mystic and medical writer. Leonardo da Vinci not only created the world’s most famous painting but is also credited with inventing the parachute, helicopter and tank. Erasmus Darwin, physician, poet and propounder of the evolutionary theory more rigorously developed by his grandson, penned a 4,400-line treatise that straddled all branches of science and technology and used a whimsical rhyming couplet structure footnoted with his scientific observations. Though polymaths are at risk of becoming an endangered species, recent interest in interdisciplinary studies may yet save them and their rhizomatic tendencies.

As to superiority, rhizomatic approaches let people access material in ways that are most intuitive for them and explore the links and pathways that interest them most. But while this can lead to more engaged learning, it can also be inefficient and leave knowledge gaps that more structured approaches would avoid. In other words, it can be both over- and under-inclusive.

I’ll illustrate this with an example from my own discipline, law—specifically, the “implied undertaking” rule in litigation.[1] Litigating parties have “discovery rights,” an aspect of which is that they can get copies of each other’s documents that are relevant to the litigation. In Canada, such rights are generally accompanied by an implied undertaking not to use the documents for a purpose outside the litigation (also framed as “collateral or ulterior” to the litigation). Suppose, for example, a dismissed employee sues her employer, a company, for wrongful dismissal. Through the discovery process, she obtains a report that one of the company’s officials made to a government ministry, with comments that criticize the former employee’s ethical conduct. She then starts a new lawsuit against the company official for defamation. If the lawsuit is based solely on the report obtained in the wrongful dismissal action, it will likely constitute a breach of the implied undertaking rule and be unsuccessful.[2] I say “likely” because the rule’s scope is subject to interpretation, may be overlaid with other court rules specific to a particular province, and may be subject to exceptions or court procedures in which a party can seek to be relieved of the undertaking.

Suppose I want to learn more about the rule. One way is to enter “implied undertaking” into a search tool like CanLII’s public-domain search engine or Westlaw’s proprietary product (available free with a U of S library account). This approach generates useful information—in the form of hundreds of case reports decided by judges. On skimming the cases, it becomes apparent that they are weighted according to relevance, which, in a mechanistic attempt at modelling human reasoning, means that they are ordered by the number of times the term “implied undertaking” comes up. This is neither a reliable nor an efficient way to learn about the scope of the rule or its exceptions, and even if I apply filters (for example, isolating the search to cases only in a certain province, or only at appeal-level courts), I am still likely to have to sort through dozens—perhaps hundreds—of cases. Even then, I will likely miss a lot of material because other terms are also used to refer to the concept, such as “deemed undertaking” or “undertaking as to confidentiality.” Isn’t there a better way to get an overview of the subject? Yes, and it’s arborescent.

That way is an encyclopedia article, in this case, from the Canadian Encyclopedic Digest (CED), another Westlaw product. One of the ways of accessing its material is through an alphabetically ordered, branching, hyperlinked table of contents. This requires some thinking about how the subject matter might be organized. After a bit of trial and error, the title “Discovery” seems an option. It has two headings, one for Ontario and one for the Western provinces. Scanning through the topics under either of these headings leads to the subheading for the article on the rule, as can be seen below (omitted subjects marked with an ellipsis).

Canadian Encyclopedic Digest

… D …

Discovery (Ontario)

Discovery (Western)

VII — Use of Discovery Evidence

1 — General

2 — Implied or Deemed Undertaking

Clicking on the subheading pulls up an 11-paragraph article that succinctly outlines the rule and how it works, footnoted with a couple of dozen cases for those who want to drill down to get more information. This still leaves me with a fair bit of reading to do, but not nearly as much as by following the purely rhizomatic approach.

Systems Convergence

In fact, the story I have just told is oversimplified, because it doesn’t explain how in modern knowledge systems, rhizomatic and arborescent approaches are evolving towards convergence. For example, the “Implied or Deemed Undertaking” article in the CED also has a paragraph that links to headings in the Canadian Abridgment Digests, which cover cases on the rule from across Canada in digest (condensed) form. Further, even if I had only used my simple keyword search, “implied undertaking,” I could have arrived at the digest list through the first case on my search results by clicking on a sidebar headed “Related Resources.”[3] Unfortunately, this list does not link back to the CED article on the topic, indicating that convergence is still a work in progress. But armed with the new information of a subject entitled “implied or deemed undertaking,” I can use that as a phrase in the CED search engine to find the encyclopedic article of the same name.

Other systems work in similar ways, with varying degrees of convergence. An example of a highly convergent system—one that allows for multiple searching approaches that are cross-linked to each other—is the Library of Congress Online Catalog. Its main web page allows you to choose from three searching methods: Browse, Advanced Search, and Keyword Search. Let’s suppose I want to look for books about collaboration in Wikipedia. I might start with something as simple as a Browse for subjects containing “Wikipedia” (see below).

Wikipedia Browse Subjects

This takes me to a page that lets me select from several subjects, including one just called “Wikipedia” with 24 records. In that list, I find two titles that seem to relate to my topic: Good Faith Collaboration: The Culture of Wikipedia and Global Wikipedia: International and Cross-Cultural Issues in Online Collaboration. I click on them to see their records:

Wikipedia Results

Those titles may be enough to satisfy me, but I won’t have done a very thorough job of researching if I don’t also look at the information below the book title. The first thing I notice is that two books have different Library of Congress (LC) classification numbers.

The first starts with AE100. LC Class A is “General Works,” and subclass AE is “Encyclopedias.” The second starts with ZA4482. LC Class Z is “Bibliography, Library Science” and subclass ZA is Information resources/materials. Thus similar-sounding material may be indexed under different numbers—in the one case, the focus is more on encyclopedias and in the other more on user-generated knowledge—but the LC system allows me to search on both those numbers to expand the search pie. I can do that simply by clicking on the number beside “LC classification (partial)” or by searching for it as a call number in the Browse feature (see below).

Wikipedia Browse Call Number

If I do that, about 15 records come up for AE100 (followed by higher-numbered records; as the name implies, this feature is similar to browsing books on a shelf). If I do the same thing for ZA4482, about 20 records come up. I have now expanded my search results from 24 to 35 potentially relevant records.

If searching by call number seems too geeky to you, a different option presents itself in the form of related subject headings. There are six of these for the Good Faith book and five for the Global Wikipedia book, with an overlap in the first subject heading, “Wikipedia.” All of these are hyperlinked and take me to the subject listings in the LC catalog. For instance, if I click on the subject heading “Authorship–Collaboration–Case studies” under the record for the Good Faith book, I get three further entries, including one German book that falls in the ZA4482 class and a book on textual curation that falls in the AE1 class.

In my next post, I will show how Wikipedia uses the LCC system as well as several other organizational systems that allow for the convergence of both arborescent and rhizomatic ways of knowing.


John Kleefeld is an associate professor at the College of Law and a 2017 teaching fellow at the Gwenna Moss Centre for Teaching Effectiveness, where he is coordinating a campus-wide project on integrating Wikipedia assignments into course materials. Portions of this blog series are from an article that he and a former law student wrote about using a Wikipedia assignment for class credit. See J. Kleefeld and K. Rattray, 2016. “Write a Wikipedia Article for Law School Credit—Really?” Journal of Legal Education, 65:3, 597-621.


[1] I am indebted to a colleague, Heather Jensen, for the following example.

[2] See Goodman v Rossi (1995), 24 OR (3d) 359 (CA), 1995 CanLII 1888.

[3] My first search result was Jutte v Jutte, 2007 ABQB 191, reflecting the large number of times the term “implied undertaking” appeared in the case. Under “Related Resources” is the hyperlinked heading “Abridgment digests and classifications for all levels of this case.” Clicking on that leads to the following branch of the Canadian Abridgment Digests (the numbers of cases digested for each heading are shown in parentheses):

CIV Civil practice and procedure
CIV.XII Discovery
CIV.XII.1 Introductory (413)
CIV.XII.1.d Deemed or implied undertaking (153).

Clicking on the last of these headings leads me to the digests, sorted in reverse date order.

Taking a Fresh Approach to the Course Design Institute

[social_share/]



For more than a decade, the Gwenna Moss Centre for Teaching and Learning (GMCTL) has offered the Course Design Institute (CDI). Throughout the CDI, facilitators from the GMCTL work with instructors on developing or redeveloping a course. We go through learning about your students, writing learning outcomes, choosing teaching strategies, developing assessments, and putting it all together through constructive alignment and the blueprinting of your course.

While the CDI had been an intensive four full-day experience within one week, a few years ago we revamped it to offer it in a “flipped” mode, with participants meeting face-to-face three half days over three weeks, plus completing activities and posting to the discussion forums to provide feedback to each other in BBLearn (our learning management system). This year, we’re taking that approach and modifying it again.

On day one, Tuesday May 2, we’ll meet with participants for a half day to cover knowing your students and writing learning outcomes.

On day two, Thursday May 4, participants will choose one of three options for a day-long elective. Participants may choose from indigenization, open pedagogy, or sustainability. Lunch is included on this day.

On day three, Tuesday May 9, also a half day session, we will talk briefly about the participants’ respective experiences in their day-long elective sessions, review their learning outcomes, and talk about assessment and rubrics.

On day four, Thursday May 11, again a half day session, we will discuss constructive alignment, instructional strategies, blueprinting your course, and course syllabi.

In between the sessions, participants will need to complete activities related to what has been covered or prepare for what will be covered in the next session. Following the CDI, participants will need to complete a brief reflective paper and, once they have begun blueprinting their course, meet with one of the facilitators for a one-on-one consultation.

For more information about the CDI or to apply to participate, please see the Course Design Institute page on our website or contact me at the GMCTL.

WikiProjects, Article Importance, and Article Quality: An Intimate Relationship (2/2)

[social_share/]



In this second part of this two-part article, I discuss how WikiProjects, article importance and article quality come together in Wikipedia—and how that conjuncture can help instructors and students with selecting appropriate articles for editing in Wikipedia-based course assignments.

Understanding the Article Assessment Grid

A key WikiProject output is assessment of an article’s quality and importance (see the previous post on the criteria for measuring these). Quality and importance can be plotted on a two-dimensional grid in which each cell represents a particular quality grade and a particular importance level. The cells can then be populated with the number of articles in each of these pairings, using the quality and importance data from the article Talk pages. In Wikipedia, this information is collected—and the cells are populated—with help from a bot (more on which later), allowing for the number of articles of interest to a WikiProject, and their importance and quality, to be updated over time.

The output can be seen by looking at the assessment grid for WikiProject Adoption, fostering, orphan care and displacement (“AFOD”), discussed in the last post. I’ve captured an image of the grid as of 26 February 2017 and hyperlinked it to the actual grid on the AFOD project page, which lets you view the most up-to-date version of the grid.

For now, let’s ignore the bottom part of the grid and focus on the vertical quality axis, starting with “List.” Scanning this line, we see that there are two list-class articles, both of them considered “High” importance. Indeed, these are the only List-class articles for AFOD, as reflected in the “Total” column at far-right, which also shows “2.” Moving up to “Stub,” we can see there are three stub-class articles of high importance, 13 of mid-importance, 31 of low importance, and two that need assessing (“???”), for a total of 49 stub-class articles tracked in this WikiProject. We can keep moving up the quality axis to see the number of articles in the Start-, C-, B-, GA-, and FA-classes and their distribution across importance levels.

Now let’s look focus on the horizontal importance axis, starting with “Top.” We can see that there is one B-class article in this category, three C-class, and seven Start-class articles—for a total of 11 assessed articles in the top-importance category. Moving to the right and following the same procedure, we can see that there are 24 high-importance, 48 mid-importance, and 91 low-importance articles—with the distribution across quality levels as shown in the grid.

Assessment grid for WikiProject Adoption, fostering, orphan care and displacement (“AFOD”), as of 26 February 2017. Click on the image to see the most up-to-date version of the grid in Wikipedia.

Assessment grid for WikiProject Adoption, fostering, orphan care and displacement (“AFOD”), as of 26 February 2017. Click on the image to see the most up-to-date version of the grid in Wikipedia.

This statistical information comes together in two metrics, called WikiWork factors, that serve as a rough guide to the amount of work a given WikiProject entails. These metrics, ω and Ω (the lower- and upper-case versions of the Greek letter omega) are at the bottom of the AFOD assessment grid. The metric ω is the number of steps a WikiProject is from having all articles attain FA status; for example, an A-class article is one step away from that status, while a Stub-class article is six steps away. (List articles aren’t counted.) Multiplying the steps by the number of articles in that class and summing everything yields a ω of 986. The other metric, Ω, is a measure of relative workload: ω divided by the number of articles (again, excluding List articles).[1] It is always a number between zero and six—in this case, 4.91—with lower numbers indicating that less work is needed on average to bring an article to FA status.

Moving from the Assessment Grid to the Article Tables

The astute observer will notice that the numbers in the cells are blue, indicating a link to an active page in Wikipedia. Indeed, clicking on a number in the assessment grid to which this image links takes you to a further page that lists the article titles for the particular quality-importance pairing. (Clicking on the numbers also changes their colour, as it has done for two of the numbers in this image.) For example, clicking on the number “2” in the cell (FA, High) takes you to a table that shows the two articles that are considered “featured” (Wikipedia’s highest quality) and high-importance:

Table showing WikiProject AFOD articles rated as “High” importance and “Featured”—accessed via the (FA, High) cell in the AFOD assessment grid. Click on the image to see the most up-to-date version of the table in Wikipedia.

Table showing WikiProject AFOD articles rated as “High” importance and “Featured”—accessed via the (FA, High) cell in the AFOD assessment grid. Click on the image to see the most up-to-date version of the table in Wikipedia.

This table shows us that: (i) both articles were rated high-importance on 22 August 2009; (ii) the article “Attachment theory” became a featured article on 30 November 2009; and (iii) the article “Reactive attachment disorder” became a featured article on 18 June 2009. Clicking on the titles under the “Article” heading takes you to the current versions of the articles; clicking on the dates takes you to permanent links to the earlier versions, which, as a pink warning banner declares, “may differ significantly from the current revision.” The letters “t” and “h” are links to the Talk and History pages for the articles; the letter “l” and the heading “Score” relate to the eventual release of the article as part of the offline project, Wikipedia 1.0.

Putting it All Together: Using the Tools to Assign Articles for Editing

From the above, you can see how these tools—the assessment grid and article tables to which they lead—can be used to create a shortlist of existing Wikipedia articles for students to edit. In the AFOD assessment grid, for example, there are 62 articles rated as stub-class or start-class and considered to be top-, high- or mid-importance. This is a likely place to look for candidates for impact and improvement—though even the C- and B-class articles may also be good candidates if the aim is to achieve at least GA status. From this universe of candidates, instructors can generate a list of articles to which students can be reasonably expected to make a meaningful contribution, or from which students can choose their own articles to edit. For example, WikiProject Canadian law, of particular interest to me, has some 300 stub- or start-class articles of mid- to top-importance. The average workload, Ω, is 5.09. There is a lot of work to be done in Wikipedia—and I haven’t even begun to consider the task of writing new articles!


John Kleefeld is an associate professor at the College of Law and a 2017 teaching fellow at the Gwenna Moss Centre for Teaching Effectiveness, where he is coordinating a campus-wide project on integrating Wikipedia assignments into course materials. Portions of this blog series are from an article that he and a former law student wrote about using a Wikipedia assignment for class credit. See J. Kleefeld and K. Rattray, 2016. “Write a Wikipedia Article for Law School Credit—Really?” Journal of Legal Education, 65:3, 597-621.

[1] That is, 986 ÷ (11 + 22 + 48 + 90 +30) = 4.91.

WikiProjects, Article Importance, and Article Quality: An Intimate Relationship (1/2)

[social_share/]



In a previous post, I wrote about how WikiProject Medicine acts as a forum for determining the priority (also called importance) of specific health-related Wikipedia articles and assessing their quality (also called class). More generally, these three concepts—WikiProjects, article importance, and article quality—are crucial for instructors and students to understand if they seek to use course-based assignments to improve Wikipedia. I will address each of them in turn.

WikiProjects

A WikiProject comprises a group of collaborators who aim to achieve specific Wikipedia editing goals, or to achieve goals in a specific subject or discipline represented in Wikipedia. An example of an editing type of project is WikiProject missing encyclopedic articles, which seeks to ensure that Wikipedia “has a corresponding article for every article in every other general purpose encyclopedia.” An example of a subject-specific type of project (in addition to WikiProject Medicine), is WikiProject Adoption, fostering, orphan care and displacement (“AFOD”). It aims to improve Wikipedia’s coverage of adoption, foster care, and child abandonment. A third type of project, one that combines both editing and subject goals, is exemplified by WikiProject Biography, which “concerns the creation, development, and organization of Wikipedia’s articles about persons.”

Wikipedia has a shortcut—sometimes more than one—for each project: “WP:” followed by an acronym. So, for example, entering WP:MISSING in Wikipedia’s search box takes you the Talk page for WikiProject Missing encyclopedic articles; entering WP:AFOD takes you to the Talk page for WikiProject AFOD. Entering WP:WPBIO, WP:BIOG or WP:BIOGRAPHY takes you to the WikiProject Biography Talk page. There is even a project on WikiProjects—a meta-project, if you will—with the shortcut WP:PROJ.

A WikiProject Council tracks these projects, including activity levels and inter-project discussions. According to the most recent version of the Council’s WikiProject List, There are about 2,000 WikiProjects in English Wikipedia, with varying levels of activity and interest in articles. Indeed, a Wikipedia article will often be of interest to more than one WikiProject, and a key activity of participants is to identify the WikiProjects to which the article is of interest, along with the article’s importance and class, as assigned by those projects. This is done on the article’s Talk page. For example, the article on Barnardo’s, a British charity founded to care for vulnerable children and young people, is of interest to WikiProject AFOD, which ranks it as a high-importance, C-class article. But it is also of interest to WikiProject London, a collaborative effort to improve Wikipedia’s coverage of the city of London; that project ranks it as mid-importance and start-class. (See accompanying excerpt.)

WikiProject information and rankings from Talk page for Barnardo’s (accessed 8 February 2017). Text via CC by 4.0; Anne of Green Gables image and London image originally incorporated into text via CC by 2.0.

Article Importance (Priority)

There are five levels of priority: top, high, mid, low, and NA (meaning “not an article”—that is, something other than an article, like a template or category). There is also a level for “unknown” or “needs assessing” that appears as ??? to the reader. Each WikiProject has its own criteria for these rankings. The table below shows the criteria in WikiProject Medicine, along with an example of a Wikipedia page for each ranking.

Article importance grading scheme
Label Criteria Examples
Top priority Subject is extremely important, even crucial, to medicine. Strong interest from non-professionals around the world. Usually a large subject with many associated sub-articles. Less than 1% of medicine-related articles achieve this rating. Tuberculosis or Cancer
High priority Subject is clearly notable. Subject is interesting to, or directly affects, many average readers. This category includes the most common diseases and treatments as well as major areas of specialization. Fewer than 10% of medicine-related articles achieve this rating. Coeliac disease or Mastectomy
Mid priority Normal priority for article improvement. A good article would be interesting or useful to many readers. Subject is notable within its particular specialty. This category includes most medical conditions, tests, approved drugs, medical subspecialties, well-known anatomy, and common signs and symptoms. Cholangiocarcinoma or Cramp
Low priority Article may only be included to cover a specific part of a more important article, or may be only loosely connected to medicine. Subject may be specific to one country or part of one country, such as licensing requirements or organizations. This category includes most of the following: very rare diseases, lesser-known medical signs, equipment, hospitals, individuals, historical information, publications, laws, investigational drugs, detailed genetic and physiological information, and obscure anatomical features. Leopard syndrome or Flynn effect
NA NA means Not an Article. This label is used for all pages that are not articles, such as templates, categories, and disambiguation pages. WikiProject Medicine

Source: WikiProject Medicine/Assessment (accessed 9 February 2017)

Article Quality (Class)

There are nine quality levels or classes for the typical Wikipedia article: stub, start, C-class, B-class, GA (good article), A-class, FA (featured article), List, and FL (featured list). These categories are used by the Wikipedia Version 1.0 Editorial Team for deciding how close an article is to being distribution-quality (that is, to the goal of publishing Wikipedia articles in print, CD, DVD, or a combination thereof). Some WikiProjects also use intermediate classes, such as B+. A summary of the common classes is provided below. For more detailed criteria, see the Version 1.0 assessment page; for a categorized list of articles in each class, click on the name of the class in the leftmost column.

 Stub  The article is either very short or a rough collection of information that needs much work. Stub-class articles are adequate enough to be accepted, but risk being dropped from article status altogether. The first step in improving a Stub-class article is usually the addition of referenced reasons that show why the topic is significant.
 Start  The article is developing but quite incomplete. Deficiencies may include inadequate citation to reliable sources or non-compliance with Wikipedia’s style guidelines. Raising the article to C-class typically requires further referencing, improvement in content and organization, and attention to grammar and writing style.
 C  The article is substantial, but still lacks important content or contains irrelevant material. The article should have some references to reliable sources, but may still have significant problems or require substantial cleanup. By the time an article reaches C-class, it typically has at least some infoboxes, photographs, diagrams or other media.
 B  The article is mostly complete and without major problems, but requires some further work to reach GA status. It is properly referenced to reliable sources, using inline citations. It is balanced, reasonably well written, and has a defined structure, including a lead section. Supporting materials, such as illustrations, diagrams and an infobox, should be included where relevant and useful. The article should not assume unnecessary technical background and should either avoid or explain technical terms where possible.
 GA  The article has attained good-article status (indicated at the top of the article by the “plus sign” logo) via an official review. In addition to being well written and following style guidelines, it is verifiable, contains no original research, and has no copyright violations or plagiarism. It represents viewpoints fairly, giving due weight to each, and focuses on the topic without going into unnecessary detail. It is typically illustrated with copyright-compliant images that are appropriately captioned. The article is stable in the sense of not being subject to edit wars or content disputes, though comparison with a featured article on a similar topic may show areas where content could be further developed.
 A  The article is well organized and essentially complete, having been reviewed by impartial reviewers from a WikiProject or elsewhere. GA status is not a requirement for this level, but with further tweaking or peer review, it may also be appropriate for GA or FA status.
 FA The article has attained featured article status (shown by the “star” logo at the top of the article) by passing an official review. It exemplifies the best work on Wikipedia and is distinguished by engaging and professional standards of writing, presentation and sourcing. A concise lead summarizes the topic and prepares the reader for the detail in subsequent sections that are hierarchically arranged and presented in a table of contents. Citation is extensive and consistent. The article has images and other media, where appropriate, with succinct captions and acceptable copyright status. FA-class articles may appear on Wikipedia’s home page, in a “Today’s featured article” section.
 List  The article meets the criteria of a stand-alone list, which is an article that contains primarily a list, usually consisting of links to articles in a particular subject area. List articles are often alphabetized or chronologically ordered and may also be annotated.
 FL  The article has attained featured list status. It comprehensively covers the defined scope, usually providing a complete set of items and annotations that provide useful and appropriate information about those items.

Source: Wikipedia:Version 1.0 Editorial Team/Assessment (accessed and adapted 12 February 2017)

In my next post, I will discuss how WikiProjects, article importance and article quality come together in a format that provides a convenient basis for selecting articles to edit for Wikipedia-based course assignments.


John Kleefeld is an associate professor at the College of Law and a 2017 teaching fellow at the Gwenna Moss Centre for Teaching Effectiveness, where he is coordinating a campus-wide project on integrating Wikipedia assignments into course materials. Portions of this blog series are from an article that he and a former law student wrote about using a Wikipedia assignment for class credit. See J. Kleefeld and K. Rattray, 2016. “Write a Wikipedia Article for Law School Credit—Really?” Journal of Legal Education, 65:3, 597-621.

Wikipedia’s Gender Bias – and What Your Students Can Do About It

[social_share/]


Every system has its biases, and Wikipedia is no exception. A common criticism of Wikipedia is its male bias. Wikimedia Foundation, which runs Wikipedia, agreed with the criticism after it conducted a 2011 survey indicating that up to 90% of editors identified as male. This is a problem for a non-profit organization whose mission is “to empower and engage people around the world to collect and develop educational content … and to disseminate it effectively and globally.”

The mechanisms for the gender bias are various, complex, and the subject of several studies, recently summarized by two New York researchers. They may include the code-heavy interface, called wiki markup, that contributors initially had to use to edit articles. To the extent that wiki markup operated to inhibit female editors, the technical hurdle has largely disappeared: since April 2015, Wikipedia’s VisualEditor is available by default on the Article pages (but not the Talk pages) for about three-quarters of the language editions of Wikipedias. A more troubling and persistent concern may be Wikipedia’s sometimes hostile user culture, which I’ll discuss in a future blog post. In response to these concerns, there has been a series of efforts to increase female editorship. These include edit-a-thons, some organized by Wikimedia Foundation and some independently, to increase coverage of women’s topics in Wikipedia and to encourage more women to edit it. An example is the worldwide Art+Feminism edit-a-thon, the third of which was held last year to coincide with International Women’s Day. Events took place in nine locations across Canada, including Saskatoon, where editors focused on Saskatchewan and Indigenous women artists including Ruth Cuthand, Mary Longman, and Michelle LaVallee. Similar events are planned for the US, Canada and Europe in March 2017.

Anonymous Woman in Green

Woman in Green Dress, Anonymous, c 1825, National Museum in Warsaw. Source: Wikimedia Commons.

Educators and their students can help address the bias. In 2012, students in Alana Cattapan’s fourth-year seminar, “The Politics of the Canadian Women’s Movement,” edited, updated, and expanded various Wikipedia articles, including “Feminism in Canada.” Though often serving as a first point of reference on Canadian feminism, this Wikipedia page was underdeveloped, and Cattapan drew on her students to set about correcting this gateway article and other related ones. Librarians have also been active. In September 2016, to celebrate Science Literacy Week, Concordia University Library partnered with McGill Library to host a Women in Science Wikipedia event. The librarians gave a tutorial on how to edit Wikipedia, followed by an editing session in which participants got one-on-one help.

If you’re thinking of wading into the field and wondering where to start, you might want to look at WikiProject Women in Red. The goal of this project is to turn “red links”—internal links that lead to Wikipedia pages that don’t exist—into “blue links”—internal links that lead to actual Wikipedia articles. (More on this later.) Or if you’d rather start by having your students edit existing material, check out the contrapuntally-named Wikiproject Women in Green, an attempt to bring articles on women up to minimum “Good article” status. The project even provides a “Hot 99” list of women’s biographies to get you started—ranging from Aisha to Natalie Wood.


John Kleefeld is an associate professor at the College of Law and a 2017 teaching fellow at the Gwenna Moss Centre for Teaching Effectiveness, where he is coordinating a campus-wide project on integrating Wikipedia assignments into course materials. Portions of this blog series are from an article that he and a former law student wrote about using a Wikipedia assignment for class credit. See J. Kleefeld and K. Rattray, 2016. “Write a Wikipedia Article for Law School Credit—Really?” Journal of Legal Education, 65:3, 597-621.

What Does It Mean to Say That “Anyone Can Edit” Wikipedia?

[social_share/]



In previous posts, I argued for the benefits of having students edit or write Wikipedia articles for university credit. But if “anyone can edit,” doesn’t that make Wikipedia prone to errors, questionable content, edit wars, and vandalism? Well, yes—these things comprise Wikipedia’s dark side and can compromise efforts at using Wikipedia for student learning. For example, Sivan Lerer, mentioned in my first post, found that when her students edited Bahá’í articles that already had substantial content, they “had a difficult time merging what they wanted with what it said in the entry,” resulting in other Wikipedians undoing their edits. But even this phenomenon can be turned into experiential learning. In “Using Wikipedia to Teach Audience, Genre and Collaboration,” Allan Bilansky, who uses Wikipedia in his social informatics course, says that “an informed effort at making contributions that persist within a large community … can be an experience at being answerable to a real audience.” Responding to a student who reported that he “had made a change [to a Wikipedia article], and then it changed back,” Bilansky told the student, “it did not change back;” rather, “[s]omeone changed it back, probably for reasons we can eventually understand.”

In both theory and practice, though, Wikipedia isn’t the freewheeling editing environment that you might think it is. Wikipedia adheres to some key principles, called the five pillars: (i) Wikipedia is an encyclopedia—not a blogging forum, a social networking site, a place to publish original research, or a dictionary (but see its companion lexical project, Wiktionary); (ii) articles adopt a neutral point of view, which includes “document[ing] and explain[ing] the major points of view, giving due weight with respect to their prominence in an impartial tone;” (iii) Wikipedia is free content that anyone can use, edit, and distribute, so anything that smacks of using Wikipedia for commercial gain attracts censure; (iv) Wikipedia has a code of conduct, or “Wikiquette,” that requires editors to treat each other with civility; and (v) Wikipedia has no firm rules, which means that “principles and spirit matter more than literal wording, and sometimes improving Wikipedia requires making exceptions.”

These principles are fleshed out in specific policies and practices. For example, a Wikipedia article should abide by three core content policies: neutral point of view (NPOV), verifiability (V), and no original research (NOR). Wikipedia is an encyclopedia, after all, which means that it is entirely derivative from other works. Thus, Wikipedians are alert to articles or edits that lack citations to reliable, published sources. Not uncommonly, articles are peppered with “citation needed” admonitions inserted by watchful Wikipedians, or prefaced by warnings that the article may violate NPOV, V, or NOR and be a candidate for deletion. Yet the civility principle means that Wikipedians must be judicious with deletions or other edits. This exhortation takes practical shape in the form of rules like the “three-revert rule”: an editor may not make more than three reversions to an article on a single page—whether involving the same or different material—within a 24-hour period.

Using view history tab on Wikipedia

Click image for screencast on using Wikipedia’s View History page.

To help students imbibe these principles and rules, Bilansky created some innovative assign­ments for his social informatics course. For example, the first assignment required students to read the View History pages (also called page history, revision history or edit history) and Talk pages (also called discussion pages) of Wikipedia articles. Every article has these pages, though most users are probably only faintly aware of them. The page history lists all the article’s previous revisions, including date and time (in UTC) of each edit, the editor’s registered username or IP address, and the user’s edit summary. As Bilansky explains, these features amount to a sort of variorum edition of each article: not only do they preserve every contributor’s minutest work and let you compare two different versions, they also provide opportunities for learning about collaborative writing processes and academic research standards.

The first thing that each of Bilansky’s students had to do was find a Wikipedia article with at least one edit that didn’t persist (that is, changes made by one editor were reverted by another), review the version history, and post to a class discussion thread an explanation of why the edits they examined were undone. (Finding such an article merely requires searching for the word reverted in the revision history.) Bilansky offered prizes for the first student to find a Wikipedia policy cited on a Talk page, which created quite a buzz and an interest in finding more strange-sounding policies or prohibitions, like the endearingly named “sock puppetry” (using multiple user accounts to hide one’s tracks). All this preparatory work smoothed the way for more complex assignments, resulting in engagement on Talk pages and in some cases, collaboration with senior Wikipedia editors (all volunteers—the whole encyclopedia is a volunteer effort) who helped the students navigate their way through what can seem like terra incognita.

I will return to the theme of Wikipedia’s integrity, because there is a lot to say about it. But in my next post, I will consider another critique of Wikipedia—its documented male bias—and what instructors and students are doing about it.


John Kleefeld is an associate professor at the College of Law and a 2017 teaching fellow at the Gwenna Moss Centre for Teaching Effectiveness, where he is coordinating a campus-wide project on integrating Wikipedia assignments into course materials. Portions of this blog series are from an article that he and a former law student wrote about using a Wikipedia assignment for class credit. See J. Kleefeld and K. Rattray, 2016. “Write a Wikipedia Article for Law School Credit—Really?” Journal of Legal Education, 65:3, 597-621

 

How Students Are Learning Medicine and Collaborative Skills, And Transforming Wikipedia

[social_share/]



In my last blog post, I wrote about the wide range of disciplines represented in student Wikipedia projects. Perhaps the most ambitious effort is the Wiki Project Med Foundation, whose goal is nothing less than “to provide the sum of all medical knowledge to all people in their own language.” Started by Wikipedia enthusiast and UBC clinical professor James Heilman, the foundation is working to this goal by collaborating with various partners. These include the closely allied WikiProject Medicine, the non-profit organization Translators Without Borders, and University of California San Francisco, where fourth-year medical students have been editing Wikipedia for credit in a month-long elective course since 2013.

Amin Azzam, associate clinical professor at the UCSF School of Medicine, found that Wikipedia was second only to Google as the most frequently used source by junior physicians (!) but that there was a clear need to bring medical articles up to par. As he explains in a 2014 interview, he and his collaborators prioritized Wikipedia’s medical articles based on the number of unique visitors to the articles and the importance of the articles from a health perspective. Wikipedia also has a system for ranking article quality, ranging from “stub” to “featured article,” and the collaborators found that many articles were at the low end of the quality scale. Azzam encouraged his students to focus on the intersection of these two—high priority but low quality—and direct their efforts to improving them. Most students picked articles from this list, such as Cirrhosis and Hepatitis, while some pursued articles that held a special interest for them, like “Race and health.” Not only did they edit their chosen articles, but they reviewed articles edited by their class peers. The results, presented at a 2015 medical education conference, were impressive. As measured by Wikipedia’s own quality metrics, the students’ work resulted in improvements to most of the selected articles, and significant improvements to several.[1]

Not only has this work continued to the present, with over 50 articles improved through student work, but UCSF’s School of Pharmacy has recently joined forces with the medical school: Tina Brock, professor and associate dean of Global Health & Educational Innovations, now assigns third-year pharmacy students articles to edit from Wiki Project Pharmacology, an initiative like WikiProject Medicine. In the meantime, the Sackler School of Medicine at Tel Aviv University has also taken up the initiative. In a recent paper in Education Information Technology, educators Shani Evenstein Sigalov and Rafi Nachmias explain how their students have edited over 128 medical articles in Hebrew Wikipedia, already viewed over 1.4 million times. The paper also presents findings of a related study that focused on students’ learning experience, long-term impact and productive teaching practices.

“But surely” you might ask, “doesn’t Wikipedia’s ‘anyone can edit philosophy’ mean that all this good work can be undone?” I’ll take up that question in my next post.

[1]  Articles that went from “start class” or “C-class” to “B-class” in the first session can be considered to have improved significantly. According to the hyperlinked study, the following articles would qualify as such: “Hepatitis,” “Diabetes,” “Amyloidosis,” “Cholecystitis,” “Toxic epidermal necrolysis,” “Placental abruption,” “Therapeutic hypothermia,” “Premature rupture of membranes,” “Umbilical cord prolapse” and “Omphalitis of newborn.”


John Kleefeld is an associate professor at the College of Law and a 2017 teaching fellow at the Gwenna Moss Centre for Teaching Effectiveness, where he is coordinating a campus-wide project on integrating Wikipedia assignments into course materials. Portions of this blog series are from an article that he and a former law student wrote about using a Wikipedia assignment for class credit. See J. Kleefeld and K. Rattray, 2016. “Write a Wikipedia Article for Law School Credit—Really?” Journal of Legal Education, 65:3, 597-621.

 

The Wikipedia Manifesto

[social_share/]



wikipedia-logo-v2-svg

This blog has been updated to correct some initial errors.

A spectre is haunting academia—the spectre of Wikipedia. And while there was a time when all the old powers would have entered into an alliance to exorcise this spectre, a worldwide community of educators is now taking a radically different approach: they’re assigning students the task of editing and writing Wikipedia’s sprawling content, and giving them academic credit for doing so. In the process, they’re turning students from indiscriminate knowledge consumers to savvy knowledge creators. At the same time, they’re building an open-access and up-to-date storehouse of knowledge that, in certain areas, already rivals traditional reference works. As Clay Shirky explains, this is all part of an interconnected movement—from wikis to open textbooks to interactive mapping applications like Ushahidi—in which technology has made possible methods of collaboration that never existed before.

How did this start? For some instructors, it came from a place of despair. So it was for Sivan Lerer, who teaches an introductory course on the Bahá’í Faith at Hebrew Jerusalem University. In an interview published in December 2016, Lerer explains:

I’ve told my students, for many years, that despite its many advantages (it’s accessible and in Hebrew), Wikipedia’s not an academic source . . . I told them they can use Wikipedia in the beginning but afterwards they have to go to the Encyclopedia of Islam, or The Encyclopedia of Religions. But all my admonitions were in vain. They used only Wikipedia as their source.

Lerer also wasn’t happy with the quality of writing in her students’ exams, finding that in many of their answers, they would just “regurgitate” her lectures. Sound familiar? That’s when she learned about the education program of Wikimedia Israel, whose goals are to cultivate deeper student learning and improve the access to, and quality of, Wikipedia’s resources. With the help of education coordinator Shai Katz and Darya Kantor, an active “Wikimedian,” Lerer redesigned her course to incorporate Wikipedia assignments, from revising existing articles to adding new material, such as a Canadian student’s article on the Bahá’í community in Canada. Altogether, 18 students created or improved 17 articles in Hebrew Wikipedia, with Lerer concluding that it was a positive experience for the students, and that “[i]nstead of memorizing, they really learned.”

Students have now contributed to Wikipedia as part of their course work in agriculture and life sciences, chemistry, community history, geobiology, linguistics, mineralogy, psychology, public policy, and a host of other fields. In March 2016, Wiki Edu’s Eryk Salvaggio, who writes a blog for the Wiki Education Foundation set out five reasons why you might consider assigning such a project instead of a term paper. It’s not hard to do, but before getting into the nuts and bolts of it, we’ll look at some examples of what others are doing. In particular, I want to tell you about some medicine professors who, with their students’ help, are now editing Wikipedia to make high-quality medical knowledge freely available around the world.


John Kleefeld is an associate professor at the College of Law and a 2017 teaching fellow at the Gwenna Moss Centre for Teaching Effectiveness, where he is coordinating a campus-wide project on integrating Wikipedia assignments into course materials. Portions of this blog series are from an article that he and a former student wrote about using a Wikipedia assignment for class credit. See J. Kleefeld and K. Rattray, 2016. “Write a Wikipedia Article for Law School Credit—Really?” Journal of Legal Education, 65:3, 597-621.

Hands up! How We Increase (Or Decrease) Student Participation

[social_share/]



We design courses with many opportunities for students to learn by completing assignments, readings and answering questions in class. But does our teaching increase such behaviours or decrease them?

One lens, psychology of learning, suggests we likely do both. B. F. Skinners’ operant conditioning suggests that how we respond to student behavior can either increase (reinforce) or decrease (punish) our students actions including participating in class discussion or completing homework.

What is Operant conditioning?

As Thorndike’s Law of Effect and B.F. Skinner’s operant conditioning note we are influenced by the consequences of our actions. Good consequences encourage more of this activity, while unpleasant (or unhelpful) consequences encourage less of this activity.

Reinforcement increases the frequency of behaviours through either the addition of a pleasant stimulus (positive reinforcement) or the removal of an unpleasant stimulus (Negative reinforcement).

Punishment” decreases the frequency of valued behaviours through either the addition of an unpleasant stimulus (positive punishment) or the removal of an unpleasant stimulus (negative punishment).

What about Encouraging students to answer questions in class:

Hands up! How We Increase (Or Decrease) Students Answering QuestionsWe might beneficially use punishment to decrease of disruptive behaviours such as disruptive side conversations, interrupting classmates, or answering cell phones by adding the unpleasantness of awkwardness when we stand near by, interrupt to redirect conversation, or let silence fall during the phone call.

Our effect may also be neutral leading to attenuation where the lack of a reward results in decreased responses, including when an instructor neither confirms or discounts the response and simply says “next” until they have 3 responses regardless of correctness.

Over time, behaviours do not need to be (and should not be) actively reinforced each time to maintain higher participation or lower skipping class (see information on schedules and fixed versus variable intervals and ratios).

Experiment!

Try seeing how the number of students’ answers increases (or decreases) with different responses. Predict via the lens of operant conditioning. For example:

  • What happens if I ask questions that are too easy? -> Students likely not rewarded by answering.
  • What happens if I ask questions that are too hard? ->Students might not be able to answer and receive the explicit or implicit feedback that they are wrong.
  • What happens if I present my answer(s) on a slide after I ask them? Students might not be rewarded by answering
  • But what if I skim by pointing out all the parts they identified and building on their answer? -> Students might be rewarded and increase participation.
  • What if I summarize the readings? -> Students who read now have the frustration of listening again and having “wasted time” while students who did not read are reinforced that their decision was correct.
  • What if I have them pull out the readings or use a specific page or section for an activity -> Students who read ware rewarded by not having to quickly skim, students who did not read might experience uncertainty or struggle.

Applying operant conditioning is not about “coddling” or saying “good try” without correcting flawed knowledge, but creating a learning experience that is encouraging of participation, reading and incorporating feedback into later performance. Even when a students’ answer is incorrect there are ways to reward behaviours that lead to improvement (e.g., asking questions) and provide feedback to modify that knowledge by “rewarding” the correct bits, “punishing” incorrect parts, and because we can speak better than pigeons, suggesting how to improve.

While it is useful to be cognizant of how our actions may act to encourage or discourage specific student behaviours, self-determination is still valued and people may not want themselves or others to be treated as treating people like lab rats such as by Sheldon on The Big Bang Theory:

Resources: