Paper: pdf Slides: pdf Halfaker, A., Gieger, R. S., Morgan, J., & Riedl, J. (2013). The Rise and Decline of an Open Collaboration System: How Wikipedia's reaction to sudden popularity is causing its decline. American Behavioral Scientist 57(5) 664-688, DOI:10.1177/0002764212469365. APA @article{halfaker13rise, author={Aaron Halfaker and R. Stuart Geiger and Jonathan Morgan and John Riedl}, title={The Rise and Decline of an Open Collaboration System: How Wikipedia's reaction to sudden popularity is causing its decline}, year={2013}, month={May}, volume={57}, number={5}, pages={664--688}, journal={American Behavioral Scientist}, doi={10.1177/0002764212469365}, url={http://dx.doi.org/10.1177/0002764212469365} } bibtex Citation:

The Rise and Decline of an Open Collaboration Community: How Wikipedia's reaction to sudden popularity is causing its decline

Open collaboration systems like Wikipedia need to maintain a pool of volunteer contributors in order to remain relevant. Wikipedia was created through a tremendous number of contributions by millions of contributors. However, recent research has shown that the number of active contributors in Wikipedia has been declining steadily for years, and suggests that a sharp decline in the retention of newcomers is the cause. This paper presents data that show that several changes the Wikipedia community made to manage quality and consistency in the face of a massive growth in participation have ironically crippled the very growth they were designed to manage. Specifically, the restrictiveness of the encyclopedia's primary quality control mechanism and the algorithmic tools used to reject contributions are implicated as key causes of decreased newcomer retention. Further, the community's formal mechanisms for norm articulation are shown to have calcified against changes – especially changes proposed by newer editors.

Authors

Key Findings (tl:dr)

To deal with the massive influx of new editors between 2004 and 2007, Wikipedians built automated quality control tools and solidified their rules of governance. These reasonable and effective strategies for maintaining the quality of the encyclopedia have come at the cost of decreased retention of desirable newcomers.

The decline represents a change in the rate of retention of desirable, good-faith newcomers.

The proportion of newcomers that edit in good-faith has not changed since 2006.

These desirable newcomers are more likely to have their work rejected since 2007.

This increased rejection predicts the observed decline in retention.

Semi-autonomous vandal fighting tools (like Huggle) are partially at fault.

An increasing proportion of desirable newcomers are having their work rejected by automated tools.

These automated reverts exacerbate the predicted negative effects of rejection on retention.

Users of Huggle tend to not engage in the best practices for discussing the reverts they perform.

New users are being pushed out of policy articulation.

The formalized process for vetting new policies and changes to policies ensures that newcomers' edits do not survive.

Both newcomers and experienced editors are moving increasingly toward less formal spaces.

Summary

Gentle reader, Below is the authors' summary of a paper that was accepted to a special issue of American Behavioral Scientist on Wikis. I also provide the unabridged, pre-print of the paper if you desire more discussion and explanation. Feel free to email me with questions, and please let me know if you find an error. Enjoy!

Aaron Halfaker (aaron.halfaker@gmail.com)

According to a report published in 2009 by the Wikimedia Foundation, the number of active editors working on the English Wikipedia is declining. As the figure 1 below suggests, the number of active editors (editors with >= 5 edits/month) abruptly stopped growing in early 2007 and entered a steady, linear decline. Recent research has shown evidence that this transition is rooted in the declining retention of new editors, not a change in the retention of already-experienced old-timers (Suh, 2009). What is unclear, or was before this work, is why this sudden change in the retention of new editors took place.

This paper implicates the strategies adopted by Wikipedia editors to preserve the quality and consistency of the encyclopedia in causing the decline in retention of desirable newcomers. Below, the results are broken up into a description of three general findings.

The decline in desirable newcomers

One of the biggest open questions about Wikipedia's newcomer decline was whether it was the result of a natural decline in the quality of newcomers (where lower-quality newcomers were "encouraged" to go elsewhere) or whether changes in how Wikipedia welcomes newcomers was at fault. In order to explore this, we manually categorized the work of 2100 newcomers sampled over the history of the website. With the help of Maryana Pinchuk (Accedie), Oliver Keyes (Ironholds) and Steven Walling (Steven_Walling) we categorized these newcomers into 4 ordinal quality classes based on their first session of editing activity:

Vandals - Purposefully malicious, out to cause harm Bad-faith - Trying to be funny, not here to help or harm Good-faith - Trying to be productive, but failing Golden - Successfully contributing productiv

In the analysis, we simplify these 4 categories into ''desirable'' newcomers (good-faith & golden) and ''undesirable'' newcomers (bad-faith & vandal).

The three plots above (and the logistic regression models we built) make a few things apparent:

The proportion of desirable newcomers entering Wikipedia has not changed since 2006

These desirable good newcomers are more likely than their predecessors to have their first contributions rejected

The decline in good newcomers is the result of a decline in desirable newcomers. Undesirable newcomer retention rate (not shown) stays constant.

Efficient quality control lead to an impersonal newcomer experience

In order to maintain the quality of encyclopedic content in the face of exponential growth in the contributor community, Wikipedians developed automated (bots) and semi-automated tools (Huggle, Twinkle, etc.) to make the work of rejecting undesirable contributions waste as little effort as possible. These tools are apparently effective at their job. Recent research has shown that the time between when vandalism is posted and reverted is very short (median: ~2 minutes)(Kittur, 2007) and has been steadily falling(West, 2010). However, we hypothesized that the ways in which efficiency was achieved with these tools was part of the problem.

Recent research by Geiger et al. has shown that an increasing amount of newcomers' first message received after joining Wikipedia is business end of an automated quality control tool (Geiger, 2012). Figure 5 shows the growing use of automated tools to send messages to newcomers. Figure 6 shows that the use of these quality control tools to revert newcomers is growing and our regression models (see the paper) suggest that desirable newcomers who were reverted with them were especially less likely to continue editing.

In order to explain this effect, we looked for evidence that bot users were interacting negatively to newcomers, by examining their adherence to Wikipedia's best practices for discussions about reverted edits, the Bold, revert, discuss cycle. As figure 7 shows, Huggle users were susprisingly unlikely to respond to desirable newcomers when they questioned why they were reverted.

Calcification of rules against newcomers

Recent work studying Wikipedia's policies and guidelines has suggested that the process by which these rules and recommendations are vetted reflect community concerns and decentralization in governance participation (Beschastnikh, 2008). However, it has also been shown that experienced Wikipedians have more power over interpretation of the rules (Kriplean, 2007) and that policy and guideline creation has slowed since 2006 (Forte, 2009) We suspected that, although natural and generally beneficial, this calcification of the rules of Wikipedia would biased against newcomers' concerns.

To explore this hypothesis, we built of regression model predicting which edits to policies and guidelines were likely to be reverted. In short, this model showed us that:

Over time, newcomers to Wikipedia are substantially less likely to have their policy edits be kept

Unlike formal policies and guidelines, essays have not suffered such a crackdown

While we do not suggest that the rules of Wikipedia be open to reinterpretation by newcomers, we advocate that concern should be allocated for newcomers with a legitimate interest in changing the way that Wikipedia works. We argue that these results suggest that it is important that newcomers have a say in how they are treated.

References