Abstract The online resource Wikipedia is increasingly used by students for knowledge acquisition and learning. However, the lack of a formal editorial review and the heterogeneous expertise of contributors often results in skepticism by educators whether Wikipedia should be recommended to students as an information source. In this study we systematically analyzed the accuracy and completeness of drug information in the German and English language versions of Wikipedia in comparison to standard textbooks of pharmacology. In addition, references, revision history and readability were evaluated. Analysis of readability was performed using the Amstad readability index and the Erste Wiener Sachtextformel. The data on indication, mechanism of action, pharmacokinetics, adverse effects and contraindications for 100 curricular drugs were retrieved from standard German textbooks of general pharmacology and compared with the corresponding articles in the German language version of Wikipedia. Quantitative analysis revealed that accuracy of drug information in Wikipedia was 99.7%±0.2% when compared to the textbook data. The overall completeness of drug information in Wikipedia was 83.8±1.5% (p<0.001). Completeness varied in-between categories, and was lowest in the category “pharmacokinetics” (68.0%±4.2%; p<0.001) and highest in the category “indication” (91.3%±2.0%) when compared to the textbook data overlap. Similar results were obtained for the English language version of Wikipedia. Of the drug information missing in Wikipedia, 62.5% was rated as didactically non-relevant in a qualitative re-evaluation study. Drug articles in Wikipedia had an average of 14.6±1.6 references and 262.8±37.4 edits performed by 142.7±17.6 editors. Both Wikipedia and textbooks samples had comparable, low readability. Our study suggests that Wikipedia is an accurate and comprehensive source of drug-related information for undergraduate medical education.

Citation: Kräenbring J, Monzon Penza T, Gutmann J, Muehlich S, Zolk O, Wojnowski L, et al. (2014) Accuracy and Completeness of Drug Information in Wikipedia: A Comparison with Standard Textbooks of Pharmacology. PLoS ONE 9(9): e106930. https://doi.org/10.1371/journal.pone.0106930 Editor: Christian Lovis, University Hospitals of Geneva, Switzerland Received: April 23, 2014; Accepted: August 3, 2014; Published: September 24, 2014 Copyright: © 2014 Kräenbring et al. This is an open-access article distributed under the terms of the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original author and source are credited. Data Availability: The authors confirm that all data underlying the findings are fully available without restriction. All relevant data are within the paper and its Supporting Information files. Funding: This work was supported by a teaching fund (to A.S.) of Technische Universität München (http://www.tum.de/en). The funders had no role in study design, data collection and analysis, decision to publish, or preparation of the manuscript. Competing interests: The authors have declared that no competing interests exist.

Introduction The omnipresence of the internet has an increasing impact on higher education and on the way students access information for learning. The current generation of undergraduate students grew up in an environment in which the internet and computer-based technologies have become an integral part of life. This cohort, often referred to as “Net Generation” or “Digital Natives” is technological savvy and familiar with the world wide web as a communication platform and source of information [1]. In addition, studies have suggested differences in learning behavior and learning preferences of “Digital Natives” in comparison to preceding student generations. For instance, Oblinger and colleagues proposed that today's students are more comfortable with multimedia learning environments, prefer to be actively engaged rather than being passive consumers of information and favor immediate responses and topic-related discussions with their peers and educators, both in person and online [2]. In parallel, the world wide web has advanced from a network of static web sites to the more user-centered “Web 2.0” that supports active participation, dynamic interaction and collaborative approaches [3]. One key element of the “Web 2.0” are wikis (a term originating from the Hawaiian word for “quick”) which are web applications that allow users to collaboratively create, edit and share content. Wikis have little implicit structure and thus allow content to emerge according to the needs of the users. The most frequented wiki of the internet is the online encyclopedia Wikipedia (www.wikipedia.org). The English language version of Wikipedia (www.en.wikipedia.org) currently consists of 4.54 million articles (yearly change +8.0% or 362,933 articles) written by approx. 40,000 registered editors. Each month, the website has more than 9.5 billion page views and 2.9 million article revisions (http://stats.wikimedia.org; accessed 18. April 2014). The smaller German language edition of Wikipedia (www.de.wikipedia.org) consists of approx. 1.67 million articles (yearly change +8.0% or 133,526 articles) created by approx. 7,900 registered editors. It has approx. 1.27 billion page views and 0.6 million article revisions per month (http://stats.wikimedia.org; accessed 18. April 2014). Since its launch in 2001, Wikipedia has become one of the most visited websites and a frequently accessed source of medical information [4]. Recent studies estimated that Wikipedia is used by up to 70% of junior physicians and practitioners as information source for medical care [5]. Moreover, Wikipedia has been shown to be widely used by undergraduate medical students [6], [7]. In contrast to traditional encyclopedias that are written and edited exclusively by professionals, Wikipedia allows any user to edit content and create articles. This lack of a formal editorial review and the heterogeneous expertise of contributors often results in skepticism by both educators and students whether Wikipedia should be recommended as an information source in undergraduate medical education [8]. While a number of empirical studies has evaluated the medical content of Wikipedia (reviewed in [9]), only few researchers have addressed the quality of pharmacological information. Of these, research has focused on a limited set of drugs or on information for patients and health practitioners [10], [11]. Whether Wikipedia is a suitable source of pharmacological information for students and educators in medical education has not been investigated. In this study we provide a comprehensive analysis of accuracy and completeness of drug information in Wikipedia in comparison to standard textbooks of basic pharmacology. In addition, references, revision history and readability of the corresponding drug articles in Wikipedia were analyzed to evaluate their potential as a learning resource in undergraduate medical education.

Methods Study design One hundred drugs were randomly selected from the list of 300 curricular drugs (www.tum300.de) taught in a basic pharmacology course for undergraduate medical students in Munich (see table S1 for the list of selected drugs). The TUM300 drug list is based on the World Health Organization Model Lists of Essential Medicines and adjusted to include all major pharmacotherapeutic substance classes and drugs that are most commonly prescribed in Germany [12]. For each drug, information on indication (IND), mechanism of action (MA), adverse effects (AE), pharmacokinetics (PK) and contraindications (CI) was excerpted from two German standard textbooks of basic pharmacology [13], [14]. Information on drug-drug interactions was included in the “pharmacokinetics” category. Only information present in both textbooks (data overlap) was included in the study and served as point of reference for the analysis of the respective drug articles in the German language version of Wikipedia (www.de.wikipedia.org; last accessed on March 20th 2013) (Figure 1). An analogous approach was used to compare the data overlap of 50 drugs (see table S1) retrieved from two standard Anglo-American textbooks [15], [16] with the corresponding drug articles in the English language version of Wikipedia (www.en.wikipedia.org; last accessed on June 20th 2014). PPT PowerPoint slide

PowerPoint slide PNG larger image

larger image TIFF original image Download: Figure 1. Study design. Pharmacological information of randomly selected curricular drugs was retrieved form two standard textbooks of general pharmacology for five different categories (indication, mechanism of action, adverse effects, pharmacokinetics and contraindications). The textbook data overlap served as point of reference for the corresponding drug articles in Wikipedia. The analysis was performed separately for the German and English language version of Wikipedia and textbooks. https://doi.org/10.1371/journal.pone.0106930.g001 Assessment of accuracy and completeness Accuracy was defined as the degree of closeness to the pharmacological information of the textbooks overlap. Accuracy of information was evaluated separately for each drug and category and quantified by calculating the number of correct statements in Wikipedia divided by the total number of statements in the textbook overlap. Completeness was defined and calculated as percentage of pharmacological statements of the textbook overlap present in the respective Wikipedia articles. All statements missing in Wikipedia were re-assessed for didactic relevance in the undergraduate medical education of basic pharmacology by six lecturers of pharmacology (teaching experience >5 years) of five different universities in Germany. Didactic relevance was defined as information that is essential for the understanding of pharmacokinetic and pharmacodynamics principles and characteristics of drugs as outlined in the German Medical Licensure Act (Approbationsordnung für Ärzte) [17]. A statement was considered relevant if more than four out of six raters endorsed its relevance for teaching. Assessment of references For each Wikipedia article all references were downloaded (www.de.wikipedia.org; accessed on March 20th 2013; www.en.wikipedia.org; accessed on November 15th 2013) and grouped in the following categories: “academic journals” (publications in peer-reviewed academic journals), “scientific databases and drug info” (official drug databases and prescribing information), “textbooks”, “professional organizations”, “news media” (print or web), “commercial companies” (non-peer reviewed drug information provided by pharmaceutical companies). The total number and reference categories for all drug articles evaluated were pooled and descriptive statistical analysis performed using GraphPad PRISM 5.0 (La Jolla, CA). Number of revisions and editors Data on the total number of edits and editors (registered, anonymous and automated “bot” editors) for each drug article in the German language version of Wikipedia was provided by Wikimedia Foundation (San Francisco, CA). Descriptive statistical analysis was performed using GraphPad PRISM 5.0 (La Jolla, CA) software. Assessment of readability Readability was evaluated using the Amstad readability index, a variant of the Flesch Reading Ease Sore adapted to the analysis of German texts [18]. The Amstad index (R Amstad ) is based on the equation The range of R Amstad is 0 (very difficult) to 100 (very easy). As a second readability index the Erste Wiener Sachtextformel (R 1. WSTF ) was employed [19], which is based on the equation. R 1. WSTF = 0.1935 x MS + 0.1672 x SL + 0.1297 x lW - 0.0327 x OS - 0.875 where MS is the percentage of words with 3 or more syllables, SL the median sentence length (number of words per sentence), IW the percentage of words with> 6 characters, and OS the percentage of words with only one syllable. A low level score of R 1. WSTF is 4 (easy) whereas high levels scores of R 1. WSTF are values>12 (very difficult). For each drug, three text passages ranging between 90 and 120 words were randomly selected in the German language version of Wikipedia and in the textbooks. Text passages were edited to exclude citations, headings or references. R Amstad and R 1. WSTF were calculated using the koRpus text analysis software (http://ripley.psycho.hhu.de/koRpus). Statistics Data are presented as means ± SEM. To test the statistical difference between means of two (t-test) or more groups (one-way ANOVA with Bonferroni correction) GraphPad PRISM 5.0 (La Jolla, CA) software was used. P values <0.05 were considered statistically significant.

Discussion Since its launch in 2001 Wikipedia has become the largest collaborative online encyclopedia and is widely used by both students and educators in undergraduate medical education [4]. In this article we investigated the accuracy and completeness of drug information in Wikipedia in comparison to standard textbooks of basic pharmacology and evaluated its potential for undergraduate medical education of pharmacology. By analyzing 100 randomly selected curricular drugs in the German language version of Wikipedia, we found that the information in Wikipedia contained few factual errors (<0.3%) and comprised the majority (>93%) of curricular drug information considered essential for undergraduate teaching of basic pharmacology. The drug articles in Wikipedia had an average of 262.8±37.4 edits that were performed by an average of 142.7±17.6 editors. Finally, Wikipedia was generally well referenced with an average of 14.6±1.6 citations per article and had a readability that was on par with standard textbooks of pharmacology. Similar results were obtained for the English language version of Wikipedia. Collectively, our results suggest that Wikipedia is an accurate and informative source of drug information for undergraduate medical students. As a collaborative encyclopedia with an open editorial policy, the reliability and accuracy of information of Wikipedia has frequently been the focus of intense debate (reviewed in [9]). An example is the single-blinded study by the journal Nature that compared the accuracy of 42 articles on various scientific topics in Wikipedia and Encyclopedia Britannica [20]. The authors identified a total of four severe errors in each of the encyclopedias. Subsequently, a number of empirical studies have assessed the quality of medical information in Wikipedia pertinent to surgical procedures [21], gastrointestinal conditions [22], cancer types [23] and pathology informatics [24]. Despite different methodologies, the main conclusion of these studies was that Wikipedia articles on health topics contain few errors and are well referenced, while the information provided often lacks depth. Few studies have investigated the quality of pharmacological information in Wikipedia. Clauson et al. compared Wikipedia to the online database Medscape drug reference [10]. By using a set of predetermined questions, the authors found that Medscape provided answers to 82.5%, while Wikipedia could answer only 40%. In particular, Wikipedia articles were missing information on drug dosages, interactions and contraindications. However, it should be noted that the Wikipedia style manual for drug articles discourages detailed dosage information (http://en.wikipedia.org/wiki/Wikipedia:WikiProject_Pharmacology/Style_guide). Kupferberg and co-worker analyzed the accuracy and completeness of pharmacological information of five different statin drugs [11]. The authors concluded that the drug information is less complete, inconsistent and cannot be recommended as sole information source. It is important to note that the aim of the studies by Clauson et al. and Kupferberg and Protus was to evaluate drug information for patients or health practitioners. In contrast, the present study aimed to analyze drug information in Wikipedia from an educators' perspective and therefore used standard textbooks of pharmacology as point of reference. Pharmacological information on drugs for educational purposes differs in focus and detail of pharmacodynamic and -kinetic content. Moreover, Wikipedia is evolving rapidly. Re-evaluating the drug entries assessed by Kupferberg and Protus in 2010, we found the quality of pharmacological information significantly improved (data not shown). Therefore, the major conclusion of the present study that Wikipedia is an accurate and informative source of drug information for undergraduate medical students, is not incompatible with previously reported findings. The ease of updating and editing information is a potential strength of Wikipedia, in particular in rapidly evolving disciplines such as pharmacology. However, this may also bear the risk of errors, vandalism or manipulation. Wikipedia has implemented several safety measures and quality surveillance mechanisms to identify and repair spurious information. These include watch lists for recent changes by volunteer editors, page protection for articles that are likely to attract controversy or vandalism, or automated computer scripts (bots) that detect and repair copyright violations, simple vandalism, or spelling and grammatical errors (reviewed in [9]. The effectiveness of these quality surveillance measures has been evaluated in a study by Viegas and co-worker who analyzed the repair dynamics of substantial rewrites of vandalism in Wikipedia [25]. The authors showed that the majority of such acts were repaired within minutes by the Wikipedia community. Another study reported that 42% of damaged articles were repaired within one viewing, while 11% were still present after 100 viewings [26]. These results suggest that while the quality surveillance is remarkably effective, a residual risk remains. A middle way between a closed expert database and an open editorial system may be platforms such as TUM300 (http://www.tum300.de) that allow anonymous user feedback, yet require editorial approval of edits prior to publication. There are several limitations to this study. In lack of a “golden standard” for the accuracy and comprehensiveness of pharmacological information for undergraduate medical education, we used well established pharmacology textbooks as the point of reference. However, these resources have not been formally evaluated for these criteria. In addition, we analyzed a limited number of drugs in the German and English language version of Wikipedia, and accuracy and completeness of pharmacological information may differ for other drugs. The present study is therefore exploratory in nature and serves as basis for future confirmatory studies with larger sample sizes. It is important to note that pharmacology training of undergraduate medical students is not limited to pharmacodynamic and –kinetic characteristics of drugs. It is not the intention of this study to evaluate the potential of Wikipedia to substitute other learning media, in particular textbooks. Finally, the results of this study cannot be directly extrapolated to other areas of pharmacology (e.g. clinical pharmacology) or to other medical disciplines. Despite these limitations our results underscore that the collaborative and participatory design of Wikipedia does generate high quality information on pharmacology that is suitable for undergraduate medical education. From a didactic point of view, the possibility to actively participate in editing and contributing to information resources is likely to meet the expectations and preferences of today's student generation.

Acknowledgments The authors thank Erik Zachte of Wikimedia foundation for providing statistical data on the number of article editors and revisions in the German Wikipedia. We thank Harald Mückter (Ludwig-Maximilians-University, Munich, Germany) and Christian Mang (Johannes Gutenberg University Mainz, Mainz, Germany) for analyzing of data and critical reading of the manuscript. We are grateful to Alexander Hapfelmeier (Institute for Medical Informatics and Epidemiology, Technische Universität München, Munich, Germany) for expert statistical advice.

Author Contributions Conceived and designed the experiments: JK TMP JG SE AS. Performed the experiments: JK SM TMP JG AS. Analyzed the data: JK TMP SM JG OZ LW RM SE AS. Contributed reagents/materials/analysis tools: JK TMP JG. Wrote the paper: AS SM OZ LW RM SE.