Abstract Boyer’s framework of scholarship was published before significant growth in digital technology. As more digital products are produced by medical educators, determining their scholarly value is of increasing importance. This scoping systematic review developed a taxonomy of digital products and determined their fit within Boyer’s framework of scholarship. We conducted a broad literature search for descriptions of digital products in the medical literature in July 2013 using Medline, EMBASE, ERIC, PSYCHinfo, and Google Scholar. A framework analysis categorized each product using Boyer’s model of scholarship, while a thematic analysis defined a taxonomy of digital products. 7422 abstracts were found and 524 met inclusion criteria. Digital products mapped primarily to the scholarship of teaching (85.4%) followed by integration (7.6%), application (5.5%), and discovery (1.5%). A taxonomy of 19 categories was defined. Web-based or computer assisted learning (41%) was described most frequently. We found that digital products are well described in medical literature and fit into Boyer’s framework of scholarship and proposed a taxonomy of digital products that parallel traditional forms of the scholarship of teaching and learning. This research should inform the development of tools to examine the impact and quality of digital products.

Introduction

In 1990 Boyer redefined the scope of scholarship in higher education with the definition of four overlapping subtypes of scholarship (discovery, integration, application, and teaching) (Boyer 1990). Prior to this redefinition, scholarship was largely considered to consist only of the discovery subtype. Boyer’s influential definition paved the way for the recognition of a broader definition of scholarship that included teaching in addition to research. The explosive growth of digital products (resources used for the dissemination of information that exist primarily in digital formats) that has occurred since the internet was democratized in 1995 could not be predicted at that time (Leiner et al. 2009). Social media, online courses, blogs, podcasts and other digital products have since changed the way we teach, disseminate, and discuss scholarly ideas. Their exclusion from traditional scholarly frameworks, combined with a lack of standards to ensure their quality, may explain why they are generally not viewed as scholarship by members of the academic establishment (Brabazon 2006; Hendricks 2010; Kirkup 2010; Savage 2006).

Scholars and educators are turning to digital methods for disseminating knowledge and reaching students (Priem 2013). This has resulted in the creation of online communities of practice with benefits including: increased collaboration, enhanced knowledge dissemination, instantaneous scholarly discussion, and the generation of scholarly identity (Kirkup 2010; Gruzd, Staves, and Wilk 2010; Maitzen 2012; Shema, Bar-Ilan, and Thelwall 2012). Arguments against digital products note that they have not proven to be superior and that they require more time to develop (Cooke 2014). The increasing prominence of digital products in medical education and the time being devoted to their development makes determining their scholarly value extremely important (Cadogan et al. 2014; Matava et al. 2013; Bahner et al. 2012).

In this scoping review paper, we quantify the increasing prevalence of digital products in the medical literature, develop a taxonomy of digital products, and compare the products in the taxonomy to traditional forms of the scholarship of teaching and learning. We hope that this will increase the awareness of this growing area of educational scholarship and classify digital products so that their value can be understood within the context of their traditional parallels.

Methods

In concert with an expert librarian, an expert search strategy was developed using the Medline, EMBASE, ERIC, and PSYCHinfo databases, as they were deemed to be the most likely to provide literature on digital products used in medical education. The search was not limited by year or language, and used the keywords and keyword variations of: (student, medical OR medical student OR “internship and residency” OR intern OR resident) AND (education, medical OR education, medical, graduate OR education, medical, undergraduate OR “medical education”) AND (blog OR weblog OR microblog OR social media OR social network OR “health 2.0” OR “web 2.0” OR video OR youtube OR podcast OR vodcast OR webcast OR screencast OR wiki OR widget OR new media OR new technology OR mobile app OR app, collaborative OR cooperative behavior OR conferencing OR crowdsource OR RSS OR “really simple syndication” OR computer-assisted instruction OR web-based instruction OR “access to information” OR open access OR free access).

In addition to this traditional literature search, a previously described Google Scholar search methodology (Chan et al. 2012) was conducted for five sets of keywords: “blogging and scholarship,” “digital scholarship medicine medical,” “free open access medical education,” “medical blogging” and “’tenure and promotion blogging.” The first 500 results for each keyword set were reviewed and relevant results were added to the findings.

A title review of the abstracts was performed by one author (BT). Abstracts were excluded if (1) there was no English-language abstract, (2) they were duplicates, or (3) they clearly did not address the use of digital products in medicine. The abstracts were coded and classified with a detailed abstract review conducted by two authors (BT, JB). Upon abstract review, articles were excluded if (1) no particular digital product was described, (2) the digital product did not meet the criteria for scholarship based on Boyer’s model, or (3) upon closer inspection they met the initial exclusion criteria.

During the abstract review, two authors (BT, JB) performed both a framework analysis and thematic analysis of the digital products described in the abstracts. Two reviewers (BT, JB) classified the digital products described in the first 60 abstracts collaboratively to develop an initial taxonomy and set of definitions for the thematic analysis and to calibrate the coding schemes for the thematic and framework analyses. Subsequently a constant comparator technique was used to perform both analyses whereby classifications were made independently in batches of approximately 100 abstracts and compared. The frequent comparisons allowed the reviewers to ensure consistency within the analyses and to refine a consensus definition for each type of digital product in the thematic analysis.

When available and necessary, full manuscripts were reviewed to accurately classify the digital products and their form of scholarship. Discordant classifications were discussed by the reviewers and resolved by consensus when possible. When consensus was not reached, a third reviewer (TC) arbitrated disagreements. The third reviewer also audited the excluded abstracts to ensure that they met the review’s exclusion criteria. The year of publication of each abstract was also recorded to demonstrate the prevalence of digital products described each year.

While they were conducted concurrently, the two analyses were functionally independent. The thematic analysis was used to derive a taxonomy that defined the described all of the digital products found in the literature. Additional items were added to the taxonomy as they were found and the definitions were frequently refined to accurately describe all of the digital products effectively.

The purpose of the framework analysis was to determine if and how digital products fit into Boyer’s four types of scholarship (Boyer, 1990). Digital products were classified as one or more of Boyer’s types of scholarship: discovery (original research for the advancement of knowledge), integration (contextualizing information across disciplines or into larger intellectual patterns), application (applying knowledge dynamically to inform and test new theories in an engaged fashion), and/or teaching (systematic study of teaching and learning in the presence of learners) (Gale et al. 2013; Boyer 1990). The intraclass correlation coefficient was calculated to determine a measure of agreement.

The definitions resulting from the thematic analysis were assessed to determine if there were traditional scholarly products used for the same purpose. This comparison, while inherently subjective, was conducted to further contextualize the role of each type of digital product.

Results

The flow diagram for the literature search, title review, and abstract review is presented in Figure 1. The thematic and framework analyses were conducted on digital products described by the 524 abstracts that met the inclusion criteria. An abstract published in 1974 described the oldest digital product.

Figure 1. Diagram illustrating the number of articles excluded through the title and abstract reviews.

The number of digital products described in the published medical literature between 1974 and July 2013 is illustrated in Figure 2. The number of digital products for 2013 was projected to double because our literature search only included articles published through July 2013.

Figure 2. The number of digital products described in the medical literature over time.

Framework Analysis

Table 1 presents the results of the analysis mapping published digital products to Boyer’s framework of scholarship.(Boyer 1990) The intraclass correlation between the raters was 0.65, but disagreements were ultimately discussed to resolve consensus. Most products (85.4%) were categorized under the scholarship of teaching. The scholarship of integration (7.6%), application (5.5%), and discovery (1.5%) were described much less frequently. This table further stratifies these scholarship models based on the 19 categories of digital products, as derived by our thematic analysis. Of note, there were some products that could be classified as more than one type of scholarship.

Table 1: Types and numbers of digital products mentioned in the literature and classified using Boyer's Framework of Scholarship

Digital Product Discovery

(%) Integration

(%) Application

(%) Teaching

(%) Total Web-based or computer assisted learning 0 (0) 1 (0.5) 2 (0.9) 214 (98.6)* 217 Multi-modal products 0 (0) 7 (14.0) 3 (6.0) 40 (80.0)* 50 Social network 0 (0) 13 (38.2)* 8 (23.5) 13 (38.2)* 34 Instructional video 0 (0) 0 (0) 2 (6.5) 29 (93.5) 31 Online repository 0 (0) 8 (27.6) 4 (13.8) 17 (58.6)* 29 Podcast 0 (0) 0 (0) 0 (0) 28 (100) 28 Online course 0 (0) 0 (0) 0 (0) 27 (100)* 27 Video podcast 0 (0) 0 (0) 2 (7.5) 25 (92.6) 27 Blog 0 (0) 4 (15.4) 3 (11.5) 19 (73.1)* 26 Open access journal 8 (50)* 7 (43.8) 0 (0) 1 (6.3) 16 Wiki 0 (0) 0 (0) 0 (0) 16 (100)* 16 Website 0 (0) 1 (6.7) 2 (13.3) 12 (80)* 15 Online discussion board 0 (0) 0 (0) 0 (0) 8 (100)* 8 E-mail 0 (0) 0 (0) 0 (0) 7 (100)* 7 Application ("app") 0 (0) 0 (0) 3 (75) 1 (25) 4 Online textbook 0 (0) 0 (0) 0 (0) 2 (100)* 2 Virtual reality 0 (0) 0 (0) 0 (0) 2 (100)* 2 Search engine 0 (0) 0 (0) 1 (100) 0 (0) 1 Serious game 0 (0) 0 (0) 0 (0) 1 (100)* 1 Total 8 (1.5) 41 (7.6) 30 (5.5) 462 (85.4) 541

In table 1, the starred numbers represent the most popular type of scholarship for each product. The table includes 17 abstracts that were classified as multiple forms of scholarship, resulting in totals (541) greater than the number of abstracts reviewed (524).

Thematic Analysis

Table 2 provides a taxonomy of the digital products described in the literature and derived from the thematic analysis. Each of the 19 categories are defined with an example provided. Together, web-based learning and computer assisted learning (41%) were the most prevalent forms of digital product. A single category was created for these two types of digital products because prior to the democratization and widespread accessibility of the internet, web-based learning products were classified under the umbrella term of computer assisted learning. The significant overlap between these two terms necessitated their amalgamation into one category in our taxonomy. Social networks, instructional videos, online repositories, podcasts, online courses, video podcasts (also known as screencasts or vodcasts), and blogs had roughly similar prevalence and collectively comprised another 37% of the publications.

Table 2: Definitions and examples of digital products.

Digital Product Definition Example Applications (‘apps’) A resource downloaded to a smartphone. iRash is an application that allows users to search and learn about various rashes (Deveau and Chilukuri 2012) Blog A website used to publish information in periodic posts that are primarily text-based. A blog was created to host synopses of ‘morning report’ sessions run by chief medical residents (Bogoch et al. 2012) E-mail A common form of direct electronic messaging between a sender and one or more recipients. E-mail was used to send questions to teach residents about pediatric emergency medicine (Komoroski 1998) Instructional Video A video demonstrating a skill (ie procedure, physical exam finding, ECG or x-ray interpretation, etc). Instructional video used to teach chest tube insertion (Davis et al. 2012) Multi-modal products A product that consists of multiple digital products. An online course on evidence based medicine and critical appraisal that used video podcasts, a wiki and blogs (Tam and Eastwood 2012) Online Course A complete curriculum delivered using multiple online modalities. Differs from multi-modal products in that it is organized into a formal curriculum. The Online Genetic Testing Curriculum is a course about the ethical, legal, and social implications of genetic testing and counseling (Metcalf, Tanner, and Buchanan 2010) Online Discussion Board An online forum that allows users to post and respond to other participants. A clinical discussion board for learners to describe their rural medicine experiences (Baker, Eley, and Lasserre 2005) Online Repository An online database that resources can be drawn from and added to. A repository of images of dermatologic findings in darker-skinned patients (Ezzedine et al. 2008) Online Textbook A textbook published online. ODITEB1 (Open Distributed Text Book), an online textbook that describes the diagnosis of gastrointestinal tumours (Horsch et al. 1998) Open Access Journal A journal only available online that publishes articles without access restrictions. Various online journals have been created to decrease cost and allow open-access publication of scientific materials (Davis and Walters 2011) Podcast Audio recordings that are published periodically with the intent of disseminating knowledge. Surgery 101 podcasts are used to teach core principles to clinical clerks on their surgical rotation (White, Sharma, and Boora 2011) Search Engine Search engines used to find information online. Google, Yahoo, Dogpile, Altavista, Metacrawlers and Ask were used to find information on scleroderma renal crisis (Akbar and Yacyshyn 2009) Serious Game An online game designed to educate the players. eMedOffice, a serious game to teach practice management.(Hannig et al. 2012) Social Network An online platform that allows synchronous and asynchronous communication between individuals. Twitter used to connect teachers with learners (Forgie, Duff, and Ross 2013) Video Podcast Videos with embedded audio that are published periodically. Differs from instructional videos because it focuses on knowledge rather than skill. Video podcasts used to teach embryology (Evans 2011)



Virtual Reality A virtual environment used to present learning material. A virtual reality simulator was used to simulate medical cases (Alverson et al. 2008) Web Based Learning or Computer Assisted Learning Educational modules that may make use of multiple modalities. Web-based learning is based online while computer assisted learning is not. These modalities were combined due to substantial overlap. A web based module on pediatric pain management (Ameringer et al. 2012) A computer based application about occupational lung disease (Bresnitz, Gracely, and Rubenstein 1992) Website An online webpage that cannot be classified as any other digital product. Case Based Pediatrics is a website with a list of teaching cases for medical students and residents (Falagas, Karveli, and Panos 2007) Wiki A website that can be openly edited by end-users. Utilizes crowd-sourcing as a method for improving and revising the content. A wiki site for orthopedic cases, utilizes a scoreboard to encourage participation (Ma et al. 2008)

Historical Parallels

As demonstrated by our framework analysis, digital products can be classified within the types of scholarship described by Boyer (Boyer 1990) and most fall under teaching and learning. Following the completion of our thematic analysis, the definitions of the digital products were compared with traditional forms of the scholarship of teaching and learning. Table 3 outlines the parallels between traditional products and 18 of the 19 digital products described in the thematic analysis. No product was found that was comparable to the digital product ‘virtual reality.’

Table 3: Comparing traditional products used for the scholarship of teaching and learning to digital products that are used for this purpose

Types of teaching and learning resources Examples of

Traditional Products Examples of

Digital Products Interactive resources Small groups

Workshops Online discussion board

Social network

Wiki Independent study resources Assignments

Discussions with tutors

Group work

Laboratory work



E-mail

Online course

Serious game

Virtual reality

Web based and Computer assisted learning Audiovisual resources Lecture

Skill demonstration Podcast

Video podcast

Instructional video Point-of-care resources Guidebooks

Pocketbooks Applications (‘apps’) Written resources Textbook

Printed journals

Medical journalism Online textbook

Blog

Open access journal

Website Resource repository Library

Library classification system Online repository

Search engine

Discussion

The growing number of digital products documented in the literature (Figure 1 and 2) suggests that medical educators are increasingly using technology to engage in various forms of scholarship. While educators have discussed applying Boyer’s traditional definitions of scholarship to digital products (Heap and Minocha 2012; Pearce et al. 2010), we provide the first comprehensive framework analysis of these products.

Our framework analysis found that, following teaching and learning, integration (7.6%), application (5.5%), and discovery (1.5%) were the most frequent types of scholarship found in digital products. We suspect that the digital products were predominantly consistent with scholarship of teaching and learning because, despite Boyer’s reclassification of scholarship, educators have traditionally not had their scholarly contributions recognized. Literature that assesses their innovations is one way to receive academic recognition for their work. Educators should keep in mind that digital products can be scholarly outside of their traditional realm of teaching. For example, Boyer’s concept of application was demonstrated by the various ‘apps’ that allow translation of concepts at the point of care (Graber, Tompkins, and Holland 2009), integration was illustrated by an online textbook that synthesized multiple resources into a single resource (Horsch et al. 1998), and discovery was exemplified by open access online journals that fostered new scientific works (P. M. Davis and Walters 2011). Social networks were the most versatile product with multiple examples of their use in teaching, application, and integration.

The thematic analysis described the diversity of digital products (Table 2). Notably, web-based and computer assisted learning programs were prominently featured in the literature and there has been a recent uptake of social media (Nickson and Cadogan 2014; Cadogan et al. 2014). Social networks, in particular, seem to have impacted medical education by allowing scholars to share their digital products (Boulos, Maramba, and Wheeler 2006).

A traditional parallel was found for nearly every digital product defined in the thematic analysis. The use of digital products was particularly prominent for the scholarship of teaching and learning. This may be because of their reach, customization, and updatability. Whereas scholarly teaching was historically a fleeting event offered to a defined group (i.e. an address that was given in a lecture hall), digital products extend their reach to large numbers of learners who can access them at their convenience. This asynchrony allows learners to customize their experience (i.e. by speeding up or slowing down a lecture) and educators to update their products as needed.

That said, there is no compelling evidence that digital products are more effective for learning and they may take more time and resources to develop than traditional products (Cooke 2014). They have also been criticized for their lack of editorial oversight and review (Brabazon 2006; Kirkup 2010). These limitations may limit their widespread endorsement and utilization. Further research will be required to determine when and how they should be used.

While our results suggest that this research is increasingly being conducted, the role and value of digital products in our current academic schema for scholarship remains poorly defined, and hence, poorly acknowledged. Institutions that do acknowledge digital products as scholarship for the purpose of promotion and tenure decisions have difficulty classifying them and quantifying their value relative to other scholarly pursuits (Gruzd, Staves, and Wilk 2010; Cheverie, Boettcher, and Buschman 2009; Rockwell 2011; Ruiz, Mintzer, and Leipzig 2006). Novel ways to recognize digital products include publishing them on a platform with peer review and publication processes such as MedEdPORTAL (Ruiz, Mintzer, and Leipzig 2006; Reynolds and Candler, Christopher 2008) or conducting educational research to evaluate their efficacy (Cheston, Flickinger, and Chisolm 2013). Regardless, the amount of academic recognition for digital products is relatively low compared to the effort expended to build and maintain them and may limit their growth in the future (Anderson et al. 2013; ProfHacker 2012).

Limitations

While our literature search was intended to be as broad as possible, it is still likely that some digital products were missed since they may not have been reported in the literature. A broader review of grey and non-English literature would not have been feasible given the sheer volume of unreported products. For example, a recent report found that there were 183 English-language blogs and podcasts in emergency medicine alone (Cadogan et al. 2014). Additionally, we may have missed digital products of historic significance that were described using terms that are not applicable today. For example, CD-ROM’s were likely to have been considered digital products in the past but were not included in our literature search. Missing resources would change the number of products per year represented in Figure 2 and made our taxonomy of digital products incomplete.

The exclusion of the MedEdPORTAL database could also be considered a limitation as it publishes many digital products. However, our search explicitly attempted to quantify and describe the digital products described in the literature. MedEdPORTAL’s publications are digital products, rather than descriptions of them, and for this reason they were considered to be outside of the scope of this review.

Finally, our quantification of the rapidly increasing number of digital products described annually in the literature fails to account for the increase in literature that has been published in general (Larsen, 2010). Unfortunately, we were unable accurately quantify this growth for the body of literature that our review assessed. As the amount of research published annually is increasing (Larsen, 2010), the increase in descriptions of digital products would have been less spectacular had we been able to take this into account.

Future Directions

Since the digital products described in the medical literature fit within Boyer’s framework, we feel strongly that they should be considered alongside other forms of scholarship. However, given the ease with which some products can be created, better evaluation tools will need to be developed to determine their quality, value, and relative impact. Educator portfolios are becoming accepted as a way to provide additional detail to the traditional curriculum vitae, which sub-optimally captures the scholarly efforts of educators (Simpson et al. 2007; Baldwin, Chandran, and Gusic 2011).

In showing that digital products fall within Boyer’s framework of scholarship, our findings suggest that we should look to apply other conceptual frameworks of educational scholarship to digital products or online educational resources. Frequently, educators lean towards the criteria for assessing scholarship developed by Glassick. Assessment frameworks such as Glassick’s criteria of scholarship are manifest in the AAMC Toolbox for Evaluating Educators and could be used to evaluate these portfolios (Glassick 2000; Gusic et al. 2013). Table 3 suggests multiple parallels between traditional and digital projects for teaching and learning that could guide how digital products should fit into these portfolios. Developing a standardized approach would allow promotion committees and administrative leadership to evaluate digital and traditional educational efforts more rigorously.

Together, Boyer and Glassick’s respective frameworks provide a roadmap for educators interested in scholarship. Digital scholars must take care to ensure that their digital products warrant scholarly respect by ensuring that they stand up to the scrutiny of these recognized conceptual frameworks.

Conclusion

Digital products are increasingly being described in the medical literature. They are likely to have a substantial impact on medical education and can readily fit into Boyer’s established framework of scholarship. Our taxonomy shows clear parallels between digital and traditional products and can hopefully provide a framework for further research on digital scholarship.

References

Akbar, S, and E Yacyshyn. 2009. “Is There Relevant Information about Scleroderma Renal Crisis on Most Frequently Visited Internet Search Engines?” Journal of Rheumatology 36 (11): 2563–64.

Alverson, Dale C, Stanley M Saiki, Summers Kalishman, Marlene Lindberg, Stewart Mennin, Jan Mines, Lisa Serna, et al. 2008. “Medical Students Learn over Distance Using Virtual Reality Simulation.” Simulation in Healthcare : Journal of the Society for Simulation in Healthcare 3 (1): 10–15. doi:10.1097/SIH.0b013e31815f0d51.

Ameringer, Suzanne, Deborah Fisher, Sue Sreedhar, Jessica M Ketchum, and Leanne Yanni. 2012. “Pediatric Pain Management Education in Medical Students: Impact of a Web-Based Module.” Journal of Palliative Medicine 15 (9): 978–83. doi:10.1089/jpm.2011.0536.

Anderson, Michael G, Donna D Alessandro, Dawn Quelle, Rick Axelson, Lois J Geist, and Donald W Black. 2013. “Recognizing Diverse Forms of Scholarship in the Modern Medical College”, 120–25. doi:0.5116/ijme.51b4.730c.

Bahner, David P, Eric Adkins, Nilesh Patel, Chad Donley, Rollin Nagel, and Nicholas E Kman. 2012. “How We Use Social Media to Supplement a Novel Curriculum in Medical Education.” Medical Teacher 34 (6): 439–44. doi:10.3109/0142159X.2012.668245.

Baker, Peter G, Diann S Eley, and Kaye E Lasserre. 2005. “Tradition and Technology: Teaching Rural Medicine Using an Internet Discussion Board.” Rural and Remote Health 5 (4): 435. http://www.ncbi.nlm.nih.gov/pubmed/16305277.

Baldwin, Constance, Latha Chandran, and Maryellen Gusic. 2011. “Guidelines for Evaluating the Educational Performance of Medical School Faculty: Priming a National Conversation.” Teaching and Learning in Medicine 23 (3): 285–97. doi:10.1080/10401334.2011.586936.

Bogoch, Isaac I, David W Frost, Suzanne Bridge, Todd C Lee, Wayne L Gold, Daniel M Panisko, and Rodrigo B Cavalcanti. 2012. “Morning Report Blog: A Web-Based Tool to Enhance Case-Based Learning.” Teaching and Learning in Medicine 24 (3): 238–41. doi:10.1080/10401334.2012.692273.

Boulos, Maged N Kamel, Inocencio Maramba, and Steve Wheeler. 2006. “Wikis, Blogs and Podcasts: A New Generation of Web-Based Tools for Virtual Collaborative Clinical Practice and Education.” BMC Medical Education 6 (January): 41. doi:10.1186/1472-6920-6-41.

Boyer, E. 1990. “Scholarship Reconsidered: Priorities of the Professoriate” The Carnegie Foundation for the Advancement of Teaching: Princeton, NJ.

Brabazon, T. (2006). The Google effect: Googling, blogging, wikis and the flattening of expertise. Libri, 56(3), 157-167. doi:10.1515/LIBR.2006.157

Bresnitz, Eddy A, Edward J Gracely, and Harriet L Rubenstein. 1992. “A Randomized Trial to Evaluate a Computer-Based Learning Program in Occupational Lung Disease.” Journal of Occupational and Environmental Medicine 34 (4). http://journals.lww.com/joem/Fulltext/1992/04000/A_Randomized_Trial_to_Evaluate_a_Computer_Based.17.aspx.

Cadogan, M., B. Thoma, T. M. Chan, and M. Lin. 2014. “Free Open Access Meducation (FOAM): The Rise of Emergency Medicine and Critical Care Blogs and Podcasts (2002-2013).” Emergency Medicine Journal, (February). doi:10.1136/emermed-2013-203502.

Chan, Teresa M, Clare Wallner, Thomas K Swoboda, Katrina a Leone, and Chad Kessler. 2012. “Assessing Interpersonal and Communication Skills in Emergency Medicine.” Academic Emergency Medicine 19 (12): 1390–1402. doi:10.1111/acem.12030.

Cheston, Christine C, Tabor E Flickinger, and Margaret S Chisolm. 2013. “Social Media Use in Medical Education: A Systematic Review.” Academic Medicine : Journal of the Association of American Medical Colleges 88 (6): 893–901. doi:10.1097/ACM.0b013e31828ffc23.

Cheverie, Joan F., Jennifer Boettcher, and John Buschman. 2009. “Digital Scholarship in the University Tenure and Promotion Process: A Report on the Sixth Scholarly Communication Symposium at Georgetown University Library.” Journal of Scholarly Publishing 40 (3): 219–30. doi:10.1353/scp.0.0044.

Cooke, David. 2014. “Futurecasting in education technologies: Fun new toys and a reality check.” International Conference on Residency Education, Plenary Session. Available at: https://www.youtube.com/watch?v=XCodjukPUEc&list=UU75Z-Vvzoq41cVWMVDzsh3A. Retrieved November 25, 2014.

Davis, James S, George D Garcia, Mary M Wyckoff, Salman Alsafran, Jill M Graygo, Kelly F Withum, and Carl I Schulman. 2012. “Use of Mobile Learning Module Improves Skills in Chest Tube Insertion.” The Journal of Surgical Research 177 (1). Elsevier Ltd: 21–26. doi:10.1016/j.jss.2012.03.022.

Davis, Philip M, and William H Walters. 2011. “The Impact of Free Access to the Scientific Literature: A Review of Recent Research.” Journal of the Medical Library Association : JMLA 99 (3): 208–17. doi:10.3163/1536-5050.99.3.008.

Deveau, Michael, and Suneel Chilukuri. 2012. “Mobile Applications for Dermatology.” Seminars in Cutaneous Medicine and Surgery 31 (3). Elsevier Inc. 174–82. doi:10.1016/j.sder.2012.06.001.

Evans, Darrell J R. 2011. “Using Embryology Screencasts: A Useful Addition to the Student Learning Experience?” Anatomical Sciences Education 4 (2). Wiley Subscription Services, Inc., A Wiley Company: 57–63. doi:10.1002/ase.209.

Ezzedine, K, a Amiel, P Vereecken, T Simonart, B Schietse, K Seymons, B S Ndiaye, et al. 2008. “Black Skin Dermatology Online, from the Project to the Website: A Needed Collaboration between North and South.” Journal of the European Academy of Dermatology and Venereology : JEADV 22 (10): 1193–99. doi:10.1111/j.1468-3083.2008.02762.x.

Falagas, Matthew E, Efthymia a Karveli, and George Panos. 2007. “Infectious Disease Cases for Educational Purposes: Open-Access Resources on the Internet.” Clinical Infectious Diseases : An Official Publication of the Infectious Diseases Society of America 45 (4): 495–500. doi:10.1086/519977.

Forgie, Sarah Edith, Jon P Duff, and Shelley Ross. 2013. “Twelve Tips for Using Twitter as a Learning Tool in Medical Education.” Medical Teacher 35 (1): 8–14. doi:10.3109/0142159X.2012.746448.

Gale, Nicola K, Gemma Heath, Elaine Cameron, Sabina Rashid, and Sabi Redwood. 2013. “Using the Framework Method for the Analysis of Qualitative Data in Multi-Disciplinary Health Research.” BMC Medical Research Methodology 13 (1). BMC Medical Research Methodology: 117. doi:10.1186/1471-2288-13-117.

Glassick, Charles E. 2000. “Boyer’s Expanded Definitions of Scholarship, the Standards for Assessing Scholarship, and the Elusiveness of the Scholarship of Teaching.” Academic Medicine 75(9), 877–80. doi:10.1097/00001888-200009000-00007

Graber, Mark L, David Tompkins, and Joanne J Holland. 2009. “Resources Medical Students Use to Derive a Differential Diagnosis.” Medical Teacher 31: 522–27. doi:10.1080/01421590802167436.

Gruzd, Anatoliy, Kathleen Staves, and Amanda Wilk. 2010. “Tenure and Promotion in the Age of Online Social Media.” Proceedings of the American Society for Information Science and Technology 48 (1): 1–9. doi:10.1002/meet.2011.14504801154.

Gusic, M, J Amiel, C Baldwin, L Chandran, R Fincher, B Mavis, P O’Sullivan, et al. 2013. “Using the AAMC Toolbox for Evaluating Educators: You Be the Judge!” MedEdPORTAL. doi:10.15766/​mep_2374-8265.9313.

Hannig, Andreas, Nicole Kuth, Monika Özman, Stephan Jonas, and Cord Spreckelsen. 2012. “eMedOffice: A Web-Based Collaborative Serious Game for Teaching Optimal Design of a Medical Practice.” BMC Medical Education 12 (January): 104. doi:10.1186/1472-6920-12-104.

Heap, Tania, and Shailey Minocha. 2012. “An Empirically Grounded Framework to Guide Blogging for Digital Scholarship.” Research in Learning Technology 20 (August). doi: 10.3402/rlt.v20i0.19195.

Hendricks, Arthur. 2010. “Bloggership, or Is Publishing a Blog Scholarship? A Survey of Academic Librarians.” Library Hi Tech 28 (3): 470–77. doi:10.1108/07378831011076701.

Horsch, A., P. Hellerhoff, M. Hogg, H. Ahlbrink, T. Balbacha, Liss. T., K. Minov, and P. Gerhardt. 1998. “Concepts of a Web-Based Open Distributed Textbook for the Multimodal Diagnostics of Gastrointestinal Tumours with MRI, CT and Video-Endoscopy Addressing Students of Medicine and Students of Medical Informatics as Two Different Target Groups.” Studies in Health Technology and Informatics 52 (2): 793–97.

Kirkup, Gill. 2010. “Academic Blogging: Academic Practice and Academic Identity.” London Review of Education 8 (1): 75–84. doi:10.1080/14748460903557803.

Komoroski, E M. 1998. “Use of E-Mail to Teach Residents Pediatric Emergency Medicine.” Archives of Pediatrics & Adolescent Medicine 152 (11): 1141–46. doi:10.1001/archpedi.152.11.1141

Larsen, P. O., & von Ins, M. (2010). The rate of growth in scientific publication and the decline in coverage provided by Science Citation Index. Scientometrics, 84(3), 575-603. doi:10.1007/s11192-010-0202-z.

Leiner, Barry M, David D Clark, Robert E Kahn, Leonard Kleinrock, Daniel C Lynch, Jon Postel, Larry G Roberts, and Stephen Wolff. 2009. “A Brief History of the Internet.” Computer Communication Review 39 (5): 22–31. doi:10.1145/1629607.1629613

Ma, Zhen-Sheng, Hong-Ju Zhang, Tao Yu, Gang Ren, Guo-Sheng Du, and Yong-Hua Wang. 2008. “Orthochina.org: Case-Based Orthopaedic Wiki Project in China.” Clinical Orthopaedics and Related Research 466 (10): 2428–37. doi:10.1007/s11999-008-0396-z.

Maitzen, Rohan. 2012. “Scholarship 2.0: Blogging And/as Academic Practice.” Journal of Victorian Culture 17 (3): 348–54. doi:10.1080/13555502.2012.689502.

Matava, Clyde T, Derek Rosen, Eric Siu, and Dylan M Bould. 2013. “eLearning among Canadian Anesthesia Residents: A Survey of Podcast Use and Content Needs.” BMC Medical Education 13 (January): 59. doi:10.1186/1472-6920-13-59.

Metcalf, Mary P, T Bradley Tanner, and Amanda Buchanan. 2010. “Effectiveness of an Online Curriculum for Medical Students on Genetics, Genetic Testing and Counseling.” Medical Education Online 15 (January): 1–12. doi:10.3402/meo.v15i0.4856.

Nickson, Christopher P, and Michael D Cadogan. 2014. “Free Open Access Medical Education (FOAM) for the Emergency Physician.” Emergency Medicine Australasia 26 (1): 76–83. doi:10.1111/1742-6723.12191.

Pearce, Nick, Martin Weller, Eileen Scanlon, and Melanie Ashleigh. 2010. “Digital Scholarship Considered: How New Technologies Could Transform Academic Work.” In Education 16 (1). http://ineducation.couros.ca/index.php/ineducation/article/view/44/508.

Priem, Jason. 2013. “Beyond the Paper.” Nature 495: 437–40. doi:10.1038/495437a.

Koh, Adeline. 2012. “The Challenges of Digital Scholarship.” Chronicle of Higher Education. Retrieved from http://chronicle.com/blogs/profhacker/the-challenges-of-digital-scholarship/38103 on D

ecember 10, 2014.

Reynolds, Robby J., and S. Candler, Christopher. 2008. “MedEdPORTAL : Educational Scholarship for Teaching.” Journal of Continuing Education in the Health Professions 28 (2): 91–94. doi:10.1002/chp.

Rockwell, Geoffrey. 2011. “On the Evaluation of Digital Media as Scholarship.” Profession 2011 1: 152–68. doi:10.1632/prof.2011.2011.1.152.

Ruiz, Jorge G, Michael J Mintzer, and Rosanne M Leipzig. 2006. “The Impact of E-Learning in Medical Education.” Academic Medicine 81 (3): 207–12. doi:10.1097/00001888-200603000-00002

Savage, William W. 2006. “The Transom: You Can’t Spill Mustard on a Blog.” Journal of Scholarly Publishing 38 (1): 47–51. doi:10.1353/scp.2007.0005.

Shema, Hadas, Judit Bar-Ilan, and Mike Thelwall. 2012. “Research Blogs and the Discussion of Scholarly Information.” PloS One 7 (5): e35869. doi:10.1371/journal.pone.0035869.

Simpson, Deborah, Ruth-Marie E Fincher, Janet P Hafler, David M Irby, Boyd F Richards, Gary C Rosenfeld, and Thomas R Viggiano. 2007. “Advancing Educators and Education by Defining the Components and Evidence Associated with Educational Scholarship.” Medical Education 41 (10): 1002–9. doi:10.1111/j.1365-2923.2007.02844.x.

Tam, Chun Wah Michael, and Anne Eastwood. 2012. “Available, Intuitive and Free! Building E-Learning Modules Using Web 2.0 Services.” Medical Teacher 34 (12): 1078–80. doi:10.3109/0142159X.2012.731105.

White, J S, N Sharma, and P Boora. 2011. “Surgery 101: Evaluating the Use of Podcasting in a General Surgery Clerkship.” Medical Teacher 33 (11): 941–43. doi:10.3109/0142159X.2011.588975.