Although some studies have shown that cognitive training can produce improvements to untrained cognitive domains (far transfer), many others fail to show these effects, especially when it comes to improving fluid intelligence. The current study was designed to overcome several limitations of previous training studies by incorporating training expectancy assessments, an active control group, and “Mind Frontiers,” a video game-based mobile program comprised of six adaptive, cognitively demanding training tasks that have been found to lead to increased scores in fluid intelligence (Gf) tests. We hypothesize that such integrated training may lead to broad improvements in cognitive abilities by targeting aspects of working memory, executive function, reasoning, and problem solving. Ninety participants completed 20 hour-and-a-half long training sessions over four to five weeks, 45 of whom played Mind Frontiers and 45 of whom completed visual search and change detection tasks (active control). After training, the Mind Frontiers group improved in working memory n-back tests, a composite measure of perceptual speed, and a composite measure of reaction time in reasoning tests. No training-related improvements were found in reasoning accuracy or other working memory tests, nor in composite measures of episodic memory, selective attention, divided attention, and multi-tasking. Perceived self-improvement in the tested abilities did not differ between groups. A general expectancy difference in problem-solving was observed between groups, but this perceived benefit did not correlate with training-related improvement. In summary, although these findings provide modest evidence regarding the efficacy of an integrated cognitive training program, more research is needed to determine the utility of Mind Frontiers as a cognitive training tool.

Competing interests: Charles Dickens is employed by Aptima, Inc., which developed the Mind Frontiers game in collaboration with University of Illinois researchers Arthur Kramer and Pauline Baniqued. A version of Mind Frontiers is currently in use in studies at the University of Illinois and Florida State University. Alexandra Geyer was employed by Aptima, Inc. during the development and implementation of the current study. She is no longer affiliated with the company. There are no patents, further products in development or marketed products to declare. This does not alter the authors’ adherence to all the PLOS ONE policies on sharing data and materials.

Funding: This research was supported by the Office of Naval Research (N00014-12-C-0360 to AFK). PLB was supported by a National Science Foundation Neuroengineering IGERT Fellowship (grant no. 0903622) and a Beckman Institute Graduate Fellowship. NMW was supported by a Beckman Institute Postdoctoral Fellowship. Charles Dickens is employed by Aptima, Inc. and Alexandra Geyer was employed by Aptima, Inc. during the development and implementation of the current study. Aptima, Inc. provided support in the form of salaries for authors CD and AG, research materials to develop the Mind Frontiers game application (including the software used to record gameplay and the hardware used by the Mind Frontiers participants during the training sessions), and technical assistance for the Mind Frontiers training games (CD), but did not have any additional role in the study design, data collection and analysis, decision to publish, or preparation of the manuscript. The specific roles of these authors are articulated in the ‘author contributions’ section.

Copyright: © 2015 Baniqued et al. This is an open access article distributed under the terms of the Creative Commons Attribution License , which permits unrestricted use, distribution, and reproduction in any medium, provided the original author and source are credited

It is to be noted that while improving reasoning/Gf abilities is a main goal of the study, we hypothesize that training with Mind Frontiers may also lead to benefits in related abilities, such as attentional control and perceptual or processing speed. As these abilities are often inter-related in the literature [ 59 – 62 ], we hypothesize that the Mind Frontiers group will also show improvements in “lower-level abilities” of selective attention, divided attention, and perceptual speed, especially given the speeded and game-like implementation of the tasks. Furthermore, reasoning/Gf ability has been shown to be relatively stable in young adulthood [ 63 – 65 ], whereas other skills that are also recruited in reasoning/Gf games may be more malleable or sensitive to training.

As expectancy effects are a significant issue in cognitive training studies, we used a questionnaire to assess perceived improvement and other biases that may contribute to a placebo training effect [ 29 , 30 ]. We also employed multiple transfer tests to allow analysis at the construct level and better generalize findings to improvement in cognitive abilities. We used a set of established measures from the Virginia Cognitive Aging Project Battery [ 59 ], which is comprised of tasks validated to assess key cognitive abilities including reasoning/Gf, episodic memory, and perceptual speed. In addition, we administered neurocognitive tests to ensure comprehensive assessment of the training effects, including multiple tests of working memory, selective attention, divided attention, and task switching.

To better attribute any training-related improvements to the Mind Frontiers program, the current study employed an active control group that also involved interaction with a mobile device and multiple adaptive training games. For the active control group, we used visual and perceptual training tasks that have been shown to produce improvements in the performance of these tasks but not improvements in working memory and reasoning/Gf tests. This included three variants of a visual search paradigm previously used as an active control task in a working memory training study [ 32 ] and three variants of a change detection task that was shown not to transfer to untrained tasks [ 58 ].

Several studies employing a multiple-task training approach, often using more complex tasks or games, show promise in engendering transfer beyond the specific trained tasks [ 14 , 51 – 55 ] (but see [ 46 , 48 , 56 – 57 ]). To maximize training benefits in the current study, we employ working memory, reasoning, and task-switching training tasks similar to those previously mentioned, which have shown promise in enhancing working memory and reasoning/Gf, abilities that highly overlap in the psychometric literature. We integrate six of these tasks into a mobile training platform called “Mind Frontiers,” which modifies the surface features of the training tasks (i.e., their appearance) to unify them into a Wild West-themed game. All tasks were programmed to be adaptive in difficulty, and a scoring/reward system was added to the game to promote engagement for the duration of training, which consisted of 20 hour-and-a-half-long sessions, with each game played for approximately 12 minutes.

Improvements in reasoning/Gf have been found in several studies that employ working memory training [ 20 , 21 ], task switching training [ 22 ], and reasoning training [ 14 , 23 ], while improvements in working memory are primarily found in training studies that use working memory training tasks ([ 9 , 17 , 20 , 24 – 28 ]). Although promising, several of these experiments, which were conducted on different age groups from children to older adults, face methodological shortcomings involving small sample sizes, single tests of cognitive transfer, and the lack of a comparable active control group [ 29 – 31 ]. Training-related improvement from the dual n-back working memory paradigm for example, has often not been replicated in other laboratories [ 32 – 35 ] (but see [ 36 , 37 ]). Recent meta-analyses and reviews differ in their conclusions on the benefit of working memory training and highlight the implications of the aforementioned methodological issues [ 38 – 42 ]. More broadly, computer-based training paradigms, from video games to laboratory-based regimens, yield improvement in the trained tasks but limited transfer to other related abilities, including those similar to the trained tasks [ 14 , 23 , 43 – 50 ]. Thus, although behavioral and neural changes can be observed from training, these changes have not been shown to consistently translate to meaningful improvements outside of the training paradigm.

Cognitive training is not a new concept, despite the surge in “brain training” applications that capitalize on the marketability of programs informed by “neuroplasticity” research [ 1 ]. In any activity, prolonged experience or practice leads to proficiency in that specific process, or skilled behavior. More recently, there has been increased interest in developing training programs that lead to improvement in or “transfer” to a wider array of cognitive abilities or exercises beyond the specific task trained. In the psychology literature, this line of research is coined “cognitive training” [ 2 – 4 ] and is often associated with the goal to enhance cognition or ameliorate the age-related decline of cognitive abilities such as working memory, reasoning, and fluid intelligence (Gf), abilities that have been shown to be predict performance in academic and workplace settings [ 5 – 7 ]. Developmental researchers also employ computerized training programs in hopes of improving cognitive abilities in children [ 8 – 13 ], including those from disadvantaged backgrounds [ 14 ] and those with learning difficulties [ 15 – 19 ].

Methods

Participants Participants were recruited from the University of Illinois campus and Champaign-Urbana community through flyers and online postings advertising participation in a “cognitive training study.” Pre-screening for demographic information (e.g., sex, education, English language proficiency) and game experience was administered using a survey completed over email. A few general game experience questions in the survey were embedded with other activity questions that included the Godin Leisure-Time Exercise Questionnaire [66]. More detailed information about game play experience, history and habits were queried in a post-experiment survey. Upon passing pre-screening, an experimenter followed-up with a phone interview that assessed major medical conditions that may affect neurocognitive testing. Participants eligible for the study fulfilled at least the following major requirements: (1) between 18 and 30 years old, (2) 75% right-handedness according to the Edinburgh Handedness Inventory, (3) normal or corrected-to-normal vision and hearing, (4) no major medical or psychological conditions, (5) no non-removable metal in the body, and (6) played no more than five hours per week of video games in the last six months. All participants signed informed consent forms and completed experimental procedures approved by the University of Illinois Institutional Review Board. One hundred two participants were recruited. Ninety participants completed the study and received compensation of $15/hour. Twelve individuals who dropped out or were disqualified from the study received $7.50/hour. Demographics are summarized in Table 1. More information about study procedures is available in S1 File. PPT PowerPoint slide

PowerPoint slide PNG larger image

larger image TIFF original image Download: Table 1. Demographics. https://doi.org/10.1371/journal.pone.0142169.t001

Study Design Participants completed three cognitive testing sessions and an MRI session before and after the training intervention. The MRI data will not be presented in this paper. Assessments were completed in a fixed order. Participants were randomly assigned to the Mind Frontiers training group or the active control training group. They completed four to five training sessions per week for four to five weeks, a total of 20 sessions; each session involved completing six cognitive training tasks (games) for approximately 12 minutes each. The task order was pseudo-randomized across sessions and all subjects completed the same order during each session. Following the training period, participants completed the same four testing sessions in reverse order. More details about the training protocol can be found in S1 File.

Training Protocol All participants completed training on portable handheld devices. After the first, tenth and last training sessions, participants completed a training feedback questionnaire electronically.

Active Control The active control group also completed six adaptive training tasks in each training session (Table 2 and Fig 2). These included three variants of a visual search task and three variants of a change detection task. The visual search paradigm was derived from Redick et al. [32] and has been shown to not highly overlap (i.e., low correlations) with the working memory, reasoning, and task-switching abilities trained in Mind Frontiers [67, 68]. The change detection paradigm was obtained from Gaspar et al. [58]. Similar to the Mind Frontiers group, the active control group also completed the tasks on a portable device, the Asus Vivotab RT. The visual search tasks were programmed in E-prime 2.0 [69] and the change detection tasks were programmed in MATLAB (MathWorks™) using the Psychophysics Toolbox extensions [70, 71]. PPT PowerPoint slide

PowerPoint slide PNG larger image

larger image TIFF original image Download: Fig 2. Active control tasks. Left: Screenshots of three versions of the change detection task, from top to bottom: colored shapes, cars, letters. Right: Screenshots of three versions of the visual search tasks, from top to bottom: original visual search in Redick et al. [32], colored Ps, Ls. For publication purposes, stimuli are not drawn to scale (enlarged). https://doi.org/10.1371/journal.pone.0142169.g002

Training Feedback Questionnaire At the end of the first, tenth, and twentieth sessions, all participants were asked the following questions about each training game and were instructed to respond on a scale of 1–10: 1) How much did you enjoy/like each game? (1 = did not enjoy/like at all, 10 = enjoyed a lot), 2) How engaging was each game? (1 = least, 10 = greatest), 3) How demanding/effortful was each game? (1 = least, 10 = greatest), 4) How motivated were you to achieve the highest possible score on each game? (1 = least, 10 = greatest), and 5) How frustrating did you find the game? (1 = not at all frustrating, 10 = very frustrating).

Cognitive Assessment Protocol Before and after 20 training sessions, participants completed a battery of tests and questionnaires to assess cognitive function at pre-test and changes that may have resulted from training. The tests measured a variety of cognitive abilities, including reasoning/Gf, episodic memory, perceptual speed, working memory, and attention (Table 3). Participants also completed questionnaires regarding sleep, personality, fitness, and media usage. Following the final testing session, participants completed a post-experiment survey that assessed their feedback on the cognitive training games, the strategies employed during training, gaming experience, and expectations. The majority of the transfer tasks have been extensively used in the cognitive psychology literature (Table 3), so only brief descriptions are provided. More details about each task can be found in S1 File. PPT PowerPoint slide

PowerPoint slide PNG larger image

larger image TIFF original image Download: Table 3. Transfer Tests. https://doi.org/10.1371/journal.pone.0142169.t003

Reasoning, perceptual speed, episodic memory Except for i-Position, the tests below were obtained from the Virginia Cognitive Aging Project Battery [59], and two different versions were used for pre- and post-testing, with the sequence counterbalanced across subjects. Shipley Abstraction: Identify missing stimuli in a progressive sequence of letters, words, or numbers. Number of correctly answered items within five minutes is the primary measure. Matrix Reasoning: Select the pattern that completes a missing space on a 3 x 3 grid. Number of correctly answered items is the primary measure. Reaction time on correct trials was also analyzed. Paper Folding: Identify pattern of holes that results from a punch through folded paper. Number of correctly answered items is the primary measure. Reaction time on correct trials was also analyzed. Spatial Relations: Identify 3D object that would match a 2D object when folded. Number of correctly answered items is the primary measure. Reaction time on correct trials was also analyzed. Form Boards: Choose shapes that will exactly fill a space. Number of correctly answered items is the primary measure. Letter Sets: Determine which letter set is different from the other four. Number of correctly answered items is the primary measure. Reaction time on correct trials was also analyzed. Digit Symbol Substitution: Write corresponding symbol for each digit using a coding table. The primary measure is number of correctly answered items within two minutes. Pattern Comparison: Determine whether pairs of line patterns are the same or different. The primary measure is number of correctly answered items within 30 seconds, averaged across two sets of problems. Letter Comparison: Determine whether pairs of letter strings are the same or different. The primary measure is number of correctly answered items within 30 seconds, averaged across two sets of problems. Logical Memory: Listen to stories and recall them in detail. The primary measure is number of correctly recalled story details, summed across three story-tellings. Paired Associates: Listen to word pairs and recall the second word in a pair. The primary measure is number of correctly recalled items. i-Position: View an array of images on a computer screen and reproduce the positions of the images. Measures are proportion of swap errors (primary) and mean misplacement in pixels.

Working memory Running Span: Recall the last n items presented in a letter list that ends unpredictably. The total number of items in perfectly recalled sets is the primary measure. We also analyzed the total number of items recalled in the correct serial order, regardless of whether the set was perfectly recalled. Operation Span: Remember a sequence of letters while alternately performing arithmetic problems, then recall the sequence of letters. The total number of items in perfectly recalled sets is the primary measure. We also analyzed the total number of items recalled in the correct serial order, regardless of whether the set was perfectly recalled. Symmetry Span: Remember a sequence of locations of squares within a matrix while alternately judging symmetry, then recall order and locations of the sequence. The total number of items in perfectly recalled sets is the primary measure. We also analyzed the total number of items recalled in the correct serial order, regardless of whether the set was perfectly recalled. Visual Short-Term Memory (VSTM): Detect color change in an array of colored circles. Data was analyzed in terms of d-prime collapsed across set sizes (2, 4, 6, 8) and Cowan’s k averaged across set sizes [105]. Each set size measure is reported in S2 File. Single N-back: Determine whether the current letter presented matches the letter presented two or three items back. The primary measure of d-prime was computed separately for the 2-back and 3-back conditions. Reaction times on correct trials were also analyzed. Dual N-back (administered in the MRI): Determine whether simultaneously presented auditory and visual stimuli match stimuli presented one, two, or three items ago. The primary measure of d-prime was computed separately for the two-back and three-back conditions following procedures in [92]. Reaction times on correct trials were also analyzed.

Divided attention, selective attention, multi-tasking Trail Making: Connect numbered circles as quickly as possible by drawing a line between them in numerical order (Trails A), then connect numbered and lettered circles by drawing a line between them, alternating between numbers and letters in numerical/alphabetical order (Trails B). The difference in Trails B and Trails A completion time was the primary measure. Attention Blink: Identify the white letter (target 1) in a sequence of rapidly presented black letters, and identify whether the white letter was followed by a black “X” (target 2). The attentional blink is calculated on trials where target 1 was accurately detected, as the difference in target 2 accuracy when detection is easiest (lag 8 after target 1) and when detection is most difficult (lag 2 after target 1). Dodge: Avoid enemy missiles and destroy enemies by guiding the missiles into other enemies. Highest level reached within eight minutes of game play was analyzed. Multi-source interference task (MSIT; administered in the MRI): Determine the stimulus (digits 1, 2, or 3) that is different from the other two in a three-digit number. The flanker effect is derived by taking the difference between reaction times on incongruent and congruent trials. Only correct trials were analyzed. Flanker: Indicate the direction (right or left) of the middle arrow, which was either flanked by two arrows on each side (incongruent with oppositely oriented arrows, or congruent with similarly oriented arrows) or two horizontal lines on each side (neutral trials, no arrow head). The flanker effect is derived by taking the difference between reaction times on incongruent and congruent trials. Only correct trials were analyzed. Anti-Saccade: Identify masked letter, cued on opposite or same side. Accuracy on a block of anti-saccade trials is used as the primary measure. Psychomotor Vigilance Task (PVT): Press key as soon as zeros begin to count up. The average of the 20% slowest RTs (bottom quintile) is used for analysis. 25 boxes (Number Search): Search for stimuli in a matrix and indicate the corresponding location on blank matrix. The average score on levels with matrix rotation (levels 12–20) was analyzed. Control tower: Search through arrays using different rules (primary task) while performing several distractor tasks. Performance on the primary task (average of symbol, letter and number score minus corresponding errors) was used as the main measure. Task-Switch, Dual-Task paradigm (TSDT): Respond to simultaneously presented auditory and visual stimuli based on cued task (auditory, visual, or both). Switch costs (reaction time difference between switch and repeat trials—for single task trials only) were analyzed separately for auditory and visual stimuli, and averaged across both.

Self-report instruments Participants also completed questionnaires during the third session of pre-testing. These included the Big Five Inventory [106] and Grit Scale [107] to assess personality, the Karolinska Sleep Questionnaire [108] and Pittsburgh Sleep Quality Index [109] to gauge sleep quality, the Godin Leisure-Time Exercise Questionnaire [66] to estimate physical activity, several questions on height, weight, resting heart rate and physical activity to estimate cardiorespiratory fitness [110], and a Media Multitasking Index Questionnaire [111] to assess media usage. These questionnaires were also completed post-testing, but were not used for analyses. Analyses of whether these individual differences moderate training effects will be discussed in a separate publication. Post-experiment questionnaire: Participants completed an online survey that assessed gaming experience prior to and during the study, as well as their experience in the study. They provided feedback about their enjoyment, effort, and difficulty in playing the training games. They also elaborated on strategies they developed while playing the games. Participants provided feedback on game experience, design, and ease of use, and offered their perspective on improvements to their daily life resulting from their participation in the study (perceived self-improvement questions), including: overall intelligence, short-term/working memory, long-term memory, sustained attention, divided attention, visuomotor coordination, perception/visual acuity, multi-tasking, problem-solving, reasoning, spatial visualization, academic performance, emotional regulation, and work/school productivity. The fourteen dimensions queried in the perceived self-improvement questions were also posed in terms of general expectancy or perceived potential benefit. Finally, the survey assessed prior knowledge of cognitive training literature.