CGN_ICA test description

The CGN_ICA test is a rapid visual categorization task with backward masking33,37,38. One hundred natural images (50 animal and 50 non-animal) with various levels of difficulty were presented to the participants. Each image was presented for 100 ms followed by a 20 millisecond inter-stimulus interval (ISI), followed by a dynamic noisy mask (for 250 ms), followed by subject’s categorization into animal vs non-animal (Fig. 1). When using iPad, the categorization was done by tapping on the left or right side of the screen; when using RasPi, subjects indicated their responses by pressing either of the two pre-assigned keys on a keyboard (‘F’ vs. ‘J’). Images were presented at the center of the screen at 7 degree visual angle. For more information about rapid visual categorization tasks refer to Mirzaei et al.33.

Figure 1 The CGN_ICA test pipeline. One hundred natural images (50 animal and 50 non-animal) with various levels of difficulty are presented to the participants. Each image is presented for 100 ms followed by 20 ms inter-stimulus interval (ISI), followed by a dynamic noisy mask (for 250 ms), followed by subject’s categorization into animal vs. non-animal. Few sample images are shown for demonstration purposes. Full size image

The CGN_ICA test starts with a different set of 10 test images (5 animal, 5 non-animal) to familiarize participants with the task. These images are later removed from further analysis. If participants perform above chance (>50%) on these 10 images, they will continue to the main task. If they perform at chance level (or below), the test instructions will be presented again, and a new set of 10 introductory images will follow. If they perform above chance in this second attempt, they will progress to the main task. If they perform below chance for the second time the test will be aborted. Only in experiment 2, three participants, out of 61, were aborted from the study due to this reason, thus 58 subjects remaining in experiment 2 that are shown in Table 1.

Table 1 Summary of all the experiments. Full size table

Scientific rationale behind the CGN_ICA test

The CGN_ICA test takes advantage of millions of years of human evolution – the human brain’s strong reaction to animal stimuli39,40,41,42. Human observers are very good at recognising whether briefly flashed novel images contain the image of an animal, and previous work has shown that the underlying visual processing can be performed quickly38,43. The strongest categorical division represented in the human higher level visual cortex (known as inferior temporal cortex) appears to be that between animates and inanimates. Several studies have shown this in human and non-human primates38,39,40,44,45. Studies also show that on average it takes about 100 ms to 120 ms for humans to differentiate animate from inanimate stimuli46,47,48. Following this rationale, in the CGN_ICA test, the images are presented for 100 ms followed by a short inter-stimulus-interval (ISI), then followed by a dynamic mask. Shorter periods of ISI can make the animal detection task more difficult and longer periods reduce the potential use for testing purposes as it may not allow for detecting less severe cognitive impairments. The dynamic mask is used to remove (or at least reduce) the effect of recurrent processes in the brain49,50,51,52,53. This makes the task more challenging by reducing the ongoing recurrent neural activity that could boost subject’s performance. This leaves less room for the resilient brain to compensate for the subtle ongoing neurodegeneration in early stages of the disease.

Participants

As shown in Table 1, we conducted four different experiments; in total, 448 volunteers took part in this study. The study was conducted according to the Declaration of Helsinki and approved by the local ethics committee at Royan Institute. Informed consent was obtained from all participants.

Participants’ inclusion criteria were individuals above age 18, with normal or corrected-to-normal vision, without severe upper limb arthropathy or motor problems that could prevent them from completing the tests independently. For each participant, information about age, education and gender was also collected.

Stimulus set

We used a set of 100 grayscale natural images, half of them contained an animal. The images varied in their level of difficulty. In some images the head or body of the animal is clearly visible to the participants, which makes it easier to detect. In other images the animals are further away or otherwise presented in cluttered environments, making them more difficult to detect. Few sample images are shown in Fig. 1. Grayscale images were used to remove the possibility of some typical color blindness affecting participants’ results. Furthermore, color images can facilitate animal detection solely based on color, without fully processing the shape of the stimulus. This could have made the task easier and less suitable for detecting less severe cognitive dysfunctions.

To construct the mask, a white noise image was filtered at four different spatial scales, and the resulting images were thresholded to generate high contrast binary patterns. For each spatial scale, four new images were generated by rotating and mirroring the original image. This leaves us with a pool of 16 images. The noisy mask used in the CGN_ICA test was a sequence of 8 images, chosen randomly from the pool, with each of the spatial scales to appear twice in the dynamic mask.

Reference pen-and-paper cognitive tests

Montreal Cognitive Assessment (MoCA)

MoCA4 is a widely used screening tool for detecting cognitive impairment, typically in older adults. The MoCA test is a one-page 30-point test administered in approximately 10 minutes.

Mini-Mental State Examination (MMSE)

The MMSE3 test is a 30-point questionnaire that is used in clinical and research settings to measure cognitive impairment. It is commonly used to screen for dementia in older adults; and takes about 10 minutes to administer.

Addenbrooke’s Cognitive Examination -Revised (ACE-R)

The ACE54,55 was originally developed at Cambridge Memory Clinic as an extension to the MMSE. ACE-R is a revised version of ACE that includes MMSE score as one its sub-scores. The ACE-R5 assesses five cognitive domains: attention, memory, verbal fluency, language and visuospatial abilities. On average, the test takes about 20 minutes to administer and score.

Symbol Digit Modalities Test (SDMT)

The SDMT is designed to assess speed of information processing, and takes about 5 minutes to administer56. A series of nine symbols are presented at the top of a standard sheet of paper, each paired with a single digit. The rest of the page contains symbols with an empty box next to them, in which participants are asked to write down the digit associated with this symbol as quickly as possible. The outcome score is the number of correct matches over a time span of 90 seconds.

California Verbal Learning Test -2nd edition (CVLT-II)

The CVLT-II test57,58 begins with the examiner reading a list of 16 words. Participants listen to the list and then report as many of the items as they can recall. After that, the entire list is read again followed by a second attempt at recall. Altogether, there are five learning trials. The final score, which is out of 80, is the summation of all the correct recalls. As in the brief international cognitive assessment for multiple sclerosis (BICAMS) battery59, we only used the learning trials of the CVLT-II, which takes about 10 minutes to administer.

Brief Visual Memory Test–Revised (BVMT-R)

The BVMT-R test assesses visuo-spatial memory60,61. In this test, six abstract shapes are presented to the participant for 10 seconds. The display is removed from view and patients are asked to draw the stimuli via pencil on paper manual responses. There are three learning trials, and the primary outcome measure is the total number of points earned over the three learning trials. The test takes about 5 minutes to administer.

Experiments

We conducted four different experiments, as summarized in Table 1. The first three experiments were designed to measure the CGN_ICA correlation with a wide range of routinely used reference cognitive tests. The goal was to investigate whether the speed and accuracy of visual processing in a rapid visual categorization task is correlated with subject’s cognitive performance.

In the first and second experiments, we aimed to test CGN_ICA’s ability in assessing cognitive performance in older adults. Therefore, we used MoCA and/or ACE-R as reference cognitive tests, both of which are routinely used to screen for mild cognitive impairment (MCI) and dementia in older adults. In the first experiment, 212 volunteers participated; the CGN_ICA test was delivered via a Raspberry Pi (RaPi) platform, which is a small single-board computer, attached to a keyboard and a LCD monitor; and MoCA was used as the reference cognitive test. For the second experiment, we had 58 participants; the CGN_ICA was delivered via iPad, and both MoCA and ACE-R were used as reference tests in this experiment.

The third experiment had SDMT, BVMT-R and CVLT-II as the reference cognitive tests, measuring speed of information processing, visuo-spatial memory and verbal learning, respectively. These three tests together form the BICAMS battery, which requires about 15 to 20 minutes to administer, and is primarily used to detect cognitive dysfunction in younger adults who may suffer from multiple sclerosis (MS). 166 participants took part in this experiment. Forty-four of them were selected for a re-test as part of a second visit to assess CGN_ICA test-retest reliability. Participants for the re-test session were selected at random, while keeping the age-range, level of education, and gender ratio relatively similar to the set of participants in the first session. The CGN_ICA was delivered via an iPad platform.

All the pen-and-paper cognitive tests were administered by a healthcare professional. The administration order for CGN_ICA vs. reference cognitive tests was at random.

Finally, experiment 4 was designed to study whether the CGN_ICA test had a learning bias if taken multiple times in short intervals. Learning bias is defined as the ability to improve your test score by learning the test simply because of several exposures to the test. 12 young volunteers participated in this study. For convenience, the CGN_ICA was delivered remotely via a web platform. Participants took the CGN_ICA test every other day for two weeks.

Accuracy, speed, and CGN_ICA summary score calculations

Preprocessing

We used boxplot to remove outlier reaction times, before computing the CGN_ICA score. Boxplot is a non-parametric method for describing groups of numerical data through their quartiles; and allows for detection of outliers in the data. Following the boxplot approach, reaction times greater than q3 + w * (q3 − q1) or less than q1 − w * (q3 − q1) are considered outliers. q1 is the lower quartile, and q3 is the upper quartile of the reaction times. Where “w” is a ‘whisker’; w = 1.5.

Accuracy is simply defined as the number of correct categorisations divided by the total number of images, multiplied by a 100.

$${\rm{Accuracy}}=\frac{number\,of\,correct\,categorisations}{total\,number\,of\,images}\times 100$$ (1)

Speed is defined based on participant’s response reaction times in trials they responded correctly.

$$Speed=\,{\rm{\min }}[100,\,100\times {e}^{\frac{-meancorrectRT}{1025}+0.341}]$$ (2)

RT: reaction time

e: Euler’s number ~2.7182……

Speed is inversely related with participants’ reaction times; the higher the speed, the lower the reaction time. The reason for defining the above formula for speed, instead of using the raw reaction times, was to have a more intuitive and standardized score to report to the clinicians, scaled between 0 to 100.

The CGN_ICA summary score is a simple combination of accuracy and speed, defined as follows:

$${\rm{CGN}}\_{\rm{ICA}}\,{\rm{Score}}=(\frac{{\rm{Speed}}}{100}\times \frac{Accuracy}{100})\times 100$$ (3)

Statistical analysis

Within the manuscript, convergent validity, and test-retest reliability for the CGN_ICA test is shown with Pearson’s Correlation. P-values for Pearson’s correlation are based on a Student’s t distribution. Calculations are done using MathWorks’ statistics and machine learning toolbox (https://www.mathworks.com/help/stats/index.html).

To measure dependency of the cognitive tests with level of education, we used explained variance, defined as the square of Pearson’s Correlation between participants’ cognitive score and their level of education (i.e. number of years). Here the statistical significance was obtained by a permutation test (10,000 permutations of participants). To formally assess statistical independence, we used a non-parametric independence test, proposed by Gretton and Gyorfi62, based on 10,000 bootstrap resampling of participants.

Finally, we used a single factor analysis of variance (ANOVA) to compare average CGN_ICA scores for participants who had taken the CGN_ICA test every other day for two weeks. The goal was to see if the mean CGN_ICA scores are significantly different at any given day.