Clinical assessment of app

A clinical assessment was conducted at Children’s Healthcare of Atlanta, Emory University of School of Medicine, and Georgia Institute of Technology to relate fingernail bed color to Hgb levels. Patients with various anemia etiologies scheduled to have their Hgb levels measured via a CBC as part of their clinical care were recruited to this study (n = 265). Subjects were excluded by quality control measures if their images showed fingernail beds that were obscured or discolored due to leukonychia, nailbed injury, nail polish, darkening due to medication44, etc. Exclusions were conducted to eliminate unnecessary variables that could obfuscate algorithm development. All CBC’s were conducted using blood samples collected via venous blood draw. After patient’s blood was collected to conduct their CBCs, two images were taken of those patients’ fingernail beds. Smartphone pictures were obtained with the camera flash both on and off. All images were taken with an Apple iPhone 5s (Apple, Cupertino, CA) using all default imaging settings. Prior to imaging, the auto-focus and brightness adjustment of the smartphone camera was activated by tapping the screen in order to focus on the nailbed. To ensure consistent images, each image was taken with the smartphone at a distance of ~0.5 m from the subjects’ fingernail beds. If possible, subjects were encouraged to curl their fingers inwards with their palms facing upwards to control for possible alterations in blood flow caused by hand and finger positioning that could potentially affect the underlying color of the fingernail beds (Fig. 1). Images were taken in clinic examination rooms, where lighting conditions and room illuminants were relatively consistent. A digital light meter (Hisgadget, Union City, CA) was placed next to the subject’s fingernail beds to further ensure consistent background lighting conditions, but external digital light meter readings were not incorporated into the Hgb level calculation. An additional 72 healthy subjects from Emory University and The Georgia Institute of Technology were tested using an identical protocol. CBC’s were conducted on each subject prior to imaging and were analyzed via the same clinical hematology analyzer (Advia 2120i, Siemens, Berlin, Germany) used in the clinical study. All imaging was conducted in a room with similar lighting conditions to the clinic exam rooms, which was confirmed via digital light meter. Fingernail bed images and blood Hgb levels were analyzed in a total of 337 subjects. These subjects’ blood Hgb levels ranged between 5.9 and 16.8 g dL−1 (Supplementary Figure 8A). Subject’s ages ranged between 1 and 60 years old and had varying skin tones (Supplementary Figure 8B). 167 female subjects and 170 male subjects were enrolled in this study. In six cases, fingernail polish was discovered after informed consent had been obtained, and these subjects were excluded from testing after study enrollment. In one case, an image labeled as having been taken with the camera flash on was discovered to have been taken with the flash off, resulting in this subject’s data being excluded.

Algorithm development/image processing

Smartphone images were transferred or transmitted from the smartphone used in the study to a computer. Fingernail data, skin color data, and image metadata were extracted from fingernail bed smartphone images via MATLAB (Mathworks, Natick, MA). Regions of interest, from which fingernail and skin color data were extracted, were manually selected to ensure that fingernail irregularities were excluded from analysis. These regions of interest were selected from each finger, excluding the thumb, and were 900 pixels2, corresponding to ~10 mm2 on the fingernail. Color data were extracted from each region and averaged together across fingers for each subject. This was shown to be an acceptable method due to the low color variability between different fingers (Supplementary Figure 9). An algorithm was then written in MATLAB utilizing robust multi-linear regression with a bisquare weighting algorithm to relate the image parameter data to CBC Hgb levels for each patient (Eq. (1))45.

\({\mathrm{Hemoglobin}}_{{\mathrm{Result}}} = C + P_1 \times W_{1 + }P_2 \times W_{2 + } \ldots P_n \times W_n\) where: C = constant, W = weights determined via robust multi linear regression, and P = skin color data, and image metadata parameters.

A uniform bias adjustment factor was also added to address the inherent variability in fingernail measurement. Two distinct use models and algorithms were applied for this Hgb measurement method: (1) as a noninvasive, smartphone-based, quantitative Hgb level diagnostic requiring calibration with CBC Hgb levels that enables chronic anemia patients to self-monitor their Hgb levels, and (2) as a noninvasive, smartphone-based anemia screening test that does not require calibration with CBC Hgb levels. Sampling strategies were used to generate the algorithm depending on the specific application.

Anemia screening among the general population: To develop the algorithm as a tool to screen for anemia, the entire study population (337 subjects) was randomly split into a discovery group (237 subjects) and a testing group (100 subjects). The discovery group was used to establish the relationship between image parameters and Hgb levels via robust multi-linear regression, much like the calibration phase of the personalized calibration study. A testing group, analogous to the testing phase of the personalized calibration study, of 100 subjects was used to validate the resultant algorithm. Validation was performed by applying the smartphone algorithm to each testing image and comparing the algorithm generated Hgb result with the CBC Hgb result (i.e., determining the residual of the algorithm-based method). This process was repeated 1000 times with different, randomly-selected without replacement, discovery/testing groups to minimize residual error, thereby optimizing the parameters of the algorithm for anemia screening. Resulting data from most accurate outcome of this optimized screening algorithm is depicted in Fig. 2. Hgb measurements taken from the previously described personalized calibration study were not included in this anemia screening study.

Personalized calibration of Smartphone App: A personalized calibration approach was tested in two β-thalassemia major patients with chronic anemia currently undergoing chronic transfusion therapy, a healthy female subject with Hgb levels that fluctuated during her menstrual cycle, and a healthy male subject with consistent Hgb levels over an identical timeframe to assess the algorithm’s capability to be accurately personalized and calibrated to that individual, regardless of their diagnosis or Hgb levels. Treatment for β-thalassemia major currently comprises red blood cell transfusions to compensate for the patients’ ineffective erythropoiesis46. Hgb levels in the chronic anemia patients fall throughout a 4-week transfusion cycle, which was chosen as an appropriate time interval for this study. Smartphone Images were obtained with and without the camera flash. Prior to each imaging session, CBC Hgb levels were obtained from each subject via venipuncture. Color data and phone metadata were compiled and a relationship between image data and CBC Hgb levels was established via robust multi-linear regression. This process was repeated for each individual using data from the 4 weeks of images to create a unique calibration curve personalized for that individual. Image parameter changes associated with Hgb level fluctuations specific to each person were related to perform algorithm calibration specific to each subject, thus improving the accuracy of Hgb level estimation. After the smartphone image analysis system was calibrated for each subject, Hgb levels were measured weekly over the next 4 weeks using the newly personalized algorithm. These Hgb level measurements were then compared to the CBC Hgb levels obtained at the same time to assess accuracy. This personalized calibration occurred over a total of 8 weeks.

Hemoglobin measurement from images of fingernails

Images were taken of 50 subjects fingernails from the previously described clinical study. These subjects’ ages ranged from 1 to 62 years old. Hematologists (M.D. physicians who specialize in clinical hematology and are trained and Board Certified in the USA) were instructed to analyze each image and measure Hgb levels. For comparison, images were loaded into the app, and the Hgb measurement protocol was performed on these images. All images and analysis were taken using an iPhone 5S. It is important to note that these images were not used in the development of the underlying image analysis algorithm.

Intraclass correlation coefficient (ICC) reflects not only degree of correlation but also agreement between measurements and ranges between 0 and 1, with values closer to 1 representing stronger reliability. Reliability refers to the degree of agreement among raters. It gives a score of how much homogeneity, or consensus, there is in the ratings given by different judges or instruments. The ICC is able to incorporate the reliability of more than 2 raters-as in the case of the 5 hematologists evaluating nail beds. Patients and the physicians were assumed to be random samples from the respective populations they represent.

App development

The Hgb level measurement algorithm was incorporated into mobile apps. The open source integrated development environment (IDE) Android Studio (Google, Mountain View, CA) was used to develop a beta version of the Hgb measurement app in the Android operating system. The proprietary IDE Xcode (Apple, Cupertino, CA) was used to develop a beta version of the app in the iOS operating system.

Human subjects research statement

All experiments complied with all relevant ethical guidelines for human subject research, namely, the Declaration of Helsinki. Verbal assent and written consent were obtained from all study subjects and their parents (age permitting) in accordance with HIPAA regulations prior to partaking in the study. All experiments involving human subjects in this manuscript were approved by either the Emory University IRB (algorithm development—approval number 00081226) or the Georgia Institute of Technology IRB (skin temperature and heart rate interference—approval number H17118).

Statistical analysis

Statistical significance (p < 0.05) was determined via two-tailed Student’s t-test assuming unequal variance. All statistical tests (calculation of regression correlation coefficients and Student’s t-tests were conducted using Origin Pro 2017 student version. 95% confidence intervals for sensitivity and specificity are calculated according to the efficient-score method. A two-way random effects model was used to estimate our ICC for measuring agreement between the app and hematologists at measuring Hgb levels based on physical examination.

Code availability

The custom MATLAB code used in this study is available from the corresponding author upon reasonable request. This code is copyrighted by Emory University and Children’s Healthcare of Atlanta and is to be used only for educational and research purposes. Any commercial use including the distribution, sale, lease, license, or other transfer of the code to a third party, is prohibited. For inquiries regarding commercial use of the code, please contact Emory University’s Office of Technology Transfer.