In this study we demonstrated the use of mobile phone surveys via SMS for gathering information about health seeking behavior data during and in the immediate aftermath of an EVD outbreak; critically important for understanding the collateral effects from the EVD outbreak on public-sector primary healthcare delivery which are conjectured to greatly exceed the direct effects of EVD infections.10 A pressing concern with the use of many Internet and mobile-enabled tools, such as SMS, is that the population denominator is not representative of the general population. We investigated the population that was reached using this methodology and used propensity score matching, which is commonly used in the statistical analysis of observational data, to create balanced populations for estimating and comparing outcomes regarding birth location and accounting for the covariates that predict being in the SMS population group compared to a DHS survey in Liberia. We demonstrate that with appropriate methodological considerations, it was possible to generate matched groups with survey (DHS) data, across which to compare outcomes. Before matching, there was a greater proportion of deliveries in public or private facilities in the GP population, compared to the DHS population, which makes sense given the demographics of the GP versus DHS populations (more educated and a higher proportion in professional/technical occupations, indicating a higher socio-economic status). Once matched, we found that matched subgroups had similar patterns of health facility delivery location in aggregate (higher proportion of private-facility deliveries in the GP matched population than the DHS), and utilizing data on the date of birth, we showed that deliveries in institutional-facilities were significantly decreased during, compared to after the outbreak (p < 0.05), whereas in the unmatched population the proportion of deliveries in a private facility decreased post-outbreak. Directly assessing behaviors from individuals via SMS also enabled the measurement of public and private sector facility utilization separately; our data also suggest that use of public facility deliveries returned to baseline values after the outbreak. While our analysis was retrospective, data from mobile tools can be captured in real-time, and the conclusions of this paper indicate that prospective SMS surveying and analysis can offer a great opportunity to understand such disruptions and to inform approaches to mitigate such potential risks in real-time.

Results confirmed what has since been found in many retrospective studies of healthcare utilization in the outbreak.1,8,9,10 After matching, we found that there was a significant change in facility-based deliveries during, compared to after the outbreak. As our approach enabled gathering information about the types of facilities used (public versus private) across the nation, in our data, this change was due to a similar lowering of births happening in public facilities and private facilities. A previous study of birth locations found that the decrease was mostly due to decline in deliveries in public locations, but the study was limited to just Monrovia11 which is where the highest proportion of private facilities are located in Liberia. Our finding is in contrast to what was found in Sierra Leone, where the decline in number of deliveries in the EVD outbreak was mainly attributable to the closing of private not-for-profit hospitals, rather than government facilities.12 After the outbreak, the proportion delivering in public institutions as measured through SMS returned to the baseline (DHS) levels, and the proportion delivering in private locations also increased.

The matching approach used here could also be applied to other observational data sources. For example, GP provides a range of mobile polling options ranging from SMS to Interactive Voice Response (IVR). IVR is sometimes believed to be more effective in situations in which there is a low rate of literacy, but is more costly to administer.24 While some studies have examined the possibility of mobile surveys via IVR, we add to this literature by conducting an SMS-only study at scale. Existing work has reported attrition related to length, so IVR might be worse than SMS in some respects,17 it is hard to exactly make any comparison between the methods as SMS is designed by nature to be a shorter medium. Since there is less work using SMS at similar scales for health surveillance and we elected to test SMS polling in this context.

In any adjustment and matching exercise, there are inherent assumptions that must be recognized. The assumption of ignorability of unobservables is imposed by this inference method; in other words, we are assuming there are no unmeasured variables which confound the relationship between the treatment (surveyed via GP) and the outcome (delivery location type).25 Practically, we have attempted to ensure that covariates included do not depend on the outcome variable, introduce bias or have some tautological relationship with the outcome variables examined. While we cannot verify it, the ignorability assumption is generally assumed reasonable because matching on or controlling for the observed covariates also matches on or controls for the unobserved covariates, in so much as they are correlated with those that are observed. Further, we do recognize that unmeasured covariates can exist. We perform our study with this awareness, and with the attempt to include covariates that would describe general demographics, and that were feasible to measure using the SMS survey. There can also be unmeasured covariates due to inherent limitations of the data source. First, simply through human error there can be issues in who is reached (for example, the quota sampling for gender was not accurately implemented and in the first round of the survey only 40% of respondents were female). As with any self-reporting, there can be limitations based on the accuracy of the data; however given that we found most reports from the same User ID had consistent demographics reported (and we discarded data from any User IDs with inconsistent attributes), we have confidence in the validity of data used. There may also be biases caused by external factors that may result in a correlation between which type of healthcare facility is used and who is surveyed via GP that is spurious, not causal. Phrased differently, it is possible that people who are surveyed through GP are more likely to give birth in certain types of locations (e.g., if types of facilities available differ by location, or the telecommunications provider that GP uses represents only one group from the population—we would never be able to adjust for the unrepresented groups). Broadly, from any mobile-phone data and also as illustrated in the summary of data pre-match from GP (Table 1), we recognize that the SMS data comes from a generally more young and urban population than DHS. As described, we use the analytic methods (propensity matching), to adjust for this given that the DHS survey (amongst other surveys) are also not representative of the population. In the future, mobile phone numbers and consent could be garnered during household surveys, which can then be used for longitudinal data monitoring of a sample drawn from a known sample frame.

To improve this work, as in any effort to decrease the amount of confounding, it would be very pertinent to increase the number of covariates. This could be done by ascertaining more baseline demographic information during the survey process, even without specifically knowing prospectively what population will be reached. Future research should involve replicating this work in other situations and places, improving the scale of data collection to improve statistical significance as well as improving data matching methods with further covariates and outcomes. Additionally, the work is limited by the types of data that are available for comparison. Though we used county-level resolution for comparability with the DHS-measured outcomes, it is possible to obtain even higher geographic resolution via mobile tools. As well, “gold standard” data sources used here are outdated as the DHS data is three to four years old. We thus see mobile data collection as a useful complement to data on health and health seeking behavior collected from household surveys as demonstrated here, particularly for more nimbly capturing changes in health-seeking behaviors that traditional surveys allow. As mobile phone ownership, and even smartphone use is growing even among poorer segments of the population, and given its low cost, with appropriate methodological approaches, it can be a valuable tool for population health intelligence.