Living Goods runs a network of Community Health Promoters (CHPs) who sell health and household goods door-to-door in their communities in Uganda and Kenya. CHPs also provide basic health counseling. Living Goods also provides consulting and funding to other organizations to run similar networks in other locations. We have been considering Living Goods for a 2014 recommendation.

We’ve now spent a considerable amount of time talking to Living Goods and analyzing documents Living Goods shared with us. This post shares what we’ve learned so far and what questions we’re planning to focus on throughout the rest of our investigation. (For more detail, see our detailed interim review.)

Living Goods has successfully completed the first phase of our investigation process and we view it as a contender for a recommendation this year. We now plan (a) to make a $100,000 grant to Living Goods (as part of our “top charity participation grants,” funded by Good Ventures) and (b) continue our analysis to determine whether or not we should recommend Living Goods to donors at the end of the year.

Reasons we prioritized Living Goods

Living Goods contacted us a few months ago to inform us that the initial results from a randomized controlled trial (RCT) of its program were available. The headline result from the study was a 25% reduction in under-five mortality, a remarkable effect size.

Questions we hope to answer in our ongoing analysis

How robust is the RCT?

The authors of the RCT have not yet completed the full report on the study, so we have not been able to vet the results in detail. RCTs generally have fewer methodological issues that severely undermine the results than other types of studies, but they are not immune to these problems. We discuss potential issues with the RCT in our interim review.

The authors are seeking publication in an academic journal and the paper will be embargoed until a journal publishes it. This may mean that we are unable to discuss the details of the study before releasing our 2014 recommendations. We are unsure how strong a recommendation of Living Goods we might make if we were unable to give the details of the main evidence for its impact.

In addition, we don’t want to overemphasize the strength of the evidence provided by a single RCT (even if it has no methodological issues). Interventions such as bednets and cash transfers are supported by multiple RCTs and other evidence.

Will future work be as impactful as past work, and how will we know?

There are some reasons to think future results could be worse than RCT results: locations for the RCT were carefully selected, perhaps to maximize impact, and malaria control in Uganda may have improved in recent years. Even if the program is somewhat less effective in the future, it may still be worth supporting.

Our main concern is about both Living Goods’ and our ability to know how well the program is performing in the future. Living Goods asks CHPs to report on activities such as treatments provided and follow up visits, but because of the incentive structure and lack of audits on the accuracy of these reports, we put limited weight on these metrics. Living Goods told us that its branch managers conduct randomized follow ups with clients, but we have not see documentation from these audits (or other evidence that these checks are happening). We’re not aware of any other monitoring that Living Goods conducts on its program.

Will other funders fill Living Goods’ funding gap?

Living Goods is looking to significantly scale up its program in the next four years. It is in discussions with current funders to see if they will increase their support. It believes it may be able to fund up to two-thirds of its scale-up through these commitments. It is continuing to seek new sources of funding. We may have to make a decision about how much funding to recommend to Living Goods in 2014 before other funders make their decisions known.

If Living Goods raises more than it needs for its scale-up, it would likely use these funds to co-fund partner organizations to start networks of CHP-like agents in other countries. This would be a riskier bet for donors, and its not clear how much we can expect to learn about how these programs turn out.

Is the CHP program cost-effective?

Living Goods estimates that its program will have a cost per life saved of $4,773 in 2015, decreasing to $2,773 in 2018. We have made some adjustments to this model to generate our own estimates. We estimate that Living Goods’ cost per life saved will be roughly $11,000 in 2014-2016. Making assumptions that we would guess are particularly optimistic about Living Goods, we estimate the cost per life saved at about $3,300. Pessimistic assumptions lead to an estimate of $28,000 per life saved. (Details in our interim review.) Our work on this model is ongoing.

Our guess is that Living Goods’ program is in the same range as (though slightly less cost-effective than) the most cost-effective programs we have considered, such as bednets, deworming, and iodization.

(See our page on cost-effectiveness for more on the role these estimates play in our recommendations.)