The above chart shows the returns to various approaches to keeping kids in school, according to research conducted by the Poverty Action Lab (J-PAL) at MIT. The results are striking. Spending $100 on public awareness in Madagascar yields a whopping 40 extra years of student attendance. $100 spent on deworming in Kenya generates an additional 28+ years of student attendance. Public awareness and deworming campaigns are 10 to 40 times more cost effective than providing school meals, scholarships, or uniforms.
J-PAL has popularized the use of randomized controlled trials (RCTs) in development aid, a movement that has recently been highlighted by Nick Kristof in the NY Times. Kristof describes RCTs as the "hottest thing in the fight against poverty." And when you see striking results like those in the chart above, it is hard not to get excited.
Michael Kremer, a professor at Harvard, pioneered some of the earliest randomized trials, including those on de-worming in Kenya. He has gone on to help design and oversee USAID's new Development Innovation Ventures, which promises to seed a lot of new approaches on the condition that they be rigorously evaluated. Michael's common sense approach to finding out what works (who would have thought that de-worming would be such a high-return investment in education!) makes this USAID initiative potentially promising.
The good news about RCTs is that they can guide the design of development projects in some circumstances. For example, in the regions of Kenya where Michael did his work, the most rational allocation of education expenditures might be first to de-worm all the children. Only after that is done would it make sense to spend money on things like subsidizing uniforms. I also have argued that the expensive Millennium Village Project should be subject to an RCT "competition" with other approaches.
But promising new tools often get promoted as silver bullets, and RCTs are no exception. This inevitably causes a backlash, which in turn means that, after 15 minutes of fame, many good tools fail to be adopted to an optimal degree. Rachel Glennerster, the director of J-PAL, told Owen Barder last year that there was a risk of too much hype, and that RCTs were not feasible or desirable in all circumstances. For those interested in this topic, I recommend the full interview.
What are the limitations or even downsides of RCTs? Angus Deaton is one of the strongest critics of RCTs, for a number of conceptual and methodological reasons. He argues that any statistically valid RCT must be very narrow in terms of applicability. For example, the findings on de-worming in Kenya cannot be generalized even beyond the villages in which the experiments were run. To his point, J-PAL's chart above shows that the returns to de-worming in India are only about 10-12% of the returns to de-worming in Kenya. To the extent RCTs are so context specific, their usefulness is severely limited.
Deaton also argues that RCTs may capture the mean but not the variance of the effects that a project has on beneficiaries in the studied population. Though an initiative may on average have positive effects, it may have a negative impact on a substantial proportion (or even majority) of the beneficiary population. And vice versa: an initiative which shows a negative average impact might benefit many beneficiaries. Drawing any sweeping conclusions under these conditions, Deaton argues, is not warranted, and could even be dangerous.
Others, such as Arvind Subramanian at the Center for Global Development, argue that even if RCTs can shed light on the effect of a development project in limited circumstances, they cannot tell us anything about whether aid itself works or not.
A young researcher from within the RCT movement noted to me recently that randomized trials can only predict marginal impacts and cannot be extrapolated. For example, the effect of public information may have the effect of keeping individual kids in school one month more; providing 12 times more information will not keep the kids in school an extra 12 months, which may be the goal.
At some point, a clever economist will try to carry out a randomized controlled trial of RCTs. She will attempt to answer the question: Does the use of RCTs lead to changes in the design of aid projects that translate into improved well being for people in developing countries?
My guess is that such a meta-RCT would not show a strong positive impact, for several reasons. First is the cost of RCTs. Even if they are a gold standard (which Deaton disputes), the costs are such that they will only be able to be done on a minuscule proportion of development initiatives.
Second, the current incentive structure in the aid industry results in an attenuated link between evidence on impact and changes in project design. Aid providers face little competitive or other pressure to seek out the initiatives that have the greatest impact. In fact, Lant Pritchett argues that, under the current structure, it "pays to be ignorant" because confessing failure hurts you more than success benefits you (in terms of political and financial support).
The bottom line is that RCTs will become like Consumer Reports trials in the consumer marketplace. Consumer Reports is a useful adjunct to decision making for some things (although often I can't find the exact models they tested!) But innovation in service of improved quality and lower cost comes from market pressures arising from consumer feedback through purchasing decisions.
In the same way, the best hope for improving the impact of aid initiatives is to create much richer and more real-time feedback loops between beneficiaries and aid providers.
Instead of determining needs ex-ante through expert studies, we need to start with asking beneficiaries "What do YOU want?" New technologies and approaches enable us to do this on a far wider scale, at dramatically lower cost, than ever before. And then once a project is underway, we need to ask beneficiaries "How do YOU think it's going and what changes need to be made?" And once that project is finished, we need to ask "Given what we learned from the previous project, what is the next thing you want?" The faster we can iterate through these questions, the faster we will get to greater impact.
Naturally, experts should provide technical analysis such as RCT results to beneficiaries to help inform their responses. But in the end, we need to make the beneficiary king.
