Your Worst Fear Realized

My friend Keith Hansen sent me a link to Conan O'Brien's recent graduation speech at Dartmouth.  The speech is a refreshing change from the "Follow your dreams" and "Whatever you can conceive, you can do" pablum that we hear and read in so many places.

Here is an excerpt:

Nietzsche famously said "Whatever doesn't kill you makes you stronger." But what he failed to stress is that it almost kills you. Disappointment stings and, for driven, successful people like yourselves it is disorienting. What Nietzsche should have said is "Whatever doesn't kill you, makes you watch a lot of Cartoon Network and drink mid-price Chardonnay at 11 in the morning."

Now, by definition, Commencement speakers at an Ivy League college are considered successful. But a little over a year ago, I experienced a profound and very public disappointment. I did not get what I wanted, and I left a system that had nurtured and helped define me for the better part of 17 years. I went from being in the center of the grid to not only off the grid, but underneath the coffee table that the grid sits on, lost in the shag carpeting that is underneath the coffee table supporting the grid. It was the making of a career disaster, and a terrible analogy.

But then something spectacular happened. Fogbound, with no compass, and adrift, I started trying things. I grew a strange, cinnamon beard. I dove into the world of social media. I started tweeting my comedy. I threw together a national tour. I played the guitar. I did stand-up, wore a skin-tight blue leather suit, recorded an album, made a documentary, and frightened my friends and family. Ultimately, I abandoned all preconceived perceptions of my career path and stature and took a job on basic cable with a network most famous for showing reruns, along with sitcoms created by a tall, black man who dresses like an old, black woman. I did a lot of silly, unconventional, spontaneous and seemingly irrational things and guess what: with the exception of the blue leather suit, it was the most satisfying and fascinating year of my professional life. To this day I still don't understand exactly what happened, but I have never had more fun, been more challenged—and this is important—had more conviction about what I was doing.

How could this be true? Well, it's simple: There are few things more liberating in this life than having your worst fear realized. I went to college with many people who prided themselves on knowing exactly who they were and exactly where they were going. At Harvard, five different guys in my class told me that they would one day be President of the United States. Four of them were later killed in motel shoot-outs. The other one briefly hosted Blues Clues, before dying senselessly in yet another motel shoot-out. Your path at 22 will not necessarily be your path at 32 or 42. One's dream is constantly evolving, rising and falling, changing course. This happens in every job, but because I have worked in comedy for twenty-five years, I can probably speak best about my own profession.

Way back in the 1940s there was a very, very funny man named Jack Benny. He was a giant star, easily one of the greatest comedians of his generation. And a much younger man named Johnny Carson wanted very much to be Jack Benny. In some ways he was, but in many ways he wasn't. He emulated Jack Benny, but his own quirks and mannerisms, along with a changing medium, pulled him in a different direction. And yet his failure to completely become his hero made him the funniest person of his generation. David Letterman wanted to be Johnny Carson, and was not, and as a result my generation of comedians wanted to be David Letterman. And none of us are. My peers and I have all missed that mark in a thousand different ways. But the point is this : It is our failure to become our perceived ideal that ultimately defines us and makes us unique. It's not easy, but if you accept your misfortune and handle it right, your perceived failure can become a catalyst for profound re-invention.

So, at the age of 47, after 25 years of obsessively pursuing my dream, that dream changed. For decades, in show business, the ultimate goal of every comedian was to host The Tonight Show. It was the Holy Grail, and like many people I thought that achieving that goal would define me as successful. But that is not true. No specific job or career goal defines me, and it should not define you. In 2000—in 2000—I told graduates to not be afraid to fail, and I still believe that. But today I tell you that whether you fear it or not, disappointment will come. The beauty is that through disappointment you can gain clarity, and with clarity comes conviction and true originality.

If Not Randomized Trials, Then What?

The above chart shows the returns to various approaches to keeping kids in school, according to research conducted by the Poverty Action Lab (J-PAL) at MIT.  The results are striking. Spending $100 on public awareness in Madagascar yields a whopping 40 extra years of student attendance.  $100 spent on deworming in Kenya generates an additional 28+ years of student attendance. Public awareness and deworming campaigns are 10 to 40 times more cost effective than providing school meals, scholarships, or uniforms.

 J-PAL has popularized the use of randomized controlled trials (RCTs) in development aid, a movement that has recently been highlighted by Nick Kristof in the NY Times.  Kristof describes RCTs as the "hottest thing in the fight against poverty."  And when you see striking results like those in the chart above, it is hard not to get excited.

Michael Kremer, a professor at Harvard, pioneered some of the earliest randomized trials, including those on de-worming in Kenya. He has gone on to help design and oversee USAID's new Development Innovation Ventures, which promises to seed a lot of new approaches on the condition that they be rigorously evaluated.  Michael's common sense approach to finding out what works (who would have thought that de-worming would be such a high-return investment in education!) makes this USAID initiative potentially promising.

The good news about RCTs is that they can guide the design of development projects in some circumstances.  For example, in the regions of Kenya where Michael did his work, the most rational allocation of education expenditures might be first to de-worm all the children.  Only after that is done would it make sense to spend money on things like subsidizing uniforms.  I also have argued that the expensive Millennium Village Project should be subject to an RCT "competition" with other approaches.

But promising new tools often get promoted as silver bullets, and RCTs are no exception.  This inevitably causes a backlash, which in turn means that, after 15 minutes of fame, many good tools fail to be adopted to an optimal degree. Rachel Glennerster, the director of J-PAL, told Owen Barder last year that there was a risk of too much hype, and that RCTs were not feasible or desirable in all circumstances.  For those interested in this topic, I recommend the full interview.

What are the limitations or even downsides of RCTs?  Angus Deaton is one of the strongest critics of RCTs, for a number of conceptual and methodological reasons.  He argues that any statistically valid RCT must be very narrow in terms of applicability. For example, the findings on de-worming in Kenya cannot be generalized even beyond the villages in which the experiments were run.  To his point, J-PAL's chart above shows that the returns to de-worming in India are only about 10-12% of the returns to de-worming in Kenya.  To the extent RCTs are so context specific, their usefulness is severely limited.

Deaton also argues that RCTs may capture the mean but not the variance of the effects that a project has on beneficiaries in the studied population.  Though an initiative may on average have positive effects, it may have a negative impact on a substantial proportion (or even majority) of the beneficiary population.  And vice versa: an initiative which shows a negative average impact might benefit many beneficiaries.  Drawing any sweeping conclusions under these conditions, Deaton argues, is not warranted, and could even be dangerous.

Others, such as Arvind Subramanian at the Center for Global Development, argue that even if RCTs can shed light on the effect of a development project in limited circumstances, they cannot tell us anything about whether aid itself works or not.

A young researcher from within the RCT movement noted to me recently that randomized trials can only predict marginal impacts and cannot be extrapolated.  For example, the effect of public information may have the effect of keeping individual kids in school one month more; providing 12 times more information will not keep the kids in school an extra 12 months, which may be the goal.

At some point, a clever economist will try to carry out a randomized controlled trial of RCTs.  She will attempt to answer the question: Does the use of RCTs lead to changes in the design of aid projects that translate into improved well being for people in developing countries?

My guess is that such a meta-RCT would not show a strong positive impact, for several reasons.  First is the cost of RCTs.  Even if they are a gold standard (which Deaton disputes), the costs are such that they will only be able to be done on a minuscule proportion of development initiatives.

Second, the current incentive structure in the aid industry results in an attenuated link between evidence on impact and changes in project design.  Aid providers face little competitive or other pressure to seek out the initiatives that have the greatest impact.  In fact, Lant Pritchett argues that, under the current structure, it "pays to be ignorant" because confessing failure hurts you more than success benefits you (in terms of political and financial support).

The bottom line is that RCTs will become like Consumer Reports trials in the consumer marketplace.  Consumer Reports is a useful adjunct to decision making for some things (although often I can't find the exact models they tested!)  But innovation in service of improved quality and lower cost comes from market pressures arising from consumer feedback through purchasing decisions.

In the same way, the best hope for improving the impact of aid initiatives is to create much richer and more real-time feedback loops between beneficiaries and aid providers.

Instead of determining needs ex-ante through expert studies, we need to start with asking beneficiaries "What do YOU want?" New technologies and approaches enable us to do this on a far wider scale, at dramatically lower cost, than ever before.  And then once a project is underway, we need to ask beneficiaries "How do YOU think it's going and what changes need to be made?"  And once that project is finished, we need to ask "Given what we learned from the previous project, what is the next thing you want?"  The faster we can iterate through these questions, the faster we will get to greater impact.

Naturally, experts should provide technical analysis such as RCT results to beneficiaries to help inform their responses.  But in the end, we need to make the beneficiary king.