Air-Traffic Controllers' Lesson for Development

Guest post by Felipe Cabezas.

Thinking about the recent buzz about aid transparency on my flight from Milwaukee to Washington, DC, I wonder if we will use more information to improve programs’ effectiveness or to conduct business as usual. Of course, a large part of that depends on how we gather – and display – information.

Perhaps we should turn to air-traffic controllers for guidance.

Air-traffic controllers oversee specific airplanes within a specific flight zone – not the entire air space. When trying to land an airplane, they communicate directly with the pilot and elicit pertinent information. They already know some things (for example, current weather conditions) but rely entirely on the pilot for airplane-specific information (for example, mechanical issues). When assessing information from multiple airplanes, air-traffic controllers discuss among themselves and develop a plan to land all of the airplanes within their flight zone. But this plan continually evolves. At any moment, an unexpected problem could occur which would cause the air-traffic controllers to adjust their initial plan. And because air-traffic controllers are not flying the airplanes, they need to constantly share their updated plans with each pilot to ensure that no airplane lands unsuccessfully – or worse, collides with another. All of this occurs from the moment an airplane enters a flight zone until it reaches its terminal gate.

So, what do air-traffic controllers teach us about aid? Just as Dennis wrote earlier, actionable, transformative information comes from the right sources and is provided to the right people at the right time.

From the Right Sources: We must communicate directly with beneficiaries as well as other stakeholders who have been the primary source of information in the past. Beneficiaries have critical information that experts do not have. But experts also have information – including lessons from experiences elsewhere – that beneficiaries do not have.

To the Right People: Aid agencies and stakeholders must collaborate not only ex-ante but also during implementation. Aid agencies devise programs that are intended to help beneficiaries. Due to unexpected problems, no aid program works perfectly as designed. Stakeholders (especially beneficiaries) must provide feedback, and aid agencies must listen and readjust their programs accordingly.

At the Right Time: A donor-stakeholder feedback loop must exist in real time. Beneficiary feedback is key. An unexpected problem must be identified and corrected immediately to ensure that a program remains on course. If not, then the program may not serve – and may even harm – the beneficiaries.

Air-traffic controllers want their airplanes to land successfully and rely on a robust, real-time feedback loop to prevent a pile-up of airplanes on the tarmac. Errors have catastrophic – and visible – consequences, which is why good feedback systems have been invented.

The costs of failed aid programs are less visible but no less tragic. The good news is that the right kind of transparency can lead to feedback loops that improve program impact significantly. A recent study in a small area of Uganda showed that improving transparency around the performance of health clinics reduced infant mortality by 33 percent, thereby saving an estimated 550 lives – the same number of people that a Boeing 747 holds.

Who did this?

We shall find too that such current notions as that society ‘acts’ or that it ‘treats’, ‘rewards’, or ‘remunerates’ persons, or that it ‘values’ or ‘owns’ or ‘controls’ objects or services, or is ‘responsible for’ or ‘guilty of’ something, or that it has a ‘will’ or ‘purpose’, can be ‘just’ or ‘unjust’, or that the economy ‘distributes’ or ‘allocates’ resources, all suggest a false intentionalist or constructivist interpretation of words which might have been used without such connotation, but which almost invariably lead the user to illegitimate conclusions.  We shall see that such confusions are at the root of the basic conceptions of highly influential schools of thought which have wholly succumbed to the belief that all rules or laws must have been invented or explicitly agreed upon by somebody.

That is from Friedrich Hayak's Law, Legislation, and Liberty via Cafe Hayek (HT to Bill Easterly).  The full post is worth reading for a quick summary of some of Hayek's key insights.  Among them is this:  that rules, laws, regulations, institutions, etc were generally not invented by specific people or by a society for specific purposes.  Hayek's point is that in most cases they actually evolved organically.

A corollary is as follows: Being able to describe (a) how a system currently does function and then (b) how it should function generally does not (c) lead to the system getting changed.

Learning from Mistakes - Medical Edition.

“When you break that paradigm of litigation and give patients the chance to understand the human element of the other side — of the doctor and what they are struggling with — you find that people are far more forgiving and understanding than has been typically assumed,” said Richard C. Boothman, one of the study’s authors and the medical center’s chief risk officer, who devised and carried out the disclosure program. “We have given patients no alternative but to sue, and then we use the fact that they sue to show how opportunistic and awful they are.”

That is from an article in the NYT about a new approach being taken by the Univ. of Michigan Health System.  Instead of circling the wagons and taking refuge behind the lawyers when a mistake is made, the hospital staff admit the mistake to their patients and talk about what they have learned and what they are going to do to avoid the same mistake in the future.  Sometimes they compensate the patient or family, but that is worked out directly rather than via a lawsuit.

You really have to hand it to the University of Michigan for trying to break with normal practice, which is that doctors are discouraged from admitting or talking about mistakes, preventing others from learning from them.  The article notes:

"That openness has in turn created an environment where patient safety and patient care, not avoidance of litigation, have become the priority."

Amen.

Don't Make the Same Mistake I Did

“We dump hardware down and hope magic will happen,” said Michael Trucano, senior information and education specialist at the World Bank, whose offering to FailFaire was a list of the 10 worst practices he had encountered in his job.

Trucano won the award for best failure at a recent Failfare gathering at the World Bank that was co-hosted by MobileActive.  Earlier I blogged about the product pileup of things the aid industry invents that people don't want.  Events like this at the World Bank are encouraging, because talking about failure is the first step toward not making the same mistakes again and again.

The full story, by Stephanie Strom in the NYTimes, is here.  Trucano's own account, with many insights, is here.  A very nice post by the World Bank's Aleem Walji is at the Development Marketplace blog.

Solving the Wrong Problems

"It turned out we were solving the wrong problem."

That is Saul Griffith, a wunderkind inventor who created a new way to make eyeglasses on an inexpensive device in developing countries.  For that invention and others, Griffith won a MacArthur genius award.  The eyeglass machine was a great invention; unfortunately, the real constraint turns out to be testing eyes and writing accurate prescriptions rather than making the lenses.

Griffiths' story is highly relevant to the aid business, especially these days as new foundations and people with experience in technology are trying to help address challenges faced by the world's poorest.  A lot of smart people are trying to help, and since their expertise is in software or technology, they (naturally) try to use those tools first.

But as Griffith says in the article, "The speed with which software-based activities and web innovations catch on - text messaging, eBay, Twitter - has encouraged public perception that transformative technological change takes place almost instantaneously."  Unfortunately, this is not the case with most development challenges, as in the case of the new eyeglass machine.

A more fundamental problem is that the inventors don't have a good way to assess demand.  So they invent all sorts of things that *seem* to make sense, but turn out not to be used or adopted by the intended beneficiaries.  Over at the CGD blog, April Harding has written about the "product pileup" in the health sector, where new agencies and foundations have created products such as Malaria Rapid Diagnostic Kits and bednets.  She cites a book by Laura Frost and Michael Reich that discusses why these innovations are not being adopted, and what can be done to reduce this mismatch between supply and demand in the health sector.

With support from the Rockefeller Foundation, GlobalGiving and Innocentive have teamed up on a pilot to reverse the sequence of innovation.  First, we ask the communities what they need.  And then we advertise that need to Innocentive's virtual teams of inventors.  This pilot is described here.


Illustration source: http://www.buzzle.com/articles/small-business-financing-is-missing-the-target.html

When Supply Meets Demand for Innovation

Guest post by Britt Lake

With support from the Rockefeller Foundation and in partnership with InnoCentive, we at GlobalGiving have been piloting a way to enable communities to tell the world what problems they need help solving.  We've then been crowdsourcing solutions to these problems to bring clean water and electricity to communities in India, Uganda, Peru, and around the world.

First, we reached out to hundreds of local organizations around the world to determine their communities' biggest challenges.  The responses came pouring in: How can we build a rainwater harvesting storage tank that is appropriate for our region? How can we design an indicator that would show us when water has enough exposure to UV light to make it safe to drink? How can we create river turbines with local materials that would provide electrical power to villages in the Peruvian jungle?

We then posted five of these challenges on the InnoCentive website, which in turn broadcast them to potential inventors around the world. The initial results are promising.  Four of the five challenges have already found potential solutions. 

Take the EDGE Project.  EDGE is an organization that researches, designs and implements sustainable development projects on the Ugandan island of Lingira in Lake Victoria.  Since they were founded, they’ve been trying to find a way to make water from Lake Victoria safe to drink for the Lingira community.  They have tried boiling the water, using biosand filters, chemical water treatments, and an electrochemical system, but none of these have provided sustainable solutions for the local environment.  These existing methods make the water taste bad, require expensive replacement parts that are not found in the community, or don’t get the water completely clean.  In the two months after the challenge was posted, the EDGE project received 85 potential solutions!  The EDGE team reviewed all the submissions and selected one promising new approach - a new type of ceramic pot filter that both cleans the water and stores and protects it from re-contamination. 

Through this process we’ve been able to figure out what people need, and we’ve identified potential solutions to these challenges.  Our final step is getting these solutions funded and tested.  We’re working with our partners now to find out what it will take to get these solutions working on the ground, then we’ll use crowdfunding to actually get these solutions tested.

When Top-Down is Needed: World Bank Reform

Reforming the World Bank, by David Phillips, is the most constructive book on the topic I have read.  Phillips, a long-time World Bank staff member, spent his career advising developing country institutions how to make the changes needed to carry out their functions effectively.  Upon retirement, Phillips decided to turn his formidable analytical skills toward his own previous employer.   He has produced a refreshingly different book about reforming the largest international aid institution.

Phillips describes not only the changes that need to be made but also offers insights into how to get them done.  New aid donors and new social technologies threaten to make the Bank and other aid agencies irrelevant if these agencies do not change with the times.   The leaders of those traditional agencies would thus do well to heed the lessons of this book.

In the fourteen years I worked at the Bank and in the ten years since I left, I have read many critiques of the institution from a variety of sources - academicians, NGOs, government commissions, and journalists.  Most are arguments for the Bank to do more or less of something (engagement with civil society, environmental projects, gender projects, infrastructure projects, policy lending, trade adjustment loans, government transparency and corruption initiatives, etc).  Others argue that the Bank should change the way it works at the margin, by making more grants and fewer loans, for example.

But few critiques show much understanding (or even interest) in how to effect the changes prescribed.  That is the great strength of Phillips' book.  He has thought carefully about the forces that have led to changes in the Bank over time, and how they might be brought to bear today.  More importantly, he describes with great clarity the incentives at work at all levels in the institution.  He knows what individual staff and management are held accountable for in reality (vs the rhetoric).  And like Moises Naim, former board director of the Bank (and most recently editor of Foreign Policy Magazine), Phillips is able to elucidate the incentives facing the board of directors as well.

The crux of this book (and something not duplicated by Naim or others) is Phillips's insights into the tensions between management and the board.  Those tensions are a binding constraint to real reform and change at the Bank.  They are all the more intense because the Bank has a resident board of directors that typically meets twice a week(!).  Phillips describes the sophisticated (and rational) efforts by management to keep the board from micro-managing by flooding the board with too much information to digest. The board, in return, distrusts management, and repeatedly tries to rein in management initiatives.  This vicious cycle eats up a huge amount of staff and management time. The brainpower, relationships, and energy of board members are also sapped by the struggle with management instead of applied to development problems.

The result is an inward-looking institution whose whole adds up to less than the sum of its parts, and whose ability to change and innovate is sharply limited.  Before I left the Bank in 2000, I asked many of my colleagues, whom I admired greatly, what percentage of their skills, energy, and time got applied directly to development challenges facing their client countries.  The response I got from most of them was "twenty-five to thirty percent."  The rest of their potential was dissipated on internal processes and compliance with regulations.  Given the tremendous assets of the Bank in terms of people, money, and relationships, this squandering of potential is a real tragedy - especially for the world's poor.

On the eve of my departure from the Bank, one of the managing directors asked me for advice on his ongoing reform effort.  "What do you think are the most important things?" he asked.  I told him that in my view real change would not happen unless the Bank deliberately subjected itself to outside competition that threatened some of the Bank's own resources.  Only if the Bank had something concrete to compete against could its real progress be assessed objectively, and only then would the board and management feel compelled to develop a more productive way of working together in order to survive.

The managing director was aghast at this idea, since he felt it would threaten the very institution he was charged with protecting.  But this is short-term thinking.  In the end, it will be up to the board and the Bank's shareholders, including the US government and others, to show the courageous leadership needed to bring the Bank into the 21st century.  Without such leadership, the Bank will remain insulated from competitive forces and will continue using the majority of its resources on internal processes rather than on promoting prosperity for the world's poorest.  But if the will to lead is there, Phillips' book can help guide the way.

Mapping the World Bank's Footprint

The World Bank, in collaboration with AidData and the Development Gateway, have just announced the release of a service that allows you to use Google Earth to see exactly where Bank projects are being implemented around the world.  In seven weeks, the team (which included members from Brigham Young University, William and Mary, and Georgetown) geocoded over 12,000 project locations.  As they explain, this information was often available, but it was buried deep in hundreds of different reports.  In practice, this meant that it was unusable for the vast majority of people.

This is a big step forward.  As I have noted before, information availability per se is not enough.  The information must be available in a format that is easily usable, and on this front, the team working on this seems to have made much progress.  What encourages me about this effort is also the inter-sectoral nature of the collaboration.  It included not only World Bank experts, but also academicians and an NGO.  It had seasoned aid veterans (Jean-Louis Sarbib used to be a VP at the World Bank) working together professors and students.  Each of them brings to bear perspectives and expertise that the others don't have.  That is the wave of the future for the development field as a whole.

I was also encouraged by the talk of empowering communities themselves to take action.  Information and transparency initiatives only succeed if they enable or require people to act or change their behaviors in a way that will make a real difference.  A number of other aid transparency initiatives are underway, and as I watched the video at the link, several important drivers of success came to mind.  Initiatives that make a difference will:

1) Be as real-time as possible.  The usefulness of the data falls off sharply as a function of age.

2) Allow and even encourage two-way flows of information.  Users have to be able to comment on the information in a way that everyone can see (and act on - see point 4).

3) Have strong incentives for updating.  As I have confessed before, I have run or been involved in a couple of massive data exercises that collapsed because after the initial push there was no ongoing incentive for people to refresh the data.  This aspect is very hard, but critical.

4) Have a clear theory of how the information will affect behavior, and that theory needs to be reflected in what data is collected and how it is presented (for an obvious but compelling example, see here.)

5) Be as simple as possible. Present only the information needed to elicit behavior change.   There is always the temptation to have more bells and whistles, but if there is one thing I have learned at GlobalGiving it is that complexity and clutter kill.  Simplify, then check to see if people are using the data, and if not, then change the data or the presentation.

Aid and Aviation

This guest post is by Felipe Cabezas.

When flying, do you want to know this information?

Before I board a plane and while I’m in the air, I don’t care about aviation statistics. They fly (get it?!) out the window. I could not care less about how many planes landed safely worldwide in the past year.

Or this information?

Instead, I worry about my flight. How are the weather conditions? Are there any mechanical issues? Did the pilot have a good night’s rest? These are the factors that will determine if my plane lands safely at Reagan National Airport. Flight so-and-so that landed safely in LaGuardia a week earlier does not factor into the equation – by a long shot.

Now, that’s not to say that aviation statistics aren’t important. They’re good to know when booking a flight and assuring myself that traveling by plane is safer than by car. But the only times when I seriously consider them are before and after my flight . . . when I’m on the ground . . . safe and sound . . .

When I read articles, documents and position papers about international aid and look at accompanying graphs and charts, I wonder if we focus too much on aviation statistics. (Child mortality worldwide has decreased by X percent over the past decade.) That’s good to know, but we’re not on the ground. We’re in the air. Perhaps we should start gathering more information about specific flights and assessing it more closely to gauge how likely we are to land safely.

Scoring Points or Making Progress?

Source: http://josh-wyxl.itmblog.com/files/2010/01/how-to-win-every-argument-main_full.jpg

The smaller the stakes, the more intense the backstabbing - or so the old saying about academia goes.  But sometimes the backstabbing is bad even when the stakes are high.

I went to an aid conference last year where the speakers took turns ridiculing each other's work.   The speakers in question were all among the top in their field, and all had tenure, so to my mind they had little reason to be insecure.  And many of them are even personal friends!  The atmosphere seemed almost childish - though most everyone there was over forty.

I asked one of them afterwards why they all seemed intent on attacking and triumphantly exposing a shortcoming or flaw in the others' presentations, with the implication being the whole approach was baloney.   Why not build on what was valid in others' presentations and then refine one's own view, so as to more quickly advance their understanding of the world? 

"That's not the way it works," my friend told me.  "You advance knowledge by mud-slinging, not by listening.  That's the way it has always been and always will be."

I just can't accept this. People need to get past their own egos and actually talk to - and learn from - each other, especially when the stakes are high.  In that context, I was really happy to have a good conversation about RCTs (Randomized Controlled Trials) with some friends and highly qualified colleagues on Facebook.  I learned a lot from it, and it may even generate a paper.  I am reproducing my original post, along with their comments (with their permission) below.

Traditional Evaluations are Not Scalable

 Thursday, July 29, 2010 at 7:23am | Delete

It’s a standard trope of this blog to point out that there’s no panacea in global development. That’s true of impact evaluation, too. It’s a tool for identifying worthwhile development efforts, but it is not the only tool.  We can’t go back to assuming that good intentions lead to good results, but there must be room for judgment and experience in with the quantifiable data.

That is Alanna Shaihk guest blogging at AidWatch.   She describes two limitations to evaluation discussed by Steve Lawry of the Hauser Center at Harvard.  Excessive reliance on evaluation, Lawry says, stifles innovation and artificially constrains aid agencies to initiatives that can be easily measured with data.

I would add a third limitation.  Formal evaluations, including the gold standard of randomized controlled trials, are not scalable.  We simply do not have the time and resources to do centralized, in-depth evaluations of everything.  The only way forward is to establish a decentralized, implicit form of evaluation in which beneficiaries and other stakeholders can provide feedback about quality and relevance of aid projects.

This is how markets work.  The magazine Consumer Reports does a great job of evaluating products.  But it evaluates a miniscule proportion of all the products produced each year in the economy.  So who evaluates the other 99.9999% of products?  The consumer.  If the consumers buy a product, it keeps getting produced.  If not, it doesn't. Does this system work perfectly?  Of course not.  Does it work better than any alternative we have found?  By far.

Related articles by Zemanta

• What can we learn from randomized evaluation? (podcast) (owen.org)

• 
  
  
  
  • 
    
    
    
    
    

    Michael Clemens A lot of the opposition to RCTs seems to be opposition to a *requirement* for RCTs. How can an RCT by itself 'stifle innovation'? You can certainly stifle innovation by requiring one single evaluation method for all projects, which would be dumb. Using RCTs in the specific settings in which they can be used is just one additional way to generate information (very good quality information) to add to the marketplace.
    

    August 1 at 1:07am · ·
  • 
    
    
    
    
    

    Dennis Whittle
    

    Michael - you are completely right, and said it better than I could have - RCTs are a valuable part of a toolkit and very useful for certain things. I am making a broader point about traditional evaluations in genera, which are (a) too exp...ensive and cumbersome, (b) often suffer from the problem of "you can tell a project's position or you can tell its velocity, but not both at the same time" problem, and (c) have little or no effect on future actions. The effective feedback loop issue is the killer. The last speech I gave before I left the World Bank was the keynote at the Evaluation Unit's annual retreat. The title was "I Have a Dream," and the second line was "that project staff would come read evaluations before they started a new project."

    August 1 at 6:02am · ·
  • 
    
  • 
    
    
    
    
    

    April Harding
    

    Sorry I missed that speech Dennis. I hope you saved it 'cause operational staff still don't (or don't have time) to read impact evaluation results. You could give it again some time.
    
    My problem with RCTs (or the RCT movement) is that they a...re not, by and large, answering the policy and program relevant questions - so those expensive evaluations are generating answers to unimportant questions (or partial answers to important questions). It is my sense that this phenomenon is "crowding out" much needed research on critically important questions. Yet researchers won't shift to the more important questions because the methods you can apply aren't as rigorous - hence they won't get published, or they will lose status in their social milieu - or what ever.
    In the worse cases, researchers then take their answers to one question (e.g. slope of demand curve for a health product) and pretend to have answered the important policy question (e.g. most effective design of a program to increase of the health product). See More

    August 1 at 6:54am · ·
  • 
    
    
    
    
    

    Michael Clemens
    

    Thanks April, that is really interesting. I can be too much of a cheerleader for randomization, because I've seen firsthand that it's possible to work in an area where people have said that rigorous evaluation can't be done -- international... migration -- and yet I found several ways to do it, by finding natural experiments. But what I need to pay more attention to is that there are huge numbers of programs in which there isn't a convenient natural experiment and it would be very costly to design an experiment into it, but people still need to have a good idea of the impact. And as you say, many researchers just don't have an incentive to touch such a project.
    
    Maybe what's needed is the same evolution that took place in medicine, after that field began to adopt randomization for some purposes. In medicine researchers can publish results from Phase I trials (almost never randomized), Phase II trials (sometimes randomized), and Phase III trials (usually randomized). Many patients who need a solution *now* go for experimental treatments that are still in Phase I or II. In other words, for many patients it doesn't matter if there's ever a Phase III because if it doesn't work they won't be around to see it. And researchers have an incentive to do all three flavors because all three are publishable.
    
    So an important question is: How can the NGO/government/foundation world generate incentives for the creation of Phase I/II-style research? By creating a career path for people who do it, online journals for them to publish in, and so on, the way universities today and existing journals provide a career path for people who do exclusively Phase III-style research. This alternative research world could support those who do Phase I/II work even if the decision point for many projects must arrive before there's a Phase III, or even regardless of whether or not a Phase III is possible.
    
    Just brainstorming! And interested in what you think.See More

    August 1 at 7:39am · ·
  • 
    
    
    
    
    

    April Harding
    

    Michael, I think this is one of the most important issues we can think about - how to incentivize more research along that spectrum.
    
    To take the health program example:
    In health policy/ program design for developing countries you see two t...ypes of studies: studies doing all kinds of analysis on household survey data (driven by availability of data); and experiments w RCTs (driven by the range of things discussed here). Only a small portion of the burning questions can be answered with HHS data and experiments/ RCTs. If you look at health policy and program research in developed countries, policy decisions are informed by a vast literature (usually referred to as health services research) with standardized data from much broader sources (e.g. not just households, but also from providers and from payers and often from policy implementers. And you see a much broader range of methods applied - where the most rigorous method is selected to suit the question and data. Clearly - part of what enables this is the existence of health services research journals (and reviewers) who understand the range of methodologies and data, so that good research can get published across the range of data sources and methodologies. But another enabling factor is: the large investments made to collect and provide the standardized data in an open, user friendly format. It would make a huge difference if development assistance funding would get programmed toward providing this data (expanding on USAID's incredible contribution via the demographic and health surveys). Of course, these funds would generate results in the medium to long term - and who the heck funds initiatives with that kind of time horizon.
    
    Our pal Bill said something once, something like: there is no development economics, there is just economics. I often feel like saying something similar: there is no special field/ techniques for researching social programs and policies in developing countries, there is just high quality social policy research. We need to nudge the field away from it's love affair with RCTs (which is not to say become unconcerned with rigor) and away from it's dependence on narrow data sources - which can only shed light on a small fraction of the quesitons we need answered)See More

    August 1 at 10:53am · ·
  • 
    
    
    
    
    

    Michael Clemens Wow, this is fantastic, April, really clear. Want to write a little 'Note' on this subject with me? It would be great and people would read it.
    

    August 1 at 7:26pm · ·
  • 
    
    
    
    
    

    Dennis Whittle May I remind you both that I own the copyright to this excellent discussion since it occurred on my wall? I want 10% of the royalties and 5% of the movie rights?
    
    Dennis
    

    August 2 at 9:07am via Email Reply · ·
  • 
    
    
    
    
    

    Marc Maxson
    

    Dennis - I cringe at metaphors based on Heisenberg's 10^-34 less-than-certainty principle (maybe because I'm a scientist?). There are plenty of examples of tradeoffs, but there has got to be a better way than pseudo-quoting scientific prini...cples beyond their reasonable domain.
    
    On the metaphor to Phase I,II,III trials in medicine - it's worth paying attention to the biggest innovation to the system in the last decade. In 2005-ish, all drug trials had to be registered BEFORE they began in order to be considered as evidence in the subsequent FDA approval process. Some companies dropped marginal "me-too" drugs as a result, but more good drugs made it into FDA approval and with more reliable evidence. And best of all - the drug companies that would start and restart trials in phase III until they got the result they wanted were no longer able to bias their results.
    
    If you look back at the shouting over this rule, you'll see lots of people complaining that it would stifle innovation since drug companies would become more conservative and drop drugs before testing them all. I'm for that- it drives the cost of healthcare BACK DOWN. Isn't there a way to keep RCTs in check through a similar process to the drug RCT registry? That way negative data is always public and traceable, and RCTs are used for hypotheses that are a good bet, not a goose chase. That money could be used in some other less rigorous way for all the goose chases.See More

    August 2 at 11:05pm · ·
  • 
    
    
    
    
    

    Michael Clemens Thanks very much Marc, this is really fascinating.
    

    August 3 at 4:12am · ·
  • 
    
    
    
    
    

    Mari Kuraishi Wow. This has got to be the most substantive discussion I've seen on FB ever. Not to mention the most open-minded--and that's on any platform.
    

    August 3 at 10:22am · · 1 personLoading... ·
  • 
    
    
    
    
    

    April Harding Agree. This is a great discussion.
    Michael, let's get together to discuss writing policy a note when you are back in DC.
    
  • 
    
    
    
    August 3 at 2:56pm · ·
  • 
    
    
    
    
    

    April Harding
    

    Chris Blattman just blog commented on the issue of development research being led by data rather than bringing to bear the range of relevant methodologies (he categorizes them into quantitative and qualitative). He is discussing a recent ...paper by Bamberger, Rao and Woolcock. Seems relevant for our policy note Michael.
    I loved his recommendation: Marry someone who specializes in the "other" category of methodologies. That oughta help keep you honest. Failing that, find co-authors who fit the bill.See More

    August 3 at 9:49pm · ·
  • 
    
    
    
    
    

    Dennis Whittle
    

    Just catching up here. A couple of things. Marc, my use of the uncertainty principle metaphor is only half in jest. Using RCTs keeps the investigator from really trying to understand the phenomenon because of the need for double-blind te...chniques to avoid bias. But if the investigator really tries to understand the phenomenon by talking to the subjects, they bias the results, for a variety of reasons in my experience (which I can expand on later).
    
    Second, somewhere t I saw an analysis of the proportion of commonly accepted medical treatments that fall into Categories I, II, and III (RCTs) above. Only 1/3 or less were RCT-based. And a surprising number were Category I. This could have several implications. The distribution could be optimal in some sense. Or a number of Category I and II treatments could be useless or even harmful.
    
    Speaking of distributions, in medicine there is surprisingly little understand of the distribution of patient response to treatments. If the distribution has a very low variance, that has one implication, but if the distribution has fat tails or is even a a power distribution, then the implications are radically different.
    
    more...See More

    August 4 at 10:20am · · 1 personMarc Maxson likes this. ·
  • 
    
    
    
    
    

    Dennis Whittle
    

    ‎...(cont). Speaking of Category I, it is worth checking out Seth Roberts's blog (HT: Mari). Roberts is an academic who does a huge amount of self-experimentation to find out what works on him personally. His view is that there is far to...o little creativity brought to bear to come up with novel hypotheses about the pathways of disease. This is because it is so freakin' expensive to do RCTs. He is kind of all over the place but worth reading anyway.
    
    Third, I (manifestly) agree with Blattman about marrying someone in specializes in a different category of methodology. I can't imagine being married to someone who thought just like me. It is sometimes infuriating, like the tension in a good piece of music, but that makes the resolution all the richer.
    
    Fourth, I wonder more broadly whether there is space for some type of moderated blog ala Posner and Becker for development. The difference would be that the Posner and Becker would vary week to week. I would love to moderate something like that. Would it be at AidWatch? CGD? Something new?See More

    August 4 at 10:27am · ·
  • 
    
    
    
    
    

    April Harding I though posner and becker just took turns blogging, and occasionally commented on each other? Is that what you mean? Or do you have in mind a bigger group of bloggers? Or a blog with an overseer who blogged but also invited other bloggers on interesting topics?
    

    August 4 at 10:31am · ·
  • 
    
    
    
    
    

    Dennis Whittle Ah, my memory was that they commented on each other's blogs and had a running conversation. I am talking more about that style.
    

    August 4 at 10:34am · ·
  • 
    
    
    
    • 
      
      
      
      
      

      Marc Maxson
      

      Interesting Dennis (about Cat I,II,III). I do know these statements are true about modern medicine (and somewhat disturbing):
      
      1. Leading cause of death = going to the hospital
      2. 50% of American women over 50 were being given medicinal horse ...piss at one point, despite there being no RCT to prove it's impact. (Later the Framingham study and the Nurse's Health Study both showed estrogen replacement therapy to be marginally effective, and possibly harmful).
      3. More knowledge about medical outcomes and risk factors has come out from the Framingham, Nurse's Health, and 7-th-day Adventist studies than any RCT.
      
      I repeat! NONE of these are RCTs. All are longitudinal observations of an isolated, well-defined social group with well-controlled monitoring methods. Hence, I've been setting up the GlobalGiving Kenya story telling project to mirror these approaches.
      
      4. Isaac Asmiov: "Life is the cruelest of all teachers, because it kills every student in the end." Medicine is quite the same, although the intermediate phases appear to beat all alternatives.See More

      August 4 at 10:32pm · ·

    August 4 at 10:32pm · ·
  • 
    
    
    
    
    

    Marc Maxson April - I vote for married couples co-blogging and taking point-counterpoint. That would make for interesting reading.
    

    August 4 at 10:35pm · ·