Bingeing at the Information Buffet

My friend Jeff Koeze (who also makes the world's best peanut butter) sent me a link to a talk by Barry Schwartz at the TED conference. In his talk , Barry notes the exceptional abundance of intelligence and knowledge among those present at the conference. But he argues that good judgment, not intelligence, is the binding constraint these days.

I agree with Barry that raw intelligence and wisdom are not necessarily correlated. In fact, my own experience is that the variance in wisdom around the mean grows as raw IQ rises. Really smart people can have insights that can be breathtakingly innovative - or completely disastrous.

So I wonder about the effect of the huge increase in knowledge coming from a particular dimension of the web. I am not talking about Facebook. I am talking about sites such as TED Talks, which makes available hundreds of riveting talks by leading (and often photogenic) scientists and intellectuals. Or Edge (my favorite) , a website that publishes short papers, interviews, and debates with some of the smartest (but often less camera-ready) people in the world. Or Marginal Revolution, a blog written mainly by the super-human Tyler Cowen, who each day spews forth an alarming volume of links and his own strong opinions on everything from the economic crisis to Mexican painting to the best cheap restaurants in Northern Virginia.

I partake of these sites like I used to partake of all-you-can-eat buffets when I was in college: I cram in a huge array of different appetizers, meat, fish, and pasta courses, followed by a brownie, a piece of apple pie, and a blackberry cobbler to finish it all off. Then I stagger home, completely bloated and feeling queasy, swearing off the buffet forever. But the next week I am back, bingeing again.

We have made big strides in computing capacity - both hardware and software. Gmail went down the other day for a few hours, and it made news, because these days it is rare that even large, highly networked websites crash. Such websites can process information that is growing at an exponential rate, with nary a hiccup.

But have we made similar strides in either the hardware or software of the human brain? Ray Kurzweil argues we will augment human cognition with machine intelligence at an accelerating pace. But few would argue that the human brain's capacity is growing as fast as in the increase in information available to it.

So the question is this: how will we process all this knowledge that is available to us? The good news is that much innovation comes from the recombination of existing technologies and approaches, so the increase in knowledge about these can in principle lead to accelerating innovation. And heaven knows we are in dire need of to address the economic crisis and challenges such as climate change. I also believe in the power of the Wisdom of Crowds in many contexts; that is one of the principles on which GlobalGiving is founded.

But what can we do to both increase the mean and reduce the variance of the wisdom emanating from all this knowledge? Sometimes I wonder whether the sheer quantity of new information we now "consume" will increase the variance in our judgement and wisdom, leading to unforeseen consequences.

[GlobalGiving]

Pascal's tennis game

THE COMPUTATIONAL METHOD at the heart of Pascal’s work was actually discovered by a Chinese mathematician named Jia Xian around 1050, published by another Chinese mathematician, Zhu Shijie, in 1303, discussed in a work by Cardano in 1570, and plugged into the greater whole of probability theory by Pascal, who ended up getting most of the credit. But the prior work didn’t bother Pascal. “Let no one say I have said nothing new,” Pascal argued in his autobiography. “The arrangement of the subject is new. When we play tennis, we both play with the same ball, but one of us places it better.”

That is from The Drunkard's Walk: How Randomness Rules Our Lives by Leonard Mlodinow, an excellent book and an easy read. This passage should remind of us two things:

1) Sometimes astonishing insights can lie around for hundreds of years (in this case six centuries!) before they are widely recognized and applied.

2) Real innovation often comes about through the novel combination of existing ideas or approaches. We tend to dismiss the power of existing ideas as "old hat" or "already known." Instead, we believe that addressing our challenges or problems requires entirely new insights (preferably by lonely geniuses). That is a big mistake: combining, tweaking, and applying existing ideas or approaches is what causes most real breakthroughs.

(Thanks to Bill Easterly for recommending this book.)

[GlobalGiving]

Settle down, boys

On the web you best build an audience by organising a claque and stroking its prejudices. Extend elaborate courtesy to people you agree with and boorish contempt to those who do not get it...

That is Clive Crook in the FT calling Paul Krugman and Robert Barro on the carpet for their recent columns in the NYT and WSJ.  He goes on to say:

Economics outside the academy has become the continuation of politics by other means...Mr Krugman gives liberals the economics they want. Mr Barro gives conservatives the same service. They narrow or deny the common ground.

Krugman and Barro both responded to Clive indignantly here, and the tone of their responses only proves Clive's point. 

ButI don't agree with Clive that people like Krugman and Barro act differently inside the academy.  My own experience is that the petulance on display at academic conferences often meets or exceeds that on display in the NYT and WSJ.  It makes me wonder why so many very smart people have such fragile egos.   Sadly, those egos serve to obscure rather than advance our understanding of the world.

[GlobalGiving]

Power to the people.

Here are some slides from a presentation I gave at a conference a hosted by Bill Easterly at NYU last week. The topic was Accountability in Development Aid.  I spoke about how decentralized systems like GlobalGiving enable much more open access to the aid process - and how they offer the potential for greater accountability to the people that aid is supposed to help in the first place. We are still in the early stages, but the possibilities are great.

Accountability in Decentralized Systems

Publish at Scribd or explore others: Academic Work GlobalGiving

[GlobalGiving]

Easterly 2.0

Bill Easterly has just started a blog called "Aid Watch." And although Bill has bad judgement with respect to sports teams, you should subscribe to his blog if you want clear thinking about how to fix the international aid system.  Judging by the large number of comments on his first two posts, this blog is going to be very popular, and for good reason.

Bill is a former colleague at the World Bank.  More importantly, he is the author of The Elusive Quest for Growth and White Man's Burden, two must-reads for people who want to get a quick (and entertaining) overview of the history of international aid, why it has underperformed, and what must be done to re-orient it.

Bill is also the editor, more recently, of Reinventing Foreign Aid, a collection of essays by some of the leading thinkers and doers in the field.  Get that book, read the executive summary first, and then dip into selected chapters that are of particular interest.  (Full disclosure:  Mari Kuraishi and I also wrote a chapter called "Competing with Central Planning" in that book, and GlobalGiving is mentioned in White Man's Burden.)

One of the reasons Mari and I left the World Bank to start GlobalGiving is that people were not satisfied with the once-a-year physical Development Marketplaces we started there.  One woman from South Africa asked us: "Why can't you also start a secondary marketplace that operates 24/7 365 days a year?"

Think of Bill's blog as his own 24/7 marketplace of ideas.  I hear rumors from an inside source that he is now thinking about writing his next book.   If so, you can get a heads up on his new ideas - and even participate in debating and honing them - through his blog.  

[GlobalGiving]

5 elements of a new aid system

The race for who will head the large US aid agencies is heating up. Everyone seems to agree that changes to the system are needed. But are we up for real change, or will we settle for  marginal shifts in emphases and organization? I fear that the changes will be marginal, when we need something more fundamental.

Earlier I proposed the 2/3 : 1/3 rule. Under this rule, 2/3 of all aid money would be allocated through bottom-up, market-based mechanisms, and only 1/3 would be allocated via top-down, centrally planned mechanisms. Today the ratio is reversed, which is one reason the productivity of aid is so low.

"Market mechanism" does not necessarily mean "private sector," although it does mean that the private sector and community/civic groups would have equal standing with official aid agencies and large non-profits.

Market mechanism means that the best qualified people and organizations - from any arena - are eligible to carry out the five core elements of any aid (or philanthropy) program:

1. Decide on what problems need to be addressed
2. Decide what the best approaches or solutions to these problems are
3. Fund the initiatives or projects that embody these solutions
4. Implement and supervise the projects on the ground
5. Evaluate the impact of the projects, and how that evaluation is used

The key is to open-source the above elements, and then reward success.  Big official agencies either provide most of these elements in-house, or they rely on a small group of "beltway bandits" to do the work.  And the rewards for success and failure are very muted. 

That may have made sense at one time, when the pool of money and expertise was very limited.  But the world has changed radically in the last fifty years, and now it is time for the aid system to change as well.

[GlobalGiving]

The 2/3 : 1/3 Rule

With the new administration coming to power in the US, there is a flurry of new proposals on how to reform the aid system. However, few of them propose real change. Instead, there are proposals to increase aid to such and such issue or country. Or to strengthen such and such agency - or to appoint a strong new leader.

Few of these proposals get at the root of the problem. As Bill Easterly has pointed out, we have spent more than $2 trillion in aid over the past fifty years with not enough to show for it.

The problem is the centrally planned, expert-driven, top-down nature of the current aid system. Just like under the Soviet regime, this approach does get things done. But the quality is bad, shortages are common, and the people have little say in what gets produced.

So let me propose the 2/3 : 1/3 rule. Henceforth, 2/3 of all aid resources will be allocated through an open-access, bottom-up, market mechanism, while 1/3 of the resources will be allocated through existing top-down approaches.

I will write more about how the marketplace system would work in coming posts and columns.

[GlobalGiving]

What percentage of your brainpower are you using?

Last week I posted a trailer of the documentary of Bill Easterly's White Man's Burden.

Below is a clip where I talk about how much brainpower is wasted in top-down hierarchical systems like the World Bank:




[GlobalGiving]

The Clint Eastwood of International Development

Check out the trailer for the coming documentary for The White Man's Burden, a book by Bill Easterly. The movie is coming out in 2010, and I urge you to read the book in 2009 if you have not already.




[GlobalGiving]

"Any other result is possible"

Last night, my friend Clive lent me some music by John Martyn, an outstanding but little known Scottish singer.  Martyn's obscurity led us to talk about the influence of chance, serendipity, and randomness on success.  Much of what happens in life is influenced by unforeseen (and often unnoticed) events and factors that we do not control.

A nice study by Matthew Salganik and others shows the degree to which musicians' success is hard to predict. The key take away is: "Success was also only partly determined by quality: The best songs rarely did poorly, and the worst rarely did well, but any other result was possible."

Hit songs, books, and movies are many times more successful than average, suggesting that "the best" alternatives are qualitatively different from "the rest"; yet experts routinely fail to predict which products will succeed. We investigated this paradox experimentally, by creating an artificial "music market" in which 14,341 participants downloaded previously unknown songs either with or without knowledge of previous participants' choices. Increasing the strength of social influence increased both inequality and unpredictability of success. Success was also only partly determined by quality: The best songs rarely did poorly, and the worst rarely did well, but any other result was possible.

The full study is here. (Free registration required.) And thanks to Richard Thaler and Cass Sunstein's book Nudge for the tip.


[GlobalGiving]