James Surowiecki spoke at the SXSW Interactive conference in Austin, Texas on Saturday 11th March 2006 about the ideas in his bestselling book The Wisdom Of Crowds. Its relevance to Web 2.0 loomed large…
[WARNING. This is a motherlode post. If that bothers you, go read it on a traditional website 😉 ]
For anyone familiar with Suroweicki’s book, this session can’t have been much of an eye opener. But I only knew of it anecdotally. So I was perfect fodder for a recap of the book’s central ideas which the author duly provided.
A lot of people blogged their dissatisfaction with this session, but they should remember that his book hasn’t been out that long, and even then, plenty of us who would like to have read it already somehow haven’t found the time. On balance, I think he could have done two sessions – this one as the primer, and a second one looking at debates around or syntheses of his ideas. I would have gone to both!
James explained that he is a business columnist for the New Yorker and took the story of Francis Galton as a starting point for illustrating his thesis. Galton (1822-1911) was a British exponent of [the roundly discredited theory of] eugenics. One day at a country fair in England, as Galton tells it, people were lined up to guess the weight of an ox after it had been slaughtered and dressed.
It was a big and diverse crowd – some there were experts on cattle and many weren’t. Galton looked at the statistics and then took the average of the groups guesses – he thought it was erroneous, but the crowd’s judgement was perfect to within a pound.
A young lady’s primer…
Under the right circumstances, Surowiecki extrapolated, groups of people can be very intelligent, and can be smarter than the smartest persons within them. His interest lies in what intelligent crowds look like and what problems they can solve. He cited the latterday example of the gameshow Who Wants To Be A Millionaire? and the reliability of nominating your expert in the show when you’re unsure or bereft of an answer. Experts get it right on average two thirds of the time, but the crowd is right on 90% of occasions!
Google is a great example of the wisdom of crowds in action, he said, in terms of its page rank algorithm [although this has been challenged by some search experts], adding the caveat that it taps into the collective intelligence of webmasters. Google’s founders realised that underneath the seeming chaos of the web, is a lot of order.
The collective forecast is good
A lot of problems that we’re concerned with are about events that will or might occur in the future, he noted.
The odds in horse races almost perfectly predict the wins and placings of the horses, providing an almost perfect forecast of the future. The stock market over a period of time is almost impossible for an individual money manager to beat (ie. the individual experts against the collective intelligence of money managers).
Tradesports.com tracked the US elections and they forecast 50 out of 50 states correctly and 32 out of 34 senate seat results.
Hewlett Packard in 1998 set up an internal stock market on what printers, etc would sell, and it out-performed the internal predictions team. Eli Lilly used the same system to track which drug candidates were most likely to make it through the drug trials. They also opened up their markets to the public / the non-expert audience.
When you aggregate the wisdom, the errors tend to fall away and the intelligence remains. But there has to be genuine, bottom-up decision making. The two most important characteristics of wise groups are diversity and independence, he argued.
Better by diversity
Diverse crowds are far more likely to be intelligent than non-diverse ones, because of the diversity of cognition. This expands the range of information we have access to – which allows us to surmount barriers when we come up against them. He cited the investment club phenomenon. Groups made up of men and women outperformed the single sex groups.
But this perspective on crowds runs counter to deep-set views of expertise. The Wisdom of Crowds puts forward the case that it’s a mistake to search out one or two experts and rely on them for the answers, because experts aren’t aware of what they don’t know.
Almost across the board expert judgements are poorly calibrated, the only exceptions being those of bridge players and weathermen! One of the reasons why the internet is such a powerful tool is that it allows a diversity of knowledge and expertise.
The problem with ‘group think’ – on the other hand – is that it reflects the knowledge of a group that is homogenous. Diversity allows the group as a whole to think more keenly. The Catholic Church created the role of the Devil’s Advocate precisely with this in mind, in order to sharpen its arguments.
Diversity also allows us to get around succombing to peer pressure.
State of independence
We want people to be making judgements based on their own knowledge and wisdom. But our general experience of groups comes in one of two kinds:
On the one hand, volatile & extreme groups: stockmarkets, lynchmobs, the madness of crowds. On the other: watered down mediocrity. Why the latter? Because we often put too much of a premium on consensus. Too much emphasis is put on one judgement that we can agree on.
Paradoxically, the wisdom of crowds is created when everyone has their independence. This is difficult to come by because, firstly, human beings are by nature imitative, and in its favour, this is a remarkably functional way to achieve decision-making. The reality is that we have learned, in an evolutionary and social sense, to be aware of what others are doing. Malcolm Gladwell’s book ‘The Tipping Point’ is about imitative behaviour – which is fine for promoting brands and fan clubs, but not good for predicting important future events.
Surowiecki cited economist John Maynard Keynes’s assertion that it’s better to fail conventionally than to fail unconventionally.
Net benefits and pitfalls
The net is a classic double-edged sword when it comes to the collective intelligence, Surowiecki explained.
On the one hand, the net makes it easier to satisfy the conditions under which intelligence pertains; with its lack of filters all you need is the willingness to participate. What's more, the knowledge we are looking for isn’t always in the places that we think it should be. In turn, we often over-estimate our abilities, and it’s hard to know who the right experts are. The net can help us surmount these barriers.
The problem lies in the fact that the internet is also a great tool for breaking down independence. It’s really easy for people to only pay attention to a small set of websites [or blogs!] and links and get locked into small worlds. You end up with "circular mills", which describes the scenario where, when ants get lost, they just follow the one in front and end up walking in circles until they die.
It’s better, Surowiecki commented, for people to keep their ties weak rather than strong, and he cited ‘Bayes Theorem’ which is a way of quantifying the effect of new information on previous calculations.
Q&A and debate with the audience
Does the WOC apply to artistic or creative endeavours, where the subjectivity of the artist/creator is traditionally viewed as paramount, someone asked. Surowiecki replied that for the WOC to apply, there needs to be a right answer, and there needs to be agreement on the problem and that is needs to be solved. But there is affinity between the WOC and the benefits that artists can accrue by working collaboratively.
Can groups come up with innovations? They can be good at sorting potential innovations, Surowiecki said, but it will probably be one person who creates or invents it.
How big is a group, another asked. Even in small groups of 6-8 people, the results are more accurate than that of the expert member, he replied. Leaders have to be careful not to influence discussion in advance in this regard, and we have to be careful that the talkative people do not dominate as they tend to then become the hub of the discussion, but there’s no evidence that talkative people are more intelligent than quieter people.
There’s something of a black box element to the wisdom of crowds, he continued, because you can’t go to one person and ask them ‘why did you do this?’, so it makes us fundamentally uneasy about the Wisdom of Crowds when it comes to very large crowds [due to the complexity of group accountability – I guess…], as with Wikipedia.
Malcolm Gladwell and Surowiecki agree that intuition is very valuable. Where James disagrees with Malcolm is that he doesn’t think that experts have a good evaluation of their own biases and blind-spots.
Web 2.0 – WOC tool or WOC manifest?
Someone raised the issue of the vast sources of collective intelligence that remain untapped in terms of the education system, and Surowiecki concurred that bureaucracies in general do a very poor job of tapping into their members' intelligence.
Brain Juicer was mentioned as a good example of a WOC application that that taps into people’s views in order to judge the likely success of new products.
James left us hanging with this tantalising paradox: we live in a moment where on the one hand we see the possibilities of collective, bottom-up emergent systems, and Web 2.0 is – in a way – a way to make that work. But countering that is the trend and desire for charismatic leadership and easy answers.
What do you think of Surowiecki’s perspective?
[NB. Cross-posted on the NMK website]
More coverage of SXSW Interactive coming soon!