The Wisdom of Crowds - by James Surowiecki
ISBN: 0385721706Date read: 2008-04-16
How strongly I recommend it: 7/10
(See my list of 360+ books, for more.)
Go to the Amazon page for details and reviews.
Mind-blowing examples of how groups of diverse people acting independently are smarter than any one person in the group. Has huge implications for management, markets, decision-making, and more.
my notes
Under the right circumstances, groups are remarkably intelligent, and are often smarter than the smartest people in them. Groups do not need to be dominated by exceptionally intelligent people in order to be smart. Even if most of the people within a group are not especially well-informed or rational, it can still reach a collectively wise decision.
When our imperfect judgements are aggregated in the right way, our collective intelligence is often excellent.
We assume that the key to solving problems or making good decisions is finding that one right person who will have the answer. The argument of this book is that chasing the expert is a mistake, and a costly one at that. We should stop hunting and ask the crowd instead. Chances are, it knows. (Crowd includes the geniuses as well as everyone else.)
The best way for a group to be smart is for each person in it to think and act as independently as possible.
** Chapter 1
In general, the bigger the crowd the better.
If you run 10 different jelly-bean-counting experiments, it's likely that each time one or two students will outperform the group. But they will not be the same students each time. Over 10 experiments, the group's performance will almost certainly be the best possible. The simplest way to get reliably good answers is just to ask the group each time.
The 4 conditions that characterize wise crowds:
1. Diversity of opinion
2. Independence
3. Decentralization
4. Aggregation
If you ask 100 people to run a 100-meter race, the average time will not be better than the time of the fastest runners. It will be worse - a mediocre time. But ask 100 people to answer a question or solve a problem, and the average number will often be as least as good as the answer of the smartest member. With most things, the average is mediocrity. With decision making, it's often excellence.
** Chapter 2
(About product innovation or small companies:) What makes a system successful is its ability to generate lots of losers and then to recognize them as such and kill them off. Sometimes the messiest approach is the wisest.
Generating a diverse set of possible solutions isn't enough. The crowd also has to be able to distinguish the good solutions from the bad.
The simple fact of making a group diverse makes it better at problem solving.
Grouping only smart people together doesn't work that well, because the smart people tend to resemble each other in what they can do. The group knows less than it otherwise might. Adding in a few people who know less, but have different skills, actually improves the group's performance.
Groups that are too much alike find it harder to keep learning, because each member is bringing less and less to the table. They spend too much time exploiting, and not enough time exploring.
The fact that cognitive diversity matters does not mean that if you assemble a group of diverse but thoroughly uninformed people, their collective wisdom wil be smarter than an expert's. But if you can assemble a diverse group of people who possess varying degrees of knowledge and insight, you're better off entrusting it with major decisions rather than leaving them in the hands of one or two people, no matter how smart those people are.
A large group of diverse individuals will come up with better and more robust forecasts and make more intelligent decisions than even the most skilled "decision maker".
Seer-sucker theory : No matter how much evidence exists that seers do not exist, suckers will pay for the existence of seers.
If a group is so unintelligent that it will flounder without the right expert, it's not clear why the group would be intelligent enough to recognize an expert when it found him.
[When one person realizes they are clearly a non-conformer in a group of conformers, they tend to conform. But...] having even one other person in the group who felt as they did made the subjects happy to announce their thoughts and the rate of conformity plummeted.
** Chapter 3
The smartest groups are made up of people with diverse perspectives who are able to stay independent of each other.
You can be biased and irrational, but as long as you're independent, you won't make the group any dumber.
The more influence a group's members exert on each other, and the more personal contact they have with each other, the less likely it is that the group's decisions will be wise ones. We could become individually smarter but collectively dumber.
"Social proof" : the tendency to assume that if lots of people are doing something or believe something, there must be a good reason why.
That's why the crowd becomes more influential as it becomes bigger : every additional person is proof that something important is happening.
Following the group is a reasonable strategy, but if too many people adopt that strategy, it stops being sensible and the group stops being smart.
(Fads/Crazes:) Everyone thinks that people are making decisions based on what they know, when in fact people are making decisions based on what they think the people who came before them knew. Instead of aggregating all the information individuals have, the cascade becomes a sequence of uninformed choices, so that collectively the group ends up making a bad decision.
People are, in general, overconfident. They overestimate their ability, level of knowledge, and decision-making prowess. More overconfident when facing big problems than easy ones.
The more important a decison, the less likely a cascade (fad/craze) is to take hold. The more important the decision, the more likely it is that the groups' collective verdict will be right.
Encouraging people to make incorrect guesses actually made the group as a whole smarter.
One key to successful group decisions is getting people to pay much less attention to what everyone else is saying.
** Chapter 4
Decentralization : power does not fully reside in one central location, and many of the important decisions are made by individuals based on their own local and specific knowledge rather than by an omniscient or farseeing planner.
It fosters and is fed by specialization.
The closer a person is to a problem, the more likely he or she is to have a good solution to it.
Decentralization's great weakness is that there's no guarantee that valuable information which is uncovered in one part of the system will find its way through the rest of the system. Sometimes valuable information never gets disseminated.
A decentralized system can only produce genuinely intelligent results if there's a means of aggregating the information of everyone in the system.
** Chapter 5
Convention explains why:
- companies rarely cut wages in a recession (it violates workers' expectations and hurts moral), preferring instead to lay people off
- every major car company releases its new models in September, (even though there would be less competition if each company released its cars in different months)
- clothing retailers apply a simple 50% markup, then discount like mad if the items don't sell
- it costs you as much to see a "total limping dog" movie in its last week of release as it does a hugely popular film on opening night
About organization and coordination:
Next time you go to the supermarket looking for orange juice, it'll be there waiting, even though you didn't tell the grocer you were coming. There will be as much orange juice in the freezer as the store's customers want over the next few days, even though none of them told the grocer they were coming. The juice you buy will have been packaged days earlier, after it was made from oranges picked weeks earlier, by people who don't even know you exist. The players in that chain - shopper, grocer, wholesaler, packager, grower - are not acting on formal rules, but they are using local knowledge and making decisions not on the basis of what's good for everyone, but rather on the basis of what's good for themselves. And yet, without anyone leading them or directing them, people are able to coordinate their economic activities.
In the 50 years since Vernon Smith did his first experiment in [wisdom of crowds] and published the results, they have been replicated thousands of times in ever more complex variations. But the essential conclusion of those early tests has not been challenged : that, under the right conditions, imperfect humans can produce near-perfect results.
** Chapter 6
To solve cooperation problems - like keeping the sidewalk free of snow, paying taxes, and curbing pollution - the members of a society need to do more. They need to adopt a broader definition of self-interest than the myopic one that maximizing profits in the short-term demands. And they need to trust those around them, because in the absense of trust the pursuit of myopic self-interest is the only strategy that makes sense.
"Ultimatum game" : behavioral economics : 2 people given $10 to divide between them. One person decides what the split should be, then makes take-it-or-leave-it offer to the other person. That person can either accept the offer, in which case both players pocket their respective shares of the cash, or reject it, in which case both players walk away empty-handed. If both players are purely rational, the proposer will keep $9 and offer $1, and the responder will take it, since if he accepts he gets $1, and rejects gets none. In practice, though, this rarely happens. Instead, lowball offers, anything below $2, are routinely rejected. Think about what this means : people would rather have nothing than let the other person walk away with too much of the loot. They will give up free money to punish what they perceive as greedy or selfish behavior.
People think, in an ideal world, everyone would end up with the amount of money they deserved.
An an interesting version of the Ultimatum game, instead of assigning the proposer role randomly, the researchers made it seem as if the proposers had earned their positions by doing better on a test. In those experiments, the proposers offered significantly less money, yet not a single offer was rejected. People apparently thought that a proposer who merited his position deserved to keep more of the wealth.
In America, the people whom inequality bothers most are the rich. Americans are far more likely to believe that wealth is the result of initiative and skill, while Europeans are more likely to attribute it to luck.
The foundation of cooperation is not really trust, but the durability of the relationship. Whether the players trust each other or not is less important in the long run than whether the conditions are ripe for them to build a stable pattern of cooperation with each other.
Successful cooperation requires that people start off by being nice, willing to cooperate, but have to be willing to punish noncooperative behavior as soon as it appears. The best approach is to be "nice, forgiving, and retailiatory".
We have learned that trade and exchange are games in which everyone can end up gaining.
The benefits of being trusting and of being trustworthy are potentially immense, because a successful market system teaches people to recognize those benefits.
Merchant guilds - most notably the German Hanseatic League - protected their members against the unfair treatment from city-states by imposing collective trade embargoes against cities that seized merchant property.
Previously, trust had been the product primarily of a personal or in-group relationship. Modern capitalism made the idea of trusting people with whom you had no personal ties seem reasonable, if only by demonstrating that strangers would not, as a matter of course, betray you. Buying and selling no longer required a personal connection. It's driven instead by the benefits of mutual exchange.
I can walk into a store somewhere far from home and buy an item that was made across the world, and it will probably work well. This is true even though I may never walk in that store again. We take the reliability of the store and manufacturer for granted. But in fact, they're remarkable achievements.
Capitalism is healthiest when people believe the long-term benefits of fair dealing outweigh the short-term benefits of sharp dealing.
Collective action like political rallys : for the individual it would make more sense to let someone else do the work. Everyone has an incentive to sit on their hands, wait for someone else to do something, and free ride. Since everyone wants to be a free-rider, nothing gets done. ((But since people do get involved, like paying taxes for example, it shows that other emotions are at work.))
** Part 2 : EXAMPLES
(Scientists:) The quest for recognition ensures a steady infusion of diverse thought, since no one becomes famous for restating what's already known.
If you talk a lot in a group, people will tend to think of you as influential almost by default. Talkative people are not necessarily well liked, but they are listened to.
Group decisions are not inherently inefficient. This suggests that deliberation can be valuable when done well, even if after a certain point its marginal benefits are outweighed by the costs.
There is no point in making small groups part of a leadership structure if you do not give the group a method of aggregating the opinions of its members. If small groups are included in the decision-making process, then they should be allowed to make decisions. If an organization sets up teams and then uses them for purely advisory purposes, it loses the true advantage that a team has : namely, collective wisdom.
** Chapter 10 : The Company
You do not need consensus in order to tap into the wisdom of a crowd. The search for consensus encourages tepid lowest-common-denominator solutions which offend no one rather than exciting everyone.
Even those companies that tried to make the decision-making process more democratic thought democracy meant endless discussion rather than a wider distribution of decision-making power.
Attempting to run an entire company by command and control is a futile task. It's too costly in terms of time and requires far too much information that top executives should not be bothering with.
What gets in the way of the exchange of real information is a deep-rooted hostility on the part of the bosses to opposition from subordinates. This is the real cost of top-down approach to decision making : it confers the illusion of perfectibility upon the decision makers and encourages everyone else simply to play along.
Companies tend to pay people based on whether they do what they're expected to do. In a market, people get paid simply on what they do! Ideally, the same would be true inside a company.
This is an essential part of what markets do : encourage people to find new valuable information and then let everyone else know about it. This too is what corporations should be looking for : ways to provide their employees with the incentive to uncover and act on private information.
Even small option grants seem to instill a sense of ownership, and we know that owners are, in general, more likely to take good care of their property than renters are.
Far more important than stock options would be the elimination of rigid managerial hierarchies and the wider distribution of real decision-making power.
The more responsibilty people have for their own environments the more engaged they will be.
Allowing people to make decisions about their own working conditions makes a material difference in how they perform.
Decentralized markets work exceptionally well because the people and companies in those markets are getting constant feedback from customers. Companies that aren't doing a good job or are spending too much learn to adjust or else they go out of business.
Some academics suggest that CEOs have, at best, a minor impact on corporate performance.
The more power you give a single individual in the face of complexity and uncertainty, the more likely it is that bad decisions will get made.
Use methods of aggregating collective wisdom.
The anonymity of the markets and the fact that they yield a relatively clear solution, while giving individuals an unmistakable incentive to uncover and act on good information, means that their potential value is genuinely hard to overestimate.
The more important the decision, the more important it is that it not be left in the hands of a single person.
** Chapter 11 : Stock Market
The measure of the stock market's success is not whether stock prices are rising. It's whether stock prices are right. It's harder for the market to get prices right when there is so little money on the short side.
The psychology of investors:
- sometimes herd, preferring the safety of the company of others to make independent decisions.
- too much credence to recent and high-profile news while underestimating the longer-lasting trends or less dramatic events
- (in the same way people worry about being killed in a plane crash while not paying attention to their high cholesterol)
- fooled by randomness, believing money managers who have had a few good quarters have figured out the trick of beating the market
- find losses twice as painful as they find gains pleasurable, so hold on to stocks longer than they should, believing as long as they haven't sold it, they haven't suffered any losses
- above all, overconfident, which means they trade more than they should and end up costing themselves money as a result
- (from 1991-1996 the market returned 17.9%. active investors earned just 11.4%. They would have done better had they just sat on their hands.)
Individual irrationality can add up to collective rationality.
People want to save, and do not need a massive push to do so. What they do need is a way to make saving easier and spending harder. One way of doing this is to make enrollment in retirement plans automatic, rather than asking people to sign up for them. If people have to take action to opt out of a retirement plan rather than having to take action to opt in, they are significantly more likely to stay in the plan and more likely to save. Inertia is a powerful tool.
The idea of the wisdom of crowds is not that a group will always give you the right answer, but that on average it will consistently come up with a better answer than any individual could provide. That's why the fact that only a fraction of investors consistently do better than the market remains the most powerful piece of evidence that the market is efficient.
The healthiest markets are those that are animated by both fear and greed at the same time. Any time you sell a stock, the person who's buying it thinks differently about the future prospects of that stock. You think it's going down, he thinks it's going up. One of you will be right, but the important thing is that it's only through the interaction of those differing attitudes that the market is able to do a good job of allocating capital.
In a bubble, all of the conditions that make groups intelligent - independence, diversity, private judgement - disappear.
The price of TVs doesn't suddenly double overnight only to crash a few months later. You never end up with a situation where the fact that prices are rising makes people more interested in buying (which is what happens in a bubble). The more expensive a TV gets, the less interested people are in buying it.
(bubbles:) The kind of diversity of opinion that a healthy market depends on was replaced by a sort of single-mindedness.
Everyone was convinced the greater fool was out there.
If groups on the whole are relatively intelligent (as we know they are), there's a good chance that a stock price is actually right. Problem is that once everyone starts piggybacking on the wisdom of the group, then no one is doing anything to add to the wisdom of the group.
News reporters tend to overplay the importance of any particular piece of information. The best way to disclose public information is without hype or commentary from people in the positions of power. (Like the way the Federal Reserve announces its interest-rate hikes.)
A mob in the middle of a riot appears to be a single organism, acting with one mind.
** Chapter 12 : Government
Government kept getting bigger, since everyone had an individual interest in getting a little more from the state, and no one was looking out for the collective interest, entered into cozy arrangements with the businesses it was regulating, and that allowed economic policy to be run in the interests of powerful groups instead of the interests of the public as a whole.
Though everyone may say they are into the common good, to different individuals and groups, the common good is bound to mean different things.
** AFTERWORD:
One of the key lessons of the Wisdom of Crowds is that we don't always know where good information is. That's why, in general, it's smarter to cast as wide a net as possible, rather than wasting time figuring out who should be in the group and who should not. This idea is well suited to the internet.
The more information a group has, the better its collective judgement will be, so you want as many people with good information in a group as possible.
The Wisdom of Crowds is not an argument against experts, but against our excessive faith in the single individual decision maker.
If a group is smart enough to know whether an individual is a genuine decision-making prodigy, then the group is smart enough to not need that individual.
Even brilliant experts have biases and blind spots, so they can make mistakes. What's troubling is that, in general, they don't know when they're making those mistakes. Experts don't know when they don't know something.
That's why it's worthwhile to cast a wider net and why relying on a crowd of decision makers improves (though doesn't guarantee) your chances of reaching a good decision.
Be careful to keep the group diverse, and careful to prevent people from influencing one another too much.
The crowd's judgement is going to give us the best chance of making the right decision, and in the face of that knowledge, traditional notions of power and leadership should begin to pale. I am cautiously hopeful that they will, allowing us to begin to trust individual leaders less and ourselves more.
-----------
Wisdom of crowds works on problems where there's a true answer, or when some choices are better than other in some Platonic sense. The reason this works is that people are operating on private info, which may be bad or fragmented.
The opinions are diverse -- not consensus but disagreements.
People don't know much about what others are betting on or guessing -- not a lot of interpersonal interaction.
------------
RELATED BOOKS:
Smart Mobs by Howard Rheingold
The Evolution of Cooperation by Robert Axelrod
Iowa Electronic Markets (IEM)
TradeSports.com
NewsFutures.com