Expert Failure and The Recognition Heuristic

Libertarians have a real fear of technocratic expertise and top-down control, especially of complex systems and especially when there is no metric for success or failure. This message can be well articulated but it can also turn into hamfisted anti-elitist nonsense. It is a complex point and complex messages lose their force and complexity in any game of telephone.

For an example here’s part of a conversation between Bill Simmons and Malcolm Gladwell on a few years ago. At the time Isiah Thomas was running the New York Knicks into the ground with poor trades and draft picks and Gladwell attempts here to explain how this could happen:

Gladwell: Here’s the real question. If I was GM of the Knicks, would I be doing a better job of managing the team than Thomas? I believe, somewhat immodestly, that the answer is yes. And I say this even though it is abundantly clear that Thomas knows several thousand times more about basketball than I do. I’ve never picked up a basketball. I couldn’t diagram a play to save my life. I would put my level of basketball knowledge, among hard core fans, in the 25th percentile.

There’s a famous experiment done by a wonderful psychologist at Columbia University named Dan Goldstein. He goes to a class of American college students and asks them which city they think is bigger — San Antonio or San Diego. The students are divided. Then he goes to an equivalent class of German college students and asks the same question. This time the class votes overwhelmingly for San Diego. The right answer? San Diego. So the Germans are smarter, at least on this question, than the American kids. But that’s not because they know more about American geography. It’s because they know less. They’ve never heard of San Antonio. But they’ve heard of San Diego and using only that rule of thumb, they figure San Diego must be bigger. The American students know way more. They know all about San Antonio. They know it’s in Texas and that Texas is booming. They know it has a pro basketball team, so it must be a pretty big market. Some of them may have been in San Antonio and taken forever to drive from one side of town to another — and that, and a thousand other stray facts about Texas and San Antonio, have the effect of muddling their judgment and preventing them from getting the right answer.

I’d be the equivalent of the German student. I know nothing about basketball, so I’d make only the safest, most obvious decisions. I’d read John Hollinger and Chad Ford and I’d print out your mid-season NBA roundup and post it on my blackboard. I’d look at the box scores every morning, and watch Charles Barkley and Kenny Smith on TNT. Would I have made the disastrous Marbury trade? Of course not. I’d wonder why Jerry Colangelo — who I know is a lot smarter than I am — was so willing to part with him.

Would I have traded for Curry? Are you kidding? All I know is that Chicago is scared of his attitude and his health, and Paxson knows way more about basketball — and about Eddy Curry — than I do. Trade for Jalen Rose? No way. One of the few simple facts that basketball dummies like me know is that players in their early thirties are pretty much over the hill. And Jerome James? Please. I have no idea how to evaluate a player’s potential. But I’d look up his stastistics on and see that’s he’s been pretty dreadful his whole career, and then I’d tell his agent to take a hike.

Now would I be as good as GM as Jerry West or Joe Dumars? Of course not. But just by sitting on my hands, and being scared of looking like a fool, and taking only the safest, most conservative steps, and drafting only solid players that everybody agrees are a can’t miss, I could make the Knicks a vastly better team than they are today — as could any reasonably cautious and uninformed fan… The point is that knowledge and the ability to make a good decision correlate only sporadically, and there are plenty of times when knowledge gets in the way of judgement. That’s Thomas in a nutshell: He knows so much about basketball that he believes that he knows more than anyone else about the potential of previously undistinguished players. He thinks he can see into the true basketball soul of Jerome James. The truth is, of course, that James doesn’t have a basketball soul.

By the way, while we’re on this topic, let’s play a real world application of this. Let’s say I’m so dumb about basketball that all I know is that the best college programs in the country are Duke and UConn, and so as a GM my rule is only draft and/or trade for the first and second team players, in any given year, from those two schools. So I fire all my scouts. I disband my front office, and basically say that I cede my basketball judgment to Jim Calhoun and Mike K. What’s my team? It’s some combination of Elton Brand, Emeka Okafor, Ben Gordon, Luol Deng, Shane Battier, Mike Dunleavy, Rip Hamilton, Corey Maggette, Jay Williams, Caron Butler, Donyell Marshall and Grant Hill — which is a really wonderful team. Now, of course, in the real world I couldn’t get all those people, because lots of them were really high draft picks. But let’s say I got Brand in a trade, after Chicago soured on him, and I was lucky enough to be in the lottery for Okafor. Maggette was a 13; Hamilton and Deng were 7s; and Butler was a 10 — so at least some of them are doable, particularly since in off-years for Duke and UConn I can trade down and stockpile picks. Battier I wine and dine in the free agent market, because who wants to be stuck in Memphis? Ditto for Gordon, who, it seems, Chicago is thinking of moving anyway. Is that the best team in the league? No. It is better than the Knicks? Absolutely. The point is that clinging to a very simple rule of thumb here — that doesn’t require knowing much about basketball — can leave you looking pretty smart.

Read Goldstein’s Models of Ecological Rationality: The Recognition Heuristic for more on the mechanism being proposed here. Abstract:

The recognition heuristic, arguably the most frugal of all heuristics, makes inferences from patterns of missing knowledge. This heuristic exploits a fundamental adaptation of many organisms: the vast, sensitive, and reliable capacity for recognition. The authors specify the conditions under which the recognition heuristic is successful and when it leads to the counterintuitive less-is-more effect in which less knowledge is better than more for making accurate inferences.

There are plenty of nits to pick here. One can imagine phrasing questions to Germans that would expose their vague familiarity with US geography as much as exploit it. However, the idea that knowledge and expertise can hit a point of diminishing and even negative returns to prediction and decision-making propositions seems intuitively correct. This is exactly what we ask our experts to do in government. It is costly for a coach like Thomas to believe he knows more than the collective rest and to act on those beliefs. It got him fired. Herd conservatism and some Bayesian skepticism would have served everyone better (except the corpse of Eddy Curry). But if he hadn’t been fired Thomas would have continued his path and walked the Knicks off a cliff. He believed, that’s what believers do. We knew he failed because the Knicks lost. How will we know when government experts fail?


Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

%d bloggers like this: