Quantum thinking

I just had a feature article for New Scientist magazine covering research showing some rather peculiar connections between the mathematics of quantum theory and patterns of human decision making. I don't want to say too much more here, but would like to clarify one very important point and give some links.

I was inspired to write this article a couple years ago at a brain storming session held by the European Commission. Participants were supposed to be bold and propose radical visions about where the most promising avenues for research lay in the near future (this was in the context of information and computing technology). One Belgian researcher gave a fascinating talk on the application of quantum mathematics to human decision making, claiming that quantum logic fits actual human behaviour more closely than does classical logic. There are many famous "anomalies" -- such as the Ellsberg Paradox -- where people systematically violate the laws of classical logic and probability when making decisions of economic importance. The Belgian researcher explained that the quantum formalism is able to accommodate such behaviour, and was therefore surprisingly useful in understanding how people organize and use concepts.

What struck me then was the derision with which several other scientists (physicists) greeted this suggestion, while completely mis-understanding what the man had said. One physicist came close to screaming that this was "embarrassing mumbo jumbo" somehow linked to the idea that quantum physics underlies brain function (the idea proposed over a decade ago by Roger Penrose in his profound book Shadows of the Mind). He had dismissed the idea so quickly that he hadn't listened. The Belgian physicist had actually pointed out that he wasn't at all suggesting that quantum physics plays a role in the brain, only that the mathematics of quantum physics is useful in describing human behaviour.

This is a very important point -- the mathematics of quantum theory (the mathematics of Hilbert spaces) isn't identical with the theory and somehow owned by it, but stands quite independent of that theory and existed for at least a century before quantum theory was invented. The Belgian was saying -- this mathematics which turned out to be so useful for quantum physics is now turning out to be profoundly useful in quite another setting.

The New Scientist article is just a very brief introduction to some of the work. A few other things I found utterly fascinating while researching the article are:

1. This research paper called A Quantum Logic of Down Below which falls somewhere in between philosophy, psychology and computer science. The second author Dominic Widdows is a computer scientist at Google working on information retrieval. The paper essentially argues that philosophers historically devised classical logic and then took it as a model for what human logic must be or at least should be. They suggest this was the wrong way around. Pure logic isn't our best example of reasoning. The best example of reasoning systems is people, and so a logic of what reasoning is and can be ought to start with people rather than mathematics. This is a powerful idea. As the authors put it:

... what reasoning is (or should be) can only be read off from what reasoners are (and can be). Such a view one finds, for example in [Gabbay and Woods, 2001] and [Gabbay and Woods, 2003b], among logicians, and, also in the social scientific literature [Simon, 1957, Stanovich, 1999, Gigerenzer and Selten, 2001b]. Here the leading idea of the “new logic” is twofold. First, that logic’s original mission as a theory of human reasoning should be re-affirmed. Second, that a theory of human reasoning must take empirical account of what human reasoners are like – what they are interested in and what they are capable of.

 They then go on to argue that whatever the accurate logic of human reasoning is, it is more similar to quantum logic than to classical.

2. A second fascinating paper is more technical and describes some applications of this in computer science and information retrieval. Here the idea is that if people create concepts and texts and organize them using a quantum-style logic, then search methods based on classical logic aren't likely to search such conceptual spaces very effectively. This paper describes applications in which literature search can be improved by using quantum logic operations. Most interesting (and I did mention this in the New Scientist piece) is the use of quantum operations to generate what might be closely akin to "hunches" or "guesses" about where in a mass of textual data interesting ideas might be found -- guesses not based on logical deduction, but on something less tightly constrained and ultimately more powerful.

The Wisdom (???) of Crowds

The notion that markets aggregate the opinions of many and thereby make superior estimations of value has a very long history. It's certainly at the root of the infamous Efficient Markets Hypothesis, which claims that markets gather and process information so efficiently that price movements have no predictable patterns and prices of financial instruments always reflect something very close to the true fundamental value of the assets in question. More recently, The Wisdom of Crowds has been the driving force behind prediction markets. One one way or another, this notion lurks behind the slippery and insidious idea that "markets know best" and that pretty much everything from water distribution to higher education should be organized as a market. 

But in his bestselling book on the topic, James Surowiecki was somewhat careful at the outset to acknowledge that the idea only works in some rather special situations (not that readers paid much attention). A crowd estimating the number of marbles in a jar or the correct price of a stock will only get superior results -- superior in accuracy to the guess of any one individual, and even of experts -- if the people are on average unbiased in their estimates; it won't work if they tend systematically to estimate too high or low. Moreover, the people have to make their estimates independently of one another. Any kind of social influence, one person copying or even being slightly swayed by the actions of another, also spoils the result. Wise crowds very quickly become dumb herds.

For an idea of such broad influence, it's surprising how few experiments have been done to probe in detail around the boundaries where wise crowds become unwise, how it happens and which are the key effects. This has been rectified by an impressive set of experiments carried out by Jan Lorenz and colleagues from ETH-Zurich, and published recently in PNAS. Their idea was to use a crowd of 144 student volunteers and have them perform estimation experiments in a range of conditions. They gave the participants monetary incentives to estimate accurately, and chose questions (on things like geography and crime statistics) for which the true answers are known. Then, in some trials, participants made their estimates on their own, without having any idea about the estimations of others, and in other trials, they were either informed in complete detail of what others had estimated or given at least average information on the others' estimates. The idea was to compare how well the crowd made estimates in the absence and presence of social influence.

What the results show is that social influence totally undermines the wisdom of crowds effect, and does so in three specific ways. It's interesting to consider these in some detail to see just how this whole "wise crowd" illusion falls apart in the face of a little social influence:

1. In what the researchers call the “social influence effect,” the mere act of listening to the judgements of others led to a marked decrease in the diversity of the participants estimates. That is, the estimates of the various people become more like one another -- people adjust their views to fit more closely with others -- but this does very little to improve the collective accuracy of the crowd. In effect, people think they are sharing information, but little information actually gets shared. The figure below illustrates what happens: in successive trials, a measure of the group's opinion diversity decreases dramatically if people hear either full or average information on the estimates of others, meanwhile the collective error decreases only marginally.

2. A second and even more interesting effect is what the researchers call the “range reduction effect.” Imagine that a government tries to use the wisdom of crowds, assembling a group and surveying their opinions, hoping to get a range of views and some idea of how much consensus there is on some topic. You would hope that, if the crowd's estimate was NOT accurate, this lack of accuracy would be reflected in a wide range of estimates from the individuals -- the wide range would signal a lack unanimity and confidence. A truly bad outcome would be a crowd that at once gives a very inaccurate estimate and does so with a narrow range of opinion differences, signalling apparent strong certainty in the result. But this is precisely what the research found -- in the social influence conditions, the individuals' estimates didn't "bracket" the true answer, with some being higher and others lower. Rather, the group narrowed the range of their views so strongly that the truth tended to reside outside of the group's range -- they were both inaccurate and apparently confident at the same time.

3. Finally, and worse still, is the “confidence effect”. The researchers interviewed the participants in the different conditions, asking them how confident they were in the accuracy of the group's final consensus estimate. Social influence, while it didn't make the crowd's estimate any more accurate, did fill the participants with strong confidence and belief in improved accuracy. Think 2005, housing bubble, mortgages with no income and no assets, etc. As hard as it is to imagine that people could have believed the market could not fail to go up further, most did. And they did in large part because they saw others apparently believing the same thing.

Altogether, this careful study points more toward the idiocy of crowds than their wisdom. Social influence is hard to eradicate. Even in markets, supposedly driven by anonymous individuals making their own estimates, lots of people are reading the newspapers and news feeds and listening to analysts, and, even when not, looking to price movements and using them to infer whether someone else may know something they don't. In these experiments, social influence makes everyone think and do much the same thing, makes it likely that the consensus view aims well wide of the actual truth, and, perversely, makes everyone involved increasingly confident that the group knows what it's doing. Some kind of Wisdom.

How to erase memories

I'm not sure how much relevance this new bit of research has for finance and economics -- quite a lot, I suspect, given the social nature of interpersonal influences, which certainly affect economic outcomes -- but its conclusion is quite striking regardless. Our memories: they can be easily erased or manipulated by social influence, by people around us simply telling us outright lies (or their own false recollections).

Most of us, often, suffer from the illusion that our memories are fairly accurate, especially of things we've experienced first hand. Remember that huge football or baseball game your favourite team won or lost 10 years ago? Remember who made the biggest play? Our memories of these things aren't nearly as good as we think. Countless studies have shown that eyewitnesses are notoriously poor at remembering key facts accurately. A few years ago I got hold of an old recording of a Washington Redskins Superbowl victory from 20 years ago -- the details of which I thought were etched with prefect accuracy in my brain. I was stunned on watching to find out that I had mis-remembered who had made big plays, where they were on the field, when in the game the plays happened, and so on.

What happens in remembering sporting events happens in remembering the rest of our lives too, I'm afraid, and this new set of experiments shows just how easily our memories can be altered by the social influence of people around us. The paper is very well written and doesn't need much explanation. Micah Edelson and colleagues from the Weizmann Institute of Science in Israel had volunteers watch an eyewitness documentary, and then several days later tested their ability to recall facts without any interference, or after being presented with memories as recounted by some other individuals. From their abstract:

We examined how socially induced memory errors are generated in the brain by studying the memory of individuals exposed to recollections of others. Participants exhibited a strong tendency to conform to erroneous recollections of the group, producing both long-lasting and temporary errors, even when their initial memory was strong and accurate.

More profoundly, the study went on, using functional brain imaging, to look at the sites in the brain where these memory changes took place. The volunteers weren't merely reporting something they didn't actually remember just to fit in; their brains actually changed under social pressure, so they remembered something different.

This reminds me of another fascinating result from a few years ago in experiments run by a team led by Gregory Berns of Emory University, who re-examined the famous 1950s experiments on social conformity of Solomon Asch. Berns and colleagues did the experiment in such a way that they could tell that conforming volunteers -- who were effectively discarding their own observations in favor of those reported by others -- weren't just trying to fit in. Social pressure actually made them see the world differently, and specific mechanisms in the brain made it happen.   

As Edelson and colleagues note, this new effect has long been discussed in the social psychology literature and is known as "memory conformity" -- our memories, like other aspects of our behaviour, conform to social pressure. As I said, I don't know immediately how this fits into economics and finance, but it must have some pretty important consequences. Conformity is certainly one mechanism by which all manner of social trends get started, amplified and perpetuated.

As a British friend of mine told me in 2005 -- having just taken out a mortgage on a third speculative property purchase -- "housing prices never go down".  I don't think he had done any independent research to establish this as fact. But he had been reading and listening to the prevailing atmosphere of ideas -- and his brain had been physically altered as a result.