The NorthPath Letter · Behavioural Finance · Afternoon Edition
Ask a room of drivers to raise a hand if they are better than the average driver, and most hands go up. The arithmetic is impossible — half of any group must, by definition, sit below the median — yet the show of hands is one of the most reliable findings in psychology. The same hand goes up, silently, in the mind of almost every investor. The next crash, the next value trap, the next permanent loss of capital: these are things that happen to other people’s portfolios. Mine is different. I have done the work. My companies are the good ones. This quiet conviction that the bad outcome belongs to someone else has a name in the literature. It is optimism bias, and it is arguably the most expensive cognitive distortion an investor carries.
The bias was first pinned down with precision by the psychologist Neil D. Weinstein in a 1980 paper in the Journal of Personality and Social Psychology titled, plainly, “Unrealistic Optimism About Future Life Events” (39(5):806–820). Weinstein asked 258 college students to estimate how their own chances of experiencing forty-two life events — owning their own home, living past eighty, but also being divorced, developing a drinking problem, having a heart attack — compared with the chances of their classmates. The results were lopsided and consistent. For the desirable events, students rated their own odds well above average; for the undesirable ones, well below. Everyone, it seemed, expected a better-than-average future. Since most of the sample could not actually be above average, much of that optimism was, in Weinstein’s exact word, unrealistic.
It is worth being careful about what optimism bias is and is not, because it is easily confused with its cousin, overconfidence. Overconfidence concerns the precision of our judgments — the conviction that our estimates are more accurate, and our skill greater, than they are. Optimism bias concerns the direction of expected outcomes — the systematic belief that good things are more likely, and bad things less likely, to befall us than the base rate allows. A person can be perfectly calibrated about their own abilities and still expect the world to break their way. Daniel Kahneman, who spent a career cataloguing the failures of intuition, judged optimism the most significant of the cognitive biases, precisely because it is the one we would least wish to give up.
That last clause matters, because optimism is not simply a defect to be excised. A degree of unwarranted hope is what gets businesses founded, treatments researched and savings invested at all; Kahneman called it the engine of capitalism, and Sharot has argued that mild optimism is associated with greater persistence and resilience. The difficulty for the investor is not optimism as such but its miscalibration in a particular setting — one where the bad tail is large, occasionally permanent, and paid for with real capital. In most of life an over-hopeful forecast costs a little disappointment. In markets it can cost the principal, and principal does not regenerate the way enthusiasm does. The task, then, is not to abolish optimism but to quarantine it from the decision about price.
The architecture of a hopeful mind
Why is the default setting of human judgment tilted toward the favourable? Several mechanisms reinforce one another. The first is egocentrism. When we forecast our own future, we attend to our own intentions, our effort, our plans — vivid, detailed, first-person material — while the comparison class of other people remains abstract and faceless. We are not so much rating ourselves as better than others as failing to think concretely about others at all.
The second mechanism is the substitution of the inside view for the outside view, a distinction Kahneman and Amos Tversky drew in their 1979 work on intuitive prediction. The inside view builds a forecast from the specifics of the case in front of us: this management, this product, this thesis, and the story of how it all works out. The outside view ignores the story and asks instead about the reference class — how ventures of this kind have historically fared. Optimism lives in the inside view, where the singular narrative crowds out the distributional record. The base rate of disappointment is known; it simply feels irrelevant to my case.
The third mechanism is motivated reasoning. We are not neutral forecasters; we want certain outcomes, and wanting bends believing. Closely related is an asymmetry in how we revise beliefs. The neuroscientist Tali Sharot and colleagues have shown that people update their expectations more readily in response to good news than to bad — when told the future is brighter than expected, we adjust; when told it is darker, we largely discount the message. In a 2007 paper in Nature (“Neural mechanisms mediating optimism bias,” 450:102–105), Sharot, Riccardi, Raio and Phelps traced this tendency to heightened activity in the amygdala and the rostral anterior cingulate cortex — regions that monitor emotional salience — when subjects imagined positive futures. Optimism, in other words, is not merely a habit of thought; it is built into the machinery. Sharot has estimated that some four-fifths of the population exhibit it, and it survives high intelligence, education, and even being explicitly warned about it.
A fourth mechanism compounds the rest: the illusion of control. The more we feel we are steering an outcome — by our research, our timing, our willingness to monitor — the lower we judge its risk to be, regardless of whether our actions move the odds at all. The investor who has read every filing feels, on that account, less exposed to loss, when in truth diligence changes the quality of the decision but not the existence of the bad tail. Optimism and the illusion of control feed each other: effort breeds confidence, confidence breeds hope, and hope quietly revises the probabilities downward where the danger lies.
From the laboratory to the order book
If optimism were confined to undergraduates rating their odds of a happy marriage, it would be a curiosity. It is not. The bias is densely documented in precisely the places where capital is committed. A review by Shepperd, Waters, Weinstein and Klein in 2015 (“A Primer on Unrealistic Optimism”) confirmed that the effect replicates across cultures, across events, and across decades since Weinstein’s first study.
Consider entrepreneurs, the people whose business is to weigh the odds of a new venture. In a 1988 study in the Journal of Business Venturing (“Entrepreneurs’ Perceived Chances for Success,” 3:97–108), Arnold Cooper, Carolyn Woo and William Dunkelberg surveyed 2,994 owners of newly founded American businesses. Eighty-one per cent put their own odds of success above seventy per cent; a remarkable thirty-three per cent rated their chances at a perfect ten out of ten. This against a base rate in which a majority of new firms do not survive five years. The founders were not lying; they genuinely could not see themselves in the failure statistics.
The same tilt runs through the professional forecasting that markets rely upon. Decades of evidence show that sell-side analysts’ earnings estimates are systematically too high, and most pronounced at the longer horizons where optimism has the most room to roam (McNichols and O’Brien, 1997; Easterwood and Nutt, 1999). Corporate managers are no better insulated: Ulrike Malmendier and Geoffrey Tate, in influential papers in 2005 and 2008, showed that optimistic chief executives — identifiable by their reluctance to diversify out of their own company’s stock — over-invest when they have internal cash and overpay when they acquire, destroying value in the name of a future they were sure would arrive. And the long-run underperformance of initial public offerings, documented by Jay Ritter as far back as 1991, is consistent with issuers selling shares into investor optimism at exactly the moment that optimism is richest.
Households complete the circuit. Surveys of individual investors repeatedly find expected returns that bear little relation to the historical record and that, tellingly, rise after prices have already risen — optimism feeding on the very gains that should counsel caution. The pattern is the opposite of what a sober reading of valuation would produce, where higher prices imply lower future returns. It is exactly what optimism bias predicts: the recent past is read as a promise about one’s own future, the comparison with long-run base rates is never made, and the most hopeful estimates cluster precisely where the subsequent disappointment will be largest.
Regulators have noticed. In the United Kingdom, the Financial Conduct Authority devoted its very first Occasional Paper — “Applying behavioural economics at the Financial Conduct Authority,” by Erta, Hunt, Iscenko and Brambley, April 2013 — to the predictable mistakes consumers make, naming over-optimism and present bias among the biases that justify intervention. The FCA’s mandatory risk warnings on financial promotions are, in effect, an institutional correction for retail optimism: the regulator forces onto the page the downside the buyer would not volunteer to imagine. In the United States, the architecture is older and even more explicit. The Securities and Exchange Commission requires issuers to disclose material risk factors under Regulation S-K (Item 105), and the safe harbour for forward-looking statements created by the Private Securities Litigation Reform Act of 1995 is conditioned on the presence of “meaningful cautionary statements.” The entire edifice of mandated risk disclosure exists for one reason: optimism is the default, and the law must compel the bad outcome into print because the human author will not put it there unprompted.
Two declarations that the downside was over
The history of markets is, in part, a history of distinguished people announcing that the bad tail had been abolished. Two episodes bracket the modern era.
The first is October 1929. Irving Fisher of Yale was then the most celebrated economist in America, a pioneer of monetary theory and index numbers. On 15 October 1929, addressing the Purchasing Agents Association at a dinner in New York, he delivered the sentence that would shadow his reputation forever: stock prices, he said, had reached “what looks like a permanently high plateau” (reported in The New York Times the following morning). Within two weeks the market broke — Black Thursday on the twenty-fourth, the crash of the twenty-eighth and twenty-ninth — and it kept breaking. From its September 1929 peak near 381, the Dow Jones Industrial Average would fall roughly eighty-nine per cent to a low near 41 in July 1932. Fisher was no fool; he was an optimist whose model simply had no place for the disaster, and he is said to have lost most of his fortune in the wreckage. The plateau was real only in the sense that a cliff edge is flat right up to the drop.
The second episode is the technology mania of 1999 and 2000. Here optimism wore the costume of a “new economy” in which the old arithmetic of earnings was held to be obsolete; growth in users — eyeballs — would substitute for profit, and analysts competed to publish ever-higher targets. Households redirected their savings into companies that had never earned a cent. The NASDAQ Composite, the index of that hope, peaked at 5,048.62 on 10 March 2000 and then fell, grindingly, to a low near 1,114 by October 2002 — a decline of roughly seventy-eight per cent. The instructive part is what happened to the survivors. Several of the era’s leading companies went on to grow their revenues many times over in the two decades that followed, and yet shares bought at the March 2000 price did not recover for an investing generation. The businesses were not the error. The optimistic price paid for them was.
Three disciplines against a hopeful mind
Because optimism is wired in rather than reasoned in, the corrective cannot be an act of will. You cannot decide to be less hopeful any more than you can decide to find a familiar face unfamiliar. What works is procedure — a set of standing disciplines that force the suppressed downside into view before capital is committed.
The first discipline is to take the outside view. Before forecasting what this particular holding will return, ask what the reference class has returned: how have companies bought at this kind of multiple, in this kind of industry, at this kind of moment, actually performed across the historical record? This is reference-class forecasting, descended directly from Kahneman and Tversky’s 1979 analysis of intuitive prediction. The aim is to replace the seductive singular story with the sober distribution of outcomes, and to anchor the estimate on the base rate before the narrative is allowed to adjust it.
The second discipline is the pre-mortem, a technique developed by the psychologist Gary Klein and championed by Kahneman. Where a post-mortem examines a failure that has occurred, a pre-mortem imagines one that has not. The instruction is concrete: assume it is three years from now, the investment has been a clear disaster, and write the story of how it happened. This exercise in “prospective hindsight” has been shown to increase the number of plausible failure causes a person can generate by around thirty per cent (Mitchell, Russo and Pennington, 1989). Its power is social and psychological: by making the failure a premise rather than a possibility, it licenses the pessimism that ordinary optimism keeps locked away, and it does so before the decision rather than after the loss.
The third discipline is to write the downside case explicitly and then to demand a margin of safety against it. Optimism is, at root, an estimate of probabilities; the margin of safety is structural insurance against that estimate being wrong. By insisting on a discount to a deliberately conservative — not hoped-for — estimate of a business’s worth, the investor builds in a tolerance so that even an optimistic error leaves the capital substantially intact. Benjamin Graham’s margin of safety, viewed through this lens, is not merely a value-investing slogan; it is an engineering allowance for the predictable optimism of the human who is doing the buying.
How the durable investors built around it
The investors who have lasted longest tend to share a settled distrust of their own hopefulness. Warren Buffett is the clearest example. In the summer of 1999, near the manic top of the technology market and to an audience full of its beneficiaries at Sun Valley, Buffett walked patiently through the arithmetic of return expectations and argued that investors extrapolating the previous seventeen years of gains were heading for disappointment — a talk reproduced that November in Fortune as “Mr. Buffett on the Stock Market” (22 November 1999). His most quoted instruction, to “be fearful when others are greedy, and greedy when others are fearful,” is at bottom a rule for trading against the crowd’s optimism, and his lifelong insistence on a margin of safety is its structural counterpart. Buffett does not try to feel less optimistic; he builds his process so that optimism cannot do much damage.
Howard Marks has made the same point his life’s work. In his memo “bubble.com,” written on 3 January 2000 at almost the exact peak, Marks dismantled the optimistic case for internet valuations with a clarity that reads as prophecy only in hindsight. His enduring framework — the market as a pendulum swinging between euphoria and despair, never resting at the sensible middle — is a sustained discipline against optimism, as is his maxim that “the riskiest thing in the world is the belief that there is no risk” (The Most Important Thing, 2011). Marks insists that the investor reason from where the cycle of sentiment actually stands, not from the extrapolated hope that the good years will simply continue.
Behind both stands Graham’s parable of Mr. Market — the manic-depressive partner who, on his euphoric days, offers to buy your share of the business at absurd prices and is happy to sell you his at the same. The whole point of the parable is that you are free to ignore his optimistic quotations. Optimism bias is, in a sense, the discovery that Mr. Market lives inside each of us, and that the discipline of the durable investor is to refuse to let him set the price.
Key takeaways
- Optimism bias is the systematic expectation of a better-than-average future for oneself — good outcomes overweighted, bad outcomes underweighted relative to the base rate. First measured by Weinstein in 1980, it is nearly universal and distinct from overconfidence in one’s skill.
- It is the engine of bull-market excess. Optimistic analysts, optimistic managers, optimistic founders and optimistic households all underprice the downside at the same time, which is why the error compounds at the market level rather than cancelling out.
- The two crashes that bracket the modern era were each prefaced by an authoritative voice declaring the downside abolished — Irving Fisher’s “permanently high plateau” in 1929, and the “new economy” chorus before the seventy-eight-per-cent fall in the NASDAQ from March 2000.
- The corrective is procedural, not emotional. Take the outside view and anchor on the reference class; run a pre-mortem to license the suppressed failure case; and demand a margin of safety against a deliberately conservative estimate of worth.
- You cannot will yourself less optimistic; you can only build a process that forces the bad outcome onto the page before you commit capital. The good company is not the question. The price you would pay if you were merely hopeful is.
— Manish Goel, FCA / NorthPath Advisory OÜ / Tallinn, Estonia
|
Important. |