One observation I have made is how enthusiastic many Chinese people—especially men—are about the stock market. I have not conducted a survey, let alone a statistical test, but if I had to bet, I would expect the proportion of people participating directly in the stock market, particularly through active investing or trading rather than index funds, to be considerably higher among the Chinese people I know than among the British people I know.

In my family, among my friends and in my friends’ families, there seems to be a mysterious age at which a man suddenly becomes obsessed with stock tips, candlestick charts and the next great opportunity.

Unfortunately, I seem to have reached it.

Before going too far, I should say that I do not think there is a concrete method by which retail investors can reliably beat the market. Anyone reading solely in search of such a magic formula can stop now.

The difficulty is not simply that beating an average is hard. Active investing is a relative competition. William Sharpe’s arithmetic of active management shows that, within a defined market, the average actively managed pound must earn the market return before costs. After research expenses, management fees and trading costs, the average actively managed pound must earn less. This does not mean that no individual can outperform. It means that active investors cannot all outperform, and that the winners must, in aggregate, be offset by losers.1

Another disclaimer: although there is a meaningful distinction between investing and speculation, I will use “investor” and “trader” somewhat interchangeably in this essay. Clarifying the boundary between the two is not my purpose here, and Benjamin Graham has already done it more carefully in The Intelligent Investor.

Being right is not enough

The rule for earning uncommon profits in common stocks is simple to state. It always involves some degree of predicting the future, particularly in circumstances where the predictions of other market participants are wrong.

But predicting the future correctly is not, by itself, sufficient.

You have to be more right than the price.

Suppose a company reports excellent earnings. Its share price may still fall because investors had expected something even better. Conversely, a disappointing result may cause the price to rise if the market had already anticipated a disaster. The relevant question is not merely whether the outcome is good or bad. It is whether the outcome is better or worse than the expectations embedded in the price.

A share price is not the market declaring that one particular future will occur. It is the point at which investors with different forecasts, probabilities, time horizons and constraints are willing to trade. Around a discrete event, such as an earnings announcement or clinical-trial readout, the price can often be thought of as reflecting a probability-weighted mixture of possible outcomes.

An investor may therefore possess an edge by estimating the outcome more accurately, assigning it a better probability, understanding its financial consequences more clearly, or predicting how other investors will react to it.

Predicting the crowd

One common, although imperfect, way to distinguish trading from investing is by the horizon and object of the prediction. Warren Buffett attributed to Benjamin Graham the famous observation:

“In the short run, the market is a voting machine but in the long run it is a weighing machine.”

2

Short-term trading often involves predicting crowd behaviour: how other investors will interpret and react to new information. Many value investors regard this as either impossible or an example of the “greater fool theory”—buying an asset without regard to its underlying value in the hope that someone else will later pay an even higher price.

Although I am a devoted admirer of Charlie Munger, I do not think every short-term strategy can be dismissed in this way. Markets can be neither perfectly efficient nor completely irrational. Lasse Heje Pedersen describes them as “efficiently inefficient”: inefficient enough for opportunities to exist, but efficient enough that exploiting them remains difficult and costly.3

Catalyst-driven trading is one example. An investor might use lawful alternative data, such as satellite imagery, to estimate customer traffic at a supermarket before its results are published. In biotechnology, an investor might combine the design of a clinical trial, the biological mechanism, biomarker evidence and results from related programmes to estimate the probability of success before the readout.

The objective is not simply to determine whether supermarket sales have risen or whether a clinical trial will succeed. It is to estimate those outcomes more accurately than the market and then compare that estimate with what the current price appears to imply.

Long-Term Capital Management is a useful illustration of both the power and the limits of this approach. LTCM was founded by John Meriwether and included Myron Scholes and Robert Merton among its principals. It used mathematical models to identify temporary discrepancies between the prices of related securities and generated strong returns for several years.

In 1998, however, the Russian financial crisis and the resulting flight to liquidity drove many of those discrepancies wider rather than narrower. LTCM’s enormous leverage meant that it could not simply wait for its positions to converge. Fourteen financial institutions eventually participated in a private recapitalisation facilitated by the Federal Reserve Bank of New York.4

The lesson is not that sophisticated models cannot identify genuine mispricings. It is that predicting an eventual outcome is different from constructing a position capable of surviving the path towards that outcome.

Being right eventually is not enough if you become insolvent first.

Predicting the business

Unlike short-term traders, long-term investors—especially value investors—use the performance of the underlying business as their anchor. They assess the business model, economic moat, competitive environment, management quality, capital allocation and other factors that might determine the company’s future earning power.

Conceptually, this is not entirely different from a quantitative strategy searching for variables that are predictive of, or perhaps causally related to, a shorter-term outcome. The factors are usually less readily quantifiable, and the outcome lies much further in the future, but it remains a forecasting problem.

Traditional Graham-style investors placed greater emphasis on buying statistically cheap businesses. Buffett- and Munger-style value investors became more willing to pay a fair price for an exceptional business. The type of company being purchased changed, but the underlying objective remained the same: find a difference between the expectations implied by the current price and the future that eventually occurs.

The voting-machine metaphor can make this sound easier than it is. Over the long term, business performance may matter more than temporary sentiment. But the investor must still predict that business performance.

In fact, a longer horizon can introduce more variables rather than fewer. Technology changes. Competitors emerge. Regulations shift. Management teams are replaced. Capital is misallocated. Consumer behaviour evolves. A company can remain an excellent business but still be a poor investment because its price already assumes too much. It can also appear cheap because the market has correctly anticipated deterioration that has not yet become visible in the financial statements.

The weighing machine is useful only if you can estimate what will eventually be placed on it.

One of my favourite illustrations is the long-term contest between Mohnish Pabrai and Guy Spier. On 14 October 2018, they began a twelve-year stock-performance contest, running until 2030. Pabrai chose Sunteck Realty, an Indian property developer, while Spier chose Mastercard.5

At first glance, the contest appears to ask whether a mediocre business bought cheaply can outperform a much better business bought at a demanding valuation. But it would be a mistake to reduce Pabrai’s position to “cheap beats great”. He had a business forecast as well as a valuation argument. He believed that Sunteck’s prospects were better than its price suggested, while questioning whether Mastercard’s apparently exceptional economics would remain as durable as the market assumed in a changing payments landscape.

The example illustrates two separate points.

First, cheap is not a strategy. Sunteck could outperform only if the pessimism embedded in its price proved excessive. Mastercard could justify a high valuation if its competitive advantages and growth lasted for sufficiently long. Neither valuation could be assessed independently of a forecast about the business.

Second, even highly regarded value investors have considerable difficulty predicting long-term outcomes. Pabrai and Spier examined the same investment horizon using broadly similar value-investing principles and reached opposing conclusions. Moving the forecast from the next quarter to the next decade did not eliminate uncertainty. It merely changed what had to be predicted.

I will leave interested readers to look up the score so far, although an interim result does not settle a contest designed to end in 2030. That itself is part of the problem: a long-term thesis may take many years to evaluate, by which point it can be difficult to distinguish patience from an unwillingness to admit that the original forecast was wrong.

Cheap is not a strategy

Earning an uncommon return therefore requires more than identifying something attractive at the first level. It involves two related judgements.

The first is identifying where the market is making a mistake.

The second is understanding how that mistake will eventually become a return for the investor.

Predicting that a company will report excellent earnings is not useful if everyone already expects excellent earnings. The expectation will be priced in. What matters is whether the reality beats or falls short of that expectation.

The same applies to value investing. A low valuation may indicate that the market has made a mistake, but it may also correctly reflect deteriorating economics, poor management, excessive leverage or a permanently declining industry. The investor has to explain not only why the company appears cheap, but why it is incorrectly cheap.

Even when the market is wrong, a further question remains: what will close the gap between price and value, and when?

The answer does not always have to be a single catalyst. The gap may close through an earnings release, takeover, asset sale, restructuring, liquidation or regulatory decision. Alternatively, the business may compound its intrinsic value until the original purchase price becomes obviously inadequate. Dividends and share buybacks can also return value to shareholders without requiring the market to change its opinion immediately.

But there still needs to be a credible path from the analytical insight to the investment return. Few people can hold a cheap security for ten years while its price does nothing. More importantly, they should not necessarily do so when the lack of progress may indicate that the thesis, rather than the market, was wrong.

Joel Greenblatt’s The Little Book That Beats the Market, first published in 2005, offers a useful example. Its Magic Formula ranks companies using earnings yield and return on capital, combining valuation with a quantitative measure of business quality. Greenblatt’s original backtest reported an average annual return of 30.8% between 1988 and 2004, compared with 12.4% for the S&P 500.6

However, valuation and measured quality are still not a complete investment thesis. The formula does not explain why a particular company is mispriced, whether its quality will persist, or what will cause the gap between price and value to close. Its annual rebalancing rule provides an exit mechanism, but not a company-specific theory of correction.

The extraordinary return in the original backtest has not been reproduced consistently since the book’s publication. Later studies often find some outperformance, but the results are much less spectacular and depend heavily on the market, company size, transaction costs and implementation. Some studies have even found that earnings yield alone outperformed the complete formula.7

This does not mean that valuation or business quality has stopped mattering. It means that a screen may tell an investor where to look without explaining what the market has misunderstood or how that misunderstanding will eventually become a return.

The retail investor’s actual edge

I hope it is now apparent why active investing is such a cost-inefficient way for most retail investors to make money, regardless of the time horizon they are trying to predict.

Retail investors generally have less time, data, access and specialist support than professional investment teams. More importantly, many do not have a clear answer to the central questions:

What has the market got wrong? Why am I better placed to recognise that mistake? How will the mistake eventually become a return?

Without convincing answers, buying an individual stock is not an expression of differentiated insight. It is a bet that an unexplained personal judgement happens to be more accurate than the collection of expectations already reflected in the price.

For most people, putting their spare capital into a low-cost, diversified index fund is therefore a sound strategy. Warren Buffett has repeatedly made essentially this recommendation. In his 2013 shareholder letter, he wrote that the objective of a non-professional should be to own a broad cross-section of businesses and disclosed instructions for the relevant cash in his estate to be invested 90% in a very low-cost S&P 500 index fund and 10% in short-term government bonds.8

That does not necessarily mean that the continued growth of index investing is harmless. A strategy can be rational for each individual investor while creating more complicated collective effects when adopted at sufficient scale. We have already seen reasons to ask what widespread passive ownership might mean for price discovery and market structure, but that is perhaps the subject of another essay.9

Nor does the case for index investing mean that retail investors can never outperform.

Peter Lynch took a much more optimistic view in One Up on Wall Street:

“The amateur investor has numerous built-in advantages that, if exploited, should result in his or her outperforming the experts, and also the market in general.”

10

That should not be interpreted as a promise that ordinary investors can easily beat professional institutions. The important word is exploited. A theoretical advantage is irrelevant when the investor chooses not to use it.

Retail investors can operate with amounts of capital that are too small to matter to a large fund. They can enter a small or relatively illiquid opportunity without moving the price or worrying whether the position will ever be large enough to affect a multibillion-pound portfolio.

They may also possess specialised knowledge derived from their professions. A biologist may understand the plausibility of a drug mechanism better than a generalist analyst. An engineer may recognise the importance of an obscure component in a supply chain. A customer, supplier or industry employee may notice changes in a business before those changes become obvious in its consolidated financial statements.

Specialised knowledge does not automatically create an investment advantage. The investor must still connect that knowledge to the expectations embedded in the price. Understanding that a technology is impressive is not the same as knowing whether the company developing it is undervalued. Nevertheless, genuine domain expertise can narrow the field in which a retail investor is competing at a disadvantage.

Retail investors also face no requirement to remain fully invested, stay close to a benchmark, generate a new idea every month or justify every quiet quarter to clients. They can study a narrow group of companies, reject almost all of them and wait until an opportunity appears in which both the market’s mistake and the route to its correction can be clearly explained.

When retail investors beat the market, they rarely do so by behaving like under-equipped institutions. They do so by choosing a game in which small size, specialised knowledge or patience genuinely matters.

Constant trading usually gives these advantages away.

The retail investor who reacts to every headline, follows every popular stock and trades merely because the market is open has voluntarily entered the part of the market where institutions possess the greatest advantages. Activity creates costs, shortens the decision horizon and removes the ability to wait for an unusually favourable opportunity.

Perhaps reaching the mysterious age at which one becomes interested in stocks is not the unfortunate part. The unfortunate part is confusing enthusiasm with edge, and the desire to participate with an obligation to act.

The retail investor’s most underused advantage is not speed. It is the freedom to do nothing.

Sources

Footnotes

  1. William F. Sharpe, “The Arithmetic of Active Management,” The Financial Analysts Journal, January/February 1991. ↩

  2. Warren E. Buffett, “Berkshire Hathaway Shareholder Letter, 1987.” ↩

  3. Lasse Heje Pedersen, “Efficiently Inefficient Markets for Assets and Asset Management.” ↩

  4. Federal Reserve History, “Long-Term Capital Management: The Near-Failure of a Hedge Fund.” ↩

  5. Mohnish Pabrai, “Q&A with Peking University, Guanghua School of Management, May 1, 2019.” ↩

  6. Joel Greenblatt, The Little Book That Still Beats the Market, Wiley. ↩

  7. “The Magic Formula: Evidence and Implementation,” Research Papers in Economics and Finance. ↩

  8. Warren E. Buffett, “Berkshire Hathaway Shareholder Letter, 2013.” ↩

  9. European Central Bank, “The Rise of Passive Investing and Its Implications for Financial Stability,” Financial Stability Review, November 2024. ↩

  10. Peter Lynch with John Rothchild, One Up on Wall Street. ↩