What it Takes to Beat the Market

2021 September 14 Twitter See all posts

Alpha isn't what you think.

Can You Beat the Market?

What is Alpha?

Alpha refers to excess returns earned on investment above the average stock market return.

Eugene Fama and the Efficient Market Hypothesis

Fama sums up the theory as:

Asset prices reflect all available information.

The implication is that you can't beat the market with publicly available data. Over time Fama and Kenneth French expanded on the original findings to show statistical anomalies relating to small companies and companies with high book-to-market ratios (value stocks). These two types of stock performed better than the original model predicted, as the data stood in the 1990s. Fama and French's model predicted 90% of returns.

Is Information Perfect?

"All available information" hides a lot of complexity. There are three levels of information availability.

  1. Weak Form

    Weak form information is historical price data. Data does not show unexplained returns.

  2. Semi-Strong Form

    Adds data like quarterly reports and disclosures. Markets rapidly incorporate this information, meaning semi-strong data is not an enduring source of advantage or inefficiency.

  3. Strong Form

    One trader or firm has monopolistic access to information. Strong form information can confer an advantage but appears limited in public markets. Fama uses a market maker with an order book as an example of a company that can profit from strong form information.

Why Haven't We Found All the Oil?

The oil and gas industry provides a more tangible look at the Efficient Market Hypothesis.

There is information humans don't know. If humans were omniscient, there would be no dry holes. In real life, dry holes are typical, and returns on drilling vary wildly between projects. Information on the subsurface is limited and expensive to obtain. Available information sets prices for drilling rights.

Lessor/Lessee Information Asymmetry

If you know nothing about oil and gas, but an oil company comes knocking on your door wanting to drill for oil, you know the rights are worth something. But how do you agree on a price for drilling rights when the oil company knows more?

  1. Risk sharing.

    Oil and gas leases (drilling rights) usually have two components. The lessor receives a lease bonus before drilling. And the lessor earns a portion of the oil and gas, usually between 1/4 and 1/8. Revenue sharing solves problems with uncertainty about oil being there and adversarial risk (the oil company knows more).

  2. Ask your neighbors!

    Oil and gas landmen universally hate the local diner/coffee shop. Someone gets a crazy lease bonus or gets 1/5 instead of 3/16? Everyone knows about it.

  3. Call competing oil companies.

    If oil companies are competing for leases in an area, you can start a bidding war. If there is only one company, you have fewer options.

The result is that even though many landowners know zero about geology, lease prices quickly react to new finds and information, even in many cases of private information. Lease prices increase or decrease until the oil company is earning much closer to risk-adjusted index returns.

Beating the Market

For an oil company to earn alpha, it must develop unique geological knowledge where few other oil companies compete. Lessors will not have other parties available to learn from. Lessors can sometimes choose not to lease, but many states have laws that prevent holdouts. States want more exploration and tax receipts.

Because drilling a new well can raise the price of surrounding acreage, oil companies might buy "protection" acreage before drilling. Then they accrue the gains of land increasing in response to new information (well results). Rumors about which company is looking where and wells coming online abound. Sophisticated companies sometimes adopt strict information security practices. Code names, canary traps, and eliminating the use of USB ports on computers are not unusual. Companies with good reputations for finding oil often use fronts to hide their leasing, as speculators will try to front-run them.

It is not uncommon to see someone go to jail for stealing information about leasing targets from a company to front-run their leasing.

Even in an industry with high levels of private information, it is hard to beat risk-adjusted index returns. Markets want to be efficient.

What Alpha Might Look Like

Public Markets are Close to Efficient

One of the strange corollaries to the Efficient Market Hypothesis is that you can't tell if a market is efficient. But there is reason to believe America's public markets are very efficient. Over long periods, close to zero active managers can beat the market.

Renaissance Technologies is one of the most famous hedge funds. They employ some of the best mathematicians and scientists in the world to do quantitative modeling and predict stock prices. They are known for their Medallion Fund, which earns gobsmacking returns. But the fund has no outside investors, and it returns capital instead of compounding it. The fund's strategy focuses on short-term and events-based trading, not long-term analysis.

Renaissance's public funds have returns that look like many other hedge funds. Unremarkable even with all the mad scientists.

Hundreds of the best math minds in the world can only find a few billion dollars a year of inefficiency in market data. All evidence points to opportunities that are small and fleeting.

Asymmetric Information

Beating markets consistently requires legally acquired information others don't have.

Warren Buffet, Computer

Fans of Warren Buffett know that he is like a computer. He has a memory like an encyclopedia and can do complex math calculations in his head. Buffett is the first to admit that most of Berkshire Hathaway's value is ultimately from the insurance business. Speaking about the purchase of Berkshire's first insurance business:

“Indeed, had we not made this acquisition [National Indemnity Company], Berkshire would be lucky to be worth half of what it is today.”

The insurance business is unique because it collects premiums upfront then pays claims later. Insurance companies can invest this "float" and earn interest. You can believe in efficient markets and that Warren Buffett achieved alpha in buying insurance businesses, given how calculation-intensive computing insurance risk is. A human with a computer brain living in 1967 is an example of strong form information.

Buffett then used the low cost of capital float to buy value stocks instead of government bonds, one of the categories Fama and French found overperformed.

Value Stocks or Value Traps?

Economists debate whether the anomaly around value stocks is rational or irrational. Investors could have correctly priced value stocks because of risks around market cycles, interest rates, or anything else. Or they could have made a mistake and been irrational.

These factors may not be permanent. As economists try to add factors to improve the model fit, they are less explanatory across countries. Securities laws, market conditions, and investor knowledge could lead to changes or better pricing that remove old anomalies and add new ones.

Indeed over time, Berkshire's returns have converged with the S&P 500.

Analysts usually attribute the decline in performance to Berkshire's increasing size. Low interest rates also decrease the cost of capital advantage insurance float has.

BRK is ~2% of the US public market cap, but it also buys private and international companies, making it a tiny portion of its potential investing market. The Oracle of Omaha's power is limited.

Margin of Safety

A key concept in value investing is called "Margin of Safety." The more intrinsic value differs from market value, the bigger the margin of safety that can counteract poor luck or miscalculation. Risk lurks, too, because a company with a much higher book value than market value could be a loser that the market has correctly identified.

Buffett is famous for going to companies he was interested in and spending hours or days touring facilities, talking to managers, and asking questions. The visits helped Buffett decide whether the company was a gem or a bust and was another source of strong form information. The same method is a pillar of the Toyota Production System. The best way to learn about something complex is to "Go and See."

Buffett would only buy companies when the margin of safety was high. Once those companies dried up, he redirected capital to other places, like utilities and railroads.

Buffett: “I’ve always said about the energy business: It’s not a way to get real rich, but it’s a way to stay real rich,”

Buffett probably knows when he can earn alpha and when he cannot.

Alpha is not Omniscient

There is often a perception that alpha comes from omniscient prediction machines. It isn't true. Anyone that consistently achieves alpha has almost certainly developed unique skills and techniques that give them strong form information. Alpha is narrow, and there is no guarantee it lasts.

Future of Index Funds

A common thought is that increasing flows into index funds will make the market less efficient, giving an advantage to active investors. There is no rule that index funds only have to be the S&P 500. Funds can be an index of value companies, small caps, or any algorithm. Index funds can capture most inefficiency through investors allocating capital to undervalued indexes.

Digitization means more asset classes will be "indexed."

Conditions that Favor Alpha

These conditions are applicable for public and private markets.

  1. Low Numbers of Participants and Capital

    Even three to four knowledgeable competitors with enough capital can make a market converge to efficiency.

  2. Illiquidity

    Prices are more likely to be away from equilibrium when a seller needs liquidity.

  3. High Uncertainty

    Where uncertainty is low, there is little opportunity. Strong form information provides more edge where uncertainty is high.

All of these factors also increase risk. Returns adjust for risk. Alpha, if it exists, is going to be challenging to see statistically. Even (or especially?) to anyone that thinks they are achieving alpha. Are your returns high because you took more risk and got lucky, or because you have strong form information and advantages? Let us know when you are 90!

The Internet is Compressing Returns

Even before computers, public markets appear to have been efficient. Adequate capital, investors, and information pushed markets towards efficiency.

Many private markets were not nearly as efficient. Local real estate markets and other assets had limited information and few intelligent capitalized buyers within niches. Many buyers could gain strong form information and build a good portfolio. Sellers might not be so happy.

Online listing and information services leveled the playing field. Capitalized buyers can flood in, driving returns closer to risk-weighted index returns. Whether it is rental real estate or oil and gas properties, the alpha opportunities decreased.

Allocating Risk

Because alpha hides in risky, illiquid environments, diversification is required. You could be an alpha-earning investor and still go broke because of bad luck. Low-risk income streams or investments need to cover living expenses.

Extra capital can go towards high-risk investments where you have developed advantages. The number of these investments could be small! Many prefer a venture capital-like model that manages risk across a portfolio. A strategy like this might work while capital is scarce, but over time returns on a portfolio will converge on the risk-adjusted index return as information and capital permeate.

If you have strong form information in a few investments a year and don't need the money to eat, it may be worth taking more risk on a smaller portfolio. The optimal portfolio size depends on the characteristics of the area you are investing in and your risk tolerance.

How to Invest


Buy indexes or similar investments to cover personal spending.

Create specialized knowledge, usually in areas that already interest you, and use those to invest money you can afford to lose. Hopefully, that area of interest is something that few other investors find interesting. Investing in these areas is likely risky, meaning a good understanding of expected value and probability is required. A few winners will likely be the majority of earnings.

If alpha opportunities dry up, extra capital should go towards index-like investments until you learn more skills.

Accelerating Returns

Using labor to juice capital returns is an old strategy. It makes sense to DIY repairs on your first rental house rather than spend hard-earned capital. Cash on cash return increases, freeing capital to buy the next one.

The billionaire version of this is to add value that allows you to invest on better terms. Warren Buffett bought many of his best companies because he let the existing owners keep running the business and managed with a light touch. Nowadays, there is so much capital that demand for this service is limited. A more modern example is Y Combinator. They provide services and support for startups in exchange for advantaged investment terms. Branding matters, too. Andreessen-Horowitz probably isn't building a media arm only for fun.

Avoid Zero-Sum Games

My favorite part of the DraftKings hype was data showing only a small portion of players earned money. A few skilled players fleeced the rest. There was no rising tide of positive index returns to cushion the blow. Zero-sum is tough.

Options are also zero-sum. Some folks use options contracts to create chicken nugget markets, but there are also plenty of sharks. Again, no rising tide to bail you out.


Being the best at one skill or area is difficult. Combinations create rapidly increasing unique opportunities. Combining several skills increases your probability of being one of the top few performers. Many of these might be niche, but some get surprisingly big.

Evaluating Managers and Deals

"Hey, I'm raising a fund to invest in X. It could be a great opportunity. I'm smart, and returns could be as high as Y! I need outside capital to pursue these opportunities!"

Pitches will usually be more polished, but you get the idea. How do you know if the opportunity is worthwhile?

First, if someone can generate alpha, they will not need outside money for long. The opposite problem occurs where there is not enough alpha-generating opportunity for their capital. How often does Berkshire Hathaway issue new stock? Renaissance Technologies forced outside investors to leave Medallion Fund.

95%-99% of offered deals will be bad deals. Sometimes you will catch an investor early, or maybe they have a more marginal opportunity that they only put in 50% of the required capital. They will keep the elite investments for themselves in the vast majority of cases.

You are better off generating your prospects or investing in indexes. Most of these funds exist to allow blame for poor performance to be shifted or keep the money distanced from its owners (like trust funds).

Rich People Have Special Advantages?

I often hear regular people say they think rich people have special knowledge about investing because they are rich or part of some rich folk network. It is true, but not how you might think.

Being rich alone confers no advantage. Multi-generational wealthy families often rely on family offices run by active investors. The track record of these offices is mixed. They prevent heirs from losing money all at once, so this is fine.

SEC accredited investor rules keep competition out of some types of investments. Networks can allow partnerships that combine advantages, but these partnerships are hard to find and maintain.

Children of successful investors and business founders often inherit the same traits that brought success. Parents might pass down techniques and knowledge, whether it is specific or general concepts like expected value. It is helpful to start learning at age eight instead of age thirty because investments have long time horizons. We aren't surprised when sons of former NBA players make the league. We shouldn't be about investors, either. Of course, family offices wouldn't need to exist if this always worked. Many people aren't cut out for it.


Financial leverage increases the riskiness of an asset. It can be a tool that optimizes risk levels and helps an investor get started. Investors can also abuse it to chase higher returns. Many investments are risky enough before debt.


No law says index returns have to be positive. Believing in the Efficient Market Hypothesis is less risky than dismissing it.

Having high-risk investments in areas you have the skills, living below your means, and enough "safe" assets to cover a period of living expenses is the best way to keep your freedom and keep investing. Many others have come to the same logic. Selling labor and spending a high portion of labor income matches most lifestyles better.


Efficient Capital Markets: A Review of Theory and Empirical Work by Eugene Fama

Lease Front Running Scheme Nets over $5 Million

"Fooled by Randomness" by Nassim Nicholas Taleb

Berkshire Hathaway Annual Letters