The efficient market hypothesis is a cornerstone of classical investment theory. It asserts that the market is efficient, or instantaneous, in incorporating all public information into to the prices of securities. Under perfect informational efficiency, if a company releases higher than expected earnings, all effects of that information are expected to immediately be reflected in that company’s stock price. Buying shares of that company anytime after the earnings report is released, even within seconds, is a futile attempt to capitalize on an inflection that has already come and gone. This goes for all public information - financial news, politics, technological innovations, natural disasters, resource discoveries, etc. If all information is efficiently priced into the market, then market prices always equal intrinsic values. The only driver of price fluctuation is the introduction of new information, which is random (else, under this theory, it would have been acted upon and priced into the market already). Thus, the efficient market hypothesis implies that it is impossible to consistently outperform the average market return because there is no way to know what the market will do next without insider information. Outperforming the market is the product of randomness, not alpha.

We know that markets are not perfectly efficient in practice - the housing bubble and 2008 stock market crash are evidence of that. Investors can speculate, overreact, misinterpret information, differ on valuation assumptions, misjudge scopes of effect, and generally act irrationally by allowing emotions to get in the way of what should be logic-driven decision-making. An entire field of research called behavioral finance has emerged to study (and capitalize on) these irrationalities. Arbitrage schemes, which many hedge funds build their investing strategies around, operate by exploiting market inefficiencies.

If the efficient market hypothesis does not hold, does this mean an intelligent and informed individual investor can successfully beat the market? Well before you jump in, you should know what you’re up against.

Most of the trading volume in the stock market today, around 70% according to Wired, is the result of automated algorithmic trade executions. These programs, designed by quants and sophisticated investors, act on and react to technical trends and actionable input with split second haste. Tools like Lexicon from Dow Jones & Company, the news corporation behind the Wall Street Journal, enable these activities by releasing breaking news stories in terms that can be read, interpreted, and acted upon immediately by said programs. Wired explains:

… the professional investors subscribing to Lexicon aren’t human—they’re algorithms …They don’t need their information delivered in the form of a story or even in sentences. They just want data—the hard, actionable information that those words represent … [Lexicon] scans every Dow Jones story in real time, looking for textual clues that might indicate how investors should feel about a stock. It then sends that information in machine-readable form to its algorithmic subscribers, which can parse it further, using the resulting data to inform their own investing decisions.

With their sub-second reaction time, these algorithms make it very difficult for human investors to capitalize on public information; we can’t compete with programs that can price developments into the markets before we even hear about them. It sounds a lot like an efficient market, but it certainly is not. Interpretations are subjective, even for these algorithms, and program code can behave in unexpected ways. Wired continues:

[Algorithms] respond instantly to ever-shifting market conditions, taking into account thousands or millions of data points every second. And each trade produces new data points, creating a kind of conversation in which machines respond in rapid-fire succession to one another’s actions. At its best, this system represents an efficient and intelligent capital allocation machine, a market ruled by precision and mathematics rather than emotion and fallible judgment. But at its worst, it is an inscrutable and uncontrollable feedback loop. Individually, these algorithms may be easy to control but when they interact they can create unexpected behaviors—a conversation that can overwhelm the system it was built to navigate.

In May 2010, the NYSE experienced what has come to be known as the “flash crash,” in which a single trade tipped the scales prompting sensitive algorithms to launch a cascade of sell orders. The Dow Jones Industrial Average Index dropped 600 points in a matter of minutes.

And this isn’t an isolated incident.

In his book The Quants, reporter Scott Patterson explains that these formulas are rooted in physics and cryptography, and the imprecision and volatility in the markets stem from an underlying misconception on which these algorithms are constructed. BusinessWeek elaborates in an early 2009 article entitled “Perfect Models, Imperfect World”:

Physics, because of its astonishing success at predicting the future behavior of material objects from their present state, has inspired most financial modeling. Physicists study the world by repeating experiments again and again to discover natural forces and their almost magical mathematical laws. Galileo dropped weights from Pisa’s leaning tower. Giant teams in Geneva study what happens when protons repeatedly collide. If a law is proposed but experiments contradict its predictions, it’s back to the drawing board. The method works. The discovered laws of atomic physics are accurate to more than 10 decimal places. Financial theory has tried hard to emulate physics and discover its own elegant, universal laws. But finance and economics are concerned with the human world of monetary value. Markets are made of people who are influenced by events, by their feelings about events, and by their expectations of other people’s feelings about events. There are no fundamental laws in finance. And even if there were, there is no way to run repeatable experiments to verify them. Financial theories written in mathematical notation—aka models—imply a false sense of precision.

Effectively, the markets cannot be perfectly efficient, and hence there will always be opportunity. But competing with programs that can incorporate new information into the financial system immediately and trade in ways you cannot might warrant a change of approach. Here are a few passive strategies that would make more sense to use in new age Wall Street:

• Construct your own algorithm
• Employ returns-based style analysis to replicate a hedge fund or mutual fund
• Construct a passive portfolio using applied statistics
• Buy and hold the market
• Explore a site like Betterment

The field of investing is centered on expected risk and return. This week Amazon.com, Inc. released its second quarter earnings report which revealed a 45% increase in earnings driven by a 41% increase in sales, yet the share price dropped 13% on this news in after-hours trading. Though Amazon reported an increase in earnings, this increase was not high enough to match investors’ expectations which were built in to the intrinsic value of the securities. Thus a lower than expected earnings report, though positive, resulted in a sell off and price drop.

I’m still trying to figure out how to blog about what interests me, but I can already tell that there is no value added by regurgitating major market news that has already been dissected and analyzed by news corporations. This post will instead be didactic: I’ll touch on some market news while introducing an instrument that measures investors’ perception of market risk: the Volatility Index (VIX). The VIX measures the implied annualized volatility of the S&P 500 for the next 30 days.

Today Bloomberg reports that the VIX closed at 28.79. Using that number I can calculate that the perceived daily volatility by dividing 28.79 by the square root of 252 (approximate number of trading days in a year) and the perceived monthly volatility by dividing 28.79 by the square root of 12 (number of months in a year). I find the monthly to be 8.31%. This is the expected standard deviation; if the expected monthly return on the S&P is 3%, investors anticipate that there is a 67% chance (“it is reasonable to assume”) that the S&P will move between 11.31% (3%+8.31%) and -5.31% (3%-8.31%) of its current value in the next month. Another way of thinking of this is as the expected risk of being invested in the S&P for the next 30 days.

The market’s adjustment to information is very efficient. On the graph of the VIX above, major fluctuations correspond to major events in financial news. The spikes in May occurred because of the Greek debt crisis and the sudden loss of confidence in the European Union’s ability to hold itself together financially. The VIX rose this week, not so much because of disappointment over a single company like Amazon, but because of the financial regulation bill said to be “the most sweeping overhaul of U.S. financial-market regulations since the Great Depression.” Several aspects of the bill, such as the refusal to bail companies out in the future, increase the risk of losing an investment in the market, which hikes up in the VIX.

So how does one profit off of the VIX? There are infinite strategies to act on the VIX, directly and indirectly. Investment vehicles include VIX ETFs, options, and futures contracts. One might collar other investments when the VIX is high, and sell covered calls when it is low. The market is not perfectly efficient, so there is an opportunity to capitalize on overreactions (more on market efficiency later). Keep in mind that wild fluctuations make investing directly in the VIX is a short term strategy.

Note: writing about investing is challenging, not because the field is particularly difficult to understand (quite the opposite, actually), but because of the varying levels of experience in the readership. If this post was a review, look forward to some more advanced techniques like hedge fund return replication, CVaR-based portfolio optimization, fundamental analysis techniques, and maybe investment ideas to capitalize on major news in future posts. If this post went over your head, fear not! I’ll likely do a brief introduction to investments in a future post which will bring you up to speed.