The idea of trying to “time” the market — of trying to get in before it goes up, and get out before it goes down — has a terrible reputation.

Timing “is a wicked idea — don’t try it, ever,” wrote Charles Ellis, one of the leading lights of indexing, many years ago.

According to conventional wisdom, any attempt to time the market is fundamentally flawed. Stock markets follow a ‘random walk’, they say. No one can predict the market’s next move, so trying to do so will end up costing you money. A lot of your long-term gains will come from a few big “up” days, and these are completely unpredictable — if you are out of the market when they happen you will miss out on a lot of profits.

Money managers often push this idea to the clients. It has, from their point of view, a side benefit: It helps keeps the clients fully invested at all times, which means their assets are generating more fees.

But is the idea correct?

The simple answer: No.

Yes, most people who try to time the market end up screwing it up — they buy and sell at the wrong times — but that does not mean the idea is flawed.

On the contrary, historically, “smart” timing, based on market fundamentals, has been one of the soundest ways to beat the market and produce above-average investment returns over the long term.

What is smart timing? Simple: It is long-term timing, and it is based on following a few solid valuation metrics.

It is not about trying to trade short-term. It is not about selling stocks on Wednesday and planning to buy them back on the following Monday. It is not about obscure market technicals like “head and shoulders” formations or Bollinger bands.

It is about cutting your exposure to stocks when the market is expensive in relation to fundamentals, and keeping your exposure down—if need be, for years—until the market becomes much cheaper. It then involves increasing your exposure, and keeping it high, again for years if necessary.

The myth of ‘random’ returns

Techniques available to anyone have worked, and worked well, for over a century. That does not mean they will work in the future, but it is a strong argument in their favor.

First, let’s demolish the myth that the stock market produces entirely “random” returns—that some years it’s up, other years it’s down, that over time it just goes up, and no one can predict anything in advance.

According to long-term data tracked by New York University’s Stern School of Business, an index of the top 500 U.S. stocks has produced since the late 1920s an average return of about 9.3% a year, when you include reinvested dividends.

That sounds like a great return, and it’s the sort of thing money managers often tell their clients. But it contains two hidden nasties.

The first is that it isn’t adjusted for inflation, which means that in real spending-power terms you made several percentage points less each year. And the second thing is that those returns did not come randomly. They came in long waves—bull markets followed by bear markets followed by new bull markets. They were not random at all.

Using Stern’s data and inflation numbers from the U.S. Labor Department, I stripped out the hidden inflation in the stock market returns and looked at the “real” returns, in other words the returns in actual purchasing power. This is what actually matters. Then I looked at these returns over ten-year periods. The reason for that is that if you are an ordinary investor — rather than a trader on Wall Street — what you are usually looking for is somewhere to grow your money soundly over the medium to long term.

You can see the results in the chart near the top of this article.

These results are not random. They are nothing like random. The waves are as clear as — well, as clear as a big wave at Sunset Beach.

If you invested in the stock market in the 1940s or early 1950s, you earned spectacular returns as you cashed in from the gigantic postwar boom.

And if you invested from the late 1970s to the early 1990s, once again you earned spectacular returns in the subsequent returns due to the huge boom from 1982 through 1999. Lucky you.

But what about at the other times?

Hmmm.

If you were unlucky, or foolish, enough to invest in the late 1920s, the later 1930s, or between 1963 and 1973, you were right out of luck. Your returns were terrible. In many cases you actually lost money on the stock market, after accounting for inflation.

Not just for one or two years. Over 10 years.

So even though since the late 1920s the average ten-year “real” return to U.S. stocks (after inflation) has been about 6.4% a year, a quarter of the time it was actually less than 1.3%--a number I chose because it happens to be the guaranteed “real” return on long-term inflation-protected U.S. bonds (I own a few) available right now. When you deduct taxes and investment costs—even in low-cost funds—the actual returns earned by most investors were lower still.

Remember how people tell you the indexes will never let you down if you stick with them for five to 10 years? It’s nonsense.

Note also, please, that these 10-year figures do not include any allowance whatsoever for volatility. Someone who invested in Wall Street in 1928 and held on for 10 years earned a real return of just 0.25% a year, after accounting for inflation. But just to earn that miserable payoff he had to stick with his stocks during the biggest crash in modern history, the 90% collapse of 1929 to 1932.

If he lost his job, or even just lost some of his nerve (understandable), and trimmed his position in the meltdown, he didn’t even get his 0.25% a year. He probably lost money.

The go-with-the-flow crowd pretends that these long periods of poor performance are basically costless. “Just sit there and wait,” they say, “and the next bull market will come along in due course. Don’t try to time these things.” But that’s deeply disingenuous. While you are earning nothing in stocks, you are missing out on gains in bonds or other assets.

The full cost of waiting out these bear markets is horrendous. When you factor in the fees, the taxes, the volatility, and the opportunity cost of what you could have been earning elsewhere, the investor gets hosed.

Knowing the wrong time to invest

OK, some will say. I understand that if I invest in the stock market at the wrong time I may fare very poorly for a decade. But what help is that knowledge? It would only be useful if I were able to work out in advance when those wrong times were.

The good news? You probably can.

I say “probably” because humility is the first virtue of investing, and we can never know the future for certain. We can only apply intelligence guided by experience, and trust to strong probabilities.

There are three measures which have a strong track record of predicting whether this is a good time to make long-term investments in U.S. stocks. Some of them may even have worked since the Victorian era, though I am skeptical of stock market data going back before the World War I. Certainly they seem to have worked well since the 1920s.

Those three measures are the Cyclically-Adjusted Price-to-Earnings Ratio, or CAPE; the Cyclically-Adjusted Book-to-Market ratio; and the “q” ratio. Two of these measures — the CAPE and the q — are readily accessible to the public.

Wielding the CAPE

The CAPE has been popularized by Yale University finance professor Robert Shiller, most notably in his book “Irrational Exuberance,” in which he predicted the bear market which began in 2000. It is popularly known as the Shiller PE ratio. It helped him just win the Nobel Prize for Economics. A summary of some of his research is available here.

This metric compares the current prices of stocks, not to this year’s or last year’s per-share earnings, but to the average per-share earnings of the past 10 years (adjusted for inflation). The argument for using this measure is that it smooths out short-term booms and slumps in profits. By this measure, the S&P 500 index has historically been on an average valuation of about 16 times cyclically-adjusted earnings. When share prices have fallen a long way below that level they have proven to be a really good deal over time: Investors who got in when stocks were cheap and hung on made super returns. On the other hand, when the CAPE or Shiller PE has been much above 16, the stock market has been a much less good deal. The subsequent returns have usually been mediocre or worse.

For example a recent analysis by Mebane Faber of Cambria Investments found that, from 1881 to 2011, if you had invested in the stock market when the CAPE was below 5 — a very rare occurrence — you would have earned a spectacular 22% a year over the next five years, even after accounting for inflation. You’d become rich.

If you had invested when the CAPE was between 5 and 10, you’d have earned on average 13% a year.

On the other hand, if you had invested when the CAPE was over 20 you would have earned just 5% a year, and if you had invested when it was over 25 you would have lost money, after accounting for inflation.

The correlations are strong. So, for example, during the two golden ages the Shiller PE was low. In the 1940s and early 1950s, and again from the late 1970s to the early 1990s, the Shiller PE averaged about 12. On the other hand, in the late 1930s, and in the later 1960s, the Shiller PE frequently rose above 20. In the late 1990s, when the go-with-the-flow crowd were cheering on ”stocks for the long run” and urging you to increase your allocation, the Shiller was flashing bright red warnings above 40.

Clifford Asness, co-founder of money firm AQR Capital and one of the smartest market analysts around, has also studied the performance of the Shiller PE as a predictive tool. His conclusion? Historically, the higher the Shiller PE when you invest in the market, the lower your likely 10-year returns. The results aren’t perfect — in the real world time and chance happen to us all — but they are remarkable. “Ten-year forward average returns fall nearly monotonically as starting Shiller P/E’s increase,” he wrote to clients late last year in a quarterly report.

It is not a perfect measure, of course. The CAPE would have gotten you into stocks too early in the mid-1970s, and out again too early in the mid-1990s. But overall someone who had used the Shiller PE to guide their investment allocation to stocks over many decades would have beaten the market.

Taking cues from ‘q’

The same is also true of the “q” measure, originally studied by economics Nobel laureate James Tobin. This compares the value of U.S. company stocks with how much it would cost to replace all their assets. Back in 1999-2000, when Shiller was using the CAPE to predict the stock- market bust, British financial consultant Andrew Smithers and University of London finance professor Stephen Wright were using the q to do the same thing. They published the results in a book, “Valuing Wall Street.” Some of the research is available here.

The q and the CAPE correlate remarkably closely. Both tend to rise and fall at about the same time and the same way. The q can be tracked by looking at the Federal Reserve’s quarterly reports on money flows in the U.S. economy.

Historically, the “q” ratio averages about 0.6 to 0.7, meaning that the total value of U.S. stocks has typically averages about 60% to 70% of the cost of replacing all those companies’ assets.

Today both the Shiller PE and the q show the market well above long-term averages. The Shiller PE is about 25 and the q is 0.96. This suggests that investors should be exercising a sharp degree of caution. The corollary of this idea is that these things can take years to play out. The market can go up a long way before it comes back down—if it does.