Stock Exchange: How to Use Backtests Effectively

 

Everyone interested in managing their own money ought to keep an old idiom in mind: if it seems too good to be true, it probably is. Unfortunately, greed is a powerful motivator. It’s tempting to see a new model with an incredible backtest and think this could be the answer.

Experienced investors know that there’s often a drastic change between a model’s backtest and its first live run. You can usually find that point by checking for where the 45-degree angle increase in value drops off into sideways movement (and generally underperforms the market).

This week, we’ll take a deeper dive into how you can minimize these problems using professional techniques.

Review

Our last Stock Exchange revisited a common theme: making stock picks according to a set time frame. Our models suggested finance and software stocks in the short-term, and energy for the long-term.

Let’s turn to this week’s ideas.

This Week— *crickets*

We usually arrive to find the gang happily enjoying their weekly poker night. Instead, all we’ve got are Felix and Oscar’s weekly rankings with a “gone fishin’” note on the counter. Strange behavior.

We decided to give Vince Castelli a call to investigate. Vince is our modeling guru, a brilliant scientist who spent the bulk of his career as a civilian employee for the U.S. Navy. During his time there, he’s had hands-on experience with modeling techniques vital to national security – not something you can find in the classroom. He knows these models better than anyone; after all, he designed them.

Jeff: Vince! What is this about giving everyone the week off?

V: I didn’t give them the week off. There were new no new fresh signals.

J: Is there something wrong with the gang? They encompass five different methods. How can there be no fresh ideas?

V: A key feature of all models is recognizing the best times to trade. When volatility increases trades become less predictable.

J: What do you mean? The VIX is lower this week. Volatility is down.

V: That measure is for amateurs. Trading volatility includes both upside and downside. That bogus fear gauge emphasizes only the downside. Predictions are affected by extreme movements in either direction.

J: Since we do not have current picks to feature, maybe we could discuss how you developed these models.  There was an excellent recent article from Ben Carlson about what you cannot learn from a backtest. It reminded me of the TV ads suggesting that anyone can discover a system and trade their way to a fortune.

V:  If only it were that easy.

J:  Ben’s description made me think of some suburbanites wandering into the woods with massive chain saws.  The power of the tools far exceeds their skill in using them.

V: I see it all of the time.

J:  I would like to take up a few of Ben’s points and get your reaction.  How about this:  How many bad backtests came before the good ones?

V:  This is a great question.  Most people do not know the right questions to ask the model developer.  That one is crucial.

J:  How would you answer?

V:  Our method preserves multiple out-of-sample periods.   We develop models on our development data, saving pristine time periods for the test.  We verify that the strength of results continues.  You cannot just look at backtest results; you must know the developer’s method.

J:  Here is another good point — Data availability at the time.   Isn’t it easy to “peek ahead” or to exclude data from failing company?

V:  It certainly is.  You need to have data that includes the failed and merged companies.  The average person at home will not pay up to get this.  It introduces a deceptive, positive bias.

J: Ben also raises an interesting point about friction.  He writes:

It’s almost impossible in a backtest to completely account for costs and frictions such as taxes, commissions, market impact from trading, market liquidity, etc. Sure, you can estimate these frictions, but you never truly understand how these things will affect your bottom line until you actually have to execute buy and sell orders.

V:  This is the first point where I really disagree with him.  Why is it almost impossible?  You should definitely include commissions and a slippage factor.  If your trades are a small percentage of the market volume, the impact from trading is negligible.  Taxes vary by the type of account and the investor.

J:  Interesting point!  “Almost impossible” is strong language.  For someone who knows the ropes, this kind of test might represent real edge.

V:  That is what I do with each of my creations!

J:  Some of Ben’s other points relate to psychological factors.  The trader bailing out of the system in the face of losses.  Or concern about real money.

V:  That is strictly a matter of confidence in the system.  If it has been developed properly, you should not do a lot of fretting.

J: Thanks for joining us, Vince. I’m sure your comments will help readers make more sense of our series.

V: Any time!

Conclusion

One important point was not mentioned in Ben’s article – simplicity.  The temptation for the untrained modeler is to introduce as many variables as possible, hoping to find correlations that others have missed.  What they find is misleading. Computers are powerful enough to discover apparent links between variables when there is actually no relationship. A great model uses as few variables as possible.  The backtest may not seem as good, but the real-time trading will be much better.

Quantitative modeling is an extraordinarily complicated field. In some ways, the way to find success here is similar to finding success in the investment world as a whole. Find the right experts, learn their methods, and try to make sense of the data for yourself. Backtesting can be effective or dangerous – it depends on the skill of the developer.

Background on the Stock Exchange

Each week Felix and Oscar host a poker game for some of their friends. Since they are all traders they love to discuss their best current ideas before the game starts. They like to call this their “Stock Exchange.” (Check it out for more background). Their methods are excellent, as you know if you have been following the series. Since the time frames and risk profiles differ, so do the stock ideas. You get to be a fly on the wall from my report. I am the only human present, and the only one using any fundamental analysis.

The result? Several expert ideas each week from traders, and a brief comment on the fundamentals from the human investor. The models are named to make it easy to remember their trading personalities.

Questions

If you want an opinion about a specific stock or sector, even those we did not mention, just ask! Put questions in the comments. Address them to a specific expert if you wish. Each has a specialty. Who is your favorite? (You can choose me, although my feelings will not be hurt very much if you prefer one of the models).

Getting Updates

We have a new (free) service to subscribers to our Felix/Oscar update list. You can suggest three favorite stocks and sectors. Sign up with email to “etf at newarc dot com”. We keep a running list of all securities our readers recommend. The “favorite fifteen” are top ranking positions according to each respective model. Within that list, green is a “buy,” yellow a “hold,” and red a “sell.”  Suggestions and comments are welcome. Please remember that these are responses to reader requests, not necessarily stocks and sectors that we own. Sign up now to vote your favorite stock or sector onto the list!

Stock Valuation and Occam’s Razor

Among competing hypotheses, the one with the fewest assumptions should be selected.

William of Occam

 

The use of this principle is valuable, but not completely determinative in science.  It often has an important application in investing.

Let us consider two hypotheses.

  1. A method of valuing markets that relies upon backward-looking data, looks at replacement value, or depends upon some other fixed ratio. Put another way, all the most popular valuation metrics.
  2. A method that considers prospective earnings, expected inflation, and interest rates.

Method one has been wrong for many years.  In fact, it has been mostly incorrect for decades.  Method two has been on the right side of market moves, but still shows significant deviations.  What can we learn from Occam’s Razor?

Method One

Since this method has been mostly wrong, many explanations have been offered.  I think I left a few out, but you get the drift.

  1. Speculation
  2. Not recognizing “fundamental” risks – Euro collapse, China collapse, recession, Brexit, etc.
  3. Depending upon dubious earnings estimates
  4. Market is about to crash
  5. Method not good for market timing, but returns will be poor for the next 5, 7, 10, 12, ? years
  6. Fed intervention – money printing and pumping up the market via QE
  7. Plunge protection team
  8. European Central Banks
  9. Suckers’ Rally
  10. Myopia of the investment world – no efficient markets
  11. High Frequency Traders and Algorithms

Method Two

Since this method has been mostly right, little explanation is needed.  The expected increase in market prices and multiples is consistent with the theory.  It should continue for another 8-10% and further if forward earnings increase.

 

Question

 

Should investors accept the complex and ever-changing explanations for method one?  Or perhaps should they consider that the method itself is flawed?

 

Are You Fooled by this Chart?

Throughout the big rally investors have been bombarded for reasons not to join in.  It is very profitable to play upon fears.  You then sell page views and advertising, conferences, gold, annuities, or fancy structured products.

I have taken a much less popular path – trying to educate investors.  Even though the issues at hand are the most important, much more significant that individual stock selection, it is not “actionable investor advice.”

I disagree!  Investors should act, and they should fight fear.

As an example, one of many I see each week, let’s take a look at this chart:

Let us start with a few simple questions:

  1. What is the red line? Presumably the blue line is the S&P 500, although the chart does not have a legend.  It is apparently some measure of equity risk premium, but there is no description of how it is calculated.
  2. What is the meaning of the gap between the lines, highlighted by the dotted lines? Is a big gap bad?  A warning?  That is clearly the message of the article.
  3. Aswath Damodaran, NYU professor and valuation authority, is cited as the source. Does this chart actually appear in his work, or did the author massage it?  What is Professor Damodaran’s current viewpoint on market valuation?

These questions should have been answered in the article.  The article, warning about the market, is using the technical term “equity risk premium” to frighten people.

Conclusion

I will do the best I can with the incomplete information in the post.

The last time the gap was this wide was in 2008-09, the best time in decades to buy stocks.  Why is it now bad news?

It is definitely NOT Prof. Damodaran’s conclusion.  Concerning Prof Shiller, he writes:

Of all of his creations, I find CAPE to be not only the least compelling but also potentially the most dangerous, in terms of how often it can lead investors astray. So, at the risk of angering those of you who are CAPE followers, here is my case against putting too much faith in this measure, with much of it representing updates of my post from two years ago.

He also provides his own version of our subject chart, described by Alex Barrow.

The following chart is from NYU finance professor Aswath Damodaran. In this, he charts the P/E of bonds (blue line), the Shiller P/E for stocks (purple line) and the ratio or spread between the two (orange bars). The lower the orange bars, the greater the risk-premium spread between bonds and stocks meaning the more attractive (cheap) stocks are relative to bonds or cash.

If you think of the orange bars as the equivalent of the dotted lines in the original chart, you will see that they are the same.  This chart, of course, has a legend explaining each of the lines and bars.  The interpretation – high equity risk premium makes stocks more attractive – is clearly stated.

This is a disturbing example of how writers “on a mission” see whatever they want in charts, provide only partial information, and prey upon the unsuspecting.  The original article is from a very popular source and is republished widely.