This afternoon at the CBOE Risk Management Conference in Carlsbad, CA Amna Qaiser, Portfolio Manager from Goldman Sachs Asset Management and Olivier Sarfati, Head of US Trading Strategies from Citigroup teamed up to discuss backtesting strategies. Backtesting has always been a difficult subject for portfolio managers and traders as backtested results do not always work as well as hoped when the strategy goes live.

Qaiser’s part of this session discussed potential pitfalls but also laid out a methodology to analyze whether a backtested strategy should move to the next step of live trading. “Does the proposed strategy have an economic foundation?” and “Is there academic support for the proposal?” are among the several questions to ask before moving forward with a potential strategy that showed promise when backtested. She finished up her presentation with two case studies. The first was a FX Hedging program with the second being an approach to managing tail risk. The tail risk talk was very interesting.  She compared three strategies - The S&P Low Volatility Index, The Daily Risk Control Index and the CBOE Collar Index.  The Collar had the lowest volatility but would affect returns in large moves higher.

Olivier Saftari began his presentation discussing "September is the cruelist month to trade" a quote from the Wall Street Journal, but is it?  As his data showed, not necessarily.  Backtesting does not necessarily take into account survivorship - is AIG, Lehman, etc. accounted for in back-testing?  Are fund flows an indicator of what will happen in the market or coincidental to market performance?  Probably the latter.

Olivier said he loves Outliers, and most should not be ignored.   If an earthquake hasn't happened in 100 years, should that data be ignored in designing a building?

Olivier talked quickly so I'll give you key points of what he said:

Momentum strategies using backtesting  - slight changes should not alter results  greatly. If so, maybe not right strategy.  He said that if someone stated that if the market moves 2.34% or more that X should happen.  That means that previous stress-tests with a 2.32% or 2.33% move didn't come close to the expected result.

Live trading is sometimes the best way to test strategy, analyze real trade after the fact.

One pitfall: failing to expect the unexpected.  Failing to account for a risk that has never happened.  E.G. implied correlation above 1 in a liquidity crisis, or decoupling in the VIX complex with large move in underlying.   Solution: stress test in hypothetical crisis situation.

Using options could be useful and could be considered to help in the following areas;

1. mark to market risk

2. stress test should be able to show if overly exposed to tail risk

3. never blindly trust backtest.   Use pitfalls to test for robustness

4. understand economic rationale.   What is backtest trying to measure?  Are you altering test to get to a desired result?

Lots of information in this presentation.  Thank you Amna and Olivier.