Problem with backtesting - A good read.

Problems with backtesting Excerpt from Fooled by Randomness. The hidden role of chance in life and in the markets.' by Nassim Nicholas Taleb.

A backtester is a software program connected to a database of historical prices, which allows me to check the hypothetical past performance of any trading rule of average complexity. I can just apply a mechanical trading rule, like buy NASDAQ stocks if they close more than 1.83% above their average of the previous week, and immediately get an idea of its past performance. The screen will flash my hypothetical track record associated with the trading rule. If I do not like the results, I can change the percentage to, say 1.2%. I can also make the rule more complex, I will keep trying until I find something that works well.

What am I doing? The exact same task of looking for the survivor within the set of rules that can possibly work. I am fitting the rule on the data. This activity is called data snooping. The more I try, the more I am likely, by mere luck, to find a rule that worked on the past data. A random series will always present some detectable pattern. I am convinced that there exist a tradable security in the Western world that would be 100% correlated with the changes in the temperature in Ulan Bator, Mongolia.

An outstanding paper by Sullivan, Timmerman and white goes further and considers that the rules that may be in use successfully today may be the result of survivorship bias.

Suppose that over time, investors have experimented with technical trading rules drawn from a very wide universe in principle thousands of parameters of variety of types of rules. As time progresses, the rules that happen to perform well historically received more attention and are considered serious contenders by the investment community, while unsuccessful trading rules are more likely to be forgotten. If enough trading rules are considered over time, some rules are bound by pure luck, even in a very large sample, to produce superior performance even if they do not genuinely possess predictive power over asset returns.

I have to decry some excess in backtesting that I have closely witnessed in my private career. There is an excellent product designed just for that, called Omega TradeStation, that is currently on the market, in use by tens of thousands of traders. It even offers its own computer language. Beset with insomnia, the computerized day traders become night testers plowing the data for some of its properties. By dint of adjusting the rules the trader will hit upon hypothetical gold somewhere. Many of them will blindly believe in it.

Note : Nassim Nicholas Taleb is a Trader, Hedge fund manager, best selling author, and a Professor. His 2007 book The Black Swan was described in a review by Sunday Times as one of the twelve most influential books since World War II.

what wrong in driving a system from the data.
I guess the success of such drived system depends in what time interval it wins.

example : let assume the system is giving good results on every year with certain minimum % .
let now break the testing and check for month wise .. again let assume the system gives certain minimum %.
Then is safer to assume the system is good . though the results may not good for week wise or daily .

any comments ?

This is classic problem with inference knowledge

In the name of data driven system, you consider only one path of event among *ALL* possible paths (while some other possible paths could be equally or more probable!).

Friend,

I could not resist to share my free advice. Try and see this positively.

Making money (specially in stock market) is not about knowing 1000s of system or paths or methods ... but executing something that you know and reasonably sure of.

If you believe basic theory of technical analysis... it says past will get repeated in future (atleast on charts). Yes... one should not be very very specific about backtested parameters but an overall idea is worth knowing and other equally important aspect is start executing rather than researching other unknown but comparable paths.


As wiser people say... fun of pudding is in eating and not discussing about it.

Regards,

what wrong in driving a system from the data.
I guess the success of such drived system depends in what time interval it wins.

example : let assume the system is giving good results on every year with certain minimum % .
let now break the testing and check for month wise .. again let assume the system gives certain minimum %.
Then is safer to assume the system is good . though the results may not good for week wise or daily .

any comments ?

This is classic problem with inference knowledge

In the name of data driven system, you consider only one path of event among *ALL* possible paths (while some other possible paths could be equally or more probable!).

First a non stock example:
Babu, a guy who sees sun everyday rising on east, believes "sun will rise on east *ALWAYS*",

But astronomers know this is not the case, afterall hundreds of millions of stars explode as supernovae for every second in this universe, those exploded suns are never going to be there next day like Babu thinks ! Some data also show galactic collisions where galaxies are ripped apart every now and then (given billions n billions of galaxies are there)!
So once you understand all possible paths that universe can take, your confidence should not depend on the parameters that did well in one path! There are equally likely millions of other paths that his sun didn't take when Babu back tested his idea!
So what people with blind faith in backtests fail to see is all those *OTHER EQUALLY PROBABLE* paths!

problem #1:
The existence of invisible other possible event paths with equal or more probability while you have prepared for only one (or extreme small fraction of paths), that's the core problem with backtesting !

There is one way to find out survival quotient of your system
survival quotient = (probability of survival paths)/ (probability of all possible paths to failure!)

Essentially no-one knows how to compute the denominator, all though you have succeed in identifying numerator (partially!). One thing is for sure the denominator is extremely big! Even guys with nobel prize underestimated the denominator ! Economics Ph.Ds who predicted one in Trillion chance of failure are dumbfounded when they see those rare events occurring every 5-10 years!

The above discussion doesn't even cover blackswans, it talks just about normal other combinations of market structures that the system failed to consider!

problem #2
Here is another small story from Nassim:
A chicken trying to plot its health and growth, will see regular growth until the last day!
Seeing a chicken trying to backtest its health equity is a comedy, if you are a chicken farmer!

======= in summary =====
I think its not all waste of time to back test. Perhaps it can be used to see how 'money management' and 'position sizing' fairs well but we should always keep in mind both problem #1 and problem #2 always exist ! I think, we should position size very conservatively when our system says green and should try a small position to capture blackswan trades even when our system says red!

If a static strategy fails backtesting then it just has no hope, but we can't just hope a static strategy to work out in reality even if it has passed backtesting !
If the static strategy works then fine, I don't think it will work happily thereafter forever after you have coded it down !

I don't know what one should do to have a dynamic winning strategy which adjusts for climates and possible combination of market structures!
I guess it should depend on core principles (Game theory, Bayesian Inference, Order Flow, Position sizing, Risk Management & intuition!) not static price/volume indicators & rules, which obviously is no easy task!

Friend,

I could not resist to share my free advice. Try and see this positively.

Making money (specially in stock market) is not about knowing 1000s of system or paths or methods ... but executing something that you know and reasonably sure of.

If you believe basic theory of technical analysis... it says past will get repeated in future (atleast on charts). Yes... one should not be very very specific about backtested parameters but an overall idea is worth knowing and other equally important aspect is start executing rather than researching other unknown but comparable paths.


As wiser people say... fun of pudding is in eating and not discussing about it.

Regards,

what wrong in driving a system from the data.
I guess the success of such drived system depends in what time interval it wins.

example : let assume the system is giving good results on every year with certain minimum % .
let now break the testing and check for month wise .. again let assume the system gives certain minimum %.
Then is safer to assume the system is good . though the results may not good for week wise or daily .

any comments ?

Good one. The example of chicken is funny, but also true.

I had read 'The black Swan' and 'Fooled by Randomness' by Nassim Taleb, but did not come across the story of the chicken. Is is somewhere else.?