Thread: Backtest timeframe

I've got a question to all experienced algo traders and especially the DailyFX moderators.

After developing my strategy advisor and modifying it so that it creates the best results, i am now into analyzing its backtest results.

In those nice DailyFX webinars the moderators often point out that its important to recognize that performance on historical data doesnt give you any security that a strategy will perform in the future as well. So i have always tried to remind myself about that if i saw a nice historical performance on my strategy.

So how would you judge a strategy that performs like this:

Symbol EURUSD
Backtest period: 2002/01/01 until 2010/12/31
Return on Initial Capital: 17367 %
Return on Max Strategy Drawdown: 195
Max Strategy Drawdown (%): -27

There was only one single month that was negative.

So to me after testing it on such a long period of 8 eight years, where we had different markets like 2002 crash, ranging, 2008 crash etc., it performed very well, that should indicate clearly that its a robust strategy.

What do you think about the timeframe of a backtest? The longer it is the more sure you can be about the performance in the future?

Would you confirm the assumption, "if you have a strategy that performs in a variety of different market circumstances and over period of several years there is a high probability that this performance will remain in the future" ?

Thanks!

Originally Posted by plainv

I've got a question to all experienced algo traders and especially the DailyFX moderators.

After developing my strategy advisor and modifying it so that it creates the best results, i am now into analyzing its backtest results.

In those nice DailyFX webinars the moderators often point out that its important to recognize that performance on historical data doesnt give you any security that a strategy will perform in the future as well. So i have always tried to remind myself about that if i saw a nice historical performance on my strategy.

So how would you judge a strategy that performs like this:

Symbol EURUSD
Backtest period: 2002/01/01 until 2010/12/31
Return on Initial Capital: 17367 %
Return on Max Strategy Drawdown: 195
Max Strategy Drawdown (%): -27

There was only one single month that was negative.

So to me after testing it on such a long period of 8 eight years, where we had different markets like 2002 crash, ranging, 2008 crash etc., it performed very well, that should indicate clearly that its a robust strategy.

What do you think about the timeframe of a backtest? The longer it is the more sure you can be about the performance in the future?

Would you confirm the assumption, "if you have a strategy that performs in a variety of different market circumstances and over period of several years there is a high probability that this performance will remain in the future" ?

Thanks!

I am no expert but here is my 2 cents.
To answer your question realty would need to some more about your strategy,first what time resolution does it use, if it is less than 15min I would suggest your results are reflecting flaws in the backtesting system, similarly if the strategy uses small target profit or stop losses I would suggest the results are due to system limitations, I have myself written strategies that match or better your result but which in a live environment are incapable of ever making a profit not even for a single day.

Personally what I would do to verify the accuracy of such a result is this.

1) retest with the bar magnifier on and set to 1 minute ticks, does that still show the same level of profitability.
2) retest the last two weeks of data with bar magnifier on using tick by tick resolution, does the result match that shown in your original back test for the same period.
3) examine the original back test, look at the points where it takes profit and reopens a position, see if these make sense and that the system is not in fact opening positions at prices that would be impossible to have achieve in the real world.
4) ensure that one way or another you have allowed for the spread, either you account for it by adding a commission or by using extended testing with bid and ask data.

If you do that and you still think you have a great strategy forward test it for one week. What you can do is open ST load your strategy into a chart and then leave it open for one week, even though it is not running in auto mode it is taking the tick data and it is getting closer to a real worl situation.

If after that week it showed the same level of performance that you had in your back test then open a demo account and run it in auto mode for one week on that demo account.

If after that week it is still turning in results that match your original testing then take it live with a single lot and congratulate yourself.

Then if it runs for a month profitably reflecting your tested results you could think about using it actively.

Rob

Originally Posted by RobE

I am no expert but here is my 2 cents.
To answer your question realty would need to some more about your strategy,first what time resolution does it use, if it is less than 15min I would suggest your results are reflecting flaws in the backtesting system, similarly if the strategy uses small target profit or stop losses I would suggest the results are due to system limitations, I have myself written strategies that match or better your result but which in a live environment are incapable of ever making a profit not even for a single day.

Personally what I would do to verify the accuracy of such a result is this.

1) retest with the bar magnifier on and set to 1 minute ticks, does that still show the same level of profitability.
2) retest the last two weeks of data with bar magnifier on using tick by tick resolution, does the result match that shown in your original back test for the same period.
3) examine the original back test, look at the points where it takes profit and reopens a position, see if these make sense and that the system is not in fact opening positions at prices that would be impossible to have achieve in the real world.
4) ensure that one way or another you have allowed for the spread, either you account for it by adding a commission or by using extended testing with bid and ask data.

If you do that and you still think you have a great strategy forward test it for one week. What you can do is open ST load your strategy into a chart and then leave it open for one week, even though it is not running in auto mode it is taking the tick data and it is getting closer to a real worl situation.

If after that week it showed the same level of performance that you had in your back test then open a demo account and run it in auto mode for one week on that demo account.

If after that week it is still turning in results that match your original testing then take it live with a single lot and congratulate yourself.

Then if it runs for a month profitably reflecting your tested results you could think about using it actively.

Rob

Rob, I think you are touching on something here that is important. Likewise, I have spent many of hours creating and backtesting strategies only to find something wrong with one of my assumptions or a change in the market conditions.

I consider backtesting like watching a professional baseball game. Ideally, I would like to sit right behind home plate and get a great view of the field. So backtesting simply gets me to the right ball park...a good back test doesn't mean I have great seats...

I do believe that if you find a nice backtest, that it is wise to forward test it live in the smallest trade size. That way, if the strategy passes that test, you are getting closer to the seats behind home plate. If you want to forward test in a demo, you can do that as an intermediary step. Eventually, you should forward test in the smallest trade size so you have a more accurate picture of spread cost and slippage.

To me, I will learn more out of the forward test as that is the live environment and I'll know if it is a good strategy then.

That is a great question and I'm glad you brought it up!

OK. Thanks to both you!

Its using 15M and 60M charts, i chose them because i know form manual trading that those lower timeframe are noisy most often time.
The stop target is 10 and profit target 60. Those values came from genetic optimization.

A forward test running right now in my demo account..i am excited how it performs during the non-farm-payrolls today.

I havent used this bar maginifier so far. So this will be next thing to test. But i didnt quite unterstand why this is useful to me as i want to trade higher timeframes anway??

Originally Posted by plainv

OK. Thanks to both you!

Its using 15M and 60M charts, i chose them because i know form manual trading that those lower timeframe are noisy most often time.
The stop target is 10 and profit target 60. Those values came from genetic optimization.

A forward test running right now in my demo account..i am excited how it performs during the non-farm-payrolls today.

I havent used this bar maginifier so far. So this will be next thing to test. But i didnt quite unterstand why this is useful to me as i want to trade higher timeframes anway??

It's possible that you could have a 15-minute or hourly candle with a range outside of your 10-pip stop and or 60-pip target. Just looking at the 15-minute or hourly chart will not tell you which was hit first, so there is no way to tell if the trade was a winner or a loser. The Bar Maginifier should do that as it will show the trading within the candle.

Thanks again!

I have added the bar magnifier without a change in performance. BUT what did change everything was adding bid AND ask! Just like RobE suggested in point 4)

It converts a performance of +17367% into -86%!

Thats a huge difference to be honest!
I guess this shows that my strategy isnt what i thought it was!

So far the forward testing on my demo account shows good performance, but now i think its a matter of time until a drawdown starts.

Originally Posted by plainv

Thanks again!

I have added the bar magnifier without a change in performance. BUT what did change everything was adding bid AND ask! Just like RobE suggested in point 4)

It converts a performance of +17367% into -86%!

Thats a huge difference to be honest!
I guess this shows that my strategy isnt what i thought it was!

So far the forward testing on my demo account shows good performance, but now i think its a matter of time until a drawdown starts.

Yes, always remember to consider the bid/ask spread when back testing strategies. It is a common mistake that you will make less and less moving forward.