Retester is the powerful backtesting engine of StrategyQuant. Its main purpose is to find robust strategies that are not over-fitted to the history data that was used for their development.
Over-fitting, or curve-fitting, is one of the biggest problems, when generating a strategy. It means to have strategies too adjusted to the history data on which they were developed. Although they will show excellent results in the building process, they most certainly will fail when tested on other data (e.g. other period).
That is why robustness tests are a very important concept in machine-learning based strategy development.
StrategyQuant has many cross-checks that aim to provide the tools and the evaluation methods necessary for filtering out these over-fitted candidates.
The simpler cross checks use different market speed or market structure to test the strategy. Running a robust candidate on other timeframe (this is the market speed test) should give similar results to those from the original timeframe. The strategy should not brake or deteriorate drastically. Testing the strategy on a different symbol chart is not that straightforward as every currency pair has its own specifics. Still a robust strategy would hold at least part of its performance and not fail completely.
The What-If simulations answer the question “What would happen if some trades were missed?”. This can easily show the stability of the strategy. Its results should not depend heavily on any particular trade (or trades), because those might have been sporadic. The What-If tests can simulate scenarios like trading in certain days or time periods only, excluding top/bottom trades, making a limited number of trades per day, taking every second trade only into account, etc. A well-performing strategy should hold and maintain its good performance even if some of its best hits are removed.
Monte Carlo tests run many simulations, each with a small variation of the parameters, to give a complete understanding of the change-result interrelation. There are two groups of Monte Carlo tests developed in StrategyQuant. The Monte Carlo trades manipulations performs test using the existing trades and does not require backtesting. This makes it a fast cross check. It randomly shuffles the trades to give a different perspective of its quality. The Net profit of the strategy cannot change but you can examine different variations of max Drawdown, for example, to see how that would influence the statistics. The other test is to randomly skip trades with a defined probability. This too can show the robustness of the strategy. Like in the What-If analysis, the strategy should not be too dependent on any particular trades.
The Monte Carlo retest methods require retesting the strategies, thus are time-consuming. A wide variety of testing methods can be performed. You can choose to randomize the history data with a given probability, which is a very good way of testing the curve-fitting of the candidates. You can randomize the starting point of the strategy or the parameters.
The filtering settings are also an important part of the process. They define the success criteria for the cross-checked strategies.
Retester can act as a kind of filtering funnel to save time and select only the viable candidates.
It can rule out failed strategies along the testing sequence. This means that if a strategy doesn’t pass a check, it is dismissed and not tested further. Of course, it is all a matter of preferences, and you can always choose to test all strategies using all the checks. It is essential to configure the filters in such an optimal way, as to insure that you will find the best candidates.