Checking Strategy Performance
When you use a trading strategy, the short term success of that strategy can depend on certain factors like stop losses, take profit, position sizing and so on. A poor strategy can sometimes be masked by creative trade management. Likewise, a good strategy can execute poorly if trade management isn’t done properly.
An objective way to measure the performance of a strategy is to measure the reliability of the signals it produces and strip out all other factors. This will tell you the rate of true positives, and false positives. This is known as hypothesis testing. True positives are when a buy signal is generated and the market did go up or a sell is generated and the market did go down over a predetermined time.
The goal then is to maximize the rate of true positives, and minimize false positives and false negatives. The false cases are where a strategy generates the wrong signal, or where the market moves and the strategy fails to create any signal at all.
After you’ve created a strategy, go to the Indicators page. Open the search box, by clicking the spyglass icon. Then click the uncheck box to deselect other indicator types. Click the box that is labeled Custom Strategy.
The list should show all of the strategies that you’ve created and which are valid to run. To be valid, the strategy must have been trained. Select from the list the strategy that you want to measure, then click the icon.
On clicking the icon, you should see a chart visualization of the strategy performance. This marks the buy signals and the sell signals.
The metrics for the buy side signals are:
- True positives: The market went up and the strategy correctly signaled buy
- False positives (hold): The market was flat and the strategy falsely signaled buy
- False positives (sell): The market went down and the strategy falsely signaled buy
- Average profit: The average profit over all buy signals
- Total: The total number of buy signals measured
The metrics for the sell side signals are:
- True positives: The market went down and the strategy correctly signaled sell
- False positives (hold): The market was flat and the strategy falsely signaled sell
- False positives (sell): The market went up and the strategy falsely signaled sell
- Average profit: The average profit over all sell signals
- Total: The total number of sell signals measured
In the screen above, the true positive rate for buy signals was 78.9%. This means, of 33 signals that the strategy generated, 26 of them correctly predicted a market rise. The average market move, for all 33 cases was 7.8%. It’s important to check the average profit because for some strategies, the market move for false positives may be much bigger than that for true positives. This means losing trades potentially could lose more on average than winning trades gain. This would make for a losing trade system.
You can change the settings for the performance test. To do this, click the expand icon at the top right of the chart. The two settings below determine how the measurements work.
- Target trade time
- Target trade profit
The target trade time sets the time in chart bars, over which the check is made. For example, on a daily chart, when this is set at 50, the measurement is done over a 50-day period. The target trade profit, sets the definition for a flat, up or down market. For example, if target profit is 1%, then buys must predict a market move of at least +1% over 50-days to be considered successful. Sells must predict a market move of at least -1% to be classed as successful (true positives). Buys and sells are classed as false if the market stays with a range of plus or minus 1% over 50 days.
Unlike most other testing, hypothesis testing also accounts for draw down. For example, if a buy signal is made and the market moves up by the end of the period by at least +1%, but there was a fall in between that was greater than -1%, this would be classed as a false positive. This is because the strategy should have signalled a sell rather than a buy. Take profit and stop loss settings might have made the trade profitable, but the downside risk was greater than the upside reward.
Use the other settings above to test how the strategy performs for other markets and at other time frames. For example, the above chart shows how the same strategy performs on the Google weekly chart. The chart below shows the same result for a daily chart.
If a strategy doesn’t produce reliable signals, it’s unlikely to do well over the longer term. Classifying performance in this way should therefore be done as a baseline. It’s independent from other forms of testing like back testing and forward testing.