Originally @: https://www.backtrader.com/blog/2019-08-22-practical-backtesting-replication/practical-replication/

There have been recently some threads on reddit/r/algotrading about the success in replicating published algotrading strategies. First

Because it has been deleted here a quick summary:

The strategies do not work

If the authors claim a given strategy stopped working due to alpha decay, the tests were run against past data and it still did not work

Bottomline: It’s all overfitting, p-hacking or a tiny alpha which doesn’t need to decay because commissions do already destroy the alpha.

A replica to the replication problem was written by Artem Kaznatcheev in:

Followed by:

The first two were theoretical (even if the first mentioned having implemented the 130 strategies), with “Overfitting” offering actual code.

With so much going on, what about trying to replicate something which was published but not as a paper, and like in the “overfitting” case, with a practical approach. Something published in a well known book.

The target: “Trying to Beat Random Entry”. This is a section in Part 3, Chapter 8 of this book:

A structured approach to engage into algotrading is presented in the book with a special emphasis on: position sizing and management of the position (i.e.: when to actually exit a trade). This being a lot more important than, for example, the entry setup, which apparently most people consider the main driver.

In Chapter 8, Van K. Tharp talks to Tom Basso and says: “From what you are saying it sounds like you could make money consistently with a random entry as long as you have good exits and size your positions intelligently.” to which the answer was that he probably could.

Rules:

Coin-Flip based entry

Always in the market — Long or Short

Reentry as soon as the exit signal was given

Volatility of the market is determined by a 10-day Exponential Moving Average of the Average True Range

of the Trailing stop at distance from the close 3-times the volatility

The stop could only move in the direction of the trade

Fixed Stake (1 contract) or 1% risk model (Chapter 12 in the book)

Results

Tested against 10 markets

Fixed Stake: made money 80% of the time

1% Risk Model: made money 100% of the time

Reliability level: 38% (percentage of winning trades)

The missing bits:

Tested markets

Tested periods

If always in the market means closing the trade “today” and re-entering “tomorrow” or if it means issuing both a the closing/re-opening orders at the same time.

This is actually the easiest to overcome.

For the last two items, the book says the conversation took place in 1991 and futures were used. To be fair to the book, futures data from before 1991 will be used. 1-day price bars are also assumed, given the mention of the 10-day Exponential Moving Average

The most obvious problem seems to get the algorithm right, but in this case the book does a good job at describing the simple algorithm and the results. To complete it, let’s summarize the “Percentage Risk Model” (aka “Model 3” in the book) from Chapter 12.

Maximal Loss : limited to x% of the account value (i.e.: the percentage risk)

: limited to of the account value (i.e.: the percentage risk) Risk per Contract : with the given algorithm, it will be the initial stop distance (3 times the volatility) multiplied by the multiplier of the future

: with the given algorithm, it will be the initial stop distance (3 times the volatility) multiplied by the multiplier of the future Amount of contracts: Maximal Loss / Risk Per Contract

The replication details

The data

The CL (Crude Oil) futures data from 1985 to 1990 (i.e.: 6 full years) will be used. The contract specification has a:

Tick size: 0.01 (i.e.: 100 ticks per point)

(i.e.: 100 ticks per point) Cost per tick: $10

With this in mind, we will use a multiplier of 1000 per 1 point (100 ticks/point x 10 dollars/tick = 1000 dollars)

Commission

A 2.00 monetary units per contract per trip will be used (IB-like)

Some implementation Details

The coin flip is modeled as an indicator, to make it easy to visualize where the flips have taken place (and if several entries, for example, go in the same direction, which is to be expected from randomness)

To also have a good visualization of the stop and how it moves, the stop price calculation and logic are also embedded in a indicator. Notice that the stop calculation logic has two different stages

When a trade starts the stop price has to be set at the given distance from the close with no relation to previous stop prices

When a trade is running, the stop price is adjusted if possible to follow the trend.

Charting

The code produces two types of charts

A chart containing the details a single test run ( --plot ) option. It makes most sense to use it when running a single iteration ( --iterations 1 ) A scatter chart showing the Profit and Loss of the runs.

Sample of #1

Sample of #2 for 10 runs

Sample invocations of the script

A single run with fixed size stake of 1 and plotting

./vanktharp-coinflip.py --years 1985-1990 --fixedsize --sizer stake=1 --iterations 1 --plot **** Iteration: 1

-- PNL: 10482.00

-- Trades 49 - Won 22 - %_Won: 0.45

**** Summary of Runs

-- Total : 1

-- Won : 1

-- % Won : 1.00

**** Summary of Trades

-- Total : 49

-- Total Won : 22

-- % Total Won : 0.45

100 runs with a 1% risk model, 10 iterations and a scatter chart

(The output is shortened for practical purposes)

$ ./vanktharp-coinflip.py --years 1985-1990 --percrisk --sizer percrisk=0.01 --iterations 100 --scatter

**** Iteration: 1

-- PNL: -18218.00

-- Trades 60 - Won 24 - %_Won: 0.40

**** Iteration: 2

...

...

**** Iteration: 100

-- PNL: 111366.00

-- Trades 50 - Won 26 - %_Won: 0.52

**** Summary of Runs

-- Total : 100

-- Won : 50

-- % Won : 0.50

**** Summary of Trades

-- Total : 5504

-- Total Won : 2284

-- % Total Won : 0.41

Test Runs Mix

10 test runs of 100 iterations were done, mixing the following variables:

Fixed size stake of 1 or percentage risk model with 1%

Executing entry/exit in the same bar or in consecutive bars

Summary of results

In average, 49% of the runs made a profit. Fixed size remained close to 50% in the test runs and the percentage risk model varied more with a test run scoring a minimum of 39% profitable runs and another scoring a maximum of 65% profitable runs (out of 10)

In average, 39% of the trades were profitable (small deviation)

Recalling what the book said:

80% of profitable runs when using a fixed size stake of 1

100% of profitable runs with the 1% percentage risk model

38% of profitable trades

It would therefore seem that:

Only the last of the items was replicated.

Conclusion

As pointed out by Artem Kaznatcheev, the replication crisis can be due to:

Using the wrong data set

Failing to correctly implement the algorithm

Or it could well be that the original implementation did not follow its own rules or that not all details were published.

Note

In any case I do still personally recommend reading the book. Failing to replicate that specific case does not mean the book is not a good read and shows a practical approach at algotrading.

The complete script