The first time I tried to backtest a pairs trading strategy, I used MATLAB.
I was excited when I ran the strategy and saw great results.
When I traded the strategy live, I lost money immediately.
So I ran backtest simulations to optimize the z-score and ADF test statistic.
I ran it again and had amazing results again!
And I lost money.
Again.
And again.
And again.
It turned out that I was making a big mistake.
One that costs countless traders countless money.
I was tweaking my parameters based on random fluctuations in the data, instead of real market inefficiencies.
I learned my lesson the only way traders do:
By losing money.
In 2023, there are dozens of backtesting frameworks.
But very few efficiently optimize parameters.
And fewer still let you efficiently optimize parameters the right way:
Using walk-forward optimization.
vectorbt is a new Python library that addresses that.
Here’s how.
1,000,000 backtest simulations in 20 seconds with vectorbt
vectorbt is a package that combines backtesting and data science.
It takes a “vectorized” approach to backtesting using pandas and NumPy.
This means that instead of looping through every day of data, it operates on all the data at once.
This allows for testing millions of strategies in a few seconds.
This is important when optimizing parameters, entry and exit types, and even the statistical significance of the strategy as a whole.
To avoid overfitting, pros use a technique called walk-forward optimization.
Walk-forward optimization is a technique used to find the best settings for a trading strategy.
Today, you’ll build a simple moving average crossover strategy. Then you’ll use walk-forward optimization to find the moving average windows that result in the best Sharpe ratio. Finally, you’ll test whether the out-of-sample results are statistically greater than the in-sample results.
Imports and set up
Start by importing NumPy and vectorbt. You’ll use SciPy to test the statistical significance of the results.
import numpy as np import scipy.stats as stats import vectorbt as vbt
Then create an array of moving average windows to test and download price data.
windows = np.arange(10, 50)
price = vbt.YFData.download('AAPL').get('Close')
Build the functions
Create the data splits for the walk-forward optimization.
(in_price, in_indexes), (out_price, out_indexes) = price.vbt.rolling_split(
n=30,
window_len=365 * 2,
set_lens=(180,),
left_to_right=False,
)
This code segments the prices into 30 splits, each two years long, and reserves 180 days for the test.
Now create the functions that run the backtest.
def simulate_all_params(price, windows, **kwargs):
fast_ma, slow_ma = vbt.MA.run_combs(
price, windows, r=2, short_names=["fast", "slow"]
)
entries = fast_ma.ma_crossed_above(slow_ma)
exits = fast_ma.ma_crossed_below(slow_ma)
pf = vbt.Portfolio.from_signals(price, entries, exits, **kwargs)
return pf.sharpe_ratio()
This function builds two moving averages for each window you pass in.
Then it creates DataFrames showing where the fast-moving average crosses above the slow-moving average. These are the trade entries. It does the opposite for the trade exits.
After the backtest is run, the function returns the Sharpe ratio.
Next, you need to figure out the combination of windows that maximizes the Sharpe ratio.
def get_best_index(performance, higher_better=True):
if higher_better:
return performance[performance.groupby('split_idx').idxmax()].index
return performance[performance.groupby('split_idx').idxmin()].index
def get_best_params(best_index, level_name):
return best_index.get_level_values(level_name).to_numpy()
The first function returns the indexes in the DataFrame for the windows in each data split that maximizes the Sharpe ratio. The second function returns the window values.
Finally, create a function that runs the backtest with the windows that maximize the Sharpe ratio.
def simulate_best_params(price, best_fast_windows, best_slow_windows, **kwargs):
fast_ma = vbt.MA.run(price, window=best_fast_windows, per_column=True)
slow_ma = vbt.MA.run(price, window=best_slow_windows, per_column=True)
entries = fast_ma.ma_crossed_above(slow_ma)
exits = fast_ma.ma_crossed_below(slow_ma)
pf = vbt.Portfolio.from_signals(price, entries, exits, **kwargs)
return pf.sharpe_ratio()
This function creates the moving average values that maximize the Sharpe ratio, runs the backtest, and returns the Sharpe ratio.
Run the analysis
Start by optimizing the moving average windows on the in-sample data.
in_sharpe = simulate_all_params(
in_price,
windows,
direction="both",
freq="d"
)
The result is a DataFrame that has the Sharpe ratio for the best combination of windows for each split.
Now you can get the optimized windows and test them with out-of-sample data.
in_best_index = get_best_index(in_sharpe)
in_best_fast_windows = get_best_params(
in_best_index,
'fast_window'
)
in_best_slow_windows = get_best_params(
in_best_index,
'slow_window'
)
in_best_window_pairs = np.array(
list(
zip(
in_best_fast_windows,
in_best_slow_windows
)
)
)
Running this code gives you the parameter values for the fast-moving average and slow-moving average you can test with the out-of-sample data.
out_test_sharpe = simulate_best_params(
out_price,
in_best_fast_windows,
in_best_slow_windows,
direction="both",
freq="d"
)
The result is a DataFrame that has the Sharpe ratio for the backtest using out-of-sample test data and the window values that optimize the Sharpe ratio from the in-sample data.
Compare the results
The whole point of this analysis is to understand if the parameters you fit on the in-sample data can be used in real life to make money.
The most common issue in backtesting is overfitting to random data. (Especially when using technical analysis.)
You can run a simple t-test to understand if the out-of-sample Sharpe ratio is statistically greater than the in-sample Sharpe ratio. If it were, it would give you some measure of confidence that you did not overfit to random data.
in_sample_best = in_sharpe[in_best_index].values
out_sample_test = out_test_sharpe.values
t, p = stats.ttest_ind(
a=out_sample_test,
b=in_sample_best,
alternative="greater"
)
In this case, the p-value is close to 1 which means you cannot reject the null hypothesis that the out-of-sample Sharpe ratios are greater than the in-sample Sharpe ratios.
In other words, you are overfitted to noise.
The moving crossover is a toy example that is known not to make money. But the technique of optimizing parameters using walk-forward optimization is the state-of-the-art way of removing as much bias as possible.
And vector bt is the state-of-the-art backtesting library that makes it possible.