Introduction

There are many libraries for backtesting cistre with Python, but it is difficult for people who entered from MetaTrader to understand, so I wrote the code for backtesting after practicing Python.

Backtesting Systre with Python

However, in the first version, I wrote that it works first, so it was quite wasteful and the execution speed was slow, so I tried to improve it a little this time.

Acquisition of FX historical data

In the case of stock prices, there are many things that can be downloaded directly from Yahoo! etc., but in the case of FX, data of multiple time frames such as 5 minutes or 15 minutes may be used, so 1 minute as basic data. I want foot data.

In that case, the data is also large, so I think it is more convenient to read the data downloaded in advance. Here, download it as sample data from the following site.

HistData.com

There are several formats for the same data, but I will download the data in Generic ASCII format because I use the pandas function.

Now, let's read the downloaded EUR / USD 1-minute data for 2015'DAT_ASCII_EURUSD_M1_2015.csv' with the pandas function. However, if this is left as it is, the market price at the beginning of the week will start on Sunday, so delay it by 7 hours and start from 00:00 on Monday.

import numpy as np
import pandas as pd

dataM1 = pd.read_csv('DAT_ASCII_EURUSD_M1_2015.csv', sep=';',
                     names=('Time','Open','High','Low','Close', ''),
                     index_col='Time', parse_dates=True)
dataM1.index += pd.offsets.Hour(7) #7 hour offset

Creation of arbitrary time frame data

Next, create the data of any time frame from the data of 1 minute bar. Again, you can easily do this by using the pandas resample (), ʻohlc () `functions.

#A function that creates four-legged data in a timeframe specified by tf from df data
def TF_ohlc(df, tf):
    x = df.resample(tf).ohlc()
    O = x['Open']['open']
    H = x['High']['high']
    L = x['Low']['low']
    C = x['Close']['close']
    ret = pd.DataFrame({'Open': O, 'High': H, 'Low': L, 'Close': C},
                       columns=['Open','High','Low','Close'])
    return ret.dropna()

ohlc = TF_ohlc(dataM1, 'H') #Creation of 1-hour data

The keyword specified for tf is

Minutes-'T'
Time-'H'
Sun-'D'
Week-'W'
Month-'M'

So, for 15 minutes, just write '15T'. With this, you can create even minutes of data. Here, I tried to make 1-hour data.

Creation of technical indicators

The technical indicators used in the trading rules of Systre are created on GitHub. You can use it by fetching only indicators.py. I'll omit the source code, but I've avoided using pandas for internal iterations and used Numba, so I think there are no more time-consuming functions. ~~ However, parabolic SAR takes some time due to the algorithm. ~~

For example, a 10-bar moving average and a 30-bar moving average can be written as:

import indicators as ind #indicators.Import of py
FastMA = ind.iMA(ohlc, 10) #Short-term moving average
SlowMA = ind.iMA(ohlc, 30) #Long-term moving average

Display of technical indicators

You can use the pandas plot function to display the created technical index on the chart, but since you may want to scale it, you often see it on market-related sites HighCharts. Let's use the Python library displayed in /).

Here, I will use the easy-to-install pandas-highcharts.

Installation is

pip install pandas-highcharts

Is OK. The code to display the closing price of FX and the two moving averages is as follows.

from pandas_highcharts.display import display_charts
df = pd.DataFrame({'Close': ohlc['Close'], 'FastMA': FastMA, 'SlowMA': SlowMA})
display_charts(df, chart_type="stock", title="MA cross", figsize=(640,480), grid=True)

When you do this, you will see a chart like this.

If you display one year's worth, the moving averages will hardly overlap, so if you zoom in properly, it will look like this, and you will be able to see the difference in the moving averages.

It's quite convenient.

Trading rules for moving average crossing system

As an example of a trading system, take the classic moving average crossing system. The trading rules are

Buy signal-Short-term moving average breaks above long-term moving average
Sell signal-Short-term moving average breaks below long-term moving average

It's as simple as that. This signal is an entry signal, and the exit signal for closing a position is that the buy exit is the same as the sell entry and the sell exit is the same as the buy entry. This is the so-called transfer trading.

#Buy entry signal
BuyEntry = ((FastMA > SlowMA) & (FastMA.shift() <= SlowMA.shift())).values
#Sell entry signal
SellEntry = ((FastMA < SlowMA) & (FastMA.shift() >= SlowMA.shift())).values
#Buy exit signal
BuyExit = SellEntry.copy()
#Sell exit signal
SellExit = BuyEntry.copy()

Here, each signal is a bool type array of numpy. If you think about it normally, you would probably judge the signal while turning the time series data, but in the case of Python, it is faster to process it collectively in an array, so put the signal in an array.

Run backtest

We will finally backtest. Input historical data and the above trading signal, and output the actual trading price and profit / loss as time series data.

def Backtest(ohlc, BuyEntry, SellEntry, BuyExit, SellExit, lots=0.1, spread=2):
    Open = ohlc['Open'].values #Open price
    Point = 0.0001 #Value of 1pip
    if(Open[0] > 50): Point = 0.01 #1 pip value of cross circle
    Spread = spread*Point #Spread
    Lots = lots*100000 #Actual trading volume
    N = len(ohlc) #FX data size
    BuyExit[N-2] = SellExit[N-2] = True #Finally forced exit
    BuyPrice = SellPrice = 0.0 #Selling price
    
    LongTrade = np.zeros(N) #Buy trade information
    ShortTrade = np.zeros(N) #Sell trade information
    
    LongPL = np.zeros(N) #Profit and Loss of Buy Position
    ShortPL = np.zeros(N) #Sell position profit or loss

    for i in range(1,N):
        if BuyEntry[i-1] and BuyPrice == 0: #Buy entry signal
            BuyPrice = Open[i]+Spread
            LongTrade[i] = BuyPrice #Buy position open
        elif BuyExit[i-1] and BuyPrice != 0: #Buy exit signal
            ClosePrice = Open[i]
            LongTrade[i] = -ClosePrice #Buy position closed
            LongPL[i] = (ClosePrice-BuyPrice)*Lots #Profit and loss settlement
            BuyPrice = 0

        if SellEntry[i-1] and SellPrice == 0: #Sell entry signal
            SellPrice = Open[i]
            ShortTrade[i] = SellPrice #Sell position open
        elif SellExit[i-1] and SellPrice != 0: #Sell exit signal
            ClosePrice = Open[i]+Spread
            ShortTrade[i] = -ClosePrice #Sell position closed
            ShortPL[i] = (SellPrice-ClosePrice)*Lots #Profit and loss settlement
            SellPrice = 0

    return pd.DataFrame({'Long':LongTrade, 'Short':ShortTrade}, index=ohlc.index),\
            pd.DataFrame({'Long':LongPL, 'Short':ShortPL}, index=ohlc.index)
        
Trade, PL = Backtest(ohlc, BuyEntry, SellEntry, BuyExit, SellExit)

In the previous version, this part was divided into several stages, but this time I tried to summarize it. Originally, it would be good if we could trade only with signals, but there are cases where signals are output even if there are positions, so we also checked the existence of positions.

In the output of this function, Trade stores the buy / sell price as a positive value and the settlement price as a negative value for each buy and sell. The realized profit / loss at the time of settlement is stored in PL.

Evaluation of trading system

With the information on Trade and PL, you can roughly evaluate the trading system. For example, it looks like this.

def BacktestReport(Trade, PL):
    LongPL = PL['Long']
    LongTrades = np.count_nonzero(Trade['Long'])//2
    LongWinTrades = np.count_nonzero(LongPL.clip_lower(0))
    LongLoseTrades = np.count_nonzero(LongPL.clip_upper(0))
    print('Number of buy trades=', LongTrades)
    print('Number of winning trades=', LongWinTrades)
    print('Maximum win trade=', LongPL.max())
    print('Average win trade=', round(LongPL.clip_lower(0).sum()/LongWinTrades, 2))
    print('Number of negative trades=', LongLoseTrades)
    print('Maximum negative trade=', LongPL.min())
    print('Average negative trade=', round(LongPL.clip_upper(0).sum()/LongLoseTrades, 2))
    print('Win rate=', round(LongWinTrades/LongTrades*100, 2), '%\n')

    ShortPL = PL['Short']
    ShortTrades = np.count_nonzero(Trade['Short'])//2
    ShortWinTrades = np.count_nonzero(ShortPL.clip_lower(0))
    ShortLoseTrades = np.count_nonzero(ShortPL.clip_upper(0))
    print('Number of sell trades=', ShortTrades)
    print('Number of winning trades=', ShortWinTrades)
    print('Maximum win trade=', ShortPL.max())
    print('Average win trade=', round(ShortPL.clip_lower(0).sum()/ShortWinTrades, 2))
    print('Number of negative trades=', ShortLoseTrades)
    print('Maximum negative trade=', ShortPL.min())
    print('Average negative trade=', round(ShortPL.clip_upper(0).sum()/ShortLoseTrades, 2))
    print('Win rate=', round(ShortWinTrades/ShortTrades*100, 2), '%\n')

    Trades = LongTrades + ShortTrades
    WinTrades = LongWinTrades+ShortWinTrades
    LoseTrades = LongLoseTrades+ShortLoseTrades
    print('Total number of trades=', Trades)
    print('Number of winning trades=', WinTrades)
    print('Maximum win trade=', max(LongPL.max(), ShortPL.max()))
    print('Average win trade=', round((LongPL.clip_lower(0).sum()+ShortPL.clip_lower(0).sum())/WinTrades, 2))
    print('Number of negative trades=', LoseTrades)
    print('Maximum negative trade=', min(LongPL.min(), ShortPL.min()))
    print('Average negative trade=', round((LongPL.clip_upper(0).sum()+ShortPL.clip_upper(0).sum())/LoseTrades, 2))
    print('Win rate=', round(WinTrades/Trades*100, 2), '%\n')

    GrossProfit = LongPL.clip_lower(0).sum()+ShortPL.clip_lower(0).sum()
    GrossLoss = LongPL.clip_upper(0).sum()+ShortPL.clip_upper(0).sum()
    Profit = GrossProfit+GrossLoss
    Equity = (LongPL+ShortPL).cumsum()
    MDD = (Equity.cummax()-Equity).max()
    print('Gross profit=', round(GrossProfit, 2))
    print('Total loss=', round(GrossLoss, 2))
    print('Total profit and loss=', round(Profit, 2))
    print('Profit factor=', round(-GrossProfit/GrossLoss, 2))
    print('Average profit and loss=', round(Profit/Trades, 2))
    print('Maximum drawdown=', round(MDD, 2))
    print('Recovery factor=', round(Profit/MDD, 2))
    return Equity
Equity = BacktestReport(Trade, PL)

Number of buy trades= 113
Number of winning trades= 39
Maximum win trade= 440.4
Average win trade= 82.57
Number of negative trades= 74
Maximum negative trade= -169.4
Average negative trade= -43.89
Win rate= 34.51 %

Number of sell trades= 113
Number of winning trades= 49
Maximum win trade= 327.6
Average win trade= 78.71
Number of negative trades= 64
Maximum negative trade= -238.5
Average negative trade= -43.85
Win rate= 43.36 %

Total number of trades= 226
Number of winning trades= 88
Maximum win trade= 440.4
Average win trade= 80.42
Number of negative trades= 138
Maximum negative trade= -238.5
Average negative trade= -43.87
Win rate= 38.94 %

Gross profit= 7077.0
Total loss= -6054.4
Total profit and loss= 1022.6
Profit factor= 1.17
Average profit and loss= 4.52
Maximum drawdown= 1125.4
Recovery factor= 0.91

Here, each item is displayed, but when performing machine learning or optimization, it is only necessary to calculate the evaluation value.

You will often want to look at the asset curve as a system valuation. In the above function, the asset curve is output as ʻEquity`, so if you add the initial assets and make a graph, it will be as follows.

Initial = 10000 #Initial assets
display_charts(pd.DataFrame({'Equity':Equity+Initial}), chart_type="stock", title="Asset curve", figsize=(640,480), grid=True)

Trade chart display

Finally, let's display where you bought and sold on the chart. You can display only the points you bought and sold, but I don't know how to do it in HighCharts, so I will connect the points where the position was opened and the points where it was closed with a line.

Let's convert the Trade information to a line with the following code.

def PositionLine(trade):
    PosPeriod = 0 #Position period
    Position = False #Presence or absence of position
    Line = trade.copy()
    for i in range(len(Line)):
        if trade[i] > 0: Position = True 
        elif Position: PosPeriod += 1 #Count the duration of a position
        if trade[i] < 0:
            if PosPeriod > 0:
                Line[i] = -trade[i]
                diff = (Line[i]-Line[i-PosPeriod])/PosPeriod
                for j in range(i-1, i-PosPeriod, -1):
                    Line[j] = Line[j+1]-diff #Interpolate the duration of a position
                PosPeriod = 0
                Position = False
        if trade[i] == 0 and not Position: Line[i] = 'NaN'
    return Line

df = pd.DataFrame({'Open': ohlc['Open'],
                   'Long': PositionLine(Trade['Long'].values),
                   'Short': PositionLine(Trade['Short'].values)})
display_charts(df, chart_type="stock", title="Trade chart", figsize=(640,480), grid=True)

It will be displayed in High Charts in the same way, so try zooming appropriately.

Since it is a trading system, you can see that the Long position and the Short position appear alternately.

Summary

I wrote the code in Python to backtest the system so that the trading rules can be described only by technical indicators. The library that can display graphs from Python to HighCharts was quite convenient. There is still room for improvement, so I would like to continue writing when I feel like it.

The continuation is here. Backtesting FX Systre with Python (2)

The code posted in this article has been uploaded below. MT5IndicatorsPy/EA_sample.ipynb

Backtesting FX Systre in Python (1)