[PYTHON] Get the complete bitflyer tick data

Introduction

I logged in to bitflyer lightning for the first time in a while. Cryptocurrencies have been held since "2017" when they showed a great rise, but they are not trading. Seeing the rapid changes in the final trading value of lightnig, I felt "potential" and wanted to trade for the first time in a long time.

Resume

I used to do discretionary trading, but I don't think I'll do it again. This is because they do not have the mentality to withstand the mental load. Therefore, it is necessary to build an automatic trading system. Therefore, the first thing that is needed is data that serves as a criterion for trade logic. API To put it simply, an API is an interface that allows you to access specific data (such as the latest BTC value) by hitting a URL and execute specific instructions (such as BTC buy orders). Most crypto exchanges have this API. Of course, it is also available in bitflyer lightning. https://lightning.bitflyer.com/docs?lang=ja

According to the documentation, to get the latest value of "FX_BTC_JPY", do the following: https://api.bitflyer.com//v1/getticker?product_code=FX_BTC_JPY

• You can get the latest "FX_BTC_JPY" value by clicking the above.

Execution limit wall

I immediately ran the following from my mac.

while true; do curl https://api.bitflyer.com//v1/getticker?product_code=FX_BTC_JPY | jq . ; done

The value is returned in JSON format as shown below.

{
  "product_code": "FX_BTC_JPY",
  "timestamp": "2020-01-09T11:40:18.027",
  "tick_id": 340221409,
  "best_bid": 873088,
  "best_ask": 873160,
  "best_bid_size": 0.01325506,
  "best_ask_size": 0.026,
  "total_bid_depth": 7516.42989333,
  "total_ask_depth": 6798.27220924,
  "ltp": 873160,
  "volume": 196692.50042163,
  "volume_by_product": 196682.81646233
}

But after a while an error

{
  "status": -1,
  "error_message": "Over API limit per period, per IP address",
  "data": null
}

→ When reading the document, API calls from the same IP are limited to xxx per minute. This is for this reason. No error occurred at 1 / s.

How to overcome the barrier of execution restrictions (delusion)

While changing the public IP using AWS, execute the API from multiple Lambda, combine the results with SQS etc. and store it in DynamoDB. Sort by time when storing. It takes time and cost to build. → You can see that it is completely misguided as shown below.

How to overcome the barrier of execution restrictions (adoption plan)

I had a hard time trying to get continuous tick data, but when I look at the document, there is an API to get the execution history. The contract history is a set that includes all tick data. Use this to solve it !!! I tried to run it immediately.

curl https://api.bitflyer.com/v1/getexecutionsproduct_code=FX_BTC_JPY&count=10

result

[{"id":1510454721,"side":"BUY","price":904727.0,"size":0.02,"exec_date":"2020-01-11T00:29:56.813","buy_child_order_acceptance_id":"JRF20200111-002956-217585","sell_child_order_acceptance_id":"JRF20200111-002955-073037"},{"id":1510454720,"side":"BUY","price":904727.0,"size":0.01,"exec_date":"2020-01-11T00:29:56.797","buy_child_order_acceptance_id":"JRF20200111-002956-109098","sell_child_order_acceptance_id":"JRF20200111-002955-073037"},{"id":1510454719,"side":"BUY","price":904727.0,"size":0.01,"exec_date":"2020-01-11T00:29:56.797","buy_child_order_acceptance_id":"JRF20200111-002956-995485","sell_child_order_acceptance_id":"JRF20200111-002955-073037"},{"id":1510454718,"side":"BUY","price":904708.0,"size":0.62,"exec_date":"2020-01-11T00:29:56.797","buy_child_order_acceptance_id":"JRF20200111-002956-109097","sell_child_order_acceptance_id":"JRF20200111-002954-142777"},{"id":1510454717,"side":"BUY","price":904707.0,"size":0.383,"exec_date":"2020-01-11T00:29:56.797","buy_child_order_acceptance_id":"JRF20200111-002956-109097","sell_child_order_acceptance_id":"JRF20200111-002955-217572"},{"id":1510454716,"side":"BUY","price":904686.0,"size":0.091,"exec_date":"2020-01-11T00:29:56.797","buy_child_order_acceptance_id":"JRF20200111-002956-109097","sell_child_order_acceptance_id":"JRF20200111-002956-073060"},{"id":1510454715,"side":"BUY","price":904686.0,"size":0.01,"exec_date":"2020-01-11T00:29:56.783","buy_child_order_acceptance_id":"JRF20200111-002956-458844","sell_child_order_acceptance_id":"JRF20200111-002956-073060"},{"id":1510454714,"side":"BUY","price":904686.0,"size":0.01,"exec_date":"2020-01-11T00:29:56.607","buy_child_order_acceptance_id":"JRF20200111-002956-152005","sell_child_order_acceptance_id":"JRF20200111-002956-073060"},{"id":1510454713,"side":"BUY","price":904707.0,"size":0.01,"exec_date":"2020-01-11T00:29:56.577","buy_child_order_acceptance_id":"JRF20200111-002956-458842","sell_child_order_acceptance_id":"JRF20200111-002955-217572"},{"id":1510454712,"side":"BUY","price":904707.0,"size":0.04,"exec_date":"2020-01-11T00:29:56.547","buy_child_order_acceptance_id":"JRF20200111-002956-217578","sell_child_order_acceptance_id":"JRF20200111-002955-217572"}]

→ You can specify the number of cases with count. This time, the number was 10. It seems that there are up to 500 cases.

Visualization with python

plottest.ipynb

import requests
import json 
import matplotlib.pyplot as plt
%matplotlib inline

response = requests.get("https://api.bitflyer.com/v1/getexecutions?product_code=FX_BTC_JPY&count=100")
res_json = json.loads(response.text)

res_price_list = []
for res in res_json :
    res_price_list.append(res['price'])
    
plt.plot(res_price_list)

→ Use request instead of curl. → Extract the response of request with .text only text and convert it to list type with json.load. → The number of contracts to be acquired is 100.

result

Please try it in your environment (jupyter notebook). The graph you see should change completely each time you run it. This is due to the active trading on bitflyer lightning. Many graphs generated multiple times have something in common. It is that a relatively remarkable trend does not occur in the past 100 contract histories, but rather it behaves like a repetitive movement. I wonder if this can be incorporated into trade logic.