[PYTHON] Get the complete bitflyer tick data

Introduction

I logged in to bitflyer lightning for the first time in a while. Cryptocurrencies have been held since "2017" when they showed a great rise, but they are not trading. Seeing the rapid changes in the final trading value of lightnig, I felt "potential" and wanted to trade for the first time in a long time. スクリーンショット 2020-01-11 8.49.19.png

Resume

I used to do discretionary trading, but I don't think I'll do it again. This is because they do not have the mentality to withstand the mental load. Therefore, it is necessary to build an automatic trading system. Therefore, the first thing that is needed is data that serves as a criterion for trade logic. API To put it simply, an API is an interface that allows you to access specific data (such as the latest BTC value) by hitting a URL and execute specific instructions (such as BTC buy orders). Most crypto exchanges have this API. Of course, it is also available in bitflyer lightning. https://lightning.bitflyer.com/docs?lang=ja

According to the documentation, to get the latest value of "FX_BTC_JPY", do the following: https://api.bitflyer.com//v1/getticker?product_code=FX_BTC_JPY

Execution limit wall

I immediately ran the following from my mac.

while true; do curl https://api.bitflyer.com//v1/getticker?product_code=FX_BTC_JPY | jq . ; done

The value is returned in JSON format as shown below.

{
  "product_code": "FX_BTC_JPY",
  "timestamp": "2020-01-09T11:40:18.027",
  "tick_id": 340221409,
  "best_bid": 873088,
  "best_ask": 873160,
  "best_bid_size": 0.01325506,
  "best_ask_size": 0.026,
  "total_bid_depth": 7516.42989333,
  "total_ask_depth": 6798.27220924,
  "ltp": 873160,
  "volume": 196692.50042163,
  "volume_by_product": 196682.81646233
}

But after a while an error

{
  "status": -1,
  "error_message": "Over API limit per period, per IP address",
  "data": null
}

→ When reading the document, API calls from the same IP are limited to xxx per minute. This is for this reason. No error occurred at 1 / s.

How to overcome the barrier of execution restrictions (delusion)

While changing the public IP using AWS, execute the API from multiple Lambda, combine the results with SQS etc. and store it in DynamoDB. Sort by time when storing. It takes time and cost to build. → You can see that it is completely misguided as shown below.

How to overcome the barrier of execution restrictions (adoption plan)

I had a hard time trying to get continuous tick data, but when I look at the document, there is an API to get the execution history. The contract history is a set that includes all tick data. Use this to solve it !!! I tried to run it immediately.

curl https://api.bitflyer.com/v1/getexecutionsproduct_code=FX_BTC_JPY&count=10

result

[{"id":1510454721,"side":"BUY","price":904727.0,"size":0.02,"exec_date":"2020-01-11T00:29:56.813","buy_child_order_acceptance_id":"JRF20200111-002956-217585","sell_child_order_acceptance_id":"JRF20200111-002955-073037"},{"id":1510454720,"side":"BUY","price":904727.0,"size":0.01,"exec_date":"2020-01-11T00:29:56.797","buy_child_order_acceptance_id":"JRF20200111-002956-109098","sell_child_order_acceptance_id":"JRF20200111-002955-073037"},{"id":1510454719,"side":"BUY","price":904727.0,"size":0.01,"exec_date":"2020-01-11T00:29:56.797","buy_child_order_acceptance_id":"JRF20200111-002956-995485","sell_child_order_acceptance_id":"JRF20200111-002955-073037"},{"id":1510454718,"side":"BUY","price":904708.0,"size":0.62,"exec_date":"2020-01-11T00:29:56.797","buy_child_order_acceptance_id":"JRF20200111-002956-109097","sell_child_order_acceptance_id":"JRF20200111-002954-142777"},{"id":1510454717,"side":"BUY","price":904707.0,"size":0.383,"exec_date":"2020-01-11T00:29:56.797","buy_child_order_acceptance_id":"JRF20200111-002956-109097","sell_child_order_acceptance_id":"JRF20200111-002955-217572"},{"id":1510454716,"side":"BUY","price":904686.0,"size":0.091,"exec_date":"2020-01-11T00:29:56.797","buy_child_order_acceptance_id":"JRF20200111-002956-109097","sell_child_order_acceptance_id":"JRF20200111-002956-073060"},{"id":1510454715,"side":"BUY","price":904686.0,"size":0.01,"exec_date":"2020-01-11T00:29:56.783","buy_child_order_acceptance_id":"JRF20200111-002956-458844","sell_child_order_acceptance_id":"JRF20200111-002956-073060"},{"id":1510454714,"side":"BUY","price":904686.0,"size":0.01,"exec_date":"2020-01-11T00:29:56.607","buy_child_order_acceptance_id":"JRF20200111-002956-152005","sell_child_order_acceptance_id":"JRF20200111-002956-073060"},{"id":1510454713,"side":"BUY","price":904707.0,"size":0.01,"exec_date":"2020-01-11T00:29:56.577","buy_child_order_acceptance_id":"JRF20200111-002956-458842","sell_child_order_acceptance_id":"JRF20200111-002955-217572"},{"id":1510454712,"side":"BUY","price":904707.0,"size":0.04,"exec_date":"2020-01-11T00:29:56.547","buy_child_order_acceptance_id":"JRF20200111-002956-217578","sell_child_order_acceptance_id":"JRF20200111-002955-217572"}]

→ You can specify the number of cases with count. This time, the number was 10. It seems that there are up to 500 cases.

Visualization with python

plottest.ipynb



import requests
import json 
import matplotlib.pyplot as plt
%matplotlib inline

response = requests.get("https://api.bitflyer.com/v1/getexecutions?product_code=FX_BTC_JPY&count=100")
res_json = json.loads(response.text)

res_price_list = []
for res in res_json :
    res_price_list.append(res['price'])
    
plt.plot(res_price_list)

→ Use request instead of curl. → Extract the response of request with .text only text and convert it to list type with json.load. → The number of contracts to be acquired is 100.

result

output_7_1.png Please try it in your environment (jupyter notebook). The graph you see should change completely each time you run it. This is due to the active trading on bitflyer lightning. Many graphs generated multiple times have something in common. It is that a relatively remarkable trend does not occur in the past 100 contract histories, but rather it behaves like a repetitive movement. I wonder if this can be incorporated into trade logic.

Recommended Posts

Get the complete bitflyer tick data
Get the column list & data list of CASTable
Get the GNOME version
Get comments and subscribers with the YouTube Data API
Get Gzip-compressed data on-memory
Get the MIME Type
[Introduction to Python] How to get data with the listdir function
Get structural data from CHEMBLID
Python --bitflyer Get asset balance
Get Youtube data with python
Gzip the data by streaming
Get the number of digits
[Python] Get the previous month
Maya | Get the workspace path
Python --bitflyer Get chat content