[PYTHON] Beginners will make a Bitcoin automatic trading bot aiming for a lot of money! Part 4 [Serverless]

This time previous I made "bitcoin automatic trading bot-like" We used Lambda on AWS to create a popular ** serverless configuration **! It's in the cloud, so it keeps running semi-permanently even if my PC stops. It's just a bot!

About cloud configuration I chose serverless because I heard that the usage fee is ** very cheap ** + ** easy to manage **. [Reference]

Looking at the service I actually used this time, ... With the following configuration ** 1 $ per month !! **

Estimate

Lambda · Lambda offers 1,000,000 free requests and 400,000 GB-seconds of compute time per month for free. [Reference]

▶ If you run it once a minute for a month, 1 x 60m x 24h x 31day = 44,640 times, so ** margin **

DynamoDB · The AWS Free Tier includes 25 GB of Amazon DynamoDB storage and processing up to 200 million requests. [Reference]

** 200 million ... (゜ Д ゜) ** Can it be exceeded by individuals? (It seems that it depends on the writing and reading capacity per hour, I don't care because I won't use it so hard this time. )

cloudwatch : $ 1.00 [[Reference]] for 1 million custom events created (https://aws.amazon.com/jp/cloudwatch/pricing/)

▶ If you run it once a minute for a month, 1 x 60m x 24h x 31day = 44,640 times, so $ 1.00?

▶ After moving it for about a week, there is no sign that it will increase at all ... If you stand for a month and it is 0 yen, it will be fixed to 0 yen.

specification

The movement is as usual ** Buy when it gets cheaper, and sell when it gets expensive! It's still a stupid bot like **. I want to fix it quickly, but I can't come up with a good algorithm. .. .. ..

① Get the previous transaction amount from the DB (this amount is the basis for judgment) ② Cancel if the previous transaction remains ② If you have Bitcoin and the price goes up, sell it! (And save the transaction amount in the DB) ③ If you have Japanese Yen and the price drops, buy it! (And save the transaction amount in the DB)

Implementation

Implement the three in AWS of the above architecture diagram. (It works just by setting these three!)

1. DynamoDB --Create table
2. Lambda --Coding --Deploy to code execution environment
3. CloudWach --Set the Lambda set above in the scheduler

１．DynamoDB

Since Lambda, the code execution environment used this time, cannot save values by sequential execution, Use the database.

Create the following two tables.

① last_transaction_price --Record the transaction price ② sequence-Record the number of successful transactions

** ① last_transaction_price --Record the transaction price **

①-1. Create the following table from [Create Table]

-Table name: last_transaction_pric ・ Primary key: trade_number (numeric type)

①-2. Record registration from [Create item](manually register only the first one)

• Enter the previous transaction amount in "last_transaction_price" (If you don't remember or have never traded, enter the current price)

** ② sequence-Record the number of successful transactions **

①-1. Create the following table from [Create Table]

・ Table name: sequence -Primary key: name (character string)

①-2. Record registration from [Create item](manually register only the first one)

２．Lambda

① Coding ② Role creation ③ Create Lambda ④ Deploy ⑤ Test

** ① Coding **

Constitution This time, it is increasing in various ways. []···folder

[zaif] 　|- main.py |-[Config]-* zaif_keys.json * -Stores zaif key and secret |-[Aws]-* dynamodb_utils.py -A class that makes it easy to use the DynamoDB API of aws. |-[Define]- common_define.py -(Mainly used in DynamoDB) Class that defines and summarizes character strings |-[Exception]- error.py * -Classes that summarize unique errors |-... -[Part 1](http://qiita.com/speedkingg/items/2d874beee26106ce9f9a#2api installation)

• When creating a Python file to be used elsewhere, it cannot be read unless "\ _ \ _ init__.py" is put in the same folder. (The contents of the file are empty and there is no problem) (The description is omitted in the above configuration) Example:

■main.py

main.py

# -*- coding: utf-8 -*-
import json
from zaifapi import ZaifPublicApi  #Class that executes API that does not require authentication information published by Zaif
from zaifapi import ZaifPrivateApi  #Class that executes API that requires authentication information published by Zaif
from pprint import pprint  #For display(It displays json nicely)
from aws.dynamodb_utils import DynamoDBUtils  #Classes that operate Dynamodb
from define.common_define import CommonDefine  #Class that defines constants such as table name
from exception.error import *  #A class that defines a proprietary error

dynamodb = DynamoDBUtils()

zaif_keys_json = open('config/zaif_keys.json', 'r')
zaif_keys = json.load(zaif_keys_json)

KEY = zaif_keys["key"]
SECRET = zaif_keys["secret"]

zaif_public = ZaifPublicApi()
zaif_private = ZaifPrivateApi(KEY, SECRET)


#Here is executed when called from Lamda
def lambda_handler(event, context):
    try:
        #Get current information
        info = _get_info()
        last_price = info["last_price"]
        funds_btc = info["funds_btc"]
        funds_jpy = info["funds_jpy"]
        open_orders = info["open_orders"]

        #Acquisition of the previous transaction amount
        trade_count = _get_trade_count()
        last_transaction_data = _get_last_transaction_data(trade_count)
        last_transaction_price = last_transaction_data[CommonDefine.LAST_TRANSACTION_PRICE]
        order_id = last_transaction_data[CommonDefine.ORDER_ID]

        print('▼[info] last_transaction_price: ' + str(last_transaction_price) + ', order_id: ' + str(order_id))

        #Cancel if the transaction is on hold
        if order_id > 0 and open_orders > 0:
            try:
                print('■ Canceled.')
                pprint(zaif_private.cancel_order(order_id=order_id))
            except Exception as e:
                print(e)
            else:
                #Delete the previous transaction amount
                _delete_last_transaction_price(trade_count)
        elif open_orders == 0:
            dynamodb.update_by_partition_key(CommonDefine.LAST_TRANSACTION_TABLE,
                                             CommonDefine.TRADE_NUMBER, trade_count,
                                             CommonDefine.ORDER_ID, 0)

        #When you have btc and the price goes up
        if funds_btc != 0 and last_price > last_transaction_price:
            _sell_btc(last_price, funds_btc)

        #When you have jpy and the price drops
        #(Minimum unit (0).0001btc minutes) when you have Japanese Yen or more)
        elif funds_jpy > last_transaction_price / 10000 and last_price < last_transaction_price:
            _buy_btc(last_price, funds_jpy)

    except ResourceNotFoundError:
        raise
    except IllegalArgumentError:
        raise
    except Exception:
        raise


#Acquisition of the previous transaction amount
def _get_last_transaction_data(trade_count):
    try:
        last_transaction_data = dynamodb.get_item_by_partition_key(CommonDefine.LAST_TRANSACTION_TABLE,
                                                                   CommonDefine.TRADE_NUMBER, trade_count)

        if "Item" not in last_transaction_data:
            raise ResourceNotFoundError("last_transaction_data is not found.")

        if CommonDefine.LAST_TRANSACTION_PRICE not in last_transaction_data["Item"]:
            raise ResourceNotFoundError(CommonDefine.LAST_TRANSACTION_PRICE + " is not found.")

        if CommonDefine.ORDER_ID not in last_transaction_data["Item"]:
            raise ResourceNotFoundError(CommonDefine.ORDER_ID + " is not found.")

    except ResourceNotFoundError:
        raise
    except IllegalArgumentError:
        raise
    except Exception:
        raise

    return last_transaction_data["Item"]


def _get_trade_count():
    trade_count_data = dynamodb.get_item_by_partition_key(CommonDefine.SEQUENCE,
                                                          CommonDefine.SEQUENCE_NAME, CommonDefine.TRADE_COUNT)
    if "Item" not in trade_count_data:
        raise ResourceNotFoundError("trade_count_data is not found.")

    if CommonDefine.COUNT_NUMBER not in trade_count_data["Item"]:
        raise ResourceNotFoundError(CommonDefine.COUNT_NUMBER + " is not found.")

    return trade_count_data["Item"].get(CommonDefine.COUNT_NUMBER)


def _get_info():
    info = {}

    info["last_price"] = int(zaif_public.last_price('btc_jpy')["last_price"])

    trade_info = zaif_private.get_info2()
    info["funds_btc"] = trade_info["funds"]["btc"]
    info["funds_jpy"] = trade_info["funds"]["jpy"]
    info["open_orders"] = trade_info["open_orders"]

    print('■ This is the current information.')
    print('last_price: ' + str(info["last_price"]))
    print('funds_btc: ' + str(info["funds_btc"]))
    print('funds_jpy: ' + str(info["funds_jpy"]))
    print('open_orders: ' + str(info["open_orders"]))

    return info


def _sell_btc(last_price, funds_btc):
    try:
        trade_result = zaif_private.trade(currency_pair="btc_jpy", action="ask",
                                          price=last_price, amount=funds_btc)
        print('■ I applied for the sale of Bitcoin.')
        pprint(trade_result)
        if trade_result["order_id"] == 0:
            _put_last_transaction_price(last_price, 0)
            print('■ The transaction is complete.')
        elif trade_result["order_id"] != 0:
            _put_last_transaction_price(last_price, trade_result["order_id"])
            print('■ The transaction is on hold.')

    except ResourceNotFoundError:
        raise
    except IllegalArgumentError:
        raise
    except Exception:
        raise


def _buy_btc(last_price, funds_jpy):
    try:
        #Since the API supports up to 4 digits after the decimal point, round()
        amount = round(float(funds_jpy) / last_price, 4)
        print('▼[trace]_buy_btc')
        print('amount: ' + str(amount))
        #Buy Bitcoin
        trade_result = zaif_private.trade(currency_pair="btc_jpy", action="bid",
                                          price=last_price, amount=amount)
        print('■ I applied for Bitcoin purchase')
        pprint(trade_result)
        if trade_result["order_id"] == 0:
            _put_last_transaction_price(last_price, 0)
            print('■ The transaction is complete.')
        elif trade_result["order_id"] != 0:
            _put_last_transaction_price(last_price, trade_result["order_id"])
            print('■ The transaction is on hold.')

    except ResourceNotFoundError:
        raise
    except IllegalArgumentError:
        raise
    except Exception:
        raise


#Write the transaction information to the DB
def _put_last_transaction_price(last_transaction_price, order_id):
    try:
        trade_count = _get_trade_count()

        trade_count += 1
        put_record_value = {
            CommonDefine.TRADE_NUMBER: trade_count,
            CommonDefine.LAST_TRANSACTION_PRICE: last_transaction_price,
            CommonDefine.ORDER_ID: order_id
        }
        dynamodb.put_item(CommonDefine.LAST_TRANSACTION_TABLE, put_record_value)

        dynamodb.update_by_partition_key(CommonDefine.SEQUENCE,
                                         CommonDefine.SEQUENCE_NAME, CommonDefine.TRADE_COUNT,
                                         CommonDefine.COUNT_NUMBER, trade_count)
    except ResourceNotFoundError:
        raise
    except IllegalArgumentError:
        raise
    except Exception:
        raise


#Delete the previous transaction information
def _delete_last_transaction_price(trade_count):
    try:
        trade_count -= 1
        dynamodb.delete_by_partition_key(CommonDefine.LAST_TRANSACTION_TABLE,
                                         CommonDefine.TRADE_NUMBER, CommonDefine.SEQUENCE_NAME)

        dynamodb.update_by_partition_key(CommonDefine.SEQUENCE,
                                         CommonDefine.SEQUENCE_NAME, CommonDefine.TRADE_COUNT,
                                         CommonDefine.COUNT_NUMBER, trade_count)
    except ResourceNotFoundError:
        raise
    except IllegalArgumentError:
        raise
    except Exception:
        raise

■[config] - zaif_keys.json ・ [Beginners will make a Bitcoin automatic trading bot aiming for a lot of money! Part 3](http://qiita.com/speedkingg/items/7f122e1fd90558c531f4# implementation)

■[aws] - dynamodb_utils.py -The basic methods for operating DynamoDB are summarized. ・ It is used by quoting from * main.py *.

dynamodb_utils.py

# coding:utf-8

import boto3
from exception.error import *  #A class that defines a proprietary error


class DynamoDBUtils:
    def __init__(self):
        self.dynamodb = boto3.resource('dynamodb')

    #Get all records in the specified table (up to 1 MB)
    def scan(self, table_name):
        if isinstance(table_name, str) is False or isinstance(table_name, unicode):
            raise IllegalArgumentError('item is not str/unicode')

        table = self.dynamodb.Table(table_name)
        response = table.scan()

        print('▼[trace][DynamoDB access]scan')
        print('table_name: ' + table_name)
        print('response:')
        print(str(response))

        return response

    #Json received by specifying the table(item)As a record or overwrite
    def put_item(self, table_name, item):
        if isinstance(table_name, str) is False or isinstance(table_name, unicode):
            raise IllegalArgumentError('item is not str/unicode')
        if isinstance(item, dict) is False:
            raise IllegalArgumentError('item is not dict')

        table = self.dynamodb.Table(table_name)
        response = table.put_item(
            Item=item
        )

        print('▼[trace][DynamoDB access]put_item')
        print('table_name: ' + table_name)
        print('put_item: ' + str(item))
        print('response:')
        print(str(response))

        return response

    #Get records by specifying table and primary key
    def get_item_by_partition_key(self, table_name, partition_key, partition_key_value):
        if isinstance(table_name, str) is False or isinstance(table_name, unicode):
            raise IllegalArgumentError('item is not str/unicode')

        table = self.dynamodb.Table(table_name)
        response = table.get_item(
            Key={partition_key: partition_key_value}
        )

        print('▼[trace][DynamoDB access]get_item_by_partition_key')
        print('table_name: ' + table_name)
        print('partition_key: ' + str(partition_key) + ', partition_key_value: ' + str(partition_key_value))
        print('response:')
        print(str(response))

        return response

    #Get records by specifying table and primary key / sort key
    def get_item_by_partition_key_and_sort_key(self, table_name, partition_key, partition_key_value,
                                               sort_key, sort_key_value):
        if isinstance(table_name, str) is False or isinstance(table_name, unicode):
            raise IllegalArgumentError('item is not str/unicode')

        table = self.dynamodb.Table(table_name)
        response = table.get_item(
            Key={partition_key: partition_key_value,
                 sort_key: sort_key_value}
        )

        print('▼[trace][DynamoDB access]get_item_by_partition_key_and_sort_key')
        print('table_name: ' + table_name)
        print('partition_key: ' + str(partition_key) + ', partition_key_value: ' + str(partition_key_value))
        print('sort_key: ' + str(sort_key) + ', sort_key_value: ' + str(sort_key_value))
        print('response:')
        print(str(response))

        return response

    #Update record by specifying table, primary key and update item
    def update_by_partition_key(self, table_name, partition_key, partition_key_value, update_key, update_value):
        if isinstance(table_name, str) is False or isinstance(table_name, unicode):
            raise IllegalArgumentError('item is not str/unicode')

        table = self.dynamodb.Table(table_name)
        response = table.update_item(
            Key={
                partition_key: partition_key_value
            },
            UpdateExpression='SET ' + update_key + ' = :uk',
            ExpressionAttributeValues={
                ':uk': update_value
            }
        )

        print('▼[trace][DynamoDB access]update_by_partition_key')
        print('table_name: ' + table_name)
        print('partition_key: ' + str(partition_key) + ', partition_key_value: ' + str(partition_key_value))
        print('update_key: ' + str(update_key) + ', update_value: ' + str(update_value))
        print('response:')
        print(str(response))

        return response

    #Delete record by specifying table and primary key
    def delete_by_partition_key(self, table_name, partition_key, partition_key_value):
        if isinstance(table_name, str) is False or isinstance(table_name, unicode):
            raise IllegalArgumentError('item is not str/unicode')

        table = self.dynamodb.Table(table_name)
        response = table.delete_item(
            Key={partition_key: partition_key_value}
        )

        print('▼[trace][DynamoDB access]delete_by_partition_key')
        print('table_name: ' + table_name)
        print('partition_key: ' + str(partition_key) + ', partition_key_value: ' + str(partition_key_value))
        print('response:')
        print(str(response))

        return response

■[define] - common_define.py -It is a class that defines and summarizes the character strings used mainly in DynamoDB. -Manage all together so that you can enjoy it when you want to change the column name etc. later.

common_define.py

# -*- coding: utf-8 -*-
class CommonDefine:
    def __init__(self):
        pass

    # dynamoDB table name
    LAST_TRANSACTION_TABLE = "last_transaction_price"
    SEQUENCE = "sequence"

    # dynamoDB key name
    TRADE_NUMBER = "trade_number"
    LAST_TRANSACTION_PRICE = "last_transaction_price"
    SEQUENCE_NAME = "name"
    COUNT_NUMBER = "count_number"
    ORDER_ID = "order_id"

    # other
    TRADE_COUNT = "trade_count"

■[exception] - error.py -A group of classes that summarizes unique errors. -Use when the error name you want to issue is not prepared as standard.

error.py

# -*- coding: utf-8 -*-
class ResourceNotFoundError(Exception):
    #Exception when the data to be acquired does not exist

    def __init__(self, message):
        self.message = message

    def __str__(self):
        return "Requested resource " + self.message + " not found."


class IllegalArgumentError(Exception):
    #Exception when the input argument is invalid

    def __init__(self, message):
        self.message = message

    def __str__(self):
        return "Illegal argument (" + self.message + ")"

** ② Role creation **

-Create a role to which you have assigned the necessary privileges to run Lambda.

1. IAM ▶ Role ▶ Select "Create New Role"
2. Role type selection: AWS Lambda
3. Attach Policy: AmazonDynamoDBFullAccess, AWSLambdaFullAccess
4. Enter the role name and click "Create Role"

** ③ Created by Lambda **

-Create a Lambda that actually executes the code.

1. Lambda ▶ Function ▶ Select "Create Lambda Function"
2. Click the "Function Settings" tab on the left
3. Enter a name (anything is fine) Runtime : Select python2.7 Existing role : Select the role created in " ② Create role **" Detailed settings ▶ Timeout: Increase the value to 10 seconds
4. "Next" ▶ "Create"

** ④ Deploy **

・ ** ③ Put the code on the Lambda created in ** Create Lambda **.

1. Open the "Settings" tab and modify the handler to ** main.lambda_handler ** (Meaning to call and execute main.py)
2. ** ① Coping the code group created by ** with zip *** Be careful not to become [zaif]-[zaif] -main.py! (Correct: [zaif] -main.py) **
3. Select the Code tab and upload the file

** ⑤ Test **

Press the "Save and Test" button, and if ** Execution result: Success **, it's OK!

３．CloudWach -Set the Lambda created above to be executed periodically.

1. CloudWatch ▶ Rules ▶ Creating rules
2. "Schedule" ▶ Change the value to 1 "Add target" ▶ Function *: ** Lambda function created above **
3. "Detailed settings" ▶ Enter name * ▶ "Rule settings"

in conclusion

This time the procedure was long. .. .. Thank you for your hard work! With this setting, you have a bot ** that trades BitCoin semi-permanently! Now, just think about the logic to make money! !! (๑ • ̀ ㅂ • ́) و✧ Trial and error ٩ (• ౪ • ٩)

Quote

・ Kemono Friends Logo Generator

▶ There are people in the world who make wonderful things. Thank you for using it. Thank you very much.

Correction history

・ 2017/4/27: I forgot to write the most important ** main.py **. .. .. I'm sorry. ・ 2017/4/27: MarkDown notation error correction (alt Thank you for pointing out.)