[PYTHON] Automat WEB process with Heroku + Selenium + Chrome

Automat WEB process with Heroku + Selenium + Chrome

-[Automate web process with Heroku + Selenium + Chrome](# heroku--selenium--chrome% e3% 81% a7web% e3% 83% 97% e3% 83% ad% e3% 82% bb% e3% 82% b9% e3% 82% 92% e8% 87% aa% e5% 8b% 95% e5% 8c% 96% e3% 81% 99% e3% 82% 8b) -[Introduction](#% e3% 81% af% e3% 81% 98% e3% 82% 81% e3% 81% ab) -[Purpose](#% e7% 9b% ae% e7% 9a% 84) -[Related Articles](#% e9% 96% a2% e9% 80% a3% e3% 81% 99% e3% 82% 8b% e8% a8% 98% e4% ba% 8b) -[Execution environment](#% e5% ae% 9f% e8% a1% 8c% e7% 92% b0% e5% a2% 83) -[Source Code](#% e3% 82% bd% e3% 83% bc% e3% 82% b9% e3% 82% b3% e3% 83% bc% e3% 83% 89) -[Scenario and Prerequisites](#% e3% 82% b7% e3% 83% 8a% e3% 83% aa% e3% 82% aa% e3% 81% a8% e5% 89% 8d% e6% 8f% 90% e6% 9d% a1% e4% bb% b6) -[Create Automatic Process](#% e8% 87% aa% e5% 8b% 95% e3% 83% 97% e3% 83% ad% e3% 82% bb% e3% 82% b9% e3% 81% ae% e4% bd% 9c% e6% 88% 90) -[Python API configuration](# python-api% e6% a7% 8b% e6% 88% 90) -[API Mainframe](#api% e3% 83% a1% e3% 82% a4% e3% 83% b3% e3% 83% 95% e3% 83% ac% e3% 83% bc% e3% 83% a0) -[Get FX Rate](# fx% e3% 83% ac% e3% 83% bc% e3% 83% 88% e5% 8f% 96% e5% be% 97) -[Heroku Settings](# heroku% e3% 81% ae% e8% a8% ad% e5% ae% 9a) -[Chrome and Driver Settings](# chrome% e3% 81% a8driver% e3% 81% ae% e8% a8% ad% e5% ae% 9a) -[Driver Version Settings](#driver% e3% 83% 90% e3% 83% bc% e3% 82% b8% e3% 83% a7% e3% 83% b3% e3% 81% ae% e8% a8 % ad% e5% ae% 9a)

Introduction

Although it is an article on Mac environment, the procedure is the same for Windows environment. Please read and try the environment-dependent part.

Purpose

After reading this article to the end, you will be able to:

--Automate web processes using Selenium and Chrome Driver --Set Chrome and Driver on Heroku

Web process automation

Obtain and display ** USD / JPY ** from the FX chart rate of Yahoo! Finance.

View manually from browser	Display automatically acquired data

Related articles

• Selenium
• heroku-buildpack-chromedriver.git
• heroku-buildpack-google-chrome.git

Execution environment

Source code

I think that understanding will deepen if you read while actually following the implementation contents and source code. Please use it by all means.

GitHub

Scenarios and prerequisites

1. Obtain and display ** USD / JPY ** from the FX chart rate of Yahoo! Finance.
2. It is supposed to be saved in MySQL or PostgreSQL, but DB related is out of scope for the purpose of explaining automation.
3. The API framework adopts Flask.

Creating an automated process

Python API configuration

tree.sh

python-Selenium
  ├── fx_rate
  │   ├── __init__.py
  │   └── utility.py
  └── main.py

API mainframe

main.py

import datetime
import logging
import os

from flask import Flask

from fx_rate.utility import get_fx_rate

logging.basicConfig(level=logging.INFO)
logger = logging.getLogger(__name__)

app = Flask(__name__)


@app.route('/fx-rate', methods=['GET'])
def get():
    usd_jpy = get_fx_rate()
    res = 'timestamp={}, USDJPY={}'.format(
        datetime.datetime.utcnow() + datetime.timedelta(hours=9), usd_jpy)
    logger.info(res)
    return res, 200


if __name__ == '__main__':
    host = os.getenv('HOST', '0.0.0.0')
    port = int(os.getenv('PORT', 5000))
    app.run(host=host, port=port, debug=True)

Forex rate acquisition

utility.py

import datetime

from selenium import webdriver
from selenium.webdriver.chrome.options import Options
from selenium.webdriver.common.by import By


def get_fx_rate():
    options = Options()
    options.add_argument('--headless')
    driver = webdriver.Chrome(options=options)
    driver.get('https://info.finance.yahoo.co.jp/fx/')
    usd_jpy = driver.find_element(By.ID, 'USDJPY_top_bid').text
    driver.quit()
    return usd_jpy


if __name__ == '__main__':
    usd_jpy = get_fx_rate()
    print('timestamp={}, USDJPY={}'.format(datetime.datetime.utcnow() +
                                           datetime.timedelta(hours=9), usd_jpy))

Heroku settings

Chrome and Driver settings

Add the following from ** Add buildpack ** in the Settings> Buildpacks section.

`If you deploy to Heroku, it will be installed automatically. Please register in advance. If you have already deployed or want to redeploy without changing the source code, try the empty commit below. ``

allow_empty.sh

~$ git commit --allow-empty -m "allow empty commit"
~$ git push heroku master

Driver version setting

Normally, no setting is required. If the version of ** chromedrive ** and ** google-chrome ** are different, such as when Chrome is upgraded, you need to specify the version of ** chromedrive **.

1. Click ** Reveal Config Vars ** in the Settings> Config Vars section.
2. Enter ** KEY ** and ** VALUE ** and click ** Add **.