[PYTHON] [First personal development] The story of deploying the Flask app and Twitter's automatic reply bot on Heroku

Introduction

It will be first post. I'm a PHP engineer with a year and a half programming experience.

I use AWS for my daily work, but I haven't been involved in the production environment yet, so I would like to summarize what I learned through this deployment to Heroku from the perspective of a beginner.

I will briefly introduce the application I made this time.

--Automatically reply to a specific word reply on Twitter

--Create a management screen to add / edit text for automatic reply with flask

--24-hour monitoring of specific word replies using Twitter Streaming API (Filter realtime Tweets)

--Bot is implemented using Python's twitter library

Directory structure

app/
├── web.py           #Flask app executable
├── web/             #flask app
│   ├── models/
│   ├── static/
│   ├── templates/
│   ├── views/
│   ├── __init__.py
│   ├── config.py
│   └── database.py
├── twitter.py       #Executable file of automatic reply bot
├── twitter/         #Automatic reply Bot (partially omitted)
│   ├── __init__.py
│   ├── config.py
│   └── database.py
├── Procfile
├── requirements.txt
├── runtime.txt
├── .env
├── .gitignore
├── migrations/      # Flask-Created with Migrate
└── venv/            #Local virtual environment

Regarding flask, I referred to the following article.

-Procedure to start Flask + SQLAlchemy project

-When I thought about the directory structure of Flask, I was trying Blueprints

Files needed to deploy to Heroku

It seems that you need to prepare a file to tell Heroku about the environment, necessary libraries and executable files.

requirements.txt

List the required libraries, including the version. Since I developed with venv, I output the package installed in the virtual environment to a file with the following command.

(venv)$ pip freeze > requirements.txt

List of installed libraries

requirements.txt

alembic==1.4.2
autopep8==1.5.3
cffi==1.14.0
click==7.1.2
cryptography==2.9.2
Flask==1.1.2
Flask-Login==0.5.0
Flask-Migrate==2.5.3
Flask-SQLAlchemy==2.4.3
gunicorn==20.0.4
itsdangerous==1.1.0
Jinja2==2.11.2
Mako==1.1.3
MarkupSafe==1.1.1
pycodestyle==2.6.0
pycparser==2.20
PyMySQL==0.9.3
python-dateutil==2.8.1
python-dotenv==0.14.0
python-editor==1.0.4
six==1.15.0
SQLAlchemy==1.3.18
toml==0.10.1
twitter==1.18.0
Werkzeug==1.0.1

runtime.txt

It seems that this does not have to be separate, but I will prepare it to specify the version of python. You should also be careful to specify the Versions Supported by Heroku (https://devcenter.heroku.com/articles/python-support).

runtime.txt

python-3.7.8

Procfile

Specify how to start the app. Flask apps need to use a WSGI server such as gunicorn in a production environment.

Procfile

web: gunicorn web:app --log-file=-
worker: python twitter.py

web launches a Flask instance called app in web.py. worker is executing the executable file twitter.py of the automatic reply bot.

Setting environment variables

I stumbled upon how to set different environment variables (ex. DB information) in the local environment and the production environment without rewriting the code, so I will summarize them.

Heroku environment variables

You can set environment variables with the following command. (You need to have the Heroku CLI installed (https://devcenter.heroku.com/articles/heroku-cli) to use the command)

$ heroku config:set DB_HOST=xxxxxxxxxxx

You can display the list of environment variables and check if they are set.

$ heroku config

For the setting of environment variables of MySQL DB, refer to here. -How to use MySQL on Heroku

Environment variables in the local environment

Prepare an .env file and describe the environment variables there. It's sensitive information, so don't forget to gitignore it.

.env

ENV = 'LOCAL'

# DB
DB_HOST = 'xxxxx'
DB_NAME = 'xxxxx'
DB_USER = 'xxxxx'
DB_PASSWORD = 'xxxxx'

# Session
SESSION_SECRET_KEY = 'xxxxx'

Read environment variables on the app side

Read the environment variables in app / web / config.py as shown below and configure the settings on the application side.

config.py

"""Provide Flask Config"""
import os
from os.path import join, dirname
from dotenv import load_dotenv

dotenv_path = join(dirname(__file__), '../.env')
load_dotenv(dotenv_path)


class Config:

    # Flask
    if (os.environ.get('ENV') == 'LOCAL'):
        DEBUG = True
    else:
        DEBUG = False

    # Session
    SECRET_KEY = os.environ.get('SESSION_SECRET_KEY')

    # SQLAlchemy
    SQLALCHEMY_DATABASE_URI = 'mysql+pymysql://{user}:{password}@{host}/{db_name}?charset=utf8'.format(**{
        'user': os.environ.get('DB_USER'),
        'password': os.environ.get('DB_PASSWORD'),
        'host': os.environ.get('DB_HOST'),
        'db_name': os.environ.get('DB_NAME')
    })
    SQLALCHEMY_TRACK_MODIFICATIONS = False
    SQLALCHEMY_ECHO = False


Config = Config

With this setting, you can read environment variables in both Heroku and local environments. For the handling of environment variables in Python, I referred to here. ・ How to write environment variables that you don't want to put on [GitHub] Python

How to implement 1 reply per account per day

Since it is not good for Twitter API to automatically reply many times in a short time, we made it a specification that one reply per account per day.

I was wondering how to manage the users who replied once.

Also, Dynas is restarted (circulated) at least once a day to maintain the health of applications running on Heroku. All changes to the local file system will be removed. Cycling occurs every 24 hours (plus a maximum of 216 random minutes to prevent all dynos of the application from restarting at the same time).

Automatic restart of Heroku seems to be done every 24 hours, so this time it is on the Python file specified as worker (always looping) I decided to manage the replied users in the list of). This will initialize the list each time it is auto-rebooted so you can auto-reply again every 24 hours.

twitter.py

replied_user_list = [] #Manage replying users in a list

#Keep monitoring for 24 hours (program keeps running)
for tweet in twitter_stream.statuses.filter(language='ja', track=tracking_text):
    #Below, a specific word (tracking)_Processing when a tweet containing text) is detected
    # ...
    #Auto-reply will add to the list
    replied_user_list.append(user_id)
    # ...

Problem that worker sleeps when web sleeps

Heroku's Free Dyno plan automatically sleeps if you don't access the web app for 30 minutes. There is nothing wrong with this, but it seems that when web goes to sleep, so does worker. [https://devcenter.heroku.com/articles/free-dyno-hours#dyno-sleeping]

Apps that only utilise a free worker dyno do not sleep, because they do not respond to web requests. Be mindful of this as they may run 24/7 and consume from your pool of hours.

This is quite a problem as the auto-reply bot will stop responding if the worker goes to sleep.

Therefore, there seems to be a workaround such as sending a request periodically as follows to prevent web from sleeping. · Four ways to keep Heroku's free dyno running 24 hours a day

In this case, both web and worker dyno will continue to operate 24 hours a day, exceeding the free dyno time of 1000 hours / month.

As a result, I decided to stop web except when needed. (Easy to launch on the Heroku dashboard) The worker alone never sleeps, so the auto-reply bot can now run 24 hours a day.

Summary

It was my first time to develop a web application with python, and I thought I would write about that, but this time I mainly wrote about what I stumbled upon when deploying to Heroku.

I keenly realized that I needed to acquire more knowledge other than coding. In the future, I would like to study AWS and become an Associate of AWS Solutions Architect.

Since this is my first technical blog, I would appreciate your advice on non-technical issues such as being difficult to read. Of course, I would appreciate your technical suggestions.