[PYTHON] I made a SlackBot that notifies me of AtCoder contest information every week

AtCoder Contest Information We have created a Slackbot to notify you of AtCoder Contest Information.

(1) Obtaining Bot API token and installing slackbot

You need to get an API token and install slackbot before you can create a Slackbot. These two are described in SlackBot development with Python ① "Getting API key and simple response". I just did that. If you execute as written and send an appropriate character string in conversation with the bot, the wording set in DEFALUT_REPLY will be returned as shown below.

(2) Web scraping

Since I confirmed that Slackbot works, I will scrape it and collect AtCoder contest information. Below is the code.

scrape.py

from urllib import request
from bs4 import BeautifulSoup
import re
import datetime


#Returns a time / URL pair
def scrape_active():
    '''
A function that returns information about the contest you are currently holding
    '''
    re_contests=[]
    #Receive information on that URL
    url="https://atcoder.jp"
    html=request.urlopen(url)
    #Analyze page information with parser
    soup=BeautifulSoup(html, "html.parser")
    #contest-table-Information on the contest currently being held is included in the div tag with the id of active
    contests1=soup.find("div",id="contest-table-active")
    #If there was no contest(Because None will be included in soup)
    if contests1 is None:
        return re_contests
    contests2=contests1.find("tbody")
    contests3=contests2.find_all("tr")
    #re_Store the contest url and end date and time in contests
    for c in contests3:
        re_contests_sub=[]
        #Store the url of the contest page in url2
        d=c.find("a",href=re.compile("contests"))
        url2=url+d.get("href")
        html2=request.urlopen(url2)
        soup2=BeautifulSoup(html2, "html.parser")
        #class is a reserved word in Python, so class_
        sftime=soup2.find_all("time",class_="fixtime-full")
        #Re only the end time_contests_Store in sub
        re_contests_sub.append(sftime[1]+"End")
        #The url of the contest page is also re_contests_Store in sub
        re_contests_sub.append(url2)
        re_contests.append(re_contests_sub)
    return re_contests


def scrape_upcoming():
    '''
A function that returns information about contests held within a week
    '''
    re_contests=[]
    url="https://atcoder.jp"
    html = request.urlopen(url)
    soup = BeautifulSoup(html, "html.parser")
    #contest-table-The div tag with the id of upcoming contains information on the contest to be held within a week.
    contests1=soup.find("div",id="contest-table-upcoming")
    #If there was no contest(Because None will be included in soup)
    if contests1 is None:
        return re_contests
    contests2=contests1.find("tbody")
    contests3=contests2.find_all("tr")
    #Get today's date and time(Monday)
    w=datetime.datetime.today()
    #Re the url and start date and time of the contest_Store in contests
    for c in contests3:
        re_contests_sub=[]
        d1=c.find("time")
        #Slice and pass only the part that contains up to the minute
        #Return from string to datetime object with strtotime function
        t=strtotime(d1.text[:16])
        #You don't have to store contests that aren't there by Sunday of the week
        if (t-w).days>=7:
            break
        #Use the timetostr function to unify the format
        #Store the start date and time of the contest
        re_contests_sub.append(timetostr(t)+"start")
        d2=c.find("a",href=re.compile("contests"))
        #The url of the contest page is also re_contests_Store in sub
        re_contests_sub.append(url+d2.get("href"))
        re_contests.append(re_contests_sub)
    return re_contests

def strtotime(date_sub):
    '''
Return as datetime object
    '''
    return datetime.datetime.strptime(date_sub,'%Y-%m-%d %H:%M')

def timetostr(date_sub):
    '''
Return datetime object as str object
    '''
    W=["Month","fire","water","wood","Money","soil","Day"]
    return ('%d-%d-%d(%s) %d:%s'%(
        date_sub.year,date_sub.month,date_sub.day,W[date_sub.weekday()],date_sub.hour,str(date_sub.minute).ljust(2,"0")
    ))

I've written what I'm doing in the comment out, but if you have any questions about how to use the function, please see the page that I referred to.

Referenced page

Summary of scraping basics at Beautiful Soup [for beginners] Basics of Beautiful Soup + Requests Python date type datetime

(3) Posting to Slack

We will post the collected contest information to Slack.

run.py

from slackbot.bot import Bot
from slacker import Slacker
import slackbot_settings
import scrape
import datetime

def make_message(channel,slack,s,message):
    '''
    slack.chat.post_Send a message using messag.
    '''
    for i in s:
        message=message+"\n"+i[0]+"\n"+i[1]+"\n"
    #pos_You can post to slack with message
    #The channel you want to post to channel
    #The message you want to post in message
    #as_By setting user to True, the url will be expanded and posted.
    slack.chat.post_message(channel, message, as_user=True)

def info(channel,slack):
    #Scraping first and storing the contest information
    s1=scrape.scrape_active()
    s2=scrape.scrape_upcoming()
    #No contest information(0 in length)If so, send a message that the contest does not exist
    if len(s1)!=0:
        make_message(channel,slack,s1,"*[List of contests being held]*")
    else:
        slack.chat.post_message(channel,"*No contests are being held*",as_user=True)
    if len(s2)!=0:
        make_message(channel,slack,s2,"*[This week's contest list]*")
    else:
        slack.chat.post_message(channel,"*There is no contest this week*",as_user=True)

def main():
    #Don't forget to add the bot application to that channel before running the bot
    channel="Competitive Pro"
    #API token is slackbot_settings.Save to py
    slack = Slacker(slackbot_settings.API_TOKEN)
    #Confirmation that it is Monday
    if datetime.datetime.today().weekday()==0:
        info(channel,slack)
    bot = Bot()
    bot.run()

if __name__ == "__main__":
    main()

See commenting out as in (2). (Because it becomes long and difficult to read if explained in detail)

Referenced page

chat.postMessage The story that the URL posted on Slack's Incoming Webhooks was not expanded Making a bot with Slack API 2: Message posting

(4) Deploy to Heroku

Up to (3), you should be able to run it locally by hitting python run.py. Next, deploy to Heroku. (Heroku is like a free server) Heroku itself is not difficult to handle if you think of it as a remote repository for Git, so if you study Git, you should know how to use it. (Progate's Git chapter is recommended as a whole.) Also, I will push files to Heroku using Git, but as described in the reference article, let's create three files, Procfile, requirements.txt, runtime.txt, and then push. .. (If you do not create it, various errors will be spit out and it will be empty.)

Referenced page

Create Slackbot in Python Notes on making SlackBot made with python resident

(5) Setting environment variables

If you follow the article referred to in (1), a file called slackbot_settigns should be created and two variables, API_TOKEN and DEFAULT_REPLY, should be defined. However, if this is left as it is, the token value will be obtained when the source code leaks, and the bot will be hijacked, so it is necessary to make it an environment variable without writing the token value in the source code. Locally, you can type the command ʻexport environment variable name = value, but if you want to set the environment variable for heroku, set the environment variable by setting heroku config: set environment variable name = value`. I can do it. You can also set environment variables in Config Vars in Settings for your application on Heroku. This allows you to set environment variables, and rewriting slackbot_settigns as shown below completes the environment variable settings.

slackbot_settings.py

import os

#Get environment variables
API_TOKEN=os.environ["API_TOKEN"]
DEFAULT_REPLY=os.environ["DEFAULT_REPLY"]

For how to use os.environ, refer to Get / Add / Overwrite / Delete Environment Variables with Python (os.environ). Please give me.

Referenced page

[Python] How to deploy a Python program on Heroku → It also describes how to use Heroku.

(6) Scheduler registration

Finally, when you register the scheduler, all automation is complete. First, I'm adding a scheduler as an add-on, but I need to register a credit card. Make sure to register from your account page. Once registered, you can add the scheduler to your add-on with $ heroku addons: create scheduler: standard and open the page for registering with the scheduler with heroku addons: open scheduler. If you press "Add job" on the opened page, you can execute it regularly by registering the time interval to move and the command you want to execute. (By the way, if you run it without registering it in the scheduler, it will be executed about every 6 hours.) Also, in my case, I wanted to run it once a week, so I changed it so that it runs every day and the program does the desired processing only once a week. (Also note that the time is in UTC, so you need to add nine hours to UTC time to convert it to Japan time.)

Referenced page

SlackBot development with Python ⑧ "Regular program execution with Heroku Scheduler"