[PYTHON] Creating a LINE BOT to notify you of additional AtCoder contests using AWS

Are you turning off notifications for apps other than LINE? I'm cutting. I wanted you to be notified on LINE when the schedule of the AtCoder contest was added, so I created a LINE BOT with Mr JJ.

What I created

We will search for contests scheduled at 00:00 every hour and BOT will notify you when new contests are added.

There is also a function that returns when you send a message. (bonus)

You can add friends to the created LINE BOT by clicking the button below.

Backend configuration

All backend processing is done on AWS (Amazon Web Services). The operation on AWS can be divided into the following two.

-[Contest Notification Section](#Contest Notification Section) --Search for AtCoder contests and notify new contests -[LINE Response Section](LINE Response Section) --Respond when a message is sent to the BOT

Describes the features implemented for each section.

Contest notification section

Contest notification section flow

1. Lambda function: ** AtCoder_contest_Search ** scrapes the AtCoder homepage to get a list of scheduled contests.
2. ** AtCoder_contest_Search ** gets the notified contest list JSON file from S3.
3. Calculate the difference set and extract unannounced contests. If the unannounced contest is empty, it ends.
4. Send unannounced contest data to your Lambda function: ** LINE_contest_notify **.
5. Notify the data sent ** LINE_contest_notify ** in broadcast format using line-bot-sdk.
6. Upload the contest data obtained in [1] to S3

Scraping function

Python3

def get_contests():
    #Fetch data from URL
    url = "https://atcoder.jp/home?lang=ja"
    html_data = requests.get(url)

    #html perspective
    soup = BeautifulSoup(html_data.text, "html.parser")
    
    tags = ["upcoming", "recent"]
    ret_dic = {}
    for tag in tags:
        div = soup.find("div", id="contest-table-"+tag)
        table = div.find("table")
        times  = table.find_all("time")
        titles = table.find_all("a")

        ret_list = []
        for i in range(len(times)):
            dic = {}
            dic["start"] = times[i].text
            dic["title"] = titles[i*2+1].text
            dic["url"] = "https://atcoder.jp" + titles[i*2+1].get("href")
            ret_list.append(dic)
        ret_dic[tag] = ret_list
    return ret_dic

Only * ret_dic ["upcoming"] * is used in this BOT.

Get files from S3

Python3

def get_data_s3():
    #Notified contest json file on S3"contests.json"Loading
    
    client = boto3.client("s3")
    try:
        response = client.get_object(Bucket = "bucket_name", Key = "json_file_name")
    except Exception as e:
        print(e)
    return json.loads(response["Body"].read().decode())

The return value of the * get_object * method is a dictionary, and the key "Body" contains the contents of the file.

File upload to S3

Python3

def put_data_s3(save_dic):
    #New notified contest json file S3 uploader
    
    text = json.dumps(save_dic, ensure_ascii = False)
    s3 = boto3.resource("s3")
    s3.Object("bucket_name", "json_file_name").put(Body = text)

When converting the dictionary to JSON format text, if the value of * ensure_ascii * is set to False, The Japanese in the JSON file becomes easier to read and the file size can be reduced.

Broadcast message transmission

Python3

def DatetimeToString(date):
    if not isinstance(date, datetime.datetime):
        date = datetime.datetime.strptime(date, "%Y-%m-%d %H:%M:%S%z")
    week_list = "Monday, Tuesday, Wednesday, Thursday, Friday, Saturday and Sunday"
    return f"{date.strftime('%m/%d')}({week_list[date.weekday()]}){date.strftime('%H:%M')}"

def MakeMessageText(event):
    text = DatetimeToString(event["start"]) + "\n" + event["title"] + "is held!\n" + event["url"]
    return text

def main(event, context):    
    line_bot_api = LineBotApi("channel_access_token")
    message = MakeMessageText(event)
    line_bot_api.broadcast(TextSendMessage(text=message))

    # ***Processing to finish the main function***

Create a message format for the contest start time with the * DatetimeToString * function Created the format of the message to be notified by the * MakeMessageText * function.

You can notify all your BOT friends by simply calling the * LineBotApi * class * broadcast * method of line-bot-sdk. ..

LINE response section

LINE response section flow

1. When the LINE BOT receives the LINE message, it uses the Webhook to POST the message content to the API Gateway.
2. POST to API Gateway triggers Lambda function: ** LINE_reply_message ** to be executed.
3. ** LINE_reply_message ** sends a LINE reply message using line-bot-sdk.

Send reply message

Python3

def main(event, context):
    handler = WebhookHandler("channel_secret")
    line_bot_api = LineBotApi("channel_access_token")
    body = event["body"]

    #Added reply function to handler
    @handler.add(MessageEvent)
    def SendReply(line_event):
        message = MakeReplyMessage(line_event.message.text)
        line_bot_api.reply_message(line_event.reply_token, TextSendMessage(text=message))

    try:
        handler.handle(body, event["headers"]["X-Line-Signature"])
    except #Error handling

    # ***Processing to finish the main function***

The * MakeReplyMessage * function returns a reply message as a character string according to the message content received by the BOT. We especially recommend reading the line-bot-sdk documentation for this part.

At the end

We introduced the main functions we created. Line-bot-sdk is very convenient because you can easily operate BOT. I have little knowledge about AWS and the design is quite rough, but if I feel like it, I will add and improve functions in the future. (maybe)