[PYTHON] [LINE Notify API, AWS] Regularly send buzzing tweets to group LINE

What to make this time

For group LINE with friends We will create a mechanism to send tweets that are buzzing at the scheduled time.

LINE Notify API

By linking with LINE Notify, LINE users will be able to easily receive service notifications.

[Source]: LINE Notify

It seems.

When you actually use it, you will receive a message from the account called LINE Notify like this.

Preparation for LINE Notify

Follow the steps in the link below to get LINE Notify ready for use. [Super easy] Try using LINE notify

Get buzzing tweets

I was wondering if I should do my best with the buzzing tweets using Twitter API, It seemed to be painful, so I decided to scrape from the existing ranking site.

Twitter popularity ranking site → [TwTimez]

Code (Python)

import requests
from bs4 import BeautifulSoup

#Get the most vigorous tweets
def bazz_get():
    #URL to access
    url = "http://www.twtimez.net/index.html"

    #Access the URL
    html = requests.get(url)

    #Handle html with Beautiful Soup
    soup = BeautifulSoup(html.text, "html.parser")

    try:
        for detalis in soup.find(class_="details details2"):
            print(detalis.get("href"))
            return str(detalis.get("href"))
    except:
        return "Some kind of error"

#Send a message to Line
def send_line(Bazz):

    notify_url = "https://notify-api.line.me/api/notify"
    token = "Access token"
    headers = {"Authorization": "Bearer " + token}

    message = "\r\n" + Bazz
    payload = {"message":  message}

    requests.post(notify_url, headers=headers, params=payload)


if __name__ == "__main__":
    send_line(bazz_get())

Scraping

What do you care about? It's my first time. It was a little bit if I used Beautiful Soup.

In the following part, I am pulling the class and tag with the desired information from html.

    for detalis in soup.find(class_="details details2"):
        print(detalis.get("href"))
        return str(detalis.get("href"))

Find the class or tag that has the information you want by following the procedure you do with GIF. You can see the html of the page by pressing F12 Key.

If there is no problem so far, a message will be sent to LINE after execution.

AWS We will create a mechanism to automatically send a message when it is scheduled using AWS services.

Maybe it's free. If you receive an invoice later, pay the study fee and stop with haste.

AWS Lambda

AWS Lambda Explainer Video

You can run your code without having to think about the server. You only pay for the computing time you actually use.

It seems. I will insert the code I wrote this time here.

Amazon CloudWatch

Amazon CloudWatch Explainer Video

Observability of AWS resources and applications on AWS and on-premises

It seems. I don't know. In short, I think you can easily monitor something you made and do various things. You can periodically execute the function created on Lambda at the set time.

Actual procedure

Sign in to the Lambda console and select Create Function Give it a name and proceed to the next.

It seems that you can upload each folder with ~~ Zip, I haven't made that much, so I'll copy and paste the code above into the function. ~~

It didn't work because I was using a module. Furthermore, you need to add or modify the code.

Also, upload the code as a .zip file instead of editing the code inline.

Code modified for AWS

import requests
from bs4 import BeautifulSoup


def bazz_get():
    #URL to access
    url = "http://www.twtimez.net/index.html"

    #Access the URL
    html = requests.get(url)

    #Handle html with Beautiful Soup
    soup = BeautifulSoup(html.text, "html.parser")

    try:
        for detalis in soup.find(class_="details details2"):
            print(detalis.get("href"))
            return str(detalis.get("href"))
    except:
        return "Some kind of error"


def send_line(Bazz):

    notify_url = "https://notify-api.line.me/api/notify"
    token = "Access token"
    headers = {"Authorization": "Bearer " + token}

    message = "\r\n" + Bazz
    payload = {"message":  message}

    requests.post(notify_url, headers=headers, params=payload)


def bot(event, lambda_context):
    send_line(bazz_get())

Below is the function that is actually called by Lambda. It would have been nice to have a separate Script that only executes it, It works fine and I have no plans to expand it in the future, so I did the following. You need an argument to call it with Lambda.

def bot(event, lambda_context):
    send_line(bazz_get())

Write the Python Script name and the method name you want to call in the upload destination handler by connecting them with ..

Upload the modular Zip file to Lambda

First, save the module in any directory.

First, move to the directory you want to save.

Run at command prompt

cd any directory

After that, save the module used in Python Script in the current directory.

Run at command prompt

pip install beautifulsoup4 -t .
pip install requests -t .

Next is Zip compression. Compress the Python Script and module you want to upload. If you perform Zip compression on the folder that stores everything, an extra folder for one layer will be created. Select all and compress like a GIF image.

CloudWatch Events

Next, select Add Trigger to work with CloudWatch. (You can also use the approach of signing in to the CloudWatch console and making separate settings.)

Schedule settings

Please refer to the link below for how to set the schedule (when to execute the code). Schedule Expressions for Rules

Sign in to CloudWatch from the console If you create a new rule, it will look like the image below. It is easy to understand because it shows when to execute.

UTC (GMT) and JST

As you can see from the image above, the word GMT is written after the execution time.

It seems that there is a 9-hour difference from the time in Japan, so it seems that you have to set the time in consideration of that time difference.

[Reference link]: [AWS] Explanation of the format of CloudWatch cron expression or rate expression

Impressions of everyone in Group LINE

All were full shikato. why?

2019/12/07 postscript

Actually, tweets with a large number of RTs and favos are all campaign-type. .. .. I was angry because Lawson's RT campaign flowed to the group line every day.

Reference link

Program that tells you what you want to know in the morning on LINE (Python) [Python] How to use BeautifulSoup / Basic method list | Scraping I tried using AWS Lambda with Python. Explanation of library loading and precautions for environment variables.