[PYTHON] Make a guy who will notify you when Oniken is on sale

This article is a relay article of 2021 New Year Advent Calendar Tech Connect! of Link Information Systems. Tech Connect! Is relayed by group members of engineer.hanzomon as an advent calendar to start on its own.


This article is for people who missed the sale during this time.

It is a common one that only integrates the scraped result with Slack, but in order to differentiate it from other articles, Google Cloud Platform (hereinafter GCP), Microsoft Azure (hereinafter Azure), Amazon Web Services (hereinafter AWS) Let's deploy each one.

What to make

Go to the My Nintendo Store Oniken page once a day and link to Slack if it's on sale.

to make

Check robots.txt in My Nintendo Store

Check https://store-jp.nintendo.com/robots.txt. There is no problem because "/ list/software /" was not disallowed.

Slack settings

Open the add screen of the app

slack01.png

Added Incoming Web Hooks

slack02.png

slack03.png

Specify the notification destination channel

slack04.png

Make a note of the webhook URL

Change the name to something that is easy to understand. slack05.png

Preparation of scripts etc.

Scripts for scraping and integration with Slack

The one who just has to move for the time being.

import json
import os

import requests
from bs4 import BeautifulSoup

res = requests.get(os.environ['TARGET_URL'])
soup = BeautifulSoup(res.text, 'html.parser')
div = soup.select_one('div.productDetail--detail__pricePrice')

if div is not None:
    item = {
        "text": div.text.strip()
    }
    json_data = json.dumps(item)
    res = requests.post(os.environ['WEBHOOK_URL'], json_data)

Scraping

Get the following information (price) with div = soup.select_one ('div.productDetail--detail__pricePrice'). If you don't get this, it's not on sale and will not work with Slack. s01.png

Linkage to Slack

The part of res = requests.post (os.environ ['WEBHOOK_URL'], json_data). Just post the json data to the webhook URL.

This time it's a simple message, but if you want to make something complicated, I think you should try various things with Block Kit Builder.

requirements.txt

beautifulsoup4==4.9.1
requests==2.24.0

Deploy to GCP

Open

Cloud Functions Responsible for scraping and linking with Slack.

Select Cloud Functions

gcp01.png

Create function

gcp02.png

various settings

"Basic" as you like. Set the "trigger" as shown in the image and "save" it. ** Make a note of the URL as you will use it later ** gcp03.png

Setting runtime environment variables

Name Value
TARGET_URL URL of Oniken page
WEBHOOK_URL Slack's Webhook URL that you noted earlier

After setting, click "Next". gcp04.png

Enable Build API

gcp05.png

Edit script

Please note that it is not a copy and paste. gcp06.png

Edit requirements.txt

After copying and pasting, "deploy". gcp07.png

Confirm deployment complete

This will take some time. gcp08.png

Cloud Scheduler Use it to run Cloud Functions once a day. ** You can create up to 3 jobs with a free frame **

Select Cloud Scheduler

gcp09.png

Create a job

There is a region selection screen after this screen, but I forgot to take a screenshot. Please choose your favorite place. gcp10.png

various settings

Name, description, etc. as you like.

Item Value
Target HTTP
URL Cloud Functions trigger URL created earlier
HTTP method GET

After setting, "Create".

gcp11.png

End of deployment to GCP

This is the end of deploying to GCP.

Deploy to Azure

Open

Environment

With this configuration, you can't write code on the Azure portal, so you'll need to do a lot of preparation. I will omit the details because it seems to be long.

Functions app (Azure Functions)

Creating a resource

az01.png

Select a function app

az02.png

various settings

As you like except the red frame. Others are "confirm and create" by default. az03.png

Create

az04.png

"Move to resource" when the deployment is complete. az05.png

Setting environment variables

Set the following two and "Save".

Name Value
TARGET_URL URL of Oniken page
WEBHOOK_URL Slack's Webhook URL that you noted earlier

az06.png

Create function

It is an operation with Visual Studio Code.

Sign in to Azure

az07_01.png

Project creation

This time, we will work in the azure_sample folder created on Desktop. az07_02.png

select a language

az08.png az09.png

Select a function trigger

Select "Timer trigger". az10.png

Give it a name

az--.png

Specify schedule

It is specified to run at 01:00 every day. az11.png

Edit script

After waiting for a while, a file called __init__.py will be opened, so rewrite it with the contents prepared in advance. az12.png

Edit requirements.txt

There is requirements.txt in the project folder (azure_sample this time), so edit it. az13.png

Deploy

It will take some time. az14.png

It is deployed properly. az15.png

End of deployment to Azure

This is the end of deploying to Azure. Thanks to the Timer trigger, it was possible to realize it by the function application alone.

Deploy to AWS

Open

Lambda

Go to the function creation screen

aws01.png aws02.png

Creating a function

You can name the function as you like. After selecting options and runtime, "Create Function". aws03.png

Create a zip file

python library from bs4 to urllib3. lambda_function.py is a script prepared in advance for lambda.

oniken.zip
 ├ bs4
 ├ certifi
 ├ chardet
 ├ idna
 ├ requests
 ├ soupsieve
 ├ urllib3
 └ lambda_function.py

lambda_function.py


import json
import os

import requests
from bs4 import BeautifulSoup

def lambda_handler(event, context):
	res = requests.get(os.environ['TARGET_URL'])
	soup = BeautifulSoup(res.text, 'html.parser')
	div = soup.select_one('div.productDetail--detail__pricePrice')

	if div is not None:
	    item = {
	        "text": div.text.strip()
	    }
	    json_data = json.dumps(item)
	    res = requests.post(os.environ['WEBHOOK_URL'], json_data)
	    
	    return "success"

Upload zip file

aws04.png aws05.png

Setting environment variables

aws06.png

Set the following two and "Save".

Name Value
TARGET_URL URL of Oniken page
WEBHOOK_URL Slack's Webhook URL that you noted earlier

aws07.png

trigger

Add trigger

aws08.png

Event Bridge settings

It is specified to run at 02:00 every day. aws09.png

End of deployment to AWS

This is the end of deploying to AWS. It was troublesome because I had to upload the library myself. (GCP, Azure just write requirements.txt)

Operation check

Notifications were scheduled to arrive at 00:00 (GCP), 01:00 (Azure), and 02:00 (AWS), but Azure and AWS are off by about 9 hours. GCP set the time zone in Cloud Scheduler, but by the way, I feel that there was no such setting in Azure and AWS. result.png

Azure Looking at the log, it was running at 01:03 (UTC). Is there a 3 minute gap?

AWS Looking at the log, it was also running at 02:00 (UTC).

Although

This is my homework because it has moved.

Summary

Azure was overwhelmingly easy if the environment could be built. Timer trigger is great.

type Easy Usually Difficult
Environment is not built GCP AWS Azure
Environment built Azure GCP AWS

reference

--Create a function in Azure using Visual Studio Code and Python --Configure function app settings in Azure | Microsoft Docs --Python AWS Lambda Deployment Package --AWS Lambda

I'm looking for someone to figure out why the Oniken page in the Microsoft Store is that way.

** What is that? ** ** Quoted from Microsoft Store Oniken Page.

Inspired by the harsh gameplay of 8-bit action/platform in the post-inventory future controlled by sinister companies, the demon sword is against enemies, traps, and challenging boss gauntlets. Striking the agility and sword.

As you can see from a little reading, Japanese is strange.

** Find the translation source ** Quoted from Microsoft Store Oniken Page. The same text on the publisher's product page.

In a post-apocalyptic future controlled by a sinister corporation, ninja mercenary Zaku is the last hope for humanity. Inspired by the demanding gameplay of 8-bit action/platformers, Oniken pits your agility and swordplay against a gauntlet of enemies, traps, and challenging bosses.

The "table of contents" was the "apocalyptic".

** Get translated ** As far as the original text (Japanese page) is seen, it seems that some kind of translation will be done, so I will ask you to translate the text of the English page on some sites.

-Google Translate

In the post-apocalyptic future dominated by sinister companies, Ninja Mercenary Zaku is the last hope for humanity. Inspired by the demanding gameplay of 8-bit action/platformers, Oniken defeats agility and swordsmanship against enemies, traps and challenging boss gauntlets.

-DeepL Translator

In a post-apocalyptic future dominated by evil companies, the ninja mercenary Zaku was humanity's last hope. Inspired by the rigorous gameplay of 8-bit action/platformers, "Oniken" uses agility and swordsmanship to break through obstacles such as enemies, traps, and bosses.

In a post-apocalyptic future dominated by sinister companies, Ninja Mercenary Zaku is the last hope of mankind. Inspired by the tough gameplay of 8-bit action/platforms, Oniken pits your agility and sword play against enemies, traps, and challenging boss gauntlets.

Ninja mercenary Zaku is the last hope for humanity in the backward movement after being dominated by an ominous enterprise. Inspired by the tough game of 8-bit action/small plates, the demon sword swiftly pours swords on enemies, traps, and challenging bosses, and swords on its abdomen.

** Google Translate results are close to the original ** Except for DeepL Translator, the three results are almost similar, and Google Translate is quite close to the original text except for the Ninja Mercenary Zaku part. Let's see the difference by arranging the original text and the result of Google Translate side by side.

--Original

Inspired by the ** harsh ** gameplay of 8-bit ** action/** platform ** in the future ** controlled ** by the sinister company ** after inventory ** Being done, the demon sword will ** hit ** your ** agility and ** sword ** against ** enemies, traps, ** and ** challenging bosses ** gauntlets **. **.

--Google Translate (excluding the Ninja Mercenary Zaku part)

In the future ** dominated ** dominated by sinister companies ** after the end , Oniken was inspired by ** demanding ** gameplay of 8-bit action/ platformers ** ** Defeats ** agility and ** swordsmanship ** against ** gauntlets ** of enemies, traps, and challenging bosses **.

It seems that there is only a level difference that the translation result has changed with the update of Google Translate.

** Deepening mystery ** I've come to the point that the text on the Japanese page is probably based on the results of Google Translate, but there remains a mystery that I can't understand.

--Tablebook I have no idea what and how the apocalypse becomes a table of contents. English → (Translation) → Japanese → (Speech to Text) → (Speech to Text) → If you use Japanese, there is a possibility that you will make a mistake in hearing at Speech to Text, but you have to do something like this. It doesn't make sense.

--The disappeared ninja mercenary Zaku Zaku is the main character of Oniken. When translating

in conclusion Please let me know if anyone can get closer to the answer.

Recommended Posts

Make a guy who will notify you when Oniken is on sale
[Stada] A script that will call you when Pokemon GO is released
Use Heroku in python to notify Slack when a specific word is muttered on Twitter
A solution when you can't start project Django on Windows
When you want to hit a UNIX command on Python
Write a script in Shell and Python to notify you in Slack when the process is finished