Recently, it's a hassle to download Twitter images one by one while working and bring them to Google Photos. It is a record that if you are an engineer, you can automate it.
I just want an image of an idol who wants to download a Twitter image. It has no other purpose.
Get tweets regularly → Download if there is an image → Upload the downloaded image to Google Photos.
Google Cloud Functions https://cloud.google.com/functions/?hl=ja
A service that executes a specific function. It's more like a small program than running a large program. This time, the program has about 120 lines, so I chose this one.
If the price is not good, it will not exceed the permanent free frame (I think that even if you make a lot of functions, it will not exceed it). Since this program causes communication outside the Google API (Tweet acquisition and image download), we are taking measures to reduce the execution frequency. However, since we do not know where to look at network usage, it is difficult to know how much communication is actually occurring even now that we have started operation.
The language set for the function this time is python. The reason is that I have been using it for business recently. There are other languages that can be used, so let's check before starting development (I will not write it here because it may change)
The price list is here.
Google Cloud Scheduler https://cloud.google.com/scheduler/?hl=ja
It's easy to connect with Google Cloud Functions, so I'll leave it to this guy for regular execution. The fee is determined by the number of jobs, but it is a service that is cheap and easy to use even if it exceeds the free frame.
The price list is here.
This time we will use GCP, Google API (Photos), Twitter API and various APIs, so we will make various settings.
https://cloud.google.com/?hl=ja
This is not particularly troublesome to set up, so it finishes quickly.
Google Photos API https://developers.google.com/photos
You need to enable the API from around the URL above. The activation itself is easy, but you need to authorize using OAuth to call the API. Here Needs to be obtained until the Refresh Token is obtained, so let's do it in advance.
Twitter API https://developer.twitter.com/content/developer-twitter/ja.html
This was the most troublesome. The operation is easy, but since the application explanation in English is required when registering as a developer, the explanation must be written in English for a certain length. I have the impression that the policy when creating an app changes considerably, so please be careful when working.
There was information that Japanese was fine, but as of January 2020, English was mandatory.
Even people who are not very English can break through by translating simple Japanese into Google Translate, so you don't have to think too hard. It's definitely more troublesome than writing in Japanese.
The examination itself was completed in an instant. Maybe it depends on how you choose your options, so it may be different for each person.
The program itself is listed on github, so please see that for the completed version.
If you create an environment variable or .env, you can execute it locally.
In terms of processing, I use python-twitter to get the tweet of USER_LIST defined in the environment variable, and if there is an image, I just upload it from the download.
Due to the specifications of the Google Photos API, there is no problem even if you give the same image (probably judged by the file name), so if there is an image in the tweet you got, you have uploaded it with the same procedure. It's useless communication, but I do it because I'm not in trouble for the time being. If it is true, it would be better to make the program save the id somewhere and get it from there, but I gave priority to the release because it seems to take a little time.
It is recommended to save it in Firebase because it can be executed locally because it is often done for free using GCP system.
Official is very easy to understand how to use Firebase from python code, so please refer to this. Please refer to the Article for how to test locally.
Cloud SQL doesn't have Always Free, so I gave up this time.
2020/02/01 postscript Due to the specifications of Twitter, if you do not set tweet_mode = "extended", you will get chopped text. Please note that the media will not be set properly for such a Tweet, so you must set Tweet_mode = "extended".
The source of Github has been updated.
Note that event triggers require arguments for the starting function.
pub/sub => event, context http => request, response
is. I forgot and got hooked a little. Also, since only two files, MAIN.PY and REQUIREMENTS.TXT, can be set, it must be implemented within that range. Writing a complicated program is painful.
If you need a log, print is fine, so let's prepare it.
If everything can be done locally, I think it's easier to confirm that it works locally. It seems that it can be done on Cloud Functions, but as mentioned above, it is impossible to encounter a mysterious error.
2020/04/17 postscript The test on Cloud Functions was because the timeout setting was set by default. For programs with long execution times, the test here will result in an error. However, it seems that the execution itself is done, so you can check in the log whether you can run completely.
The memory settings change depending on the program, but with this program, 256MB was enough. Triggers are created in Cloud Pub / Sub. You can set a maximum timeout of 518 seconds, so you can rest assured that you set the maximum for the time being. This time the program took about 200 seconds, so I did it in 60 seconds and timed out.
Just set the trigger to pub / sub and the topic to the value set in Google Cloud Functions. The time to execute is written by the cron method. I set it only occasionally, so I don't feel like learning how to write it, but there is a site like this, so let's use it.
If you want to run it right away, press the button and it will run, so make sure it succeeds.
It's easy to run small programs on the server on a regular basis, so it's great for automating small things. This time, it took about 3 hours to implement and 1 hour to set up, so it was very meaningful to be able to do hobbies, practical benefits and study in a short time.
There are still issues around Firebase and Cloud Functions testing, but we'll work on them as soon as we know them. I think I will write an article when it is over.
Recommended Posts