[PYTHON] How to automatically start a GCP preemptible VM

How to automatically start a GCP preemptible VM

GCP has a VM called Preemptible VM that stops in up to 24 hours at the cost of a low price. https://cloud.google.com/compute/docs/instances/preemptible?hl=ja

VMs that are stopped after receiving the preempt notification will not start automatically. The following is a description of the tools that enable automatic startup using shutdown scripts and Cloud Functions.

Tool overview

--Preemptible VM is a tool to start because it does not start automatically after stopping. --It is designed to start only when you receive a preempt notification so that it will not start if you stop it manually. --The execution environment is python3.7.

Tool behavior

  1. Set shutdown-script on the VM. It runs when the VM is stopped, checks if it has received a preempt notification, writes TRUE or FALSE to a file and uploads it to GCS.
  2. Run Cloud Functions triggered by a GCS file update.
  3. Function starts the VM only if the preempt notification is TRUE. Sequence.png

Setup steps

  1. Download the source code

    git clone [email protected]:cloud-mainte/auto-boot-preemtible-vm.git
    
    or 
    
    git clone https://gitlab.com/cloud-mainte/auto-boot-preemtible-vm.git
    
  2. Create a GCS bucket --Create a bucket to place the shutdown script and place the shutdown script. --Rewrite the BUCKET variable in the script as appropriate.

    $ gsutil mb -l ASIA-NORTHEAST1 gs://<backet name>
    $ gsutil cp shutdown-script.sh gs://<backet name>/
    $ gsutil ls -l gs://<backet name>
    

--Create a bucket to store the status of whether or not you received a preempt notification.

```
$ gsutil mb -l ASIA-NORTHEAST1 gs://<backet name>
```
  1. Set shutdown-script on the target instance

    gcloud compute instances add-metadata <instance name> \
        --metadata shutdown-script-url=gs://<backet name>/shutdown-script.sh
    
  2. Set the instance launcher in Cloud Functions --Rewrite bucket in main.py as appropriate.

    gcloud functions deploy boot_preemptible_instance --region=asia-northeast1 -- 
    entry-point boot_instance --runtime python37 --trigger-event 
    google.storage.object.finalize --trigger-resource gs://<backet name>
    

Operation check

  1. Try stopping the instance manually.

$ gcloud compute instances stop $ gcloud compute instances list $ gsutil cat gs:///preempted_status.out {"preempted_status":"FALSE", "instance_name":"", "zone":"asia-northeast1-a", "project":""} ```

If stopped manually, it will be " preempted_status ":" FALSE ".
Check the log.

```
$ gcloud logging read "resource.type=\"cloud_function\" AND resource.labels.function_name=\"boot_preemptible_instance\" AND resource.labels.region=\"asia-northeast1\"" --format="table(timestamp, textPayload)" |head -10
TIMESTAMP                       TEXT_PAYLOAD
2020-08-11T04:38:26.982711475Z  Function execution took 1894 ms, finished with status: 'ok'
2020-08-11T04:38:26.979Z        Preempted status is FALSE.
 :
2020-08-11T04:38:25.090798409Z  Function execution started
```
  1. Try to imitate the Preempt notification.
$ cat preempted_status.out
{"preempted_status":"TRUE", "instance_name":"<instance name>", "zone":"asia-northeast1-a", "project":"<project name>"}
$ gsutil cp preempted_status.out gs://<backet name>/
$ gsutil cat gs://<backet name>/preempted_status.out
{"preempted_status":"TRUE", "instance_name":"<instance name>", "zone":"asia-northeast1-a", "project":"<project name>"}

Check the log.

$ gcloud logging read "resource.type=\"cloud_function\" AND resource.labels.function_name=\"boot_preemptible_instance\" AND resource.labels.region=\"asia-northeast1\"" --format="table(timestamp, textPayload)" |head -12
TIMESTAMP                       TEXT_PAYLOAD
2020-08-11T05:02:51.640069088Z  Function execution took 15066 ms, finished with status: 'ok'
2020-08-11T05:02:51.637Z        Instance started.
2020-08-11T05:02:51.637Z        Instance starting.. status = RUNNING
2020-08-11T05:02:46.246Z        Instance starting.. status = STAGING
2020-08-11T05:02:40.862Z        Instance starting.. status = TERMINATED
2020-08-11T05:02:39.373Z        Instance status is now TERMINATED
2020-08-11T05:02:38.959Z        Preempted status is TRUE. Boot up the instance.
 :
2020-08-11T05:02:36.577339221Z  Function execution started

The instance has started.

$ gcloud compute instances list
NAME                         ZONE               MACHINE_TYPE   PREEMPTIBLE  INTERNAL_IP     EXTERNAL_IP    STATUS
<instance name>  asia-northeast1-a  g1-small       true         XX.XX.XX.XX   XX.XX.XX.XXX  RUNNING

Recommended Posts

How to automatically start a GCP preemptible VM
How to fix a bug that jupyter notebook does not start automatically
I want to send a business start email automatically
How to call a function
How to hack a terminal
How to start the program
How to run the Export function of GCP Datastore automatically
How to make a Japanese-English translation
How to put a symbolic link
How to make a slack bot
How to create a Conda package
How to make a crawler --Advanced
How to make a recursive function
How to make a deadman's switch
How to create a Dockerfile (basic)
[Blender] How to make a Blender plugin
How to delete a Docker container
How to make a crawler --Basic
How to create a config file
How to batch start a python program created with Jupyter notebook
How to create a clone from Github
How to build a sphinx translation environment
How to create a git clone folder
Qiita (1) How to write a code name
How to draw a graph using Matplotlib
[Python] How to convert a 2D list to a 1D list
[Colab] How to copy a huge dataset
How to install a package using a repository
[Ubuntu] How to execute a shell script
How to get a stacktrace in python
How to create a repository from media
How to make a Backtrader custom indicator
How to choose a Seaborn color palette
How to test on a Django-authenticated page
How to make a Pelican site map
How to run a Maya Python script
How to automatically upload .gpx files to Strava
How to calculate the volatility of a brand
How to start Python (Flask) when EC2 starts
How to read a CSV file with Python 2/3
How to disguise a ZIP file as a PNG file
How to use GCP trace with open Telemetry
A simple example of how to use ArgumentParser
How to share a virtual environment [About requirements.txt]
How to send a message to LINE with curl
How to create a Python virtual environment (venv)
How to code a drone using image recognition
How to open a web browser from python
How to clear tuples in a list (Python)
How to hold a competition at Coda Lab
How to draw a 2-axis graph with pyplot
How to embed a variable in a python string
How to create a function object from a string
How to draw a 3D graph before optimization
How to develop a cart app with Django
How to create a JSON file in Python
How to generate a Python object from JSON
How to call Cloud API from GCP Cloud Functions
How to implement a gradient picker in Houdini
How to make a QGIS plugin (package generation)
How to write a ShellScript Bash for statement