Various memos when using terraform
Terraform to understand in 10 minutes
The point is, when creating a cloud environment, instead of creating a procedure manual and creating it manually, It seems to be a tool that tries to do all that with code. (If you can manage it with code, it may be easier to manage the version of the environment itself, or you can eliminate personality.)
I tested the scraping script with Colaboratory and For launching a server quickly when you want to run it all night
cloudshell Terraform is installed in the default cloudshell docker image, so you don't need to create a custom image
Docker image pushed to GCR
Use selenium and beautiful soup to put scraping results into a pandas dataframe Dockerfile that can be used when you want to insert from pandas to BigQuery
GCP JSON key file and scraping python file must be in the same directory as Dockerfile
Dockerfile
FROM python:3
ARG project_dir=/selenium/
ARG credential_json=<JSON key file name>
ADD requirements.txt $project_dir
ADD $credential_json $project_dir
ADD <Scraping python file> $project_dir
ENV GOOGLE_APPLICATION_CREDENTIALS ./$credential_json
WORKDIR $project_dir
RUN apt-get update
RUN apt-get install -y vim less python3-selenium
RUN pip install -r requirements.txt
CMD ["/bin/bash"]
requirements.txt
selenium==3.141.0
requests==2.23.0
pandas==1.0.4
bs4==0.0.1
google-cloud-bigquery==1.21.0
pandas-gbq==0.11.0
lxml==4.5.1
Create a file (tf file) that defines resources in an appropriate directory
main.tf The main file that sets things up for GCE
main.tf
provider "google" {
credentials = "${file("<JSON key file>")}"
project = "${var.project_name}"
region = "${var.region}"
}
data "google_compute_network" "default" {
name = "default"
}
resource "google_compute_instance" "apps-gcp-terraform" {
name = "selenium-docker"
machine_type = "n1-standard-1"
zone = "${var.zone}"
boot_disk {
auto_delete = true
initialize_params {
image = "${var.boot_image_name}"
type = "pd-standard"
}
}
metadata = {
gce-container-declaration = "${var.docker_declaration}"
}
network_interface {
network = "default"
access_config {
}
}
service_account {
email = "<Service account>"
scopes = ["cloud-platform"]
}
}
variables.tf File that handles parameters
When creating a GCE instance from Docker image, use a boot image called container-optimized image You can check the current container-optimized image version with the following command.
gcloud compute images list --project cos-cloud --no-standard-images
variables.tf
variable "project_name" {
type = string
default = "<Project ID>"
}
variable "region" {
type = string
default = "us-central1"
}
variable "zone" {
type = string
default = "us-central1-a"
}
variable "boot_image_name" {
type = string
default = "projects/cos-cloud/global/images/cos-stable-81-12871-119-0"
}
variable "docker_declaration" {
type = string
default = "spec:\n containers:\n - name: selenium-docker\n image: '<docker image>'\n stdin: false\n tty: true\n restartPolicy: Always\n"
}
Execute the following command in the directory where the tf file is located
When such a message appears Terraform has initialized, but configuration upgrades may be needed.
terraform 0.12upgrade ↑ It seems that it will update the tf file to the latest version nicely
terraform plan Check if the tf file is correct If an error occurs, fix the tf file
terraform apply Up to gcp You will be asked if you want to execute it on the way, so enter yes
Do you want to perform these actions? Terraform will perform the actions described above. Only 'yes' will be accepted to approve.
Enter a value: yes
google_compute_instance.apps-gcp-terraform: Creating... google_compute_instance.apps-gcp-terraform: Still creating... [10s elapsed] google_compute_instance.apps-gcp-terraform: Creation complete after 13s
Apply complete! Resources: 1 added, 0 changed, 0 destroyed.
If it looks like ↑, it's OK
Then enter the created GCE instance
docker container ls
Then the container is running, so
docker container exec -it container ID/bin/bash
Enter the container with commands such as
nohup python scraping python file> log.txt 2>&1 &
Run the program
terrafrom destroy Do you really want to destroy all resources? Terraform will destroy all your managed infrastructure, as shown above. There is no undo. Only 'yes' will be accepted to confirm.
Enter a value:yes
Will be deleted by
Error: Cannot get auth token: Metadata server responded with status 404 When when It seems that you can not get tokens from GCE's metadata server,
So, if you execute the following command in the instance,
curl "http://metadata.google.internal/computeMetadata/v1/instance/service-accounts/default/token" \
-H "Metadata-Flavor: Google"
↓ message appears Service account not enabled on this instance
So, when I added the service account attachment to main.tf, it was OK ↓
main.This part of tf
service_account {
email = "<Service account>"
scopes = ["cloud-platform"]
}
terraform documentation Create and configure an instance Use variables in Terraform GCP Infrastructure Construction Automation Road part1 ~ Terraform Introduction ~ Deploy a web app on Google Compute Engine with Terraform. Terraform 0.12 has been released, so I upgraded it [Try building a GCP environment with Terraform](https://techblog.gmo-ap.jp/2017/11/16/terraform%E3%81%A7gcp%E7%92%B0%E5%A2%83% E3% 82% 92% E6% A7% 8B% E7% AF% 89% E3% 81% 97% E3% 81% A6% E3% 81% BF% E3% 82% 8B /) scopes param for service_account of google_compute_instance should be optional
Recommended Posts