1. 1. You can continuously acquire data at low cost with GCP's permanent free frame etc.

GCP (Google Cloud Platform) has a free tier that can be used for the first 3 months, and even though there are restrictions on regions and resources, you can use the server permanently for free. Google Compute Engine 1 f1-micro instance (US Region only, excluding Northern Virginia [us-east4]) 30 GB HDD (total period), 5 GB snapshot (total period) 1 GB downlink network from North America to all regions (excluding China and Australia) Google Cloud Storage 5 GB Regional Storage (US Region only, excluding Northern Virginia [us-east4]) 5,000 Class A operations 50,000 Class B operations 1 GB downlink network from North America to all regions

There are other services available, but this time I'd like to use Python to scrape and store data in cloud storage.

Please note that may not be completely free due to the 1GB limit on network usage as mentioned above.

2. 2. How to operate CGS from GCE in Python

It is assumed that each instance of GCE and GCS has already been created.

First, install the library for Python on an instance of GCE (Ubuntu, etc.).

$ pip install --upgrade google-cloud-storage

2-1. Import the library for operating GCS

This is necessary for all operations.

from google.cloud import storage
client = storage.Client()
bucket_name="test_bucket"
bucket = client.get_bucket(bucket_name)

2-2. How to operate objects in the bucket

2-2-1. Get object list

for blob in client.list_blobs(bucket_name):
    print(blob.name)

2-2-2. Creating an instance of an object

blob = bucket.blob("sample.txt")

2-2-3. Downloading the object

blob.download_to_filename("sample1.txt")

2-2-4. Uploading objects

blob.upload_from_filename("sample2.txt")

3. 3. Example

Here's how to upload data from GCE to GCS.

from google.cloud import storage
client = storage.Client()
bucket_name = "test-bucket"
bucket=client.get_buket(bucket_name)#Get bucket object
blob = bucket.blob("test.txt")#Create an instance of an object
blob.download_to_filename("sample.txt")#download

[PYTHON] Data accumulation with GCP's permanent free frame