GCP (Google Cloud Platform) has a free tier that can be used for the first 3 months, and even though there are restrictions on regions and resources, you can use the server permanently for free. Google Compute Engine 1 f1-micro instance (US Region only, excluding Northern Virginia [us-east4]) 30 GB HDD (total period), 5 GB snapshot (total period) 1 GB downlink network from North America to all regions (excluding China and Australia) Google Cloud Storage 5 GB Regional Storage (US Region only, excluding Northern Virginia [us-east4]) 5,000 Class A operations 50,000 Class B operations 1 GB downlink network from North America to all regions
There are other services available, but this time I'd like to use Python to scrape and store data in cloud storage.
First, install the library for Python on an instance of GCE (Ubuntu, etc.).
$ pip install --upgrade google-cloud-storage
This is necessary for all operations.
from google.cloud import storage client = storage.Client() bucket_name="test_bucket" bucket = client.get_bucket(bucket_name)
for blob in client.list_blobs(bucket_name): print(blob.name)
blob = bucket.blob("sample.txt")
Here's how to upload data from GCE to GCS.
from google.cloud import storage client = storage.Client() bucket_name = "test-bucket" bucket=client.get_buket(bucket_name)#Get bucket object blob = bucket.blob("test.txt")#Create an instance of an object blob.download_to_filename("sample.txt")#download