[PYTHON] [Cloud Functions] Automatically decompress GZIP files placed in Storage

Preamble

• I want to automatically decompress when a GZIP file is added (updated) to Google Cloud Storage
• I searched Google Cloud Storage API Reference, but I can't find any description about GZIP decompression.
• There is no choice but to seek to utilize Python's GZIP library.

Source

I modified the code (reference link-3) that decompresses the Zip file that my predecessor dropped. It is assumed that a single file is Gzip-compressed with Cloud Functions (Python 3.7) launched by a Cloud Storage trigger.

main.py

from google.cloud import storage
import io,gzip

def hello_gcs(event, context):

    client = storage.Client()
    bucket_name = event['bucket']
    bucket = client.get_bucket(bucket_name)
    blob_name = event['name']
    blob = bucket.blob(blob_name)

    data = io.BytesIO(blob.download_as_string())
    with gzip.open(data) as gz:
        file = gz.read()
        blob_decompress = bucket.blob(blob_name.replace('.gz',''))
        blob_decompress.upload_from_string(file)

Big problem

• Maximum memory allocated to Cloud Functions is 2GB (at the time of posting)
• If the processing exceeds the memory capacity, an error will naturally occur.

⇒ At the time of the GZIP file to be placed, the capacity could be 2GB or more, so it ended up being a crap. ..

Reference link

1. Cloud Functions Guide: Google Cloud Storage Trigger
2. [I created a serverless job that loads data into BigQuery with event drive when a file is placed in GCS](https://dev.classmethod.jp/articles/cloud-functions-gcs-trigger-load- data2bigquery /)
3. [How do I unzip the .Zip file on Google Cloud Storage? ](Https://www.it-swarm.dev/ja/google-cloud-platform/google%E3%82%AF%E3%83%A9%E3%82%A6%E3%83%89%E3% 82% B9% E3% 83% 88% E3% 83% AC% E3% 83% BC% E3% 82% B8% E3% 81% A7zip% E3% 83% 95% E3% 82% A1% E3% 82% A4% E3% 83% AB% E3% 82% 92% E8% A7% A3% E5% 87% 8D% E3% 81% 99% E3% 82% 8B% E3% 81% AB% E3% 81% AF% E3% 81% A9% E3% 81% 86% E3% 81% 99% E3% 82% 8C% E3% 81% B0% E3% 82% 88% E3% 81% 84% E3% 81% A7% E3% 81% 99% E3% 81% 8B% EF% BC% 9F / 837268170 /)