[PYTHON] Access Cloud Storage from your Compute Engine instance

Introduction

I want to access Cloud Storage from a GCE instance. Since it is accessed from inside the project, Method using OAuth 2.0 directly Simpler [Service account]( Use https://cloud.google.com/compute/docs/authentication).

Authentication

First, get an access token from the metadata server.

import json
import urllib2

METADATA_SERVER = "http://169.254.169.254/computeMetadata/v1/instance/service-accounts"
SERVICE_ACCOUNT = "default"

req = urllib2.Request("{0}/{1}/token".format(METADATA_SERVER, SERVICE_ACCOUNT))
req.add_header("Metadata-Flavor", "Google")

data = json.load(urllib2.urlopen(req))

token = data["access_token"]
token_type = data["token_type"]

Access using tokens

Use the received token to access Cloud Storage.

Get object

from apiclient import discovery

sp = discovery.build("storage", "v1")
req = sp.objects().get_media(bucket=<BUCKET>, object=<PATH>)
req.headers["Authorization"] = "{0} {1}".format(token_type, token)
res = req.execute()

Sample code used ʻoauth2_client.AccessTokenCredentials`, but you can also access it by the above method.

Uploading objects in chunk format

from apiclient import discovery
from apiclient.http import MediaIoBaseUpload

sp = discovery.build("storage", "v1")
# fp:Data to upload, mimetype: MIME type
media_body = MediaIoBaseUpload(fp, mimetype, resumable=True)

req = sp.objects().insert(bucket=<BUCKET>, body=dict(name=<PATH>), media_body=media_body)
req.headers["Authorization"] = "{0} {1}".format(token_type, token)
req.execute()

reference

Recommended Posts

Access Cloud Storage from your Compute Engine instance
Use Cloud Datastore from Compute Engine
Access Google Cloud Storage from Python (boto) using your service account and key file (p12)
Using Cloud Storage from Python3 (Introduction)
Run Cloud Dataflow (Python) from App Engine
Access Django's development server from your browser
Operate Sakura's cloud object storage from Python
Firebase: Use Cloud Firestore and Cloud Storage from Python
PySpark: Set up PySpark on your Alibaba Cloud CentOS instance
From python to running instance on google cloud platform