This article is the 10th day article of Pathee Advent Calendar 2019.
Hello. I'm Kunii, an engineer.
I've summarized how to upload / download files to ** AWS S3 ** / ** Azure BlobStorage ** / ** GCP Cloud Storage ** with ** Python **.
I used to upload / download files to Azure Blob Storage with Python, and that's why I decided to try other storage services together.
Somehow I chose AWS S3 / GCP Cloud Storage, It's Python because our backend is implemented in Python.
For each storage service, we will summarize in the following items.
·Advance preparation ・ ** Connection information settings ** ・ ** Module installation ** ・ ** Create a box to store files ** ・ ** Upload the file to the created box ** ・ ** Download the file from the created box ** ・ ** Delete the created box **
Every storage service has a module, so it's fairly easy to use.
Python 3.6.4 macOS 10.15.1
AWS S3
You will need ** Access Key ID ** and ** Secret Access Key ** as connection information. Create an access key ID and secret access key by referring to Create AWS Access Key.
Set the access key ID and secret access key in the environment variables with the following command.
$ export AWS_ACCESS_KEY_ID="Access key ID"
$ export AWS_SECRET_ACCESS_KEY="Secret access key"
Install ** boto3 **.
$ pip install boto3
For each subsequent operation on AWS S3, ** s3_resource ** created with the following code will be the base.
import boto3
s3_resource = boto3.resource('s3', region_name='ap-northeast-1')
In AWS S3, the box that stores files is called a ** bucket **. You can create a bucket with the following code.
#Creating a bucket
s3_client = s3_resource.meta.client
s3_client.create_bucket(Bucket='The name of the bucket to create', CreateBucketConfiguration={'LocationConstraint': 'ap-northeast-1'})
It seems that there are various setting items in CreateBucketConfiguration, but for the time being, it can be created only by LocationConstraint (constraint of the region where the bucket is created).
#File upload
s3_resource.Bucket('The name of the bucket to upload to').upload_file('Path of the file to upload', 'File name at the upload destination')
#File download
s3_resource.Bucket('The name of the bucket from which you downloaded it').download_file('File name to download', 'Download destination file path')
#Delete bucket
s3_client = s3_resource.meta.client
s3_client.delete_bucket(Bucket='The name of the bucket to delete')
In order to delete a bucket, you need to delete all the files uploaded to the bucket and leave them empty. If you try to delete a bucket when the bucket is not empty, an error will occur and you will not be able to delete it.
You can delete the uploaded file with the following code, but I only know how to delete it by specifying the file name ... (There is also a delete_objects method, but it seems that you have to specify the file name after all)
#Delete file
s3_client = s3_resource.meta.client
s3_client.delete_object(Bucket='The name of the bucket that contains the files to delete', Key='File name to delete')
Azure BlobStorage
You will need the storage account ** connection string ** as the connection information.
Sign in to the Azure portal and do Create Storage Account (https://docs.microsoft.com/en-us/azure/storage/common/storage-quickstart-create-account?tabs=azure-portal) .. Refer to here Get (copy) the connection string of the storage account.
Set the connection string in the environment variable with the following command.
$ export CONNECT_STR="Connection string"
Install ** azure-storage-blob **.
$ pip install azure-storage-blob
Subsequent operations on Azure Blob Storage are based on ** blob_service_client ** created with the following code.
import os
from azure.storage.blob import BlobServiceClient
connect_str = os.getenv('CONNECT_STR')
blob_service_client = BlobServiceClient.from_connection_string(connect_str)
In Azure BlobStorage, the box that stores files is called a ** container **. You can create a container with the following code.
#Creating a container
container_client = blob_service_client.create_container('The name of the container to create')
#File upload
blob_client = blob_service_client.get_blob_client(container='The name of the container to upload to', blob='The name of the file at the upload destination')
with open('Path of the file to upload', 'rb') as uploaded_file:
blob_client.upload_blob(uploaded_file)
#File download
blob_client = blob_service_client.get_blob_client(container='The name of the container from which it was downloaded', blob='The name of the file to download')
with open('File path at the download destination', 'wb') as downloaded_file:
downloaded_file.write(blob_client.download_blob().readall())
You can delete a container without deleting all the files you uploaded to the container (although you may have deleted them in the method).
#Delete container
container_client = blob_service_client.get_container_client('The name of the container to delete')
container_client.delete_container()
GCP CloudStorage
As connection information, you will need the key of the account (** service account key **) to access the cloud storage called the service account.
[Here](https://sleepless-se.net/2018/05/22/googlecloudstorage%E3%81%A7python%E3%81%8B%E3%82%89%E3%83%95%E3%82% A1% E3% 82% A4% E3% 83% AB% E3% 82% 92% E3% 82% 84% E3% 82% 8A% E3% 81% A8% E3% 82% 8A% E3% 81% 99% Create a service account key by referring to E3% 82% 8B% E6% 96% B9% E6% B3% 95 / # API) and download the JSON file of the service account key.
Set the service account key in the environment variable with the following command.
$ export GOOGLE_APPLICATION_CREDENTIALS="Path of downloaded JSON file"
Install ** google-cloud **.
$ pip install --upgrade google-cloud
The ** storage_client ** created with the following code will be the basis for each subsequent operation on Google Cloud Storage.
from google.cloud import storage
storage_client = storage.Client()
In GCP Cloud Storage, the box that stores your files is called a ** bucket **. You can create a bucket with the following code.
#Creating a bucket
bucket = storage_client.create_bucket('The name of the bucket to create')
#File upload
bucket = storage_client.get_bucket('The name of the bucket to upload to')
blob = bucket.blob('The name of the file at the upload destination')
blob.upload_from_filename(filename='Path of the file to upload')
#File download
bucket = storage_client.get_bucket('The name of the bucket that contains the file to download')
blob = bucket.blob('The name of the file to download')
blob.download_to_filename('Download destination file path')
#Delete bucket
bucket = storage_client.get_bucket('The name of the bucket to delete')
bucket.delete(force=True)
In order to delete a bucket (similar to AWS S3), you need to delete all the files you uploaded to the bucket and leave them empty. If you try to delete a bucket when the bucket is not empty, an error will occur and you will not be able to delete it.
By setting ** force = True **, the bucket will be emptied and then deleted.
By the way, deleting the file is as follows.
#Delete file
bucket = storage_client.get_bucket('The name of the bucket that contains the files to delete')
blob = bucket.blob('The name of the file to delete')
blob.delete()
Recommended Posts