[PYTHON] Use boto3 to mess with S3

Introduction

In IoT related projects, it was necessary to download and upload files in S3 from the gateway (hereinafter GW), so try implementing it using python (2.7) and boto3 (AWS SDK for python). I did. I will keep a memorandum of the procedure at that time.

final goals

1. You will be able to download the files in your bucket
2. You will be able to upload files to a specific directory

Let's get started

installation of boto3

Log in to GW with ssh connection and install boto3 with pip. (GW has python2.7 pip installed)

sudo pip install boto3

If you want to specify a specific version, install as follows.

sudo pip install boto3==1.0.0

IAM creation

When using boto3, it is necessary to set a policy in IAM. This time I created an IAM user with S3FullAccess set. If necessary, set the minimum necessary policies such as reading only or narrowing down the buckets that can be accessed.

• This step is omitted

AWS CLI settings

Set the access key information for the IAM user you created earlier.

AWS CLI installation

sudo pip install awscli

Configure settings

aws configure
AWS Access Key ID: xxxxxxxxxxxxxxxxx
AWS Secret Access Key: xxxxxxxxxxxxxxxxx
Default region name: ap-northeast-1
Default output format:

This completes the settings. You can check the settings in "~ / .aws / credentials" and "~ / .aws / config".

Import boto3 in python

In order to import bote3 and mess with S3 files, start writing as follows.

boto3_test.py

# -*- coding: utf-8 -*-

import boto3

s3 = boto3.resource('s3')

If you want to use other services, write as follows. (DynamoDB example)

boto3_test.py

# -*- coding: utf-8 -*-

import boto3

# S3
s3 = boto3.resource('s3')

# DynamoDB
dynamo = bot3.resource('dynamodb')

Actually mess with S3 files

Download from S3

Use download_file (). When downloading fuga.txt in the bucket name hoge

boto3_test.py

# -*- coding: utf-8 -*-

import boto3

s3 = boto3.resource('s3')

bucket = s3.Bucket('hoge')
bucket.download_file('fuga.txt', 'fuga.txt')

The bucket is the same, if you want to download fuga.txt in the folder hogehoge

boto3_test.py

#---abridgement---

bucket = s3.Bucket('hoge')
bucket.download_file('hogehoge/fuga.txt', 'fuga.txt')

Upload to S3

Use upload_file (). When uploading fuga.txt in the bucket name hoge

boto3_test.py

#---abridgement---

bucket = s3.Bucket('hoge')
bucket.upload_file('fuga.txt', 'fuga.txt')

The bucket is the same, if you upload fuga.txt in the folder hogehoge

boto3_test.py

#---abridgement---

bucket = s3.Bucket('hoge')
bucket.upload_file('fuga.txt', 'hogehoge/fuga.txt')

You can now download and upload files in S3. By the way, it is to display each object information in the bucket

boto3_test.py

#---abridgement---

bucket = s3.Bucket('hoge')
print bucket.name

A list is displayed with.

Finally

I had a hard time because I hadn't touched python so much before using boto3, but the official documentation was pretty good. It was helpful. On the contrary, those who do not show rejection of English should read the official document. Sweat