[PYTHON] Try server-side encryption on S3 with boto3

When uploading an object to S3 with a Python script I investigated how to encrypt on the server side.

S3 and object encryption

Reference: There are roughly two types.

Client-Side Encryption（CSE）

The client side encrypts the data before sending it to the server for storage. -> Data is protected during transmission and reception with the server (S3).

Server-Side Encryption（SSE）

The client sends the data as is, and the server encrypts it before storing it. -> Data is protected while stored on the server (S3).

The latter requires less effort on the client side to think about encryption. It is easy to handle because it is transparently decrypted even when downloaded. __ * Sensitive information should of course be encrypted before sending and receiving __

S3 and Server-Side Encryption

Reference: There are different types of SSE options in S3.

	SSE-S3	SSE-KMS	SSE-C
Encryption key	Keys managed behind the scenes by S3	Keys managed by AWS KMS service	User-managed keys
Feature	Since everything is managed by AWS, there is little effort. However, the access authority to the key cannot be manipulated, and the key can only be used to save data in S3.	You can flexibly change the access authority to the key itself by operating IAM users and roles. SSE-Although it has a smaller turn than S3, the number of setting items increases.	You can choose the key to use and the storage location. It has the highest degree of freedom, but you need to ensure the secure storage of your keys yourself.

I wanted to leave the role to the AWS side as much as possible, so I tried SSE-S3 and SSE-KMS this time.

Prior confirmation in Management Console

When I try to save an object to a bucket in the Management Console Along the way, you will be given an encryption option.

"Amazon S3 master-key" stands for SSE-S3. Even if you select it, there are no particular choices.

"AWS KMS master-key" means AWS-KMS. Once selected, you will be given a choice of keys to use.

--"aws / s3" is the key that KMS tries to create for S3 services. --"test_key" is the key created by KMS (I created it in advance) --If you select Custom KMS ARN, you can specify the key in the description of ARN.

If you try to save it with SSE-S3, it will be displayed like this on the object details screen.

Do this from the script (boto3).

Script description

Because there is no description of for Python in [Document] of S3 (http://docs.aws.amazon.com/ja_jp/AmazonS3/latest/dev/UsingServerSideEncryption.html) I wrote it with reference to other languages and boto3 documentation. The execution environment is Lambda's Python 3.6, but I think that the writing style and behavior do not change much anywhere.

Rewrite the role nicely so that you can access the S3 and KMS keys.

SSE-S3

`SSE-S3`


import boto3

def lambda_handler(event, context):
    
    file_name = 'test.txt'

    #Can be used temporarily'/tmp'Create a file in
    with open('/tmp/' + file_name, 'w') as f:
        f.write('hoge')
    
    #When uploading a file, specify the encryption method for ExtraArgs
    response = boto3.client('s3').upload_file(
        Filename='/tmp/' + file_name,
        Bucket='nanakenashi-test',
        Key=file_name,
        ExtraArgs={'ServerSideEncryption': 'AES256'})

    return True

It's very simple, with only more static arguments when uploading. At the moment (2017/09/16), it seems that the only encryption method is ʻAES256`. You can see that the actually saved object has the same shape as before.

SSE-KMS

When using the default key for S3

Rewrite only ʻExtraArgs` in the above script.

`SSE-KMS (using default key for S3)`


ExtraArgs={
    'ServerSideEncryption': 'aws:kms',
    }

The saved object is encrypted with the default key for S3. (Since this key was created at this timing, I think it is different from the key for SSE-S3)

However, this key is not much different from SSE-S3 because the setting cannot be changed.

When using a key created by KMS

`SSE-KMS (using the key created by KMS)`


#Add the ID of the key to use
ExtraArgs={
    'ServerSideEncryption': 'aws:kms',
    'SSEKMSKeyId': 'ea41458h-0c2o-496g-b92e-67441d771282'
    }

You can see that it is encrypted with the key created in advance.

Supplement

SSE-S3 and SSE-KMS /AmazonS3/latest/dev/UsingKMSEncryption.html)

With server-side encryption, only object data is encrypted. Object metadata is not encrypted.

Also

If you need server-side encryption for all objects stored in your bucket You can use bucket policies.

(It's difficult Japanese ...) In other words, it is possible to prohibit the saving of unencrypted objects. Setting this bucket policy makes it easier to maintain a secure state.

Summary

To make data storage in S3 more secure I tried several options for server-side encryption.

By the way, for SSE-C, refer to Around here. If you do, you can use it.