[Cloudian # 8] Try setting the bucket versioning with Python (boto3)
Introduction
Cloudian is fully compatible with S3, so it's convenient to use the AWS SDK! Last time tried deleting the bucket in Python (boto3).
This time, I will use Python (boto3) to set the versioning of the object storage bucket.
Bucket versioning settings
Enable the versioning function and check its operation. It is assumed that the bucket "pythonbucket2" has already been created and exists.
1. Enable bucket versioning / put_bucket_versioning ()
Pass the bucket name to change the versioning state and the state to change (here ‘Enabled’) to the argument of put_bucket_versioning () and execute it to enable the versioning function.
In the following example, the status of the versioning function of the bucket "pythonbucket2" is changed to Enabled ('Status': ‘Enabled’).
test1.py
import boto3
client = boto3.client(
's3',
endpoint_url='https://xxx.yyy.com'
)
#Bucket name:Enable versioning feature for pythonbucket2
client.put_bucket_versioning(
Bucket='pythonbucket2',
VersioningConfiguration={'Status': 'Enabled'}
)
{'ResponseMetadata': {'RequestId': '9dad3e00-0e30-1dbc-a754-06bdfcde1d5e',
'HostId': '',
'HTTPStatusCode': 200,
'HTTPHeaders': {'date': 'Sun, 13 Dec 2020 22:30:33 GMT',
'x-amz-request-id': '9dad3e00-0e30-1dbc-a754-06bdfcde1d5e',
'content-length': '0',
'server': 'CloudianS3'},
'RetryAttempts': 0}}
** Caution ** Once the bucket versioning feature is enabled, it cannot be disabled thereafter. If you want to suspend the versioning function, pass "Suspended" to the Status of VersioningConfiguration and execute put_bucket_versioning () as described later.
2. Check the versioning status of the bucket / get_bucket_versioning ()
Execute it by passing the bucket name to check the versioning status to the argument of get_bucket_versioning ().
In the following example, the versioning state of the bucket "pythonbucket2" is acquired.
test1.py
import boto3
client = boto3.client(
's3',
endpoint_url='https://xxx.yyy.com'
)
#Bucket name:Get versioning state of pythonbucket2
client.get_bucket_versioning(Bucket='pythonbucket2')
{'ResponseMetadata': {'RequestId': '9dad3e02-0e30-1dbc-a754-06bdfcde1d5e',
'HostId': '',
'HTTPStatusCode': 200,
'HTTPHeaders': {'date': 'Sun, 13 Dec 2020 22:30:36 GMT',
'x-amz-request-id': '9dad3e02-0e30-1dbc-a754-06bdfcde1d5e',
'content-type': 'application/xml;charset=UTF-8',
'content-length': '161',
'server': 'CloudianS3'},
'RetryAttempts': 0},
'Status': 'Enabled'}
3. Upload the file to a versioned bucket
Try uploading the file multiple times with the same key to the versioned bucket using upload_file () from S3 Transfers above.
In the following example, the file is uploaded to the bucket "pythonbucket2" with the versioning function enabled four times in total with "10mb.dat" set as the key.
upload1.py
import boto3
from boto3.s3.transfer import S3Transfer
from boto3.s3.transfer import TransferConfig
client = boto3.client(
's3',
endpoint_url='https://xxx.yyy.com'
)
config = TransferConfig(
multipart_threshold = 8 * 1024 * 1024,
max_concurrency = 10,
multipart_chunksize = 8388608,
num_download_attempts = 10,
max_io_queue = 100
)
#Creating an S3Transfer object
transfer = S3Transfer(client, config)
#【First time】
transfer.upload_file('fileup/10mb.dat', 'pythonbucket2', '10mb.dat')
#【Second time】
transfer.upload_file('fileup/10mb.dat', 'pythonbucket2', '10mb.dat',
extra_args={
'ACL': 'public-read',
'Metadata': {
'Purpose': 'boto3 demo 1',
'Engineer': 'yamahiro',
'Company': 'nw'
},
'ServerSideEncryption': 'AES256'
}
)
#【Third time】
transfer.upload_file('fileup/10mb.dat', 'pythonbucket2', '10mb.dat',
extra_args={
'ACL': 'public-read',
'Metadata': {
'Purpose': 'boto3 demo 2',
'Engineer': 'miki',
'Company': 'nw'
},
'ServerSideEncryption': 'AES256'
}
)
#【4th】
transfer.upload_file('fileup/10mb.dat', 'pythonbucket2', '10mb.dat',
extra_args={'ServerSideEncryption': 'AES256'}
)
Add user-defined metadata when uploading extra_args = {Metadata': {'name':'value', .....}}
Specify encryption (AES256) when uploading 'ServerSideEncryption': 'AES256'
4. List versioned objects / list_object_versions ()
To list the versioned (not currently the latest version) objects (files), run list_object_versions ().
Display without processing
In the following example, the bucket name in the argument of list_object_versions () Only (Bucket ='pythonbucket2') is set and all return values are displayed.
client.list_object_versions(Bucket='pythonbucket2')
{'ResponseMetadata': {'RequestId': '9dad3e3e-0e30-1dbc-a754-06bdfcde1d5e',
'HostId': '',
'HTTPStatusCode': 200,
'HTTPHeaders': {'date': 'Sun, 13 Dec 2020 22:35:45 GMT',
'x-amz-request-id': '9dad3e3e-0e30-1dbc-a754-06bdfcde1d5e',
'x-gmt-policyid': 'b46db4b3ebb2180b046ad1065c9702e1',
'x-amz-bucket-region': 'region1',
'content-type': 'application/xml;charset=UTF-8',
'content-length': '1674',
'server': 'CloudianS3'},
'RetryAttempts': 0},
'IsTruncated': False,
'KeyMarker': '',
'VersionIdMarker': '',
'Versions': [{'ETag': '"669fdad9e309b552f1e9cf7b489c1f73-2"',
'Size': 10485760,
'StorageClass': 'STANDARD',
'Key': '10mb.dat',
'VersionId': 'fe14c26d-16ea-60bf-a754-06bdfcde1d5e',
'IsLatest': True,
'LastModified': datetime.datetime(2020, 12, 13, 22, 31, 14, 996000, tzinfo=tzutc()),
'Owner': {'ID': '27b8e84694ca0b529d5379049564ebe1'}},
{'ETag': '"669fdad9e309b552f1e9cf7b489c1f73-2"',
'Size': 10485760,
'StorageClass': 'STANDARD',
'Key': '10mb.dat',
'VersionId': 'fe14c26d-18b5-d1ef-a754-06bdfcde1d5e',
'IsLatest': False,
'LastModified': datetime.datetime(2020, 12, 13, 22, 31, 11, 985000, tzinfo=tzutc()),
'Owner': {'ID': '27b8e84694ca0b529d5379049564ebe1'}},
{'ETag': '"669fdad9e309b552f1e9cf7b489c1f73-2"',
'Size': 10485760,
'StorageClass': 'STANDARD',
'Key': '10mb.dat',
'VersionId': 'fe14c26d-1a6a-119f-a754-06bdfcde1d5e',
'IsLatest': False,
'LastModified': datetime.datetime(2020, 12, 13, 22, 31, 9, 126000, tzinfo=tzutc()),
'Owner': {'ID': '27b8e84694ca0b529d5379049564ebe1'}},
{'ETag': '"669fdad9e309b552f1e9cf7b489c1f73-2"',
'Size': 10485760,
'StorageClass': 'STANDARD',
'Key': '10mb.dat',
'VersionId': 'fe14c26d-1e9b-382f-a754-06bdfcde1d5e',
'IsLatest': False,
'LastModified': datetime.datetime(2020, 12, 13, 22, 31, 2, 93000, tzinfo=tzutc()),
'Owner': {'ID': '27b8e84694ca0b529d5379049564ebe1'}}],
'Name': 'pythonbucket2',
'Prefix': '',
'MaxKeys': 1000,
'EncodingType': 'url'}
Since list_object_versions () returns so much information, the following examples will narrow down the output.
Narrow down to display information about versioned objects
In the following example, only the bucket name (Bucket ='pythonbucket2') is set in the argument of list_object_versions (), and the output is narrowed down to the key, version ID, and last modification date and time.
for version in client.list_object_versions(Bucket='pythonbucket2')['Versions']:
print(version['Key'],version['VersionId'], version['LastModified'].strftime("%Y/%m/%d %H:%M:%S"))
10mb.dat fe14c26d-16ea-60bf-a754-06bdfcde1d5e 2020/12/13 22:31:14
10mb.dat fe14c26d-18b5-d1ef-a754-06bdfcde1d5e 2020/12/13 22:31:11
10mb.dat fe14c26d-1a6a-119f-a754-06bdfcde1d5e 2020/12/13 22:31:09
10mb.dat fe14c26d-1e9b-382f-a754-06bdfcde1d5e 2020/12/13 22:31:02
5. Pause bucket versioning/put_bucket_versioning ()
Pass the bucket name to change the versioning state and the state to change (here ‘Suspended’) to the argument of put_bucket_versioning (), and execute it to suspend the versioning function.
In the following example, the versioning function of the bucket "python bucket 1" is suspended (VersioningConfiguration = {'Status':'Suspended'}).
client.put_bucket_versioning( Bucket='pythonbucket2', VersioningConfiguration={'Status': 'Suspended'} )
{'ResponseMetadata': {'RequestId': '9dad3e02-0e30-1dbc-a754-06bdfcde1d5e',
'HostId': '',
'HTTPStatusCode': 200,
'HTTPHeaders': {'date': 'Sun, 13 Dec 2020 22:30:36 GMT',
'x-amz-request-id': '9dad3e02-0e30-1dbc-a754-06bdfcde1d5e',
'content-type': 'application/xml;charset=UTF-8',
'content-length': '161',
'server': 'CloudianS3'},
'RetryAttempts': 0},
'Status': 'Enabled'}
After executing the above put_bucket_versioning (), the status of the versioning function of the bucket "pythonbucket1" is displayed, and it is confirmed that the versioning function is paused.
client.get_bucket_versioning(Bucket='pythonbucket2')
{'ResponseMetadata': {'RequestId': '9dad3e44-0e30-1dbc-a754-06bdfcde1d5e',
'HostId': '',
'HTTPStatusCode': 200,
'HTTPHeaders': {'date': 'Sun, 13 Dec 2020 22:36:23 GMT',
'x-amz-request-id': '9dad3e44-0e30-1dbc-a754-06bdfcde1d5e',
'content-type': 'application/xml;charset=UTF-8',
'content-length': '163',
'server': 'CloudianS3'},
'RetryAttempts': 0},
'Status': 'Suspended'}
Summary
I tried to set the versioning of the bucket with Python (boto3).
Next time, I would like to operate object storage/Cloudian in various ways with Python.
Recommended Posts