Upload files to Aspera that comes with IBM Cloud Object Storage (ICOS) using SDK (Python version)
How to upload a file to IBM Cloud Object Storage (ICOS). This time, we will operate Aspera that comes with ICOS using the Python version of the SDK.
Environment used
- macOS Mojave 10.14.6、Python2.7.10
- Windows7 Ultimate、Python2.7.16
3 series is fine, but currently the SDK only supports up to 3.6, so I checked with 2 series.
Setting up the Python SDK
Please set up referring to here. GitHub：IBM-Cloud/data-lake/upload/cos-upload
After setting up pip, run install.sh, For the Windows version, I copied the contents of the file directly and ran the command.
install.sh
pip install --upgrade pip
pip install --upgrade setuptools
pip install "requests>=2.22.0,<2.23.0" "ibm-cos-sdk>=2.5.4" "cos-aspera>=0.1.163682"
IQOS information confirmation
Log in to the IBM Cloud console (web) to verify your ICOS service credentials.
The service credentials can be found in the bucket credentials created in ICOS.
Run
The command example is as follows.
cos-upload.py <endpoint> <apikey> <bucket_name> <prefix> <file>
Run on Mac
$ python cos-upload.py s3.jp-tok.cloud-object-storage.appdomain.cloud "api-key" test-bucket "" test-upload-python-mac.txt
Initialize COS and Aspera Transfer Manager using endpoint: https://s3.jp-tok.cloud-object-storage.appdomain.cloud
Upload file to COS: /tmp/test-upload-python-mac.txt => test-upload-python-mac.txt
Upload file to COS completed.
Run on Windows
C:\tmp>py -2 D:\Workspace\cos-upload.py s3.jp-tok.cloud-object-storage.appdomain.cloud "api-key" test-bucket "" test-upload-python-win.txt
Initialize COS and Aspera Transfer Manager using endpoint: https://s3.jp-tok.cloud-object-storage.appdomain.cloud
Upload file to COS: C:\tmp\test-upload-python-win.txt => test-upload-python-win.txt
Upload file to COS completed.
Arrival confirmation
Confirm that the upload was successful with the ibmcloud command.
//Confirm that it was uploaded successfully
$ ibmcloud cos list-objects --bucket test-bucket
OK
Found 3 There are objects in the bucket'test-bucket':
Name Last modified date and time Object size
test.txt Feb 28, 2020 at 02:27:12 14 B
test-upload-python-mac.txt Mar 05, 2020 at 20:05:11 20 B
test-upload-python-win.txt Mar 05, 2020 at 21:21:50 20 B
Afterword
I think the points of the Python version are "construction of execution environment" and "understanding the concept of parameter specification at runtime".
I was running on 3.7 series without noticing Require for a while, so I was having trouble with install.sh, but when I implemented it on 2.7 series, it was solved brilliantly, so I lost time! I felt that. Also, I don't think there are many cases where Python runs on Windows, but I tried it because it was a big deal, and it worked, so I added it.
The run-time parameters <prefix> and <file> are songwriters, and as you can see, the file wasn't uploaded as intended at first, which was a bit difficult to understand.
For example, if you specify the full path for <file>,
/tmp/upload/test.txt Mar 03, 2020 at 21:21:50 20 B
A file with the path name will be uploaded.
So, moving to the directory of the file you want to send and calling cos-upload.py from there does not have to be tricky, so it may be easier to do. I think.
Reference link
--Upload and publish files to IBM Cloud Object Storage (ICOS) with curl command -Upload files to IBM Cloud Object Storage (ICOS) using CLI (ibmcloud command) -Upload files to Aspera that comes with IBM Cloud Object Storage (ICOS) using SDK (Java version)