Python code for writing CSV data to DSX object storage

A reminder about how to read files on object storage from Python code on a Notebook in one of Watson Data Platform's core services, Data Science Experience (https://datascience.ibm.com/) (DSX). This is a sequel to Python code for reading CSV data from DSX object storage, and this time it is the code for writing.

In the case of the previous reading, the data to be read was clear and the authentication information was obtained from that data, but this time the purpose is to draw the result of the calculation process, so there is a container corresponding to the project. Even so, I cannot get the credentials from the file name. So what should I do?

When I log in to DSX and look for credentials to write to the container from the Data Service, I can't find it anywhere. However, you can get container credentials by logging in from Bluemix PaaS and looking for object storage information.

From the list of services, go to DSX-Object Storage. スクリーンショット 2017-04-20 14.00.23.png

When the DSX-ObjectStorage service credential is displayed, the service credential is created for each container, that is, for each project, so copy and paste the service credential of the corresponding container to the DSX Notebook to use it. スクリーンショット 2017-04-20 14.09.15.png

The service credentials on the Bluemix side don't include the container name, so just add that part manually and you're ready to go.

from io import BytesIO  
import requests
import numpy as np
import json


# Object storage credentials
credentials_5 = {
 'auth_url':'https://identity.open.softlayer.com',
 'project':'object_storage_bc6cdc85_586e_4581_8a09_8f01f7bdf3ed',
 'project_id':'2a9de4c1d50944a49f1a46dd53394158',
 'region':'dallas',
 'user_id':'********************************',
 'domain_id':'fb119f3e1bc0469dad2b253b317ec7ea',
 'domain_name':'952993',
 'username':'***********************************************',
 'password':"********************",
'container':'DefaultProjecttakarajpibmcom' <--- Add by hand
}


def Write_CSV_to_ObjectStorage(credentials, npdata,label, fileName):  
   """This functions returns a StringIO object containing
   the file content from Bluemix Object Storage V3."""

   csvData = ""
# Label data written to the first line of CSV
   c = 0
   for col in label:
       if c > 0:
           csvData = csvData + ','
       csvData = csvData + col
       c = c + 1
   csvData = csvData + "\n"
   
# Convert Numpy array data to CSV format text data
   rows, cols = np.shape(npdata)
   for row in a:
       c = 0
       for col in row:
           if c > 0:
               csvData = csvData + ','
           csvData = csvData + str(col)
           c = c + 1
       csvData = csvData + "\n"
   
#Object storage authentication
   url1 = ''.join(['https://identity.open.softlayer.com', '/v3/auth/tokens'])
   data = {'auth': {'identity': {'methods': ['password'],
           'password': {'user': {'name': credentials['username'],'domain': {'id': credentials['domain_id']},
           'password': credentials['password']}}}}}
   headers1 = {'Content-Type': 'application/json'}
   resp1 = requests.post(url=url1, data=json.dumps(data), headers=headers1)
#Exit when authentication error occurs
   if resp1.status_code != 201:
       return False, resp1.status_code
   resp1_body = resp1.json()

# Write to object storage
   for e1 in resp1_body['token']['catalog']:
       if(e1['type']=='object-store'):
           for e2 in e1['endpoints']:
               if(e2['interface']=='public'and e2['region']=='dallas'):
                   url2 = ''.join([e2['url'],'/', credentials['container'], '/', fileName])
   s_subject_token = resp1.headers['x-subject-token']
   headers2 = {'X-Auth-Token': s_subject_token, 'accept': 'application/json'}
   resp2 = requests.put(url=url2, headers=headers2, data = csvData )
   result = True
   if resp2.status_code != 201:
       print "ERROR ", resp2.status_code
       result = False
   return result, resp2.status_code
   

# Main

# Data generation
x = np.linspace(0,1,60)
y = np.linspace(10,11,60)

# Two-dimensional array of changes (x, y) to a single array
a = np.stack((x, y), axis=-1)
print a
# Data label
label = ["Time","Data"]

# Save to object storage
rslt = Write_CSV_to_ObjectStorage(credentials_5, a, label,"testDataSet.csv")
print rslt
ew

Explanation of how to use

The first argument is the credentials. The second argument is an array of num.py in the data array and will be inserted in the first line of the CSV file with the label of the data column. Next is the file name (the name of the object stored in the object storage container).

rslt = Write_CSV_to_ObjectStorage(credentials_5, a, label,"testDataSet.csv")

The return value will be 201 Created if successful. Otherwise it will fail.

Recommended Posts

Python code for writing CSV data to DSX object storage
Reading CSV data from DSX object storage Python code
Try writing JSON format data to object storage Cloudian/S3
Python> Output numbers from 1 to 100, 501 to 600> For csv
How to use "deque" for Python data
[Cloudian # 1] Try to access object storage with AWS SDK for Python (boto3)
How to access data with object ['key'] for your own Python class
Writing logs to CSV file (Python, C language)
Write CSV data to AWS-S3 with AWS-Lambda + Python
Character code for reading and writing csv files with python ~ windows environment ver ~
Rewrite Python2 code to Python3 (2to3)
Before writing Python code
Use data class for data storage of Python 3.7 or higher
[For beginners] How to study Python3 data analysis exam
[Introduction for beginners] Reading and writing Python CSV files
[Python] Change the Cache-Control of the object uploaded to Cloud Storage
Python code memo for yourself
How to specify Cache-Control for blob storage in Azure Storage in Python
[Python / Ruby] Understanding with code How to get data from online and write it to CSV
Stylish technique for pasting CSV data into Excel with Python
Write to csv with Python
Python for Data Analysis Chapter 2
[Python] Sample code for Python grammar
[Introduction to Python] Combine Nikkei 225 and NY Dow csv data
[Python] Introduction to graph creation using coronavirus data [For beginners]
~ Tips for beginners to Python ③ ~
Introduction to Python For, While
A python script that converts Oracle Database data to csv
Convert python 3.x code to python 2.x
Python for Data Analysis Chapter 3
Python environment construction 2016 for those who aim to be data scientists
How to configure CORS settings for Azure storage service in Python
[Cloudian # 2] Try to display the object storage bucket in Python (boto3)
[Python] How to store a csv file as one-dimensional array data
Prolog Object Orientation for Python Programmers
[Python] How to FFT mp3 data
[Python] Write to csv file with Python
Minimum grammar notes for writing Python
Output to csv file with Python
Python course for data science_useful techniques
VS Code snippets for data analysts
Notes for me python csv graph
Python CSV file reading and writing
Preprocessing template for data analysis (Python)
A note for writing Python-like code
Data formatting for Python / color plots
Reading and writing CSV with Python
Easy Python to learn while writing
An introduction to Python for non-engineers
File upload to Azure Storage (Python)
[Cloudian # 3] Try to create a new object storage bucket with Python (boto3)
[Data science basics] I tried saving from csv to mysql with python
Data analysis: Easily apply descriptive and inference statistics to CSV data in Python
Copy data from Amazon S3 to Google Cloud Storage with Python (boto)
Error due to UnicodeDecodeError when reading CSV file with Python [For beginners]
Cloud Pak for Data object operation example in Python (WML client, project_lib)
Tips for Python beginners to use the Scikit-image example for themselves 6 Improve Python code
Try to extract specific data from JSON format data in object storage Cloudian/S3
[Python] How to convert db file to csv
Data input / output in Python (CSV, JSON)
Python visualization tool for data analysis work