The Data Store Connector (DSC) provided by Watson IoT Platform is excellent in that it can automatically collect information collected from IoT devices in an external database.
It can be set easily by using the Application SDK and sample code provided by IBM, but there is a feeling that there is a little lack of information in practical operation, so I will summarize the results of my own research.
Python 3.7 wiotp-sdk 0.11.0 (can be installed with pip3)
This time, Cloudant is used as the connection destination of DSC, and it is assumed that the following work has been completed. --Create an IBM Cloud account --Create and run Watson IoT Platform services --Create and run Cloudant services
There are many detailed writings on how to work, so I will omit them here. There are many in Qiita, so let's search.
Next, generate an authentication key and token for Watson IoT Platform. This is used for authentication when the code to be executed uses wiotp-sdk to call the API provided by Watson IoT Platform.
See below for more information. https://developer.ibm.com/jp/tutorials/iot-generate-apikey-apitoken/
In particular, tokens are displayed only when they are generated, so make a note of the character string printed on the screen.
Follow the steps below to check your username and password in your Cloudant service credentials. https://cloud.ibm.com/docs/Cloudant?topic=Cloudant-creating-an-ibm-cloudant-instance-on-ibm-cloud&locale=ja#locating-your-service-credentials
Note) It seems that it offers two types of authentication methods that are recently Cloudant, but the code handled in this article must be selected when "Use both legacy credentials and IAM" is selected when provisioning (creating) the service. Execution does not work.
The username and password confirmed here are passed to the Watson IoT Platform through the Service Binding of the Application SDK and are used for authentication when the Watson IoT Platform writes data to Cloudant.
The link below contains sample code to set up a DSC to connect to Cloudant.
https://www.ibm.com/support/knowledgecenter/SSQP8H/iot/platform/reference/dsc/cloudant.html
Rewrite the username and password in serviceBinding to the ones confirmed in 2.3 and execute.
# export WIOTP_AUTH_KEY='kkkkkkkkkkk'
# export WIOTP_AUTH_TOKEN='tttttttttttttttt'
# python sample.py
In the above execution example, the sample code (sample.py) is executed using the authentication key and token generated in 2.2. Set the authentication key and token in WIOTP_AUTH_KEY and WIOTP_AUTH_TOKEN respectively before executing the sample code.
By executing the sample code as in 2.4, all the events reported to Watson IoT Platform will be saved in the DB in Cloudant.
If you read the contents of the sample code a little more carefully,
--Generate Service Binding --Generate Connector by linking to Service Binding --Generate Destionation and Rule as Connector properties
You can see that some DSC objects are created from nothing like.
It will bring you from the state to the point where it works for the time being, but it seems that you will need a procedure to check or change the current configuration in order to use it on the premise of actual operation.
Below, I will try some steps that seem to be the minimum necessary to master DSC for the time being.
It is essential for operation to understand what the current settings are. I wrote a program to display a list of Service Bindings and DSCs.
ApplicationClient.ServiceBindings, ApplicationClient.dsc, etc. are looped by iterator, but be careful because the usability is different from the normal list. For example, when I try to check how many are stored using len (), 0 is returned.
ls_dsc.py
import wiotp.sdk.application
options = wiotp.sdk.application.parseEnvVars()
appClient = wiotp.sdk.application.ApplicationClient(options)
print("Service Bindings")
for s in appClient.serviceBindings:
print("ID: " + s.id)
print("")
print("DSC")
for c in appClient.dsc:
print("ID: " + c.id)
print("serviceId: " + c.serviceId)
for d in c.destinations:
print("Destination.name: " + d.name)
for r in c.rules:
print("Rules.id: " + r.id)
print("Rules.name: " + r.name)
print("Rules.destinatioName: " + r.destinationName)
print("Rules.typeId: " + r.typeId)
print("Rules.eventId: " + r.eventId)
Below are the results of running in my environment.
# export WIOTP_AUTH_KEY='kkkkkkkkkkkkkkkkkkk'
# export WIOTP_AUTH_TOKEN='tttttttttttttttttt'
# python ls_dsc.py
Service Bindings
ID: d2b141eb-2390-4fa0-b253-24b14c465298
ID: 5aa553e6-692c-45fd-ba2c-a717a635d5bb
ID: 85cb36b6-eda7-48b0-bc5a-6eb2c2f03619
ID: ae5a431d-98f6-4c9b-9ffc-a3ba9206770e
ID: 4f905b29-70ce-4f01-a2c6-9feb1c2d3e4a
ID: 137becb6-502a-4919-83da-079a434a7175
DSC
ID: 5db2d160f7e0960025221b51
serviceId: 4f905b29-70ce-4f01-a2c6-9feb1c2d3e4a
Destination.name: events
ID: 5db2d1b8f7e0960025221b52
serviceId: 137becb6-502a-4919-83da-079a434a7175
Destination.name: events
Rules.id: 5db2d1bbf7e0960025221b53
Rules.name: allevents
Rules.destinatioName: events
Rules.typeId: *
Rules.eventId: *
I modified the sample code of 2.4 so that only event notifications are sent to Cloudant, but as a result of trial and error in the middle, unnecessary Service Binding and DSC remain.
There are two DSCs, but the one listed first (ID = 5db2d160f7e0960025221b51) is the one whose Rules seem empty and do not work. It remains incomplete because the sample code terminated abnormally before the Rule was created. The other one (ID = 5db2d1b8f7e0960025221b52) has both Destination / Rules already created and works.
Six Service Bindings have been created, but the valid DSC uses the one pointed to by dsc.serviceId (ID = 137becb6-502a-4919-83da-079a434a7175), the remaining five. Is still unnecessary.
If you use this ls_dsc.py, you will be able to grasp the DSC setting status. Currently, only the minimum properties are displayed, but I think that you can add properties to be displayed as needed.
Now that it can be displayed, we will delete unnecessary objects.
I searched for a way to remove unwanted objects using the Python SDK, but couldn't find one. If you delete it with REST API, it's OK, so I will delete it directly using curl.
It seems that the Service Binding used by the DSC cannot be deleted, so delete the unnecessary DSC first, and then delete the unnecessary Service Binding.
See the links below for more information on the REST API. https://docs.internetofthings.ibmcloud.com/apis/swagger/v0002/historian-connector.html#/Services
For username / password used for authentication, specify the authentication key and token created in 2.2. Replace oooooo in the URL specified by the REST API with the orgnization id of Watson IoT Platform.
# curl -X DELETE --user 'kkkkkkkkkkkkkkkkkkk:tttttttttttttttttt' https://oooooo.internetofthings.ibmcloud.com/api/v0002/historianconnectors/5db2d160f7e0960025221b51
# curl -X DELETE --user 'kkkkkkkkkkkkkkkkkkk:tttttttttttttttttt' https://oooooo.internetofthings.ibmcloud.com/api/v0002/s2s/services/5aa553e6-692c-45fd-ba2c-a717a635d5bb
# curl -X DELETE --user 'kkkkkkkkkkkkkkkkkkk:tttttttttttttttttt' https://oooooo.internetofthings.ibmcloud.com/api/v0002/s2s/services/d2b141eb-2390-4fa0-b253-24b14c4652
# curl -X DELETE --user 'kkkkkkkkkkkkkkkkkkk:tttttttttttttttttt' https://oooooo.internetofthings.ibmcloud.com/api/v0002/s2s/services/85cb36b6-eda7-48b0-bc5a-6eb2c2f03619
# curl -X DELETE --user 'kkkkkkkkkkkkkkkkkkk:tttttttttttttttttt' https://oooooo.internetofthings.ibmcloud.com/api/v0002/s2s/services/ae5a431d-98f6-4c9b-9ffc-a3ba9206770e
# curl -X DELETE --user 'kkkkkkkkkkkkkkkkkkk:tttttttttttttttttt' https://oooooo.internetofthings.ibmcloud.com/api/v0002/s2s/services/4f905b29-70ce-4f01-a2c6-9feb1c2d3e4a
# export WIOTP_AUTH_KEY='kkkkkkkkkkkkkkkkkkk'
# export WIOTP_AUTH_TOKEN='tttttttttttttttttt'
# python ls_dsc.py
Service Bindings
ID: 137becb6-502a-4919-83da-079a434a7175
DSC
ID: 5db2d1b8f7e0960025221b52
serviceId: 137becb6-502a-4919-83da-079a434a7175
Destination.name: events
Rules.id: 5db2d1bbf7e0960025221b53
Rules.name: allevents
Rules.destinatioName: events
Rules.typeId: *
Rules.eventId: *
After deleting it, I ran ls_dsc.py again to check the result. Unnecessary Service Binding and DSC have been deleted and it is refreshing.
Now that we know that Service Binding and DSC can be deleted, it is possible to change various settings by "delete" and "create new". I was able to confirm the minimum functions required for installing and setting DSC, but it seems difficult to "delete" and "create new" every time, so I will also check how to update the configuration.
It is likely that you will want to change the configuration of the DSC once generated due to trial and error at the initial stage of implementation or specification changes. It seems that various properties specified when Service Binding, DSC, Destinations and Rules included in DSC are generated can be changed by using the update () method.
At this time, no document with specifications has been found. I just read the source code and found out. The code to change the Rule is shown below as an example, but please note that it may not work due to future changes in the Python SDK.
update_rule.py
import wiotp.sdk.application
options = wiotp.sdk.application.parseEnvVars()
appClient = wiotp.sdk.application.ApplicationClient(options)
# Get connector instance with its ID (=5db2d1b8f7e0960025221b52) as a key
c = appClient.dsc['5db2d1b8f7e0960025221b52']
# Update rule (ID=5db2d1bbf7e0960025221b53)
s = {
'deviceType' : '*',
'eventId' : 'bar'
}
c.rules.update(ruleId='5db2d1bbf7e0960025221b53', ruleType='event', name="allevents", destinationName="events", description="A sample rule", enabled=True, selector=s)
In the above example, only eventId is changed from'*' to'bar'.
For appClient.dsc, specify the DSC ID (5db2d1b8f7e0960025221b52 in this example) as the key to get the DSC instance corresponding to the ID. You can update the Rule by calling update () on the rules of this instance. Specify the ID of the Rule you want to update in the first argument of this update () method, and set the properties of the Rule you want to update in the rest. It seems that the property that is not updated is also called by setting the current value. It is a little difficult to specify the selector argument, but when I read the source code and the REST API specifications called by the Python SDK, it seems to set it like this.
Below are the execution results.
# export WIOTP_AUTH_KEY='kkkkkkkkkkkkkkkkkkk'
# export WIOTP_AUTH_TOKEN='tttttttttttttttttt'
# python ls_dsc.py
Service Bindings
ID: 137becb6-502a-4919-83da-079a434a7175
DSC
ID: 5db2d1b8f7e0960025221b52 <- !!!DSC ID
serviceId: 137becb6-502a-4919-83da-079a434a7175
Destination.name: events
Rules.id: 5db2d1bbf7e0960025221b53 <- !!!Rule ID
Rules.name: allevents
Rules.destinatioName: events
Rules.typeId: *
Rules.eventId: * <- !!!Change before
# python update_rule.py <- !!!Change
# python ls_dsc.py
Service Bindings
ID: 137becb6-502a-4919-83da-079a434a7175
DSC
ID: 5db2d1b8f7e0960025221b52
serviceId: 137becb6-502a-4919-83da-079a434a7175
Destination.name: events
Rules.id: 5db2d1bbf7e0960025221b53
Rules.name: allevents
Rules.destinatioName: events
Rules.typeId: *
Rules.eventId: bar <- !!!After change
I introduced how to check / delete / change DSC settings using the Python SDK.
Again, there are some methods that IBM does not explicitly guide, and all use of the code illustrated in this article is at your own risk. Also note that future changes to the Python SDK implementation may cause these codes to stop working.
The surest thing is to implement it using the REST API, but using the Python SDK is simpler and easier to code. I hope that the Python SDK will evolve into a more user-friendly form in the future.
The following Qiita article may be helpful for how to set using the REST API, so if you are interested, please refer to it. https://qiita.com/Motonaga/items/6304f5f66f63cb566943