Manipulate riak from python

Overview

Work with riak 2.0.x from python 3.4. The python client documentation can be found at http://basho.github.io/riak-python-client/index.html.

Preparation

Install riak

You can install riak 2.0.4 with the brew command. The brew command seems to be behind the latest, so if you want to update it, please do it yourself.

% brew install riak

Make settings to change the file descriptor. It seems that it will not start unless these are implemented.

/Library/LaunchDaemons/limit.maxfiles.plist


<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd">
  <plist version="1.0">
    <dict>
      <key>Label</key>
        <string>limit.maxfiles</string>
      <key>ProgramArguments</key>
        <array>
          <string>launchctl</string>
          <string>limit</string>
          <string>maxfiles</string>
          <string>65536</string>
          <string>65536</string>
        </array>
      <key>RunAtLoad</key>
        <true/>
      <key>ServiceIPC</key>
        <false/>
    </dict>
  </plist>

/Library/LaunchDaemons/limit.maxproc.plist


<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE plist PUBLIC "-//Apple/DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd">
  <plist version="1.0">
    <dict>
      <key>Label</key>
        <string>limit.maxproc</string>
      <key>ProgramArguments</key>
        <array>
          <string>launchctl</string>
          <string>limit</string>
          <string>maxproc</string>
          <string>2048</string>
          <string>2048</string>
        </array>
      <key>RunAtLoad</key>
        <true />
      <key>ServiceIPC</key>
        <false />
    </dict>
  </plist>

Reboot the OS here.

Check the settings after startup.

% launchctl limit
cpu         unlimited      unlimited      
filesize    unlimited      unlimited      
data        unlimited      unlimited      
stack       8388608        67104768       
core        0              unlimited      
rss         unlimited      unlimited      
memlock     unlimited      unlimited      
maxproc     2048           2048           
maxfiles    65536          65536   

riak start

If you display pong, it's OK.

% riak start
% riak ping
pong

Change settings

Set allow_mult to true. First, check the location of the configuration file that was automatically generated at startup.

$ riak config effective | grep allow_mult
buckets.default.allow_mult = false

The above output information was false, so change it to true.

$ sudo echo "buckets.default.allow_mult = true" >> /usr/local/Cellar/riak/2.0.4/libexec/etc/riak.conf

Restart riak and make sure it is true.

$ riak restart
$ riak config effective | grep allow_mult
buckets.default.allow_mult = true

Install riak client

Install the official riak-python-client as well.

% pip install riak

Riak as KVS

Simple example

% ipython
import riak
client = riak.RiakClient()
bucket = client.bucket('manifestations')
obj = bucket.new('4873113938', data={
  'title': 'First Python 3rd Edition',
  'creator': 'Mark Lutz',
  'isbn': '4873113938'
}).store() # -> <riak.riak_object.RiakObject at 0x10a663898>

obj = bucket.new('4797372273', data={
  'title': 'Fun Ruby 4th Edition',
  'creator': 'Masayoshi Takahashi',
  'isbn': '4797372273'
}).store() # -> <riak.riak_object.RiakObject at 0x10a6b4470>

print(bucket.get('4873113938').data) # -> {'creator': 'Mark Lutz', 'title': 'First Python 3rd Edition', 'isbn': '4873113938'}

bucket.get_keys() # -> ['4873113938', '4797372273']

print(bucket.multiget(['4873113938','4873113938'])) # -> [<riak.riak_object.RiakObject object at 0x10a6b45f8>, <riak.riak_object.RiakObject object at 0x10a787a90>]

bucket.delete('4873113938')
print(bucket.get('4873113938').data) # -> None

bucket.get_keys() # -> ['4797372273']

Use the counter

A function that allows you to increase or decrease the numerical value. It helps to be standard. It's a mystery that you have to store () after incrementing ...

$ riak-admin bucket-type create counters '{"props":{"datatype":"counter"}}'
$ riak-admin bucket-type activate counters
$ ipython
import riak
from riak.datatypes import Counter
client = riak.RiakClient()
bucket = client.bucket_type('counters').bucket('manifestations')
counter = Counter(bucket, 'manifestations')
counter.value # -> 0
counter.increment()
counter.store()
counter.value # -> 1

Use full-text search function (Yokozuna)

Preparation

In the initial state, the search function (search) should be turned off, but let's check it for the time being. If off, change the setting and restart riak.

$ riak config effective | grep "search ="
search = off
$ sudo sed -e 's/search = off/search = on/' -i.back /usr/local/Cellar/riak/2.0.4/libexec/etc/riak.conf
$ riak restart
ok
$ riak ping
pong
$ riak config effective | grep "search ="
search = on

Setting

It's a hassle, but I'll set it.

% wget https://raw.githubusercontent.com/basho/yokozuna/develop/priv/default_schema.xml
% ipython
import riak
client = riak.RiakClient()
f = open('default_schema.xml')
content = f.read()
f.close()
schema_name = '_yz_default'
client.create_search_schema(schema_name, content)
client.create_search_index('manifestation_index','_yz_default')

Create a bucket-type.

$ riak-admin bucket-type create manifestation_index '{"props":{"search_index":"manifestation_index"}}'
manifestation_index created
$ riak-admin bucket-type activate manifestation_index
manifestations has been activated

I will try data registration and search.

$ ipython
import riak
client = riak.RiakClient()
bucket = client.bucket_type('manifestation_index').bucket('books')

book = bucket.new('4621088629', {'title_s': 'History of oriental astronomy', 'isbn_s': '4621088629', 'price_i': 1080, 'creator_s': 'Tsuko Nakamura', 'date_of_publication_dts': '2014-10-25T00:00:00Z'})
book.store()

book = bucket.new('4864102538', {'title_s': 'Space probe', 'isbn_s': '4864102538', 'price_i': 4320, 'creator_s': 'Philip Seguela', 'date_of_publication_dts': '2013-08-06T00:00:00Z'})
book.store()

book = bucket.new('4422420046', {'title_s': 'The most beautiful element picture book in the world', 'isbn_s': '4422420046', 'price_i': 4104, 'creator_s': 'Theodore Gray', 'date_of_publication_dts': '2010-10-22T00:00:00Z'})
book.store()

book = bucket.new('4087472221', {'title_s': 'The Summit of the Gods <above>', 'isbn_s': '4087472221', 'price_i': 788, 'creator_s': 'Baku Yumemakura', 'date_of_publication_dts': '2000-08-01T00:00:00Z'})
book.store()

book = bucket.new('408747223X', {'title_s': 'The Summit of the Gods <below>', 'isbn_s': '408747223X', 'price_i': 864, 'creator_s': 'Baku Yumemakura', 'date_of_publication_dts': '2000-08-01T00:00:00Z'})
book.store()

book = bucket.new('4163537406', {'title_s': 'Hostile waters', 'isbn_s': '4163537406', 'price_i': 119, 'creator_s': 'Peter Haksozen', 'date_of_publication_dts': '1998-01-01T00:00:00Z'})
book.store()

bucket.get_keys() # -> ['4422420046',
 '4163537406',
 '4087472221',
 '408747223X',
 '4621088629',
 '4864102538']

results = client.fulltext_search('manifestation_index', 'title_s:God*')
for doc in results['docs']:
  bucket = client.bucket_type(doc['_yz_rt']).bucket(doc['_yz_rb']) 
  b = bucket.get(doc['_yz_rk']) 
  print(b.data)
  
# -> {'creator_s': 'Baku Yumemakura', 'title_s': 'The Summit of the Gods <above>', 'price_i': 788, 'date_of_publication_dts': '2000-08-01T00:00:00Z', 'isbn_s': '4087472221'}
{'creator_s': 'Baku Yumemakura', 'title_s': 'The Summit of the Gods <below>', 'price_i': 864, 'date_of_publication_dts': '2000-08-01T00:00:00Z', 'isbn_s': '408747223X'}

results = client.fulltext_search('manifestation_index', 'price_i:[* TO 1000]')
for doc in results['docs']:
  bucket = client.bucket_type(doc['_yz_rt']).bucket(doc['_yz_rb']) 
  b = bucket.get(doc['_yz_rk']) 
  print(b.data)
  
# -> {'creator_s': 'Baku Yumemakura', 'title_s': 'The Summit of the Gods <above>', 'price_i': 788, 'date_of_publication_dts': '2000-08-01T00:00:00Z', 'isbn_s': '4087472221'}
{'creator_s': 'Baku Yumemakura', 'title_s': 'The Summit of the Gods <below>', 'price_i': 864, 'date_of_publication_dts': '2000-08-01T00:00:00Z', 'isbn_s': '408747223X'}
{'creator_s': 'Peter Haksozen', 'title_s': 'Hostile waters', 'price_i': 119, 'date_of_publication_dts': '1998-01-01T00:00:00Z', 'isbn_s': '4163537406'}  

Finally

ConflictError error

If you use the siblings property of the instance variable that is riak.riak_object.RiakObject, you will get multiple items in the list. In this case, delete it with the delete method and then re-register. I wish there was a way to delete all at once ...

Recommended Posts

Manipulate riak from python
sql from python
Manipulate BigQuery tables from a Python client
Use thingsspeak from python
Touch MySQL from Python 3
Operate Filemaker from Python
Use fluentd from python
Access bitcoind from python
Changes from Python 3.0 to Python 3.5
Changes from Python 2 to Python 3.0
Python from or import
Use MySQL from Python
Run python from excel
Install python from source
Operate neutron from Python!
Operate LXC from Python
Force Python from Fortran
Use BigQuery from python.
Execute command from python
[Python] Read From Stdin
Use mecab-ipadic-neologd from python
Manipulate excel files from python with xlrd (personal notes)
Flatten using Python yield from
Call CPLEX from Python (DO cplex)
Deep Python learned from DEAP
Post from Python to Slack
Grammar features added from Python3.6
Cheating from PHP to Python
Make MeCab available from Python3
Information obtained from tweet_id (Python)
OCR from PDF in Python
Run illustrator script from python
Use MySQL from Anaconda (python)
Anaconda updated from 4.2.0 to 4.3.0 (python3.5 updated to python3.6)
Study from Python Hour4: Object-oriented ②
Query Athena from Lambda Python
Access Oracle DB from Python
Study from Python Hour3: Functions
Manipulate spreadsheets locally using Python
Stop Omxplayer from Python code
Manipulate various databases with Python
Switch from python2.7 to python3.6 (centos7)
Connect to sqlite from python
Install pyenv from Homebrew, install Python from pyenv
Study from Python Hour4: Object-oriented ①
Python naming convention (from PEP8)
With skype, notify with skype from python!
Use e-Stat API from Python
Register redmine issue from Python
Call Matlab from Python to optimize
Execute Python script from batch file
Call a Python function from p5.js.
Python: Exclude tags from html data
Use Stanford Core NLP from Python
Touch a Python object from Elixir
Python
Hit treasure data from Python Pandas
Using Rstan from Python with PypeR
Get data from Quandl in Python
Install Python from source with Ansible
Create folders from '01' to '12' with python