[Introduction to Python3 Day 19] Chapter 8 Data Destinations (8.4-8.5)

8.4 NoSQL data store

--Created for the purpose of processing very large datasets, flexible data definitions, support for custom data processing, etc.

8.4.1 dbm family

--The dbm format is a key-value store (multiple key / value pairs) --The nature of dbm datastores --A value can be assigned to the key, and the assigned value is automatically saved in the DB on the disk. --You can get the value from the key.

#"c"The mode is literate
>>> db=dbm.open("definitions","c")
#Assign values to keys just like a dictionary.
>>> db["mustard"]="yellow"
>>> db["ketchup"]="red"
>>> db["pesto"]="green"
>>> db.close
<built-in method close of _dbm.dbm object at 0x101bb0290>
>>> db.close()
#Check if the rewritten version is actually saved.
>>> db=dbm.open("definitions","r")
#Keys and values are stored as bytes.
>>> db["mustard"]
b'yellow'

8.4.2 memcached

--A fast in-memory cache server for keys and values. --Often used as DB pre-processing or for storing web server session data. --After connecting, you can do the following. --Get and set the value with the specified key --Value increment, decrement --Key deletion --Data is not persistent and may disappear from the oldest. ** Prevents throwing away old data and running out of memory. ** **

8.4.3 Redis

--As a general rule, store it in memory on the data structure server. --The following points are different from memcached. --Data can be saved on disk, so old data can be kept. --There are also data structures other than simple strings.

8.4.3.1 String

error

>>> import redis
#Connect to the Redis server.
>>> conn=redis.Redis()
>>> conn.keys("*")
#An error has occurred.

――I got the following error and investigated it for about an hour. ⇨ ** The error was dealt with by installing redis with Homebrew. ** **

Workaround

During handling of the above exception, another exception occurred:

$ brew search redis
==> Successfully started `redis` (label: homebrew.mxcl.redis)

result

>>> import redis
>>> conn = redis.Redis()
>>> conn.keys()
[]
#set()Write value by.
>>> conn.set("secret","ni!")
True
>>> conn.set("caratst","24")
True
>>> conn.set("fever","101.5")
True

#Read the value using the key.
>>> conn.get("secret")
b'ni!'
>>> conn.get("caratst")
b'24'
>>> conn.get("fever")
b'101.5'

#setnx()The method sets the value unless the key exists.
>>> conn.setnx("secret","icky-icky-icky-ptang-zoop-boing!")
False
>>> conn.get("secret")
b'ni!'

#getset()The method returns the original value and sets the new value.
>>> conn.getset("secret","icky-icky-icky-ptang-zoop-boing!")
b'ni!'
>>> conn.get("secret")
b'icky-icky-icky-ptang-zoop-boing!'

#getrange()Extracts a substring.
>>> conn.getrange("secret",-6,-1)
b'boing!'

#setarrange()Replaces a substring.
>>> conn.setrange("secret",0,"ICKY")
32
>>> conn.get("secret")
b'ICKY-icky-icky-ptang-zoop-boing!'

#mset()Use to get multiple keys at the same time.
>>> conn.mset({"pie":"cherry","cordial":"sherry"})
True

#mget()Use to get multiple values at once.
>>> conn.mget(["fever","caratst"])
[b'101.5', b'24']

#The key is deleted.
>>> conn.delete("fever")
1
>>> conn.incr("caratst")
25
>>> conn.decr("caratst",10)
15
>>> conn.set("caratst","24")
True
>>> conn.incr("caratst",10)
34
>>> conn.decr("caratst")
33
>>> conn.decr("caratst",15)
18
>>> conn.set("fever","101.5")
True
>>> conn.incrbyfloat("fever")
102.5
>>> conn.incrbyfloat("fever",0.5)
103.0

#decrbyfloat()Not, but a negative increment.
>>> conn.incrbyfloat("fever",-2.0)
101.0

8.4.3.2 List

--Redis list can only store strings.

#Use lpush to insert at the beginning.
>>> conn.lpush("zoo","bear")
1
#Insert multiple elements at the beginning.
>>> conn.lpush("zoo","alligator","duck")
3
#linsert()Insert before the value using.
>>> conn.linsert("zoo","before","bear","behavor")
4
#linsert()Insert after the value using.
>>> conn.linsert("zoo","after","bear","aaaar")
5
#lset()Insert at the specified position using.
>>> conn.lset("zoo",2,"marmoset")
True

#rpush()Insert at the end using.
>>> conn.rpush("zoo","marmoset")
6
#lindex()To get the offset value.
>>> conn.lindex("zoo",3)
b'bear'
#lrange()Get the value of the offset range specified using.
>>> conn.lrange("zoo",0,2)
[b'duck', b'alligator', b'marmoset']
#ltrim()Pruning the list using. Only the elements in the specified offset range remain.
>>> conn.ltrim("zoo",1,4)
True
>>> conn.lrange("zoo",0,-1)
[b'alligator', b'marmoset', b'bear', b'aaaar']

8.4.3.3 hash

--Redis hashes are very similar to Python dictionaries, but can only store strings.

#hmset()Do to song hash using,Set the re field at the same time.
>>> conn.hmset("song",{"do":"aA","re":"About a deer"})
True
#hset()Use to set the value of one field to a field in the hash.
>>> conn.hset("song","mi","a note to follow me")
1
#hget()To get the value of one field.
>>> conn.hget("song","mi")
b'a note to follow me'
#hmget()Use to get the values of multiple fields.
>>> conn.hmget("song","re","do")
[b'About a deer', b'aA']
#hkeys()Get the keys for all fields of the hash using
>>> conn.hkeys("song")
[b'do', b're', b'mi']
#havals()Get the values of all fields in the hash using
>>> conn.hvals("song")
[b'aA', b'About a deer', b'a note to follow me']
#Get the number of fields in the hash
>>> conn.hlen("song")
3
#hegetall()Use to get the values and keys of all the fields in the hash.
>>> conn.hgetall("song")
{b'do': b'aA', b're': b'About a deer', b'mi': b'a note to follow me'}
#hsetnx()Use to set the field if you don't already have a key.
>>> conn.hsetnx("song","fa","asdf")
1

8.4.3.4 Set

--The Redis set is very similar to the Python set.

#Add one or more values to the set.
>>> conn.sadd("zoo2","a","b","c")
3
#Get the number of values in the set.
>>> conn.scard("zoo2")
3
#Get all the values of the set.
>>> conn.smembers("zoo2")
{b'a', b'c', b'b'}
#Remove the value of the set.
>>> conn.srem("zoo2","c")
1

#Set creation
>>> conn.sadd("zoo3","ni","b","a")
3
#Get the intersection.
>>> conn.sinter("zoo2","zoo3")
{b'a', b'b'}
#Get the intersection and set fowl_Store the result in zoo.
>>> conn.sinterstore("fowl_zoo","zoo2","zoo3")
2
>>> conn.smembers("fowl_zoo")
{b'a', b'b'}

#Creating a union
>>> conn.sunion("zoo2","zoo3")
{b'a', b'b', b'ni'}
#Create a union and set the results f_Store in zoo.
>>> conn.sunionstore("f_zoo","zoo2","zoo3")
3
>>> conn.smembers("f_zoo")
{b'a', b'b', b'ni'}

#sdiff()Create a difference set using.(Things that are in zoo2 but not in zoo3)
>>> conn.sdiff("zoo2","zoo3")
{b'c'}
#Create a difference set and set the results zoo_Store in sale.
>>> conn.sdiffstore("zoo_sale","zoo2","zoo3")
1
>>> conn.smembers("zoo_sale")
{b'c'}

8.4.3.5 Sorted set

--The most applicable Redis data types are sorted sets or zsets. --zset is a unique set of values, but each value also has a floating point number called ** score **.

--I checked for the error'str'object has no attribute'items' but couldn't resolve it. I did the following to resolve it.

--When I checked, I saw on some sites that the error was resolved by upgrading setup-tools, so I went.

result

>>> import time
>>> now=time.time()
>>> now
1579936585.194324
>>> conn.zadd("logins","smeagol",now+(5*60))
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File 
    return iter(x.items())
AttributeError: 'str' object has no attribute 'items'

Practice

$ pip install --upgrade 
Successfully installed setuptools-45.1.0

8.4.3.6 bits

>>> days=["2013-02-24","2013-02-25","2013-02-26"]
#Give the user ID.
>>> big_spender=1089
>>> tire_kicker=40459
>>> late_joiner=550212
#Set the bit for the user ID logged in that day.
>>> conn.setbit(days[0],big_spender,1)
0
>>> conn.setbit(days[0],tire_kicker,1)
0
>>> conn.setbit(days[1],big_spender,1)
0
>>> conn.setbit(days[2],big_spender,1)
0
>>> conn.setbit(days[2],late_joiner,1)
0
#Who are the visitors on each day of the three days?
>>> for day in days:
...     conn.bitcount(day)
... 
2
1
2
>>> conn.getbit(days[1],tire_kicker)
0
>>> conn.getbit(days[1],big_spender)
1

#How many users log in every day?
>>> conn.bitop("and","everyday",*days)
68777
>>> conn.bitcount("everyday")
1
#Who is it? → big_spender
>>> conn.getbit("everyday",big_spender)
1
>>> conn.getbit("everyday",tire_kicker)
0
>>> conn.bitop("or","alldays",*days)
68777
>>> conn.bitcount("alldays")
3

8.4.3.7 Cache and expiration date

--All Redis keys have a lifetime and expiration date. (By default, these are forever.) --You can use the expire () function to tell Redis how long to keep the key. (Value is in seconds) --The expireat () command invalidates the key at the specified Unix time.

>>> import time
>>> key="now you see it"
>>> conn.set(key,"but not fot long")
True
>>> conn.expire(key,5)
True
>>> conn.ttl(key)
-2
>>> conn.get(key)
>>> time.sleep(6)
>>> conn.get(key)
>>> 

8.5 Review assignment

8-1 Substitute the string "This is a test of the emergency text system" into the variable test1 and write the contents of test1 to a file called text.txt.

>>> test1="""This is a test of the emergency text system"""
>>> fout=open("text.txt","wt")
>>> fout.write(test1)
43
>>> fout.close()

8-2 Open a file called text.txt and read its contents into the test2 variable. Are test1 and test2 the same?

>>> with open("text.txt","rt") as infile:
...     test2=infile.read()
... 
>>> test1==test2
True

8-3 Save the following text in a file called books.csv.

>>> a="""auther,book
... J R R Tolkien,The Hobbit
... Lynne Truss,"Eats, Shoots & Leaves"
... """

8-4 Use the csv module and its DictReader method to read the contents of books.csv into the books variable and display the contents.

>>> import csv
>>> with open("books.csv","rt") as fout:
...     books=csv.DictReader(fout)
...     for book in books:
...         print(book)
... 
OrderedDict([('auther', 'J R R Tolkien'), ('book', 'The Hobbit')])
OrderedDict([('auther', 'Lynne Truss'), ('book', 'Eats, Shoots & Leaves')])

8-5 Create a CSV file called books.csv using the following lines.

>>> text="""title,author,year
... The Weirdstone of Brisingamen,Alan Garner,1960
... Perdido Street Station,China Miéville,2000
... Thud!,Terry Pratchett,2005
... The Spellman Files,Lisa Lutsz,2007
... Small Gods,Terry Pratchett,1992
... """

>>> with open("books.csv","wt") as fout:
...     fout.write(text)
... 
202

8-6 Create SQLiteDB called books.db using sqlite3 module, and create table called book with fields title, author, book in it.

>>> import sqlite3
>>> conn=sqlite3.connect("books.db")
>>> curs=conn.cursor()
>>> curs.execute("""CREATE TABLE book(title text,author text,year int)""")
<sqlite3.Cursor object at 0x1029e8960>
>>> db.commit()

8-7 Read books.csv and insert the data into the book table.

>>> import csv
>>> import sqlite3
>>> ins_str ="insert into book values(?,?,?)"
>>> with open("books.csv","rt") as infile:
...     books=csv.DictReader(infile)
...     for book in books:
...         curs.execute(ins_str,(book["title"],book["author"],book["year"]))
... 
<sqlite3.Cursor object at 0x1029e8960>
<sqlite3.Cursor object at 0x1029e8960>
<sqlite3.Cursor object at 0x1029e8960>
<sqlite3.Cursor object at 0x1029e8960>
<sqlite3.Cursor object at 0x1029e8960>
>>> conn.commit()

8-8 Select the title column of the book table and display it in alphabetical order.

#The data was inserted by mistake the first time, so when I tried again, the previous data remained. In addition, I found out that SQLite cannot delete data, but it is possible to delete a table.
>>> sql="select title from book order by title asc"
>>> for row in conn.execute(sql):
...     print(row)
... 
('    ',)
('        Perdido Street Station',)
('        Small Gods',)
('        The Spellman Files',)
('        The Weirdstone of Brisingamen',)
('        Thud!Terry Pratchett',)
('Perdido Street Station',)
('Small Gods',)
('The Spellman Files',)
('The Weirdstone of Brisingamen',)
('Thud!',)

8-9 Select all columns in the book table and display them in order of publication year.

>>> for row in conn.execute("select * from book order by year"):
...     print(row)
... 
('        Thud!Terry Pratchett', '2005', None)
('    ', None, None)
('        The Weirdstone of Brisingamen', 'Alan Garner', 1960)
('The Weirdstone of Brisingamen', 'Alan Garner', 1960)
('        Small Gods', 'Terry Pratchett', 1992)
('Small Gods', 'Terry Pratchett', 1992)
('        Perdido Street Station', 'China Miéville', 2000)
('Perdido Street Station', 'China Miéville', 2000)
('Thud!', 'Terry Pratchett', 2005)
('        The Spellman Files', 'Lisa Lutsz', 2007)
('The Spellman Files', 'Lisa Lutsz', 2007)

8-10 Let's connect to the DB of books.db of sqlite3 created in 8-6 using the sqlalchemy module. Select the title column as in 8-8 and display them in alphabetical order.

>>> import sqlalchemy as sa
>>> conn=sa.create_engine("sqlite:///books.db")
>>> sql="select title from book order by title asc"
>>> rows=conn.execute(sql)
>>> for row in rows:
...     print(row)
... 
('    ',)
('        Perdido Street Station',)
('        Small Gods',)
('        The Spellman Files',)
('        The Weirdstone of Brisingamen',)
('        Thud!Terry Pratchett',)
('Perdido Street Station',)
('Small Gods',)
('The Spellman Files',)
('The Weirdstone of Brisingamen',)
('Thud!',)
>>> 

8-11 Install the Redis server and Python's redis library. Then create a Redis hash called test with count (1), name ("Fester Besteretester") fields to display all the fields of test.

#Don't forget to restart brew!
$ brew services restart redis

>>> import redis
>>> conn=redis.Redis()

>>> conn.hmset("test",{"count":"1","name":"Fester Besteretester"})
True

#The output is a bytes variable.
>>> conn.hgetall("test")
{b'count': b'1', b'name': b'Fester Besteretester'}

Let's increment the count field of 8-12 test and display the result.

>>> conn.hincrby("test","count")
2
>>> conn.hget("test","count")
b'2'

Impressions

I learned more and understood better. After that, I would like to deepen my understanding through documentation and online research.

References

"Introduction to Python3 by Bill Lubanovic (published by O'Reilly Japan)"

"Redis document" http://redis.shibu.jp

"Install Redis on macOS." https://weblabo.oscasierra.net/redis-macos-install-homebrew/