RPC completed with standard Python3 modules

table of contents

--There are too many types of RPC in Python now --Value that can be executed with the standard library --Code description --Try moving it

There are too many types of RPC in Python today

As of 2020, many RPCs have been released as OSS, but there are too many types, and the associated increase in selection cost, evaluation cost, and learning cost are personal issues. I feel that.

Since time is precious when working as a member of society, I'm studying libraries that appear one after another and learning how to use them when doing PoC quickly or closing APIs etc. It is not appropriate when making a good product.

In Python, the manager class exists in multiprocessing, and high-performance Remote Procudure Call can be implemented by operating interprocess communication during multiprocessing over the network.

Remote Procudure Call (hereinafter, RPC) is another computer on the network that calls or processes some kind of processing to perform large-scale distributed processing. Convenient.

Value that can be executed in the standard library

I've used a lot of libraries, but I don't have the maintainers for those libraries, I don't have the system to accept pull requests, or I don't have the time. Many "looks good" libraries have been left behind in the past.

If possible, it's library-independent and easy to implement, so it can be used as long as Python exists (or isn't deprecated in Python).

Code description

An element that can be divided into server and client, where server listens for an instruction and client issues an instruction.

server

from multiprocessing.managers import BaseManager as Manager
import os

#Assuming in-memory KVS
obj = {}
def get(k):
    print('get', k)
    return obj.get(k)

def put(k, v):
    obj[k] = v
    print('put', k,v)

#Get uname for server(Functions that understand Linux and MacOS)
def get_uname():
    print('get_uname')
    return str(os.uname())

if __name__ == "__main__":
    port_num = 4343
    Manager.register("get", get) #Register the function used for standby
    Manager.register("put", put)
    Manager.register("get_uname", get_uname)
    manager = Manager(("", port_num), authkey=b"password") #Instructions can be accepted from anywhere by leaving the host name blank. Password can be set
    manager.start()
    input("Press any key to kill server".center(50, "-")) #Enter something and you're done
    manager.shutdown()

client

from multiprocessing.managers import BaseManager as Manager
Manager.register("get") #Register function
Manager.register("put")
Manager.register("get_uname")

if __name__ == "__main__":
    port_num = 4343

    manager = Manager(address=('25.48.219.74', port_num), authkey=b"password")
    manager.connect()
    print('get', manager.get('a')) #None should be back
    print('put', manager.put('a', 10)) # a ->Set 10
    print('get', manager.get('a').conjugate()) #10 should be back, (For primitive type etc., retrieve the value with the conjugate function)
    print('get_uname', manager.get_uname()) #I'm running the client on MacOS, but Linux on the Server should come back

Try to actually move it

I ran the above program from Linux (Ubuntu) as server, MacOS (darwin) as client, and from a convenience store cafe to my home PC.

The result was as expected, and I was able to make it work arbitrarily.

With this, for example, it seems that you can process YouTube views, Twitter trend keywords, etc., and you can efficiently date without running BigQuery or RedShift frequently. You can aggregate the data.

It's important to learn new great tools, but it's also important to keep the existing ones small, and the overall cost of achieving what you want to do is very low, so this person is definitely welcome. We would appreciate it if you could consider the law.

Bonus: YouTube viewer count code

I forked the client and tried it on the assumption that there was a lot of access from multiple people.

cleint

from concurrent.futures import ProcessPoolExecutor
import random
from multiprocessing.managers import BaseManager as Manager
Manager.register("get")  #Register function
Manager.register("inc")

def extract(x):
    if hasattr(x, 'conjugate'):
        return x.conjugate()
    else:
        return x


def hikakin_watch(num):
    port_num = 4343
    manager = Manager(address=('127.0.0.1', port_num), authkey=b"password")
    manager.connect()
    for i in range(1000):
        try:
            now = extract(manager.get('hikakin'))
            print(now)
            manager.inc('hikakin')
        except Exception as exc:
            print(exc)

if __name__ == "__main__":
    with ProcessPoolExecutor(max_workers=5) as exe:
        exe.map(hikakin_watch, list(range(5)))
    
    port_num = 4343
    manager = Manager(address=('127.0.0.1', port_num), authkey=b"password")
    manager.connect()
    now = extract(manager.get('hikakin'))
    print(now)

server

from multiprocessing.managers import BaseManager as Manager
import os

#Assuming in-memory KVS
obj = {}
def get(k):
    if k not in obj:
        obj[k] = 0
    return obj.get(k)

def inc(k):
    obj[k] += 1

if __name__ == "__main__":
    port_num = 4343
    Manager.register("get", get) #Register the function used for standby
    Manager.register("inc", inc)
    manager = Manager(("", port_num), authkey=b"password") #Instructions can be accepted from anywhere by leaving the host name blank. Password can be set
    manager.start()
    input("Press any key to kill server".center(50, "-")) #Enter something and you're done
    manager.shutdown()

We obtained 5000 as the expected output, and found that exclusive control can be properly performed even for parallel access.

It seems that it can be used for counting the number of views on YouTube.