This time, I was planning to post a post titled "Solving High Equations with Integer Type = Distributed Parallel Processing =". However, when I was writing a socket communication program for posting, there was an "unexpected event" and I decided to report it first.
This "unexpected event" was due to the "TCP-related kernel parameters" in this title. At first, it was a program that could send and receive long integer type data, so the list of numbers to be sent was getting bigger and bigger. At first, the amount sent on the client side and the amount received on the server side were exactly the same. However, from a certain point, the phenomenon that the amount of reception on the server side became smaller than the amount of transmission from the client side began to appear. The progress is shown below. It is a model with 8GB of memory of Raspberry Pi 4B, and uses Thonny included in Raspberry Pi OS (2020-AUG-23, 64bit beta version).
The server program and client program for tcp socket communication are shown below.
tcp-server.py
import socket
import time
s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
s.bind(('192.168.2.20', 60020))
s.listen(5)
clientsocket, address = s.accept()
print(f"Connection from {address}")
list0en = b''
list0en = clientsocket.recv(1016)
list0str = list0en.decode("utf-8")
len_list0str = int(list0str)
print(len_list0str)
initial_time=time.time()
q, mod = divmod(len_list0str, 1016)
list1stren = b''
if mod > 0:
q += 1
for i in range(q):
list1stren += clientsocket.recv(1016)
print('recv finished')
list1str = list1stren.decode("utf-8")
print(str(time.time() - initial_time))
if len_list0str==len(list1str):
print('length OK')
clientsocket.close()
s.close()
print(list1str)
time.sleep(10)
tcp-client.py
import socket
import time
s1 = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
s1.connect(('192.168.2.20', 60020))
tmp_0=1234567890123456
list0 = []
Su16b=800#14,400byte OK
#Su16b=1000#18,000byte loss
#Su16b=10000#180,000byte loss
#Su16b=100000#1.75MB loss
#Su16b=600000#10MB loss
for i in range(Su16b):
list0.append(tmp_0)
list0str = str(list0)
len_list0str = len(list0str)
len_list0str_str = str(len_list0str)
s1.send(bytes(len_list0str_str, 'utf-8'))
q, mod = divmod(len_list0str, 1016)
for i in range(q):
temp_21str = list0str[1016*i:1016*(i+1)]
s1.send(bytes(temp_21str, 'utf-8'))
if mod > 0:
temp_21str = list0str[1016*q:]
s1.send(bytes(temp_21str, 'utf-8'))
s1.close()
time.sleep(10)
There is Su16b = 800 # 14,400byte in the client program, but I was able to send and receive without problems so far. Next, when the transmission amount was increased to Su16b = 1000 # 18,000 bytes, the number of bytes received on the server side was small. Even if Su16b = 10000 is further increased, the number of bytes received on the server side is insufficient. If the transmission amount is slightly more than 14,400 bytes, the reception amount will not match. In addition, the server program should have closed the socket, but the 60020 port is still in use. I restarted the Raspberry Pi and recovered. When I run the server program again and receive it, the amount of reception does not match again, and the port of 60020 is still in use repeatedly.
There are two problems here. One is that the 60020 port on the server side remains in use. The other is that the amount received is less than the amount sent. We will solve it separately below.
This time, I solved it by restarting the Raspberry Pi itself.
The first thing I came up with was a buffer leak that I often see. Furthermore, I paid attention to "If the transmission amount increases a little more than 14,400 bytes, the reception amount will not match." I expected to run out of receive buffer. So I checked it with the sysctl command, which shows the kernel parameters. I haven't touched on the kernel parameters, so it was a little daunting.
net.core.rmem_default=212992
net.core.wmem_default=212992
net.ipv4.tcp_rmem=4096 131072 6291456
net.ipv4.tcp_wmem=4096 16384 4194304
Since the default of the receive buffer rmem is 131,072 bytes, it seems that it has nothing to do with the buffer leakage from around 14,400 bytes. On the other hand, the default of the send buffer wmem is 16,384 bytes. It is close to buffer leakage from around 14,400 bytes. I assumed that it was a send buffer leak, not a receive buffer leak. Originally, the receive buffer is automatically expanded by default, so it seems that the receive buffer does not leak. Assuming the send buffer is leaking, you can increase the send buffer.
First of all, I had read the following page of socket programming on python.org in advance. It was written to "make sure to close the socket". He added, "(believe it or not) you can't always receive all five characters in a single recv ... Under heavy networks, code that doesn't use two recv loops I wrote the program with reference to the part of "It will be useless in a blink of an eye" because it may be a high load. Socket Programming HOWTO Author: Gordon McMillan
And this is the buffer leakage phenomenon. First, I wondered if I could change the kernel parameters. Web server tuning is a common occurrence. I am changing the kernel parameters steadily. I also referred to the following page. Linux kernel tuning .github It also included tuning for public web servers such as syn attack protect. However, this time it is a closed LAN in the back, and TCP communication that also decided the transmission and reception method. Syn attacks for public servers are not relevant. This time, we changed only the send / receive buffer related items due to the increase in transmission volume.
sudo mousepad /etc/sysctl.conf
I opened the sysctl.conf file in the etc folder with a mousepad and added the following 6 lines.
net.core.rmem_default=4194304
net.core.wmem_default=4194304
net.core.rmem_max=16777216
net.core.wmem_max=16777216
net.ipv4.tcp_rmem=4096 4194304 16777216
net.ipv4.tcp_wmem=4096 4194304 16777216
After restarting the Raspberry Pi itself, when I ran the server program and client program mentioned above, I was able to send and receive accurately up to 10MB. However, since the buffer MAX is about 16MB, I also found that if the transmission amount exceeds 10MB, it is necessary to divide and transmit.
** Problem of changing kernel parameters this time: ** I never browse the internet with Raspberry Pi. But let's say you browse the Internet with this setting. For example, suppose you browse a web page with 100 types of small 10kbyte image icons. For the request of 100 images placed on the web page, 100x send buffer 4,194,304 bytes and 100x receive buffer 4,194,304 bytes, totaling about 840MB of memory will be used instantaneously. However, if you use too much memory, each socket will be automatically reduced to 4,096 bytes of min, so you can avoid freezes. However, you will have to receive a 10kbyte image file with a receive buffer of 4,096 bytes. Maybe there will be a missed reception. It's some kind of bumpy tuning. From now on, it seems that the default of 4194304 bytes this time is usually about 1/40 of this, about 131072 bytes at the maximum. Also, with Raspberry Pi models with 1GB and 500MB of memory, tuning will be more delicate due to the small amount of memory, so tuning is likely to become even more difficult.
This time, we have changed the TCP related settings of the kernel parameter that we have never touched. For the purpose of sending and receiving long integer type data with a large number of digits, I intend to change only the minimum range that does not cause any problems, but I do not understand. It was a little daunting to post, but this time I reported it as one phenomenon that occurred during program creation and one solution that I tried.
During the bloated program with the amount of transmission and reception, the phenomenon of separation due to buffer leakage began to occur, and it was found that the signs of the setting sun were beginning to be glimpsed in the program that seemed to be proceeding smoothly.
Thank you for watching until the end.
This time, I agree with the purpose of Qiita and post it, so you are free to copy or modify the published program. There are no copyright issues.
However, when using the Raspberry Pi 4B, a particularly large heat sink is required for the CPU. In the case of this program, where LAN communication is infrequent, the LAN chip does not get hot. However, if the calculation time continues, a considerable amount of power is used as can be seen from the intense heat generated by the CPU. The small black chip behind the power supply UCB C type also gets hot, so you also need a fan flow.
Next time, in "Solving high equations with integer type = Distributed parallel processing =", we plan to try to shorten the calculation time by distributed processing using socket communication of 4 Raspberry Pis.