Python <-> Interprocess communication between C languages
Due to various circumstances, I decided to write a program in C language, but I thought it would be very difficult to write the interface part in the same C, so I made the interface in Python and connected the programs with IPC etc. I thought about it. At first, I thought it would be okay to pack / unpack the C language structure on my own, but since there are some that my predecessors made, can I use ** Protocol Buffer **? So I decided to research and try it.
Protocol Buffer(protobuf) Official Protocol buffer support is as follows: (As of 11/11/2018)
proto2
proto3 (+ above)
proto2 and proto3 are not compatible. Qiita also has an article that explains in detail the outline of proto2, proto3, protobuf, etc., so please read it.
「Proto2 vs Proto3」：https://qiita.com/ksato9700/items/0eb025b1e2521c1cab79
Generate proto file
sample.proto
message DataStructs{
optional uint32 id = 1;
optional uint32 ip_address = 2;
optional uint32 port_num = 3;
}
message IpcMessage {
enum Errors {
SUCCESS = 200;
ERROR_BAD_REQUEST = 400;
ERROR_NOT_FOUND = 404;
ERROR_SERVER_ERROR = 500;
ERROR_SERVICE_UNAVAILABLE = 503;
}
// error_code refers to the above Errors.
optional Errors error_code = 1;
//You can nest messages.
optional DataStructs data = 2;
}
The above is the description of proto2. I think that even people who see it for the first time can understand it somehow.
It is convenient to add comments in the .proto file.
proto3 is not compatible due to different syntax.
protobuf-c
So, this time, I wanted to run protobuf in C language. There is no official support, but when I look it up, it seems that there is a third party project, so I will use it. protobuf-c : https://github.com/protobuf-c/protobuf-c
It seems that protobuf-c does not support proto3, so I will try it with proto2 after that.
If you think you have to compile, it seems that modern distributions have packages. It can be installed from standard repositories on both CentOS and Ubuntu.
CentOS-7.5-yum
$ yum search protobuf-c | grep x86_64
protobuf-c.x86_64 : C bindings for Google's Protocol Buffers
protobuf-c-compiler.x86_64 : Protocol Buffers C compiler
protobuf-c-devel.x86_64 : Protocol Buffers C headers and libraries
...
Ubuntu-18.04-apt
$ apt-cache search protobuf-c | grep '(protobuf-c)'
libprotobuf-c1 - Protocol Buffers C shared library (protobuf-c)
libprotobuf-c-dev - Protocol Buffers C static library and headers (protobuf-c)
libprotobuf-c1-dbg - Protocol Buffers C shared library debug symbols (protobuf-c)
protobuf-c-compiler - Protocol Buffers C compiler (protobuf-c)
This time, it will be carried out in the CentOS environment.
$ cat /etc/redhat-release && uname -r && rpm -aq | egrep '(protobuf)|(gcc)' && python -V
CentOS Linux release 7.5.1804 (Core)
3.10.0-862.14.4.el7.x86_64
gcc-4.8.5-28.el7_5.1.x86_64
protobuf-2.5.0-8.el7.x86_64
protobuf-c-devel-1.0.2-3.el7.x86_64
protobuf-c-1.0.2-3.el7.x86_64
protobuf-compiler-2.5.0-8.el7.x86_64
libgcc-4.8.5-28.el7_5.1.x86_64
protobuf-python-2.5.0-8.el7.x86_64
protobuf-c-compiler-1.0.2-3.el7.x86_64
Python 2.7.5
Compile the .proto file with protoc-c, which is a protobuf compiler for protobuf-c, generate .pb-ch and .pb-cc, I will use it from now on.
$ protoc-c sample.proto --c_out=./ && ls sample.*
sample.pb-c.c sample.pb-c.h sample.proto
It is an example of reading protobuf in C language
Before writing a program to deserialize in C language, prepare a script to serialize to standard output in Python.
First, create a Proto file for Python.
$ protoc sample.proto --python_out=. ; ls *.py
sample_pb2.py
A script that displays python serialized protobuf on standard output. This time we are talking about protobuf-c, so I will omit the details.
serialize_sample.py
#!/usr/bin/python
# -*- encoding:utf-8 -*-
import sample_pb2
import sys
message = sample_pb2.IpcMessage()
message.error_code = sample_pb2.IpcMessage.ERROR_NOT_FOUND
message.data.id=123
message.data.ip_address=(192<<24)+(168<<16)+(0<<8)+5
message.data.port_num=5060
data=message.SerializeToString()
#Output so as not to include line feed code
sys.stdout.write(data)
In c language, try deserializing from standard input.
deserialize_sample.c
#include <stdio.h>
#include "sample.pb-c.h"
int main(){
char buffer[1024];
int len=0;
FILE *fp;
//Input in Binary mode from standard input
fp=freopen(NULL, "rb", stdin);
len=fread(buffer, sizeof(char), sizeof(buffer), fp);
//Follow the definition defined in the Proto file
IpcMessage *message;
//The exact length of the serialized data is required when unpacking. NG if it is too short or too long
message=ipc_message__unpack(NULL, len, buffer);
// has_*Check if the optional item has a value in.
if(message->has_error_code)
printf("error_code : %d\n", message->error_code);
//Nested messages are also dynamically generated when packed.
if(message->data->has_id)
printf("data.id : %d\n", message->data->id);
if(message->data->has_ip_address)
printf("data.ip_address : %d.%d.%d.%d\n", (message->data->ip_address)>>24 & 0xff,
(message->data->ip_address)>>16 & 0xff,
(message->data->ip_address)>>8 & 0xff,
(message->data->ip_address) & 0xff);
if(message->data->has_port_num)
printf("data.port_num : %d\n", message->data->port_num);
//Release unpacked objects
//It seems that nested messages are also released
ipc_message__free_unpacked(message, NULL);
close(fp);
return 0;
}
Compile
$ gcc -l protobuf-c deserialize_sample.c sample.pb-c.c -o deserialize_sample
$ ./serialize_sample.py | ./deserialize_sample
error_code : 404
data.id : 123
data.ip_address : 192.168.0.5
data.port_num : 5060
I was able to successfully retrieve the protobuf message serialized in Python in C language.
Next, an example of reading Protobuf generated in C language in Python
serialize_sample.c
#include <stdio.h>
#include <stdlib.h>
#include "sample.pb-c.h"
int main(){
void *buffer;
int len=0;
FILE *fp;
//Input in Binary mode from standard input.
fp=freopen(NULL, "wb", stdout);
//There is an example of using the INIT macro, but here it is dynamically allocated by malloc.
//After malloc__Must be initialized using init
IpcMessage *message;
message=(IpcMessage *)malloc(sizeof(IpcMessage));
ipc_message__init(message);
//Unlike the time of pack, even if init is done, the nested message area is not secured.
//Reserve space for separately nested messages and init. Needs initialization
message->data=(DataStructs *)malloc(sizeof(DataStructs));
data_structs__init(message->data);
// .The element specified as optional in the proto file is has_*Flag to true.
//If it is not set to true, it will be ignored when serializing.
message->has_error_code=1;
message->error_code=IPC_MESSAGE__ERRORS__ERROR_SERVICE_UNAVAILABLE;//503
message->data->has_id=1;
message->data->id=1192;
message->data->has_ip_address=1;
message->data->ip_address=(192<<24)+(168<<16)+(0<<8)+234;
message->data->has_port_num=1;
message->data->port_num=8080;
//Serialize process, get size and malloc&Serialization process
len=ipc_message__get_packed_size(message);
buffer=malloc(len);
ipc_message__pack(message, buffer);
//Binary output to standard output
fwrite(buffer, sizeof(void), len, fp);
//Free malloc area
free(buffer);
free(message->data);
free(message);
close(fp);
return 0;
}
Compile
$ gcc -l protobuf-c serialize_sample.c sample.pb-c.c -o serialize_sample
A script that deserializes and displays the input serialized by the standard input protobuf.
deserialize_sample.py
#!/usr/bin/python
# -*- encoding:utf-8 -*-
import sample_pb2
import sys
data = sys.stdin.read()
message = sample_pb2.IpcMessage()
message.ParseFromString(data)
if message.HasField("error_code"):
print("error_code : {}".format(message.error_code))
if message.data.HasField("id"):
print("data.id : {}".format(message.data.id))
if message.data.HasField("ip_address"):
print("data.ip_address : {}.{}.{}.{}".format((message.data.ip_address>>24)&0xff,
(message.data.ip_address>>16)&0xff,
(message.data.ip_address>> 8)&0xff,
(message.data.ip_address>> 0)&0xff))
if message.data.HasField("port_num"):
print("data.port_num : {}".format(message.data.port_num))
$ ./serialize_sample | ./deserialize_sample.py
error_code : 503
data.id : 1192
data.ip_address : 192.168.0.234
data.port_num : 8080
I was able to successfully retrieve the protobuf message serialized in C language with Python.
It's easy because I didn't have much information about protobuf-c in Japanese, but I've summarized the serialization method and deserialization method.
It is a sample code that is not very conscious of error handling etc., and in reality it is not a simple standard input / output when implemented with IPC, so it is necessary to reassemble with socket etc., but I think that you can understand the essence. think···.
Until now, I didn't really understand the value of protocol buffer, and to be honest, I only thought "I should dump it with json".
Certainly, if it is necessary to define in a form that is easy for people to understand, such as REST / API, it is good to define it in json or yaml, but what to do with the data format between different processes, different programming languages, etc. I found it very useful as a solution to the problem.
Furthermore, if it is a low-level language (compared to recent languages) such as C language, json or yaml is not supported as standard, and its calculation speed is not wasted as much as possible. I understand it.
Recommended Posts