Grpc Preface
gRPC is a high-performance, open source remote procedure call (RPC) framework developed by Google. It communicates based on the HTTP/2 protocol and uses Protocol Buffers (protobuf) as the interface definition language, which can be regarded as a protocol.
GRPC can be used for communication between various services, blocking underlying details (such as programming languages, operating systems, etc.)
Since one of my go backends (or not go) needs to implement neural network-related functions, I want to call a python service, so I thought of using grpc.
The first contact will be introduced from 0 to functional implementation.
Thanks for the official documentation
/docs/languages/go/quickstart/
Main process
Define .proto file: Create a .proto file that defines the message type and service interface for graph data and GCN results.
Generate gRPC code: Use the Protocol Buffers compiler protoc to generate gRPC code for Go and Python based on .proto files.
Implement Python server: Implement a gRPC server in Python, receive graph data from the Go client, perform GCN processing, and return processing results.
Implementing the Go client: Implementing the gRPC client in Go, sending graph data to the Python server, and receiving processing results.
Install the protoc interpreter
grpc works through the protocol pb (protocolbuffer). First, install the interpreter of protoc and add its bin folder to the environment variable.
address:/protocolbuffers/protobuf/releases
After downloading, unzip it to any folder location, and then add the unzipped bin folder to the environment variable.
Open cmd and enter protoc and return it, and it will be OK.
In order to use some protoc instructions in the project to work properly, copy the file under the bin file to the C:\Windows\System32 folder.
By the way, in order to highlight the proto file in vscode, the official plugin can be installed: protobuf, author: pbkit
Go-side work
First install the go grpc library
go get -u /grpc
Then install an interpreter that can automatically parse the proto file into a go file:
go install /protobuf/cmd/protoc-gen-go go install /grpc/cmd/protoc-gen-go-grpc
Define the proto file:
where go_package specifies the generated file location and the package name where the file is located. "." means that the file is generated in the current directory, ";" means parameter division, and the subsequent proto is to place the generated file in the proto package.
There are more optional parameters, please refer to the official one
go_package: Specifies the package path when generating Go code, in the format option go_package = "package_path";, used to place the generated code in the specified Go package.
java_package: Specifies the package path when generating Java code, the format is option java_package = "package_path";, which is used to place the generated code in the specified Java package.
java_outer_classname: Specifies the external class name when generating Java code, in the format option java_outer_classname = "ClassName";, used to specify the external class name of the generated Java class.
cc_generic_services: Specifies whether to generate C++ general services, in the format option cc_generic_services = true; or option cc_generic_services = false;, default is
true。
cc_enable_arenas: Specifies whether to enable C++ arenas memory management, in the format option cc_enable_arenas = true; or option cc_enable_arenas = false;, default to false.
optimize_for: Specifies optimization options, which can be SPEED, CODE_SIZE or LITE_RUNTIME, the format is option optimize_for = SPEED;, and the default is SPEED.
deprecated: Specifies that the message or field is outdated, in the format option deprecated = true;.
rpc_timeout: Specifies the timeout time of the gRPC call, the format is option rpc_timeout = "10s";, indicating that the timeout is 10 seconds.
The protocol file is specifically defined as:
syntax = "proto3";
option go_package = ".;proto";
message Node {
string id = 1;
repeated float features = 2;
}
message Edge {
string source_id = 1;
string target_id = 2;
}
message GraphData {
repeated Node nodes = 1;
repeated Edge edges = 2;
}
message GCNResult {
map<string, float> node_scores = 1;
}
service GCNService {
rpc ProcessGraph(GraphData) returns (GCNResult);
}
I'm using it to process graph data here.
Next generate gRPC code:
protoc --go_out=. --go_opt=paths=source_relative your_proto_file.proto
protoc --go-grpc_out=. --go-grpc_opt=paths=source_relative your_proto_file.proto
Two files and gcn_grpc will be generated.
The file generated by the former contains all the go codes of the PB protocol and writes the data format as the structure of the go. Contains padding, serialization, search request and response message types.
The latter contains the methods defined in the client's use and the interface type to call. and the interface type to be implemented by the server. Contains methods to create client and server services.
Please note that both files must be generated whether the go backend is the client or the server that requests grpc.
Create a client using client := (conn)
Finally, complete the GRPC request code on the Go side. Each request is created to create a GRPC connection. After the request is completed, defer is disconnected:
func GCN_request() (map[string]float32, error) {
conn, err := ("localhost:9999", (()))
if err != nil {
// Failed to connect to gcnserver(py) by grpc
return nil, err
}
defer ()
client := (conn)
// Create an instance of graph data G_example := &{
Nodes: []*{
{Id: "node1", Features: []float32{0.1, 0.2, 0.3}},
{Id: "node2", Features: []float32{0.4, 0.5, 0.6}},
},
Edges: []*{
{SourceId: "node1", TargetId: "node2"},
},
}
// Send a request and receive a response result, err := ((), G_example)
if err != nil {
// Error calling ProcessGraph
return nil, err
}
return , err
}
At the end, don't forget to create a route to call this method.
Working on python
py also installs grpc related libraries
pip install grpcio pip install grpcio-tools
The code related to py's server-side code grpc is relatively concise, and the focus is on data processing. Keep its service port consistent with the request port of the grpc client.
Some of the code is taken from the official example.
import grpc
from concurrent import futures
from proto import gcn_pb2_grpc, gcn_pb2
class GCNServicer(gcn_pb2_grpc.GCNServiceServicer):
def ProcessGraph(self, request, context):
# Process graph data using GCN and return result
# example: return gcn_pb2.GCNResult(node_scores={"node1": 0.5, "node2": 0.8})
def server():
server = ((max_workers=10))
gcn_pb2_grpc.add_GCNServiceServicer_to_server(GCNServicer(), server)
server.add_insecure_port('[::]:9999')
()
print('gRPC server is enabled, port is 9999...')
server.wait_for_termination()
if __name__ == '__main__':
server()start up
Once you do the above work, you will be done. Just start two services.
go run
python
Then access the defined route address, as shown below, nice~
This is the article about using grpc to implement communication between golang backend and python services. For more related communication content between go python services, please search for my previous articles or continue browsing the related articles below. I hope everyone will support me in the future!