Preface
You may at first glance, connecting two processes through the TCP/IP layer will feel terrible, but it may be much simpler in Go language than you think. I won’t say much below, let’s take a look at the detailed introduction together.
Application scenarios for sending data in TCP/IP layer
Of course, in many cases, not most cases, it is undoubtedly better to use higher-level network protocols, because the gorgeous APIs can be used, and they hide a lot of technical details. Now, depending on different needs, there are many choices, such as message queue protocol, gRPC, protobuf, FlatBuffers, RESTful website API, websocket, etc.
However, in some special scenarios, especially small projects, choosing any other method will feel too bloated, not to mention that you need to introduce additional dependencies.
Fortunately, using the standard library's net package to create simple network communication is no more difficult than you can see.
Because there are two simplifications in Go language.
Simplification 1: Connection is io stream
The interface is implemented, and the interface. Therefore, TCP connections can be treated like other io streams.
You might think: "Okay, I can send strings or byte shards in TCP, which is very good, but what should I do when I encounter complex data structures? For example, are we encountering structure-type data?"
Simplification 2: Go language knows how to effectively decode complex types
When it comes to sending encoded structured data over the network, the first thing that comes to mind is JSON. But wait a moment - the Go standard library encoding/gob package provides a way to serialize and issue Go data types, which does not require adding string tags to structures and Go incompatible JSONs, or wait for use to parse text into binary data.
Gob encoding and decoding can directly operate the io stream, which perfectly matches the first simplification.
Next, we will implement a simple App through these two simplified rules.
The goal of this simple app
This app should do two things:
- Send and receive simple string messages.
- Send and receive structures through gob.
The first part, sending simple strings, will demonstrate how easy it is to send data over a TCP/IP network without the help of advanced protocols.
The second part, a little deeper, sends complete structures over the network, using strings, shards, maps, and even recursive pointers to themselves.
Thank you for the fact that you have a gob package, you need to do this without any effort.
Client
Structure to be sent �
testStruct structure
| ^
V |
gob encoding ---------------------------------> gob decoding
| ^
V |
Send ==================== Network========================= Receive
Basic elements of sending string data through TCP
On the sending side
Sending a string requires three simple steps:
- Open the connection to the corresponding receiving process.
- Write string.
- Close the connection.
The net package provides a pair of ways to implement this function.
- ResolveTCPAddr(): This function returns the TCP terminal address.
- DialTCP(): Similar to dial-up for TCP networks.
Both methods are defined in the src/net/ file of the go source code.
func ResolveTCPAddr(network, address string) (*TCPAddr, error) {
switch network {
case "tcp", "tcp4", "tcp6":
case "": // a hint wildcard for Go 1.0 undocumented behavior
network = "tcp"
default:
return nil, UnknownNetworkError(network)
}
addrs, err := ((), network, address)
if err != nil {
return nil, err
}
return (network, address).(*TCPAddr), nil
}
ResolveTCPAddr() receives two string parameters.
- network: It must be the TCP network name, such as tcp, tcp4, tcp6.
- address: TCP address string. If it is not a literal IP address or the port number is not a literal port number, ResolveTCPAddr will resolve the incoming address to the address of the TCP terminal. Otherwise, a pair of literal IP addresses and port numbers are passed as addresses. The address parameter can use the host name, but it is not recommended because it will return at most one IP address of the host name.
ResolveTCPAddr() receives a string representing the TCP address (for example, localhost:80, 127.0.0.1:80, or [::1]:80, both representing the 80 port of the machine), and returns (pointer, nil) (if the string cannot be parsed into a valid TCP address, it will return (nil, error)).
func DialTCP(network string, laddr, raddr *TCPAddr) (*TCPConn, error) {
switch network {
case "tcp", "tcp4", "tcp6":
default:
return nil, &OpError{Op: "dial", Net: network, Source: (), Addr: (), Err: UnknownNetworkError(network)}
}
if raddr == nil {
return nil, &OpError{Op: "dial", Net: network, Source: (), Addr: nil, Err: errMissingAddress}
}
c, err := dialTCP((), network, laddr, raddr)
if err != nil {
return nil, &OpError{Op: "dial", Net: network, Source: (), Addr: (), Err: err}
}
return c, nil
}
The DialTCP() function receives three parameters:
- network: This parameter is the same as the network parameter of ResolveTCPAddr, and must be the TCP network name.
- laddr: A pointer of type TCPAddr, representing the local TCP address.
- raddr: A pointer of type TCPAddr, which represents a remote TCP address.
It will connect to dial-up two TCP addresses and return this connection as an object (return error if connection fails). If we don't need to have too much control over the Dial settings, then we can use Dial() instead.
func Dial(network, address string) (Conn, error) {
var d Dialer
return (network, address)
}
The Dial() function receives a TCP address and returns a general one. This is enough for our test cases. However, if you need only the available functions on TCP connections, you can use TCP variants (DialTCP, TCPConn, TCPAddr, etc.).
After successfully dialing, we can treat the new connection equally with other input and output streams as described above. We can even wrap the connection into it, so we can use various ReadWriter methods such as ReadString(), ReadBytes, WriteString, etc.
func Open(addr string) (*, error) {
conn, err := ("tcp", addr)
if err != nil {
return nil, (err, "Dialing "+addr+" failed")
}
// Wrap the object into return ((conn), (conn)), nil
}
Remember to call the Flush() method after buffering Writer is written, so that all data will be flushed to the underlying network connection.
Finally, each connection object has a Close() method to terminate the communication.
fine tuning
The Dialer structure is defined as follows:
type Dialer struct {
Timeout
Deadline
LocalAddr Addr
DualStack bool
FallbackDelay
KeepAlive
Resolver *Resolver
Cancel <-chan struct{}
}
- Timeout: The maximum number of times dialing is waiting for the connection to end. If Deadline is set at the same time, it can fail earlier. There is no timeout by default. When using TCP and dialing hostnames with multiple IP addresses, the timeout is divided between them. The operating system can force an earlier timeout with or without using the timeout. For example, TCP timeout is usually around 3 minutes.
- Deadline: It is the absolute point in time when dialing is about to fail. If Timeout is set, it may fail earlier. A value of 0 indicates that there is no deadline, or that it depends on the operating system or uses the Timeout option.
- LocalAddr: is the local address used when dialing an address. This address must be of the type that is fully compatible with the network address to be dialed. If nil, a local address will be automatically selected.
- DualStack: This property enables RFC 6555 compatible"Happy Eyeballs"Dial, when the network is tcp, the host in the address parameter can be resolved by IPv4 and IPv6 addresses. This allows the client to tolerate a address family's network regulations slightly broken.
- FallbackDelay: When DualStack is enabled, specify the time to wait before a fallback connection is generated. If set to 0, the default delay is 300ms.
- KeepAlive: Specifies the time to stay active for active network connections. If set to 0, keep-alive is not enabled. This field will be ignored by network protocols that do not support keep-alive.
- Resolver: Optional, specifying the alternative resolver to use.
- Cancel: Optional channel, its closure means that the dial should be cancelled. Not all dialing types support dialing cancellation. Deprecated, DialContext can be used instead.
There are two available options to fine-tune.
Therefore, the Dialer interface provides two options that can be fine-tuned:
- DeadLine and Timeout options: Timeout setting for unsuccessful dialing.
- KeepAlive options: Manage the life span of the connection.
type Conn interface {
Read(b []byte) (n int, err error)
Write(b []byte) (n int, err error)
Close() error
LocalAddr() Addr
RemoteAddr() Addr
SetDeadline(t ) error
SetReadDeadline(t ) error
SetWriteDeadline(t ) error
}
The interface is a general network connection that is flow-oriented. It has the following interface methods:
- Read(): Read data from the connection.
- Write(): Write data to the connection.
- Close(): Close the connection.
- LocalAddr(): Returns the local network address.
- RemoteAddr(): Returns the remote network address.
- SetDeadline(): Sets the connection-related read and write deadline. It is equivalent to calling SetReadDeadline() and SetWriteDeadline() at the same time.
- SetReadDeadline(): Sets the timeout deadline for future read calls and currently blocked read calls.
- SetWriteDeadline(): Sets the timeout deadline for future write calls and currently blocked write calls.
The Conn interface also has deadline settings; there are (SetDeadLine()) for the entire connection, and there are also specific read and write calls (SetReadDeadLine() and SetWriteDeadLine()).
Note that deadline is a fixed time point (wallclock) Unlike timeout, they will not be reset after new activity. Therefore, each activity on the connection must be set up with a new deadline.
The sample code below does not use deadline, because it is simple enough that we can easily see when it gets stuck. When Ctrl-C, we manually trigger deadline tool.
On the receiving end
The steps at the receiving end are as follows:
- Turn on listening on the local port.
- When the request arrives, a spawn goroutine is generated to process the request.
- In goroutine, read the data. The response can also be sent optionally.
- Close the connection.
Listening needs to specify the port number for local listening. Generally speaking, the listening application (also called server) announces the listening port number. If a standard service is provided, then use the relevant port corresponding to this service. For example, web services usually listen to 80 to servo HTTP and 443 ports to servo HTTPS requests. SSH guardians listen to port 22 by default, and WHOIS service uses port 43.
type Listener interface {
// Accept waits for and returns the next connection to the listener.
Accept() (Conn, error)
// Close closes the listener.
// Any blocked Accept operations will be unblocked and return errors.
Close() error
// Addr returns the listener's network address.
Addr() Addr
}
func Listen(network, address string) (Listener, error) {
addrs, err := ((), "listen", network, address, nil)
if err != nil {
return nil, &OpError{Op: "listen", Net: network, Source: nil, Addr: nil, Err: err}
}
var l Listener
switch la := (isIPv4).(type) {
case *TCPAddr:
l, err = ListenTCP(network, la)
case *UnixAddr:
l, err = ListenUnix(network, la)
default:
return nil, &OpError{Op: "listen", Net: network, Source: nil, Addr: la, Err: &AddrError{Err: "unexpected address type", Addr: address}}
}
if err != nil {
return nil, err // l is non-nil interface containing nil pointer
}
return l, nil
}
The core part of the net package implementation server is:
() Create a new listener on the given local network address. If you only pass the port number to it, for example ":61000", the listener will listen to all available network interfaces. This is quite convenient, because computers usually provide at least two active interfaces, loopback interface and at least one real network card. If this function succeeds, return Listener.
The Listener interface has an Accept() method to wait for the request to come in. It then accepts the request and returns the caller a new connection. Accept() is generally called in a loop and can serve multiple connections at the same time. Each connection can be processed by a separate goroutine, as shown in the following code.
Code section
Rather than having the code push some bytes back and forth, I want it to demonstrate something more useful. I want it to send different commands to the server with different data carriers. The server should be able to identify each command and decode command data.
In our code, the client will send two types of commands: "STRING" and "GOB". They all terminate with line breaks.
The "STRING" command contains a line of string data, which can be processed through simple read and write operations in bufio.
The "GOB" command consists of a structure that contains some fields, contains a shard and map, and even a pointer to its own. As you can see, when running this code, there is nothing unusual about gob packages being able to move this data over our network connection.
We basically have some ad-hoc protocols here (ad-hoc protocol: ad-hoc protocol: ad-hoc, specific purpose, ad-hoc, project-based), and both the client and the server follow it, followed by a line break, and then data. For each command, the server must know the exact format of the data and know how to handle it.
To achieve this goal, the server code will be implemented in two steps.
- Step 1: When the Listen() function receives a new connection, it will generate a new goroutine to call handleMessage(). This function reads the command name from the connection, querys the appropriate processor function from the map, and then calls it.
- Step 2: The selected processor function reads and processes the command line data.
package main
import (
"bufio"
"encoding/gob"
"flag"
"/pkg/errors"
"io"
"log"
"net"
"strconv"
"strings"
"sync"
)
type complexData struct {
N int
S string
M map[string]int
P []byte
C *complexData
}
const (
Port = ":61000"
)
Outcoing connections
Using a transmit connection is a snapshot. Satisfies and interfaces, so we can view TCP connections as much as any other Reader and Writer.
func Open(addr string) (*, error) {
("Dial " + addr)
conn, err := ("tcp", addr)
if err != nil {
return nil, (err, "Dialing " + addr + " failed")
}
return ((conn), (conn)), nil
}
Open the connection to the TCP address. It returns a TCP connection with a timeout and wraps it into a buffered ReadWriter. Dial up remote process. Note that the local port is allocated in real time (on the fly). If you must specify a local port number, use the DialTCP() method.
Enter the connection
This section involves the preparation of incoming data. According to the ad-hoc protocol we introduced earlier, command name + line break + data + line break. Natural data is related to specific commands. To handle this situation, we create an Endpoint object with the following properties:
- It allows one or more processor functions to be registered, each function can handle a special command.
- It schedules the specific command to the relevant processor function according to the command name.
First we declare a HandleFunc type, which is the function type that receives a pointer value, that is, the processor function we want to register for each different command later. The parameters it receives are connections wrapped using the ReadWriter interface.
type HandleFunc func(*)
Then we declare an Endpoint structure type, which has three properties:
- listener: ()Returned Listener object.
- handler: Used to save the map of registered processor functions.
- m: A mutex that solves the problem of multi-goroutine insecure in maps.
type Endpoint struct {
listener
handler map[string]HandleFunc
m // Maps are not thread-safe, so mutexes are needed to control access.}
func NewEndpoint() *Endpoint {
return &Endpoint{
handler: map[string]HandleFunc{},
}
}
func (e *Endpoint) AddHandleFunc(name string, f HandleFunc) {
()
[name] = f
()
}
func (e *Endpoint) Listen() error {
var err error
, err = ("tcp", Port)
if err != nil {
return (err, "Unable to listen on "+().String()+"\n")
}
("Listen on", ().String())
for {
("Accept a connection request.")
conn, err := ()
if err != nil {
("Failed accepting a connection request:", err)
continue
}
("Handle incoming messages.")
go (conn)
}
}
// handleMessages reads the connection to the first newline character. Based on this string, it calls the appropriate HandleFunc.
func (e *Endpoint) handleMessages(conn ) {
// Wrap the connection to the buffered reader for easy reading rw := ((conn), (conn))
defer ()
// Read from the connection until EOF is encountered. Expect the next input to be the command name. Calls the registered processor for the command.
for {
("Receive command '")
cmd, err := ('\n')
switch {
case err == :
("Reached EOF - close this connection.\n ---")
return
case err != nil:
("\nError reading command. Got: '" + cmd + "'\n", err)
}
// Trim the extra carriage returns and spaces in the request string - ReadString does not remove any newlines.
cmd = (cmd, "\n ")
(cmd + "'")
// Get the appropriate processor function from the handler map and call it.
()
handleCommand, ok := [cmd]
()
if !ok {
("Command '" + cmd + "' is not registered.")
return
}
handleCommand(rw)
}
}
The NewEndpoint() function is the factory function of Endpoint. It only initializes the handler map. To simplify the problem, assume that the ports our terminal listener are pretty fixed.
The Endpoint type declares several methods:
- AddHandleFunc(): Use mutex to safely add processor functions that handle specific types of commands to handle handler attributes.
- Listen(): Start listening on all interfaces of the terminal port. Before calling Listen, at least one handler function must be registered through AddHandleFunc().
- HandleMessages(): wrap the connection with bufio, then read it in two steps, first read the command plus the line break, and we get the command name. Then, the processor function corresponding to the registered command is obtained through the handler, and then the function is scheduled to perform data reading and parsing.
.Notice:How to use dynamic functions above. Find the specific function according to the command name, and then assign this specific function to handleCommand. In fact, this variable type is HandleFunc type, that is, the processor function type declared earlier.
At least one processor function is required before calling the Listen method of Endpoint. Therefore, we define two types of processor functions below: handleStrings and handleGob.
The handleStrings() function receives and processes the processor function in our instant protocol that only sends string data. The handleGob() function is a complex structure that receives and processes the sent gob data. handleGob is a little more complicated. In addition to reading data, we need to decode the data.
We can see that when we use ('n') twice in a row, we read the string, and when we encounter a new line stop, we save the read content to the string. Note that this string contains the end line break.
In addition, for ordinary string data, we directly use bufio to wrap the connected ReadString to read. For complex gob structures, we use gob to decode data.
func handleStrings(rw *) {
("Receive STRING message:")
s, err := ('\n')
if err != nil {
("Cannot read from connection.\n", err)
}
s = (s, "\n ")
(s)
-, err = ("Thank you.\n")
if err != nil {
("Cannot write to connection.\n", err)
}
err = ()
if err != nil {
("Flush failed.", err)
}
}
func handleGob(rw *) {
("Receive GOB data:")
var data complexData
dec := (rw)
err := (&data)
if err != nil {
("Error decoding GOB data:", err)
return
}
("Outer complexData struct: \n%#v\n", data)
("Inner complexData struct: \n%#v\n", )
}
Client and server functions
Everything is ready, we can prepare our client and server functions.
The client function connects to the server and sends STRING and GOB requests.
The server starts listening for requests and triggers the appropriate processor.
// Called when the application uses the -connect=ip address
func client(ip string) error {
testStruct := complexData{
N: 23,
S: "string data",
M: map[string]int{"one": 1, "two": 2, "three": 3},
P: []byte("abc"),
C: &complexData{
N: 256,
S: "Recursive structs? Piece of cake!",
M: Map[string]int{"01": "10": 2, "11": 3},
},
}
rw, err := Open(ip + Port)
if err != nil {
return (err, "Client: Failed to open connection to " + ip + Port)
}
("Send the string request.")
n, err := ("STRING\n")
if err != nil {
return (err, "Could not send the STRING request (" + (n) + " bytes written)")
}
// Send STRING request. Send the request name and send the data.
("Send the string request.")
n, err = ("Additional data.\n")
if err != nil {
return (err, "Could not send additional STRING data (" + (n) + " bytes written)")
}
("Flush the buffer.")
err = ()
if err != nil {
return (err, "Flush failed.")
}
// Read the response
("Read the reply.")
response, err := ('\n')
if err != nil {
return (err, "Client: Failed to read the reply: '" + response + "'")
}
("STRING request: got a response:", response)
// Send a GOB request. Create an encoder to convert it directly into the request name. Send GOB
("Send a struct as GOB:")
("Outer complexData struct: \n%#v\n", testStruct)
("Inner complexData struct: \n%#v\n", )
enc := (rw)
n, err = ("GOB\n")
if err != nil {
return (err, "Could not write GOB data (" + (n) + " bytes written)")
}
err = (testStruct)
if err != nil {
return (err, "Encode failed for struct: %#v", testStruct)
}
err = ()
if err != nil {
return (err, "Flush failed.")
}
return nil
}
The client function is executed when specifying the connect flag when executing the application. This can be seen in the following code.
The following is the server program server. The server listens for incoming requests and schedules them to the registered specific related processor according to the request command name.
func server() error {
endpoint := NewEndpoint()
// Add processor function ("STRING", handleStrings)
("GOB", handleGOB)
// Start monitoring return ()
}
main function
The main function below can start both the client and the server, depending on whether the connect flag is set. If this flag is not available, the process will be started as the server and listen for incoming requests. If there is a flag, start as the client and connect to the host specified by this flag.
Both processes can be run on the same machine using localhost or 127.0.0.1.
func main() {
connect := ("connect", "", "IP address of process to join. If empty, go into the listen mode.")
()
// If the connect flag is set, enter client mode if *connect != '' {
err := client(*connect)
if err != nil {
("Error:", (err))
}
("Client done.")
return
}
// Otherwise enter server mode err := server()
if err != nil {
("Error:", (err))
}
("Server done.")
}
// Set the field flag for loggingfunc init() {
()
}
How to get and run the code
Step 1: Get the code. Note that the -d flag automatically installs binary to the $GOPATH/bin directory.
go get -d /appliedgo/networking
Step 2: cd to the source code directory.
cd $GOPATH/src//appliedgo/networking
Step 3: Run the server.
go run
Step 4: Open another shell, enter the source code directory (step 2), and then run the client.
go run -connect localhost
Tips
If you want to modify the source code slightly, here are some suggestions:
- Run client and server on different machines (in the same LAN).
- Beef up complexData with more maps and pointers, and see how gob deals with it.
- Start multiple clients at the same time to see if the server can handle them.
2017-02-09: Map is not thread-safe, so if you use the same map in different goroutines, you should use a mutex to control the map's access.
In the above code, map has been added before the goroutine is started, so you can safely modify the code and call AddHandleFunc() when the handleMessages goroutine is already running.
Summary of the knowledge learned in this chapter
- bufio application.
- Gob application.
- Maps are not safe when shared among multiple goroutines.
Reference link
- TCP/IP network original text
- Simple TCP server and client
- gob data
Summarize
The above is the entire content of this article. I hope that the content of this article has certain reference value for everyone's study or work. If you have any questions, you can leave a message to communicate. Thank you for your support.