What is a sticking problem
Recently, I am writing the Socket layer using Golang and found that sometimes the receiver will read multiple packets at once. So by looking up the information, I found that this is the legendary TCP sticking problem. The following is to reproduce this problem by writing code:
Server code server/
func main() {
l, err := ("tcp", ":4044")
if err != nil {
panic(err)
}
("listen to 4044")
for {
// Listen to a new connection, create a new goroutine and hand it over to the handleConn function to handle conn, err := ()
if err != nil {
("conn err:", err)
} else {
go handleConn(conn)
}
}
}
func handleConn(conn ) {
defer ()
defer ("closure")
("New Connection:", ())
result := (nil)
var buf [1024]byte
for {
n, err := (buf[0:])
(buf[0:n])
if err != nil {
if err == {
continue
} else {
("read err:", err)
break
}
} else {
("recv:", ())
}
()
}
}
Client code client/
func main() {
data := []byte("[This is a complete packet]")
conn, err := ("tcp", "localhost:4044", *30)
if err != nil {
("connect failed, err : %v\n", ())
return
}
for i := 0; i <1000; i++ {
_, err = (data)
if err != nil {
("write failed , err : %v\n", err)
break
}
}
}
Running results
listen to 4044
New connection: [::1]:53079
recv: [This is a complete data packet][This is a complete data packet][This is a complete data packet][This is a complete data packet][This is a complete data packet][This is a complete data packet][This is a complete data packet][This is a complete data packet][This is a complete data packet][This is a complete data packet][This is a complete data packet][This is a complete data packet][This is a complete data packet][This is a complete data packet][This is a complete data packet][This is a complete data packet][This is a complete data packet][This is a complete data packet][This is a complete data packet][This is a complete data packet][This is a complete data packet][This is a complete data packet][This is a complete data packet][This is a complete data packet][This is a complete data packet][This is a complete data packet][This is a complete data packet][This is a complete data packet][This is a complete data packet][This is a complete data packet][This is a complete data packet][This is a complete data packet][This is a complete data packet][This is a complete data packet][This is a complete data packet][This is a complete data packet][This is a complete data packet][This is a complete data packet][This is a complete data packet][This is a complete data packet][This is a complete data packet][This is a complete data packet][This is a complete data packet][This is a complete data packet][This is a complete data packet][This is a complete data packet][This is a complete data packet][This is a complete data packet][This is a complete data packet][This is a complete data packet][This is a complete data packet][This is a complete data packet][This is a complete data packet][This is a complete data packet][This is a complete data packet][This is a complete data packet][This is a complete data packet][This is a complete data packet][This is a complete data packet][This is a complete data packet][This is a complete data packet][This is a complete data packet][This is a complete data packet][This is a complete data packet A complete data packet][This is a complete data packet][This is a complete data packet][This is a complete data packet][This is a complete data packet][This is a complete data packet][This is a complete data packet][This is a complete data packet][This is a complete data packet][This is a complete data packet][This is a complete data packet][This is a complete data packet][This is a complete data packet][This is a complete data packet][This is a complete data packet][This is a complete data packet][This is a complete data
recv: �][This is a complete data packet][This is a complete data packet][This is a complete data packet][This is a complete data packet][This is a complete data packet][This is a complete data packet]
recv: [This is a complete packet]
recv: [This is a complete packet]
recv: [This is a complete data packet][This is a complete data packet][This is a complete data packet]
recv: [This is a complete packet]
...Omit other...
From the server console output, we can see that there are three types of outputs:
- One is a normal packet output.
- One is that multiple data packets are "sticked" together, and we define this read packet as a sticky packet.
- One is that a data packet is "disassembled" to form a broken packet. We define this packet as a half packet.
Why do half-pack and sticky pack appear?
- The client sends packets too fast for a period of time, and the server has not completed all of them. Then the data will be accumulated and a sticky packet will be generated.
- The defined read buffer is not large enough, and the data packet is too large or due to sticky packets, the server cannot read all at once, generating half a packet.
When should I consider dealing with half-pack and sticky packs?
A TCP connection is a long connection, that is, a connection is connected to send data multiple times.
The data sent each time is structured, such as JSON format data or the protocol of the data packet is defined by ourselves (the packet header contains the actual data length, protocol magic number, etc.).
Solution
- Fixed-length separation (each packet is the maximum length, and special characters are used to fill it when it is insufficient), but when the data is insufficient, transmission resources will be wasted
- Use specific characters to split the packet, but if the data contains split characters, a bug will occur.
- Adding a length field to the data packet makes up for the shortcomings of the above two ideas. It is recommended to use
Unpacking demonstration
Through the above analysis, we should use the third idea to solve the problem of unpacking and sticking.
Golang's bufio library provides us with Scanner to solve this type of split data problem.
type Scanner
Scanner provides a convenient interface for reading data such as a file of newline-delimited lines of text. Successive calls to the Scan method will step through the 'tokens' of a file, skipping the bytes between the tokens. The specification of a token is defined by a split function of type SplitFunc; the default split function breaks the input into lines with line termination stripped. Split functions are defined in this package for scanning a file into lines, bytes, UTF-8-encoded runes, and space-delimited words. The client may instead provide a custom split function.
Simply put, it is:
Scanner provides a convenient interface for reading data. Continuous calls to the Scan method will get the "tokens" of the file one by one, skipping the bytes between tokens. The specification of token is defined by a function of type SplitFunc. We can provide custom splitting instead.
Next, let's take a look at what a function of type SplitFunc looks like:
type SplitFunc func(data []byte, atEOF bool) (advance int, token []byte, err error)
Examples of use provided on the Golang official website document 🌰:
func main() {
// An artificial input source.
const input = "1234 5678 1234567901234567890"
scanner := ((input))
// Create a custom split function by wrapping the existing ScanWords function.
split := func(data []byte, atEOF bool) (advance int, token []byte, err error) {
advance, token, err = (data, atEOF)
if err == nil && token != nil {
_, err = (string(token), 10, 32)
}
return
}
// Set the split function for the scanning operation.
(split)
// Validate the input
for () {
("%s\n", ())
}
if err := (); err != nil {
("Invalid input: %s", err)
}
}
So, we can rewrite our program like this:
Server code server/
func main() {
l, err := ("tcp", ":4044")
if err != nil {
panic(err)
}
("listen to 4044")
for {
conn, err := ()
if err != nil {
("conn err:", err)
} else {
go handleConn2(conn)
}
}
}
func packetSlitFunc(data []byte, atEOF bool) (advance int, token []byte, err error) {
// Check whether the atEOF parameter and the four bytes of the packet header are 0x123456 (the magic number of the protocol we define) if !atEOF && len(data) > 6 && .Uint32(data[:4]) == 0x123456 {
var l int16
// Read out the length of the actual data in the data packet (size is 0 ~ 2^16) ((data[4:6]), , &l)
pl := int(l) + 6
if pl <= len(data) {
return pl, data[:pl], nil
}
}
return
}
func handleConn2(conn ) {
defer ()
defer ("closure")
("New Connection:", ())
result := (nil)
var buf [65542]byte // Since there are only two bytes that identify the length of the packet, the maximum packet is 2^16+4 (magic number)+2 (length identification) for {
n, err := (buf[0:])
(buf[0:n])
if err != nil {
if err == {
continue
} else {
("read err:", err)
break
}
} else {
scanner := (result)
(packetSlitFunc)
for () {
("recv:", string(()[6:]))
}
}
()
}
}
Client code client/
func main() {
l, err := ("tcp", ":4044")
if err != nil {
panic(err)
}
("listen to 4044")
for {
conn, err := ()
if err != nil {
("conn err:", err)
} else {
go handleConn2(conn)
}
}
}
func packetSlitFunc(data []byte, atEOF bool) (advance int, token []byte, err error) {
// Check whether the atEOF parameter and the four bytes of the packet header are 0x123456 (the magic number of the protocol we define) if !atEOF && len(data) > 6 && .Uint32(data[:4]) == 0x123456 {
var l int16
// Read out the length of the actual data in the data packet (size is 0 ~ 2^16) ((data[4:6]), , &l)
pl := int(l) + 6
if pl <= len(data) {
return pl, data[:pl], nil
}
}
return
}
func handleConn2(conn ) {
defer ()
defer ("closure")
("New Connection:", ())
result := (nil)
var buf [65542]byte // Since there are only two bytes that identify the length of the packet, the maximum packet is 2^16+4 (magic number)+2 (length identification) for {
n, err := (buf[0:])
(buf[0:n])
if err != nil {
if err == {
continue
} else {
("read err:", err)
break
}
} else {
scanner := (result)
(packetSlitFunc)
for () {
("recv:", string(()[6:]))
}
}
()
}
}
Running results
listen to 4044
New connection: [::1]:55738
recv: [This is a complete packet]
recv: [This is a complete packet]
recv: [This is a complete packet]
recv: [This is a complete packet]
recv: [This is a complete packet]
recv: [This is a complete packet]
recv: [This is a complete packet]
recv: [This is a complete packet]
...Omit other...
Summarize
The above is the entire content of this article. I hope that the content of this article has certain reference value for your study or work. Thank you for your support.