Detailed explanation of the redis replication ring buffer algorithm

Redis'sCopy ring buffer(Replication Backlog) is to implement the master-slave nodeIncremental synchronizationThe core mechanism of (Partial Resynchronization).

Its essence is aFixed-size memory ring queue, used to temporarily store the most recently propagated write commands of the master node.

When the slave node is temporarily disconnected and reconnected, if the required data is still in the buffer, the master node can directly send incremental data to avoid the overhead of full synchronization.

1. The function of ring buffer

1. Incremental synchronization
   When the node is disconnected and reconnected, priority is given to trying to recover the lost data from the buffer to avoid full synchronization (RDB transmission).
2. Reduce the impact of network jitter
   When the network is unstable, the buffer retains the most recent data, improving the system's fault tolerance.
3. Efficient memory management
   A fixed-size ring structure avoids unlimited memory growth, and old data will be overwritten by new data.

2. The core field of the ring buffer

In RedisINFO replicationIn the output, fields related to the ring buffer include:

3. Principles of ring buffer algorithm

1. Data structure

The buffer is aCharacter array, logically considered a ring (similar to a loop queue).

Implicit management through two pointers:

• Write pointer:correspondmaster_repl_offset, indicates the location of the latest written to the master node.
• Starting pointer:correspondrepl_backlog_first_byte_offset, indicates the starting position of the earliest data in the buffer.

2. Write data

Each time the master node propagates a write command:

• Append commands to the buffer.
• renewmaster_repl_offset(Increase the byte length of the command).
• If the buffer is full (repl_backlog_histlen == repl_backlog_size), overwrite the old data and move the starting pointer forward (repl_backlog_first_byte_offsetincrement).

3. Coverage mechanism

• Trigger condition:whenmaster_repl_offset - repl_backlog_first_byte_offset > repl_backlog_size。
• Coverage behavior: New data covers old data,repl_backlog_first_byte_offsetPush forward to ensure the buffer size is fixed.

4. Synchronization logic when reconnecting from nodes

When the slave node reconnects the master node:

Send your ownslave_repl_offset(Last offset copied).

Master node check:

• ifslave_repl_offsetexist[repl_backlog_first_byte_offset, master_repl_offset]Within range:
• Incremental synchronization: Extract from the bufferslave_repl_offset + 1arrivemaster_repl_offsetThe data between them is sent to the slave node.
• otherwise:
• Fully synchronized: Generate an RDB snapshot and transmit all data.

4. Configuration optimization suggestions

Buffer size (repl-backlog-size)

• It needs to be adjusted according to the network environment and data writing rate.
• Formula suggestions：Buffer size ≥ Maximum disconnection time × Average write rate。
• For example: If the network may be disconnected for up to 60 seconds and the master node writes 10KB per second, the buffer will be set to at least60s × 10KB = 600KB(The actual suggestion is slightly larger).

Buffer retention time (repl-backlog-ttl)

• The default is 3600 seconds (1 hour), which indicates the time when the master node retains the buffer when there is no slave node connection.
• If all slave nodes are disconnected for a long time, the buffer will be released after the timeout to save memory.

V. Sample Scenario

Assume that the buffer size is 1000 bytes, the initial state:

repl_backlog_first_byte_offset = 1
master_repl_offset = 1
repl_backlog_histlen = 0

Write 500 bytes of data

• master_repl_offsetBecome501，repl_backlog_histlen = 500。
• The buffer is not full, and the starting point pointer remains unchanged.

Write 600 bytes of data

• Total space required500 + 600 = 1100, exceeds the buffer size (1000).
• Overwrite old data, start pointer forward to101(cover the first 100 bytes).
• repl_backlog_first_byte_offset = 101，master_repl_offset = 1101，repl_backlog_histlen = 1000。

Reconnection from node disconnection

• If from the nodeslave_repl_offset = 800：
• exist[101, 1101]Within the range, trigger incremental synchronization.
• If from the nodeslave_repl_offset = 50：
• Not within range, trigger full synchronization.

6. Summary

Redis's ring buffer significantly improves the robustness and performance of master-slave replication through efficient memory management and offset tracking mechanisms.

Reasonable configurationrepl-backlog-sizeand monitoringrepl_backlog_histlenIt is the key to avoiding full synchronization.

The above is personal experience. I hope you can give you a reference and I hope you can support me more.