introduction
In high concurrency applications, data access performance is often one of the key bottlenecks in system performance. As a high-performance in-memory database, Redis is widely used in caching, session storage, rankings and other scenarios.
However, in some scenarios where a large number of Redis commands are required, the accumulation of network round-trip delays (RTT) may significantly affect performance.
To solve this problem, Redis provides pipeline technology, allowing clients to send multiple commands to the server at once and obtain all results in a network interaction, thereby greatly improving operational efficiency.
1. Redis pipeline technology principle
Redis pipeline is a network communication optimization technology that allows clients to send multiple command requests to the Redis server without waiting for the response of the previous command, and the last time they get the response results of all commands at once.
In standard Redis operations, each command execution follows the "request-response" pattern:
- The client sends commands to the server
- Server processing commands
- The server returns the response to the client
- Client receives response
In this mode, each command requires a full network round trip, and when a large number of commands are executed, the network latency accumulates exponentially.
When using pipeline technology:
- The client sends multiple commands to the server at once
- The server processes all commands in sequence
- The server returns all command responses at once
- The client receives all responses at once
2. Why do you need a Redis pipeline?
1. Performance Advantages
Network latency is usually one of the main bottlenecks in Redis operations. In a typical Redis operation, the command execution time may be only a few microseconds, but the network round trip delay may reach several milliseconds, hundreds of times the command execution time.
2. Applicable scenarios
Redis pipelines are particularly suitable for the following scenarios:
Special note: Pipeline does not guarantee atomicity, requires Transaction
- Batch query or update
- Scenarios of executing a large number of simple commands
- High latency network environments that require reduced network round trips
3. Implementing Redis pipeline in Java
There are many Redis clients in the Java ecosystem, commonly used ones include Jedis, Lettuce, and Redisson.
1. Implement Redis Pipeline with Jedis
Jedis is one of the earliest and widely used Redis Java clients, providing an intuitive API.
First, add Jedis dependencies:
<dependency>
<groupId></groupId>
<artifactId>jedis</artifactId>
<version>4.3.1</version>
</dependency>
Example of basic pipeline usage:
import ;
import ;
import ;
import ;
import ;
import ;
public class JedisPipelineExample {
public static void main(String[] args) {
// Create a Jedis connection try (Jedis jedis = new Jedis("localhost", 6379)) {
// Execute multiple commands without using pipelines long startTime = ();
for (int i = 0; i < 10000; i++) {
String key = "key" + i;
String value = "value" + i;
(key, value);
}
long endTime = ();
("It takes time to execute 10,000 SET commands without using the pipeline: " + (endTime - startTime) + "ms");
// Execute multiple commands using pipeline startTime = ();
Pipeline pipeline = ();
for (int i = 0; i < 10000; i++) {
String key = "key" + i;
String value = "value" + i;
(key, value);
}
// Execute the pipeline and get all responses (); // Or use() to get all return values endTime = ();
("It takes time to execute 10,000 SET commands using the pipeline: " + (endTime - startTime) + "ms");
}
}
}
Get command results in the pipeline:
public void pipelineWithResults() {
try (Jedis jedis = new Jedis("localhost", 6379)) {
Pipeline pipeline = ();
// Send multiple commands and save the response Map<String, Response<String>> responseMap = new HashMap<>();
for (int i = 0; i < 10; i++) {
String key = "key" + i;
(key, "value" + i); // Set some values for testing first
// Save the response object to the map (key, (key));
}
// Execute pipeline ();
// Processing results for (<String, Response<String>> entry : ()) {
(() + ": " + ().get());
}
}
}
2. Use Lettuce to implement Redis pipeline
Lettuce is another popular Redis Java client, based on Netty, providing an asynchronous and responsive programming model.
Add Lettuce dependencies:
<dependency>
<groupId></groupId>
<artifactId>lettuce-core</artifactId>
<version>6.2.</version>
</dependency>
Example of basic pipeline usage:
import ;
import ;
import ;
import ;
import ;
import ;
import ;
public class LettucePipelineExample {
public static void main(String[] args) {
// Create Redis client RedisClient redisClient = ("redis://localhost:6379");
try (StatefulRedisConnection<String, String> connection = ()) {
// Get the asynchronous command API RedisAsyncCommands<String, String> commands = ();
// By default, Lettuce is automatically pipelined, here we manually control batch processing (false);
// Record the start time long startTime = ();
// Create a list of asynchronous results List<RedisFuture<?>> futures = new ArrayList<>();
// Send multiple commands for (int i = 0; i < 10000; i++) {
String key = "key" + i;
String value = "value" + i;
((key, value));
}
// Flash out all commands to the Redis server ();
// Wait for all commands to complete for (RedisFuture<?> future : futures) {
try {
();
} catch (InterruptedException | ExecutionException e) {
();
}
}
// Record the end time long endTime = ();
("It takes time to execute 10,000 SET commands using the Lettuce pipeline: " + (endTime - startTime) + "ms");
// Restore automatic refresh (true);
} finally {
// Close the client ();
}
}
}
More complex Lettuce pipeline operation examples:
public void lettucePipelineWithDifferentCommands() {
RedisClient redisClient = ("redis://localhost:6379");
try (StatefulRedisConnection<String, String> connection = ()) {
RedisAsyncCommands<String, String> commands = ();
(false);
// Set the hash structure for expiration time RedisFuture<String> hmsetFuture = ("user:1000",
("name", "John Doe",
"email", "john@",
"age", "30"));
RedisFuture<Boolean> expireFuture = ("user:1000", 3600);
// Increment counter RedisFuture<Long> incrFuture = ("visitsCounter");
// Add multiple elements to the collection RedisFuture<Long> saddFuture = ("activeUsers", "1000", "1001", "1002");
// Get the collection size RedisFuture<Long> scardFuture = ("activeUsers");
// Flash out all commands ();
try {
// Get and process the results ("HMSET Results: " + ());
("EXPIRE Results: " + ());
("INCR Results: " + ());
("SADD Results: " + ());
("SCARD Results: " + ());
} catch (InterruptedException | ExecutionException e) {
();
}
(true);
} finally {
();
}
}
3. Use Redis Pipeline in Spring Boot
In Spring Boot apps, you can easily use pipelines with Spring Data Redis:
import ;
import ;
import ;
import ;
import ;
@Service
public class RedisPipelineService {
@Autowired
private RedisTemplate<String, String> redisTemplate;
public void executePipelinedOperations() {
List<Object> results = ((RedisCallback<Object>) connection -> {
// Perform multiple operations in the pipeline ().set("key1".getBytes(), "value1".getBytes());
().set("key2".getBytes(), "value2".getBytes());
().get("key1".getBytes());
().hSet("hash1".getBytes(), "field1".getBytes(), "value1".getBytes());
().hGetAll("hash1".getBytes());
// Return null, the result will be returned by the executePipelined method return null;
});
// Processing results ("Pipeline execution result:");
for (int i = 0; i < (); i++) {
("result " + i + ": " + (i));
}
}
}
4. Redis pipeline best practices and precautions
1. Pipeline usage suggestions
Batch size control
Commands in the pipeline accumulate in the client buffer, so too large batches can cause memory problems. It is recommended that each batch be controlled between 1000-10000 commands.
// Process a large number of commands in batchespublic void executeBatchOperations(List<String> keys, Jedis jedis) {
int batchSize = 1000;
for (int i = 0; i < (); i += batchSize) {
Pipeline pipeline = ();
int end = (i + batchSize, ());
for (int j = i; j < end; j++) {
((j));
}
();
}
}
Use in conjunction with transactions
The Redis pipeline itself does not guarantee atomicity. If atomicity is required, it can be used in conjunction with transactions (MULTI/EXEC).
public void pipelineWithTransaction(Jedis jedis) {
Pipeline pipeline = ();
(); // Start a transaction
("key1", "value1");
("key2", "value2");
("counter");
(); // Submit transaction (); // Submit the pipeline}
Exception handling
If there is an error in the command in the pipeline, the exception will not be immediately thrown, but will be thrown when executing sync() or syncAndReturnAll(). Be sure to handle exceptions well.
public void safeExecutePipeline(Jedis jedis) {
Pipeline pipeline = ();
try {
for (int i = 0; i < 1000; i++) {
("key" + i, "value" + i);
}
();
} catch (Exception e) {
((),e);
// Error recovery logic }
}
2. Things to note
Memory consumption
Command responses in the pipeline will accumulate in client memory. Pay attention to client memory pressure when using large batches.
Network timeout
Execution of a large number of commands in a pipeline may cause network timeout, so the client timeout time should be reasonably configured.
// Set a longer timeoutpublic void configureTimeoutsForPipeline() {
Jedis jedis = new Jedis("localhost", 6379);
().setConnectionTimeout(30000); // 30 seconds connection timeout ().setSoTimeout(30000); // 30 seconds operation timeout
// Perform large-scale pipeline operations...
();
}
Compare with Lua scripts
For complex operations that require atomicity, you can also consider using Lua scripts instead of pipelines + transactions.
public void luaScriptVsPipeline(Jedis jedis) {
// Use Lua script to perform atomic operations String script = "('SET', KEYS[1], ARGV[1]); " +
"('SET', KEYS[2], ARGV[2]); " +
"return ('INCR', KEYS[3])";
Object result = (script,
("key1", "key2", "counter"),
("value1", "value2"));
("Lua script execution result: " + result);
}
Pipeline is incompatible with publish subscriptions
Pipeline cannot be used for Redis's publish subscription operations.
5. Summary
Redis pipeline technology significantly improves the performance of Redis operations by reducing the number of network round trips, making it particularly suitable for batch operation scenarios.
It should be noted that although Redis pipelines can significantly improve performance, they should also pay attention to their limitations, such as not ensuring atomicity, and possibly increasing client memory pressure.
In practical applications, appropriate technical combinations should be selected according to the specific scenario, such as pipeline + transaction, pipeline + Lua script, etc., to obtain the best performance and reliability balance.
The above is the detailed explanation of the code that implements Redis pipeline technology in Java. For more information about Java's implementation of Redis pipeline, please pay attention to my other related articles!