3 strategies for generating distributed global unique IDs based on Redis

In distributed system design, global unique ID is a basic and critical component. With the expansion of business scale and the evolution of system architecture to microservices, traditional stand-alone self-increasing IDs can no longer meet the needs. Highly concurrency and high availability distributed ID generation solutions have become a necessary condition for building a reliable distributed system.

Redis has high performance, atomic operation and simple and easy-to-use features, so we can realize the generation of global unique IDs based on Redis.

The core requirements of distributed ID

An excellent distributed ID generation solution should meet the following requirements

Global uniqueness: Ensure that the ID is not duplicated in the entire distributed system
high performance: Can quickly generate IDs and support high concurrency scenarios
Highly available: Avoid single point of failure and ensure continuous service availability
Trends are increasing: The generated ID is roughly increasing, which is convenient for database indexing and sharding
Security (optional): It does not contain sensitive information and is not easy to be speculated and forged

1. Simple self-increment ID based on INCR command

principle

This is the most direct way to implement Redis distributed ID, using RedisINCRCommand atomically increments a counter to ensure unique ID in a distributed environment.

Code implementation

import ;
import ;

@Component
public class RedisSimpleIdGenerator {
    private final RedisTemplate&lt;String, String&gt; redisTemplate;
    private final String ID_KEY;
    
    public RedisSimpleIdGenerator(RedisTemplate&lt;String, String&gt; redisTemplate) {
         = redisTemplate;
        this.ID_KEY = "distributed:id:generator";
    }
    
    /**
      * Generate the next ID
      * @return Unique ID
      */
    public long nextId() {
        Long id = ().increment(ID_KEY);
        if (id == null) {
            throw new RuntimeException("Failed to generate id");
        }
        return id;
    }
    
    /**
      * Generate ID for the specified service
      * @param bizTag Business tags
      * @return Unique ID
      */
    public long nextId(String bizTag) {
        String key = ID_KEY + ":" + bizTag;
        Long id = ().increment(key);
        if (id == null) {
            throw new RuntimeException("Failed to generate id for " + bizTag);
        }
        return id;
    }
    
    /**
      * Get the current ID value but not increment
      * @param bizTag Business tags
      * @return Current ID value
      */
    public long currentId(String bizTag) {
        String key = ID_KEY + ":" + bizTag;
        String value = ().get(key);
        return value != null ? (value) : 0;
    }
}

Pros and cons

advantage

The implementation is extremely simple, only one Redis operation is required
ID is strictly incremented, suitable as the database primary key
Support multi-service ID isolation

shortcoming

Redis single point of failure will cause ID generation service to be unavailable
Master-slave switching may cause ID duplication
Cannot include business meaning

Applicable scenarios

Auto-increment primary key generation for small and medium-sized systems
Business scenarios that require ID continuity
Applications for single data center deployment

2. Bulk ID generation based on Lua scripts

principle

A batch of IDs can be obtained at one time through Lua scripts to reduce the number of network round trips. Clients can allocate IDs in memory in sequence, significantly improving performance.

Code implementation

import ;
import ;
import ;

import ;
import ;
import ;
import ;
import ;

@Component
public class RedisBatchIdGenerator {
    private final RedisTemplate&lt;String, String&gt; redisTemplate;
    private final String ID_KEY = "distributed:batch:id";
    private final DefaultRedisScript&lt;Long&gt; batchIncrScript;
    
    // Batch size    private final int BATCH_SIZE = 1000;
    
    // Local counter and lock    private AtomicLong currentId = new AtomicLong(0);
    private AtomicLong endId = new AtomicLong(0);
    private final Lock lock = new ReentrantLock();
    
    public RedisBatchIdGenerator(RedisTemplate&lt;String, String&gt; redisTemplate) {
         = redisTemplate;
        
        // Create Lua script        String scriptText = 
            "local key = KEYS[1] " +
            "local step = tonumber(ARGV[1]) " +
            "local currentValue = ('incrby', key, step) " +
            "return currentValue";
        
         = new DefaultRedisScript&lt;&gt;();
        (scriptText);
        ();
    }
    
    /**
      * Get the next ID
      */
    public long nextId() {
        // If the current ID exceeds the allocation range, re-acquire a batch        if (() &gt;= ()) {
            ();
            try {
                // Double check to prevent repeated acquisition of multi-threads                if (() &gt;= ()) {
                    // Execute Lua script to get a batch of IDs                    Long newEndId = (
                        batchIncrScript, 
                        (ID_KEY),
                        (BATCH_SIZE)
                    );
                    
                    if (newEndId == null) {
                        throw new RuntimeException("Failed to generate batch ids");
                    }
                    
                    // Set a new ID range                    (newEndId);
                    (newEndId - BATCH_SIZE);
                }
            } finally {
                ();
            }
        }
        
        // Assign the next ID        return ();
    }
    
    /**
      * Generate ID for the specified service
      */
    public long nextId(String bizTag) {
        // In actual projects, independent counters and ranges should be maintained for each business tag        // This is simplified, using only different Redis keys        String key = ID_KEY + ":" + bizTag;
        
        Long newEndId = (
            batchIncrScript, 
            (key),
            (1)
        );
        
        return newEndId != null ? newEndId : -1;
    }
}

Pros and cons

advantage

Significantly reduce the number of Redis network requests
Client caches ID segments, greatly improving performance
Reduce Redis Server Pressure
Support burst traffic processing

shortcoming

Implementation complexity increases
Service restart may lead to waste of ID segments

Applicable scenarios

High concurrency system, scenarios that require extremely high ID generation performance
Businesses with less stringent requirements for ID continuity
Scenarios where a small number of ID waste can tolerate

3. Redis-based segmented ID allocation (segment mode)

principle

The number segment mode is an optimized batch ID generation solution, which reduces competition among services by pre-allocating number segments (ID ranges), and at the same time introduces a dual-buffer mechanism to improve availability.

Code implementation

import ;
import ;
import ;

import ;
import ;
import ;
import ;
import ;
import ;

@Component
public class RedisSegmentIdGenerator {
    private final RedisTemplate&lt;String, String&gt; redisTemplate;
    private final String SEGMENT_KEY = "distributed:segment:id";
    private final DefaultRedisScript&lt;Long&gt; segmentScript;
    
    // Size    private final int SEGMENT_STEP = 1000;
    // Loading factor, when the current number segment is used to load the next number segment asynchronously when using this percentage    private final double LOAD_FACTOR = 0.7;
    
    // Map that stores business segment information    private final Map&lt;String, SegmentBuffer&gt; businessSegmentMap = new ConcurrentHashMap&lt;&gt;();
    
    public RedisSegmentIdGenerator(RedisTemplate&lt;String, String&gt; redisTemplate) {
         = redisTemplate;
        
        // Create Lua script        String scriptText = 
            "local key = KEYS[1] " +
            "local step = tonumber(ARGV[1]) " +
            "local value = ('incrby', key, step) " +
            "return value";
        
         = new DefaultRedisScript&lt;&gt;();
        (scriptText);
        ();
    }
    
    /**
      * Get the next ID
      * @param bizTag Business tags
      * @return Unique ID
      */
    public long nextId(String bizTag) {
        // Get or create a number segment buffer        SegmentBuffer buffer = (
            bizTag, k -&gt; new SegmentBuffer(bizTag));
        
        return ();
    }
    
    /**
      * Internal number segment buffer class, implementing dual-buffer mechanism
      */
    private class SegmentBuffer {
        private String bizTag;
        private Segment[] segments = new Segment[2]; // Double Buffers        private volatile int currentPos = 0; // The current segment location        private Lock lock = new ReentrantLock();
        private volatile boolean isLoadingNext = false; // Is the next number segment being loaded asynchronously        
        public SegmentBuffer(String bizTag) {
             = bizTag;
            segments[0] = new Segment(0, 0);
            segments[1] = new Segment(0, 0);
        }
        
        /**
          * Get the next ID
          */
        public long nextId() {
            // Get the current number segment            Segment segment = segments[currentPos];
            
            // If the current segment is empty or used up, switch to another segment            if (!() || () &gt; ()) {
                ();
                try {
                    // Double check the current segment status                    segment = segments[currentPos];
                    if (!() || () &gt; ()) {
                        // Switch to another segment                        currentPos = (currentPos + 1) % 2;
                        segment = segments[currentPos];
                        
                        // If another segment is not initialized or used up, then load synchronously                        if (!() || () &gt; ()) {
                            loadSegmentFromRedis(segment);
                        }
                    }
                } finally {
                    ();
                }
            }
            
            // Check whether the next segment needs to be loaded asynchronously            long value = ();
            if (value &gt; () + (() - ()) * LOAD_FACTOR
                    &amp;&amp; !isLoadingNext) {
                isLoadingNext = true;
                // Asynchronously load the next segment                new Thread(() -&gt; {
                    Segment nextSegment = segments[(currentPos + 1) % 2];
                    loadSegmentFromRedis(nextSegment);
                    isLoadingNext = false;
                }).start();
            }
            
            return value;
        }
        
        /**
          * Loading segment from Redis
          */
        private void loadSegmentFromRedis(Segment segment) {
            String key = SEGMENT_KEY + ":" + bizTag;
            
            // Execute Lua script to get the maximum number segment value            Long max = (
                segmentScript, 
                (key),
                (SEGMENT_STEP)
            );
            
            if (max == null) {
                throw new RuntimeException("Failed to load segment from Redis");
            }
            
            // Set number range            long min = max - SEGMENT_STEP + 1;
            (max);
            (min);
            (min - 1); // Set to min-1, the first incrementAndGet returns min            (true);
        }
    }
    
    /**
      * Internal number segment class, storing range information of number segments
      */
    private class Segment {
        private long min; // Minimum value        private long max; // Maximum value        private AtomicLong value; // Current value        private volatile boolean initialized; // Is it initialized        
        public Segment(long min, long max) {
             = min;
             = max;
             = new AtomicLong(min);
             = false;
        }
        
        public long getValue() {
            return ();
        }
        
        public void setValue(long value) {
            (value);
        }
        
        public long incrementAndGet() {
            return ();
        }
        
        public long getMin() {
            return min;
        }
        
        public void setMin(long min) {
             = min;
        }
        
        public long getMax() {
            return max;
        }
        
        public void setMax(long max) {
             = max;
        }
        
        public boolean isInitialized() {
            return initialized;
        }
        
        public void setInitialized(boolean initialized) {
             = initialized;
        }
    }
}

Pros and cons

advantage

Dual Buffer design, high availability
Asynchronously load the next number segment, with higher performance
Redis access frequency significantly
Even if Redis is temporarily unavailable, an ID of a period of time can still be assigned.

shortcoming

Complex implementation and large code volume
When deploying multiple instances, the number segments obtained by each instance are not continuous.
The ID in the number segment may be wasted when restarting the service
Need to maintain state in memory

Applicable scenarios

Services with high requirements for ID generation availability
Distributed systems requiring high-performance and multi-server deployment

4. Performance comparison and selection suggestions

Strategy	performance	Availability	ID length	Implement complexity	Monotonously increasing
INCR Command	★★★☆☆	★★☆☆☆	Increment integer	Low	Strictly increment
Lua batch generation	★★★★★	★★★☆☆	Increment integer	middle	In-batch increment
Segmented ID	★★★★★	★★★★☆	Increment integer	high	In-segment increment

5. Practical optimization skills

1. Redis high availability configuration

// Configure Redis Sentinel mode to improve usability@Bean
public RedisConnectionFactory redisConnectionFactory() {
    RedisSentinelConfiguration sentinelConfig = new RedisSentinelConfiguration()
        .master("mymaster")
        .sentinel("127.0.0.1", 26379)
        .sentinel("127.0.0.1", 26380)
        .sentinel("127.0.0.1", 26381);
    
    return new LettuceConnectionFactory(sentinelConfig);
}

2. ID warm-up strategy

// Preheat ID generator when system starts@PostConstruct
public void preWarmIdGenerator() {
    // Obtain a batch of IDs in advance to ensure that the system is ready to use immediately after starting    for (int i = 0; i &lt; 10; i++) {
        try {
            ("order");
            ("user");
            ("payment");
        } catch (Exception e) {
            ("Failed to pre-warm ID generator", e);
        }
    }
}

3. Downgrade strategy

// Downgrade policy when Redis is unavailablepublic long nextIdWithFallback(String bizTag) {
    try {
        return (bizTag);
    } catch (Exception e) {
        ("Failed to get ID from Redis, using local fallback", e);
        // Use local UUID or other alternatives        return (().getMostSignificantBits());
    }
}

6. Conclusion

When choosing a suitable distributed ID generation strategy, it is necessary to comprehensively consider the system scale, performance requirements, reliability requirements and implementation complexity. No matter which solution you choose, you should pay attention to high availability design, add monitoring and early warning mechanisms to ensure the stable operation of ID generation services.

In practice, these solutions can be combined and optimized based on business needs, such as selecting different strategies for different services, or embeding business identifiers in IDs, etc., to create a distributed ID generation solution that is more suitable for your own system.

This is the article about three strategies for generating distributed global unique IDs based on Redis. For more related content related to Redis generation distributed global unique IDs, please search for my previous articles or continue browsing the related articles below. I hope everyone will support me in the future!