Best practices for redis storage breakpoints to continue file status

1. The file upload status stored in Redis

Redis provides a rich data structure that allows flexibly storing and updating various states of file uploads. Here are several common implementation methods.

Use Hash to store file status

In Redis, each file upload status can use a unique key (such asfile_idOr a combination of user ID + file name) to identify, and all data related to file upload (such as the number of uploaded bytes, total file size, uploaded chunks, etc.) are stored in a Hash table. For example:

Key: file_upload:<file_id>
Fields:
  - uploaded_size: Number of uploaded bytes
  - file_size: Total file size
  - chunks: Uploaded chunked index list
  - status: Current upload status（like "uploading", "paused", "completed"）
  - last_update_time: Last updated time

Storage block upload status

For large file chunk uploads, Redis's set (Set) or list (List) can store each uploaded chunk. for example:

Key: file_chunks:<file_id>
Set: {chunk_1, chunk_2, chunk_3, ...}

In this way, each uploaded chunk will be recorded, and the upload status can be accurately tracked and managed.

Use TTL for status expiration management

For temporary status of file upload, you can set an appropriate expiration time. For example, when the file upload is completed, the status data in Redis will be automatically cleaned:

EXPIRE file_upload:<file_id> 86400  # Set the file status to expire in one day

This avoids the long-term memory occupancy of useless data.

2. Redis is consistent with the database

Although Redis is efficient and fast, it is a memory database after all, and the stored data can be lost when the system restarts or fails. Therefore, it is particularly important to keep the breakpoint continuous transmission state in Redis consistent with the persistent data in the database.

Method 1: Synchronize regularly

The easiest way is to periodically sync the upload status in Redis to the database via a timed task such as Cron Job. You can set up a background service that scans all upload statuses in Redis every certain time (such as hourly) and writes them to the database.

Database table design：

CREATE TABLE file_upload_status (
    file_id VARCHAR(255) PRIMARY KEY,
    uploaded_size BIGINT,
    file_size BIGINT,
    chunks TEXT,  -- Store uploaded chunking information，The format is JSON
    status ENUM('uploading', 'paused', 'completed'),
    last_update_time DATETIME
);

Method 2: Real-time synchronization

If higher real-time performance is required, real-time synchronization can be adopted. Whenever the upload status of a file in Redis changes, it is immediately synchronized to the database. You can use a message queue (such as Kafka or RabbitMQ) to handle synchronous tasks asynchronously, or update synchronously directly in your code.

For example:

1. When updating the status in Redis, an asynchronous task is triggered.
2. Using Redis'sKeyspace Notifications(Keyspace Notification) to listen for changes in the Redis middle key and automatically synchronize the changes to the database.

Method 3: Double writing mechanism

The double write mechanism is to directly synchronize the database every time Redis is updated. This approach ensures that each write operation affects both Redis and the database, thus avoiding data inconsistencies.

For example, when updating file upload progress:

MULTI  # Redis transactionsHSET file_upload:<file_id> uploaded_size 1024
EXEC

-- Update the database simultaneously
UPDATE file_upload_status SET uploaded_size = 1024 WHERE file_id = '<file_id>';

Method 4: Recovery after system restart

In order to restore the upload state after the system restarts, the upload state can be loaded from the database when the system starts and synchronized to Redis. In this way, even if the service is restarted, the status of the interruption point will not be lost.

for record in ("SELECT * FROM file_upload_status WHERE status = 'uploading'"):
    (f"file_upload:{record['file_id']}", {
        "uploaded_size": record['uploaded_size'],
        "file_size": record['file_size'],
        "status": record['status']
    })

3. Consistency guarantee

To ensure data consistency between Redis and the database, we can adopt the following strategies:

1. Transaction control: Ensure that the write operations of Redis and database are completed in the same transaction to ensure data consistency.
2. Message Queue: Record Redis change events through message queues, and then synchronize them to the database by backend services, thereby avoiding performance bottlenecks caused by direct operation of the database.
3. Idepotency design: Ensure that each operation is idempotent, that is, the data will not conflict or inconsistency even if it is repeated.
4. Regular data reconciliation: Regularly compare the data in Redis and the database to ensure consistency. If inconsistencies are found, a repair mechanism can be triggered.

4. Summary

As a temporary storage, Redis can efficiently support the state management of the breakpoint continuous transmission system. Combined with timing synchronization, real-time updates or double-write mechanisms, it can ensure consistency between Redis and data in the database. When implementing it, we must also pay attention to consistency guarantees to avoid data loss caused by Redis failure or restart.

5. Code Practice

5.1 Store file upload status in Redis

First, we need to create a Hash table in Redis for the upload status of each file to record the status of the file. Suppose we are uploading a large file, using chunk uploads.

#include <hiredis/>
#include <iostream>
#include <string>

// Connect to RedisredisContext* connectToRedis() {
    redisContext* c = redisConnect("127.0.0.1", 6379);
    if (c == NULL || c->err) {
        if (c) {
            std::cerr << "Redis connection error: " << c->errstr << std::endl;
        } else {
            std::cerr << "Unable to allocate redis context\n";
        }
        exit(1);
    }
    return c;
}

// Set file upload statusvoid setFileUploadStatus(redisContext* c, const std::string& file_id, size_t uploaded_size, size_t file_size, const std::string& status) {
    redisReply* reply = (redisReply*)redisCommand(c, 
        "HSET file_upload:%s uploaded_size %zu file_size %zu status %s",
        file_id.c_str(), uploaded_size, file_size, status.c_str());
    freeReplyObject(reply);
}

int main() {
    redisContext* c = connectToRedis();
    
    std::string file_id = "file123";
    size_t uploaded_size = 5000;  // Uploaded 5000 bytes    size_t file_size = 10000;     // Total file size: 10000 bytes    std::string status = "uploading";  // Upload status: Uploading    
    // Update the file status in Redis    setFileUploadStatus(c, file_id, uploaded_size, file_size, status);

    redisFree(c);
    return 0;
}

5.2 Store uploaded chunking status

For chunked uploads, you can use Set in Redis to record uploaded chunks.

// Add uploaded chunks to Redis Setvoid addUploadedChunk(redisContext* c, const std::string& file_id, const std::string& chunk_id) {
    redisReply* reply = (redisReply*)redisCommand(c,
        "SADD file_chunks:%s %s", file_id.c_str(), chunk_id.c_str());
    freeReplyObject(reply);
}

int main() {
    redisContext* c = connectToRedis();

    std::string file_id = "file123";
    std::string chunk_id = "chunk_1";  // The first block uploaded    
    //Storing uploaded chunks into Redis Set    addUploadedChunk(c, file_id, chunk_id);
    
    redisFree(c);
    return 0;
}

5.3 Data synchronization to the database

Synchronize the state in Redis to the MySQL database to ensure consistency of persistent storage.

#include <mysql/>

// Connect to MySQL databaseMYSQL* connectToDatabase() {
    MYSQL* conn = mysql_init(NULL);
    if (conn == NULL) {
        std::cerr << "mysql_init() failed\n";
        exit(1);
    }

    conn = mysql_real_connect(conn, "localhost", "root", "password", "file_upload", 3306, NULL, 0);
    if (conn == NULL) {
        std::

cerr << "mysql_real_connect() failed\n";
        exit(1);
    }

    return conn;
}

// Synchronize file upload status to the databasevoid syncToDatabase(MYSQL* conn, const std::string& file_id, size_t uploaded_size, size_t file_size, const std::string& status) {
    std::string query = "UPDATE file_upload_status SET uploaded_size = " + std::to_string(uploaded_size) + 
                        ", file_size = " + std::to_string(file_size) + 
                        ", status = '" + status + 
                        "' WHERE file_id = '" + file_id + "'";

    if (mysql_query(conn, query.c_str())) {
        std::cerr << "MySQL query failed: " << mysql_error(conn) << std::endl;
    }
}

int main() {
    MYSQL* conn = connectToDatabase();

    std::string file_id = "file123";
    size_t uploaded_size = 5000;
    size_t file_size = 10000;
    std::string status = "uploading";

    // Synchronize file upload status to the database    syncToDatabase(conn, file_id, uploaded_size, file_size, status);
    
    mysql_close(conn);
    return 0;
}

In this way, we can achieve an efficient and stable breakpoint continuous transmission system while ensuring data consistency between Redis and the database.

This is the article about the best practices of redis storage breakpoint file transfer status. For more related redis storage breakpoint file transfer content, please search for my previous article or continue browsing the related articles below. I hope everyone will support me in the future!