Solution to Redis Big Key Problem

What is a big key problem

The value corresponding to certain keys in Redis is particularly large, or there are too many elements stored in the collection data structure (such as hash, set, zset, list). This is a big key problem. It is divided into the following three types:

The value of a single string type (String) Key is particularly large

The specific size standards vary according to the business scenario. It is generally believed that in ordinary business scenarios, if the value of a single String type is greater than 1MB, or more than 10KB in high concurrency and low latency scenarios, it may be regarded as a big key.

There are too many elements in the collection data types (such as Hash, Set, ZSet, List) or the overall data volume is too large

For example, although the number of members of a Hash type Key is only 1,000, the total size of these members' Value reaches 100MB, or the number of members of a ZSet type reaches 10,000, which is also considered a big key problem.

The memory usage of a single key is too high

For example, in the Alibaba Cloud Redis definition, the value of a String type key reaches 5MB, or the number of key members of a ZSet type reaches 10,000, which are all considered as large keys.

Negative impact

High reading cost

Due to its large size, large keys consume more time and increase latency. Especially in network transmission, large keys will occupy more bandwidth, affecting the system's response speed and effective utilization of network resources.

Writing operations are prone to blockage

When writing large keys, since Redis uses a single-threaded model to process requests, operating large keys will block the execution of other commands, resulting in the response of the entire Redis service slowing down and even unable to respond to other requests normally.

Slow query synchronous exception with master-slave

The read and write operation time of large keys may trigger slow query logging of Redis. Frequent slow query will increase the burden on the server. At the same time, in the master-slave replication scenario, the synchronization of the large key will be slower than that of the small key, which may affect the consistency and real-time of the data.

Takes up more storage space, resulting in eviction with OOM

Large Key occupies a large amount of memory space, which can easily trigger Redis's memory elimination strategy, causing important data to be accidentally removed (eliminated). In extreme cases, it may cause Redis instances to crash (OOM) due to memory exhaustion.

Unbalanced memory resources under the cluster architecture

In a Redis cluster, if there is a large key on a shard, the memory usage rate of the shard will be much higher than that of other shards, breaking the balanced state of memory usage between clusters and affecting the overall performance and stability of the cluster.

Scenarios that affect high concurrency and low latency requirements

In delay-sensitive applications, the existence of a large key will significantly increase the request processing time and reduce the system's ability to handle high concurrent requests.

Cause

Unreasonable business design

The most common reason is to directly store large amounts of data (such as large JSON objects or binary file data) in one key without reasonable splitting. This approach ignores the characteristics of Redis as an in-memory database and does not fully utilize its advantages of efficiently handling small data blocks.

Failed to deal with the value dynamic growth issue

Over time, if data is continuously added to a key's Value without a corresponding periodic deletion mechanism, reasonable expiration strategies or size limits, the size of the Value will eventually grow to a difficult level. For example, scenes such as Weibo fan lists, popular comments or live barrage can easily form a big key.

Program Bug

Sometimes, errors in software development can cause some keys to have a life cycle beyond expectations, or the number of elements they contain is abnormally increasing. For example, if the business code responsible for consuming LIST type keys fails, it may cause the members of the key to only increase and not decrease, thus forming a large key.

Find the big key

Using redis-cli--bigkeysparameter

This is the most direct way. Redis-cli, using Redis command line tool--bigkeysParameters to scan all keys in the Redis instance. It will iterate over the entire keyspace and return information for the largest key in each data type. Execute the command as follows:

redis-cli --bigkeys

This command outputs the largest key in each data type and its related information, as well as some overall statistics, such as the number of keys of different types, the average length, etc.

Leverage Redis RDB Tools

A more in-depth and customized method of analysis is to useRedis RDB ToolsThis open source tool. First, export the RDB file of Redis, and then use the tool to analyze the file to find out the big key. For example, output the first 5 keys that take up more than 128 bytes of memory to a CSV file:

rdb -c memory  --bytes 128 --largest 5 -f 

Doing so allows for more precise control of analysis conditions, such as filtering keys by size, or filtering the largest few keys by number.

Observability analysis

Tracking Redis's performance metrics such as latency, throughput, and error rates, and analyzing slow query logs is also an indirect way to discover large keys. If there is an operation that takes too long to perform, this may be caused by a large key. Some cloud service providers may also provide the function of directly viewing Top Key statistics to facilitate discovering keys with high memory footprint.

Solution

Then after finding the big key through the above methods, two problems need to be analyzed:

• Determine whether the big key is necessary for the business (if not, let it stick😠), can this issue be handled through optimization of business logic
• Check the program bug to see if there is a problem with the code writing 🙂, which causes the Key to be of abnormal size

If all the above problems are checked correctly and the major key problem has not been solved, you can consider the following optimization strategy:

😋 Avoid big key problems

The best way to solve the problem is to solve the person who asked the problem

In the early stages of business design, large keys should be avoided and only cached necessary data fields.

✂ Data splitting

Just like video shard buffering, you can consider sharding large keys into small keys for storage. For example, directly storing some detailed information (investment and weather) in all provinces, cities and districts in China will definitely produce a big key ("China"), but if stored in pieces, define a namespace, and then the province, city and district will store it separately, a small key will be formed. The specific explanation is:

• Define namespace: "China" can be used as a namespace.

• Provinces, cities and districts store:

  • Provincial information can be stored asChina:province:province ID, the value is the provincial details of the province.
  • Municipal information storage isChina:city:city ID, the value is the city's municipal details.
  • District and county-level information storage isChina:district:region ID, the value is the district-level detailed information of the district.

This can be solved by quantification. If it is an indefinite demand, that is, the Value will grow, then you can consider how many pieces of the Value will be based on the first shard, and then divide it down according to this num.

However, sharding may cause partial writing problems. For example, an inappropriate but easy to understand example. When a user submits an order, these three shards need to be written at the same time:

• Order:Info:{OrderID}-> Order basic information
• Order:Items:{OrderID}-> Order Product List
• Order:Payment:{OrderID}-> Order payment information

Assume that when performing a write operation, the following happens:

• Order:Info:{OrderID}Writing successfully
• Order:Items:{OrderID}Writing successfully
• Order:Payment:{OrderID}Writefail

This is part of the writing problem. Then during the design stage, you can add a version number to each Value for consistency check. If a version number is not matched, you will immediately return to the source, read it again, then reload and try to write again.

↪️ Change direction

Have you ever considered that this data should not be stored using String?

For example, in a web crawler, developers may use a long String to record which URLs have been visited. Each character corresponds to the hash value of a URL, but this will waste memory space. If you use Bitmap to record the accessed URL, each bit represents the hash value of a URL, which can save a lot of memory and quickly determine whether the URL has been accessed.

Or user activity statistics. If you want to record the login activity of millions of users on a large website every day, for each day, you can use an integer 32-bit (or 64-bit) bit to indicate whether all users have logged in that day, each of which represents whether a user ID is active. If String is used for storage, even if each user is active, it will take up more space due to the encoding method of String (such as UTF-8), and each character takes up at least 8 bits.

🧹 Reasonable cleaning

Just design a plan to clean it properly to avoid the accumulation of large keys.

• Delete during low peak periods. It is possible to consider cleaning the cache regularly when the service traffic is low. But it is obvious that this plan is not very reasonable. What if the business traffic is very high all day long, then a big key has been generated?
• Delete in batches regularly and regularly. Quantitative deletion is to prevent blockage.
• Asynchronous deletion. Unlike the del command, the unlink command asynchronously deletes the specified key and the associated value. That is, it adds the key to be deleted to a list to be deleted and returns immediately without blocking the client. The Redis server will asynchronously delete the keys in the list to be deleted in the background.

This is all about this article about the solution to the Redis Key problem. For more related redis key content, please search for my previous articles or continue browsing the related articles below. I hope everyone will support me in the future!