What is message queue MQ
MQ (Message Queue) message queue is a "first-in, first-out" data structure in the basic data structure. It refers to putting the data (messages) to be transmitted in a queue, and using the queue mechanism to realize message delivery - the producer generates messages and puts the messages into the queue, and then the consumer processes it.
Consumers can pull messages to the specified queue, or subscribe to the corresponding queue, and the MQ server pushes messages to them.
effect
Message queue middleware is an important component in a distributed system, mainly solving problems such as application decoupling, asynchronous messages, traffic cutting, etc., and achieving high performance, high availability, scalability and ultimate consistency architecture.
- Decoupling: A service requires multiple modules to be implemented together, or a message has multiple systems to process accordingly. It only requires that after the main service is completed, a MQ is sent, and the other modules consume MQ messages, so that the service can be realized and the coupling between modules is reduced.
- Asynchronous: After the main service is executed, the slave service will execute asynchronously through MQ, reducing the service response time and improving user experience.
- Peak cutting: In the case of high concurrency, the business is processed asynchronously, providing peak service processing capabilities to avoid system paralysis.
shortcoming
1. System availability is reduced. There are more dependent services, and the easier it is for the service to hang up. MQ paralysis needs to be considered
2. System complexity is improved. It is necessary to consider the order of message loss, repeated message consumption, and message delivery
3. Business consistency. Handling of consistency between main and slave services
Main products
The main MQ products include: RabbitMQ, ActiveMQ, RocketMQ, ZeroMQ, Kafka, IBM WebSphere, etc.
The more popular ones are RocketMQ and Kafka.
Apply to web and big data respectively
Give an example
1. Example 1
An example of an actual use of message queues (MQ) is the order processing system of an e-commerce website. In e-commerce websites, users need to go through a series of processing processes after placing an order, including order verification, inventory inspection, payment processing, logistics tracking, etc. These processing flows require collaboration between multiple modules, and each module needs to process a large amount of order data.
To solve these problems, a message queue system can be used to implement asynchronous processing of the order processing process. Specifically, order data can be sent to an order processing queue, and then the order data can be retrieved from the queue for processing by different modules. This allows the sending and receiving of order data to be decoupled, achieving high reliability, scalability and maintainability of the system.
For example, when a user places an order, the order system will send the order data to the order processing queue, and the inventory system will obtain the order data from the queue for inventory inspection, the payment system will obtain the order data from the queue for payment processing, and the logistics system will obtain the order data from the queue for logistics tracking, etc. These modules can process order data in parallel, and each module can be independently expanded and upgraded, thus achieving high availability and scalability of the system.
In addition, since the order data volume of e-commerce websites may be very large, using message queues can achieve peak cutting and valley filling and smooth processing of order requests, thereby ensuring the availability and stability of the system. In addition, using message queues can also enable cache and delay processing of order data to improve system performance and throughput.
In short, using message queues can make the order processing system of e-commerce websites more efficient, reliable and maintainable, and is an indispensable technology in modern e-commerce systems.
Example 2
Multiplayer scenes in online games are synchronized. In multiplayer games, multiple players need to synchronize the status of the game scene in real time, including character position, state, action, etc., and also need to handle complex scene logic such as interactions and collisions between players.
To achieve such real-time scene synchronization, a message queue can be used to handle the distribution and processing of game scene events. Specifically, events in the game scene (such as player movement, attack, skill release, etc.) can be encapsulated into messages, and then broadcast through message queues, so that all clients participating in the game can receive the corresponding messages and update the local game status according to the message content.
The advantage of using message queues is that it can decouple the sending and receiver of game scene events, thereby improving the scalability and maintainability of the system. At the same time, since the message queue can process a large number of messages, it can also cope with high concurrency scenarios, and can cut peaks and fill valleys to improve the stability and availability of the system.
In multiplayer games, using message queues can also implement some advanced functions, such as delayed synchronization, anti-cheating, etc. For example, events in the game scene can be cached to the message queue, and a certain delay time can be set before processing to ensure that all players can receive the same game state. In addition, message queues can be used to monitor and filter player behavior to detect and prevent cheating.
To sum up, using message queues can achieve real-time synchronization of multiplayer game scenarios and improve the scalability, maintainability and stability of the system. It is an important technology in modern game development.
Example 3
After registering, you need to send a registration email and a registration text message. There are two traditional methods.
- 1. Serial method;
- 2. Parallel method;
- 3. Message Queue
(1) Serial method: After writing the registration information into the database, send the registration email, and then send the registration text message. Only after all the above three tasks are completed will it be returned to the client. There is a problem with this being that emails and text messages are not necessary, it is just a notification, and this practice makes the client wait for something that is not necessary to wait for.
(2) Parallel method: After writing the registration information into the database, send a message while sending an email. After the above three tasks are completed, it is returned to the client. The parallel method can increase the processing time.
(3) Message Queue
After introducing the message queue, the business logic that sends emails and text messages is not necessary is processed asynchronously.
Example 4: Flow peak cutting
Flash sale activities usually cause the application to lapse due to excessive traffic. In order to solve this problem, the message queue is generally added to the front-end of the application.
After receiving the user's request, the server first writes the message queue. When the length of the message queue exceeds the maximum value, the user request will be directly discarded or redirected to the error page. The flash sale service will perform subsequent processing based on the request information in the message queue.
Publish Subscription vs MQ
Publish subscription mode and message queue (MQ) are both messaging mechanisms widely used in distributed systems, and they both have the advantage of decoupling senders and receivers.
While the two patterns are similar in some ways, they also have some important differences.
Who is responsible for managing the news?
- In publish subscription mode, the sender of the message does not need to know the receiver of the message, but sends the message to a topic, which will then be received by all subscribers who subscribe to the topic. The management of topics is usually the responsibility of the publisher.
- In a message queue, the sender of the message sends the message to the queue, and the subscriber subscribed to the queue consumes in the order of messages in the queue. Queue management is usually managed by the message queue system.
How to distribute messages?
- In publish subscription mode, messages are distributed through topics and all subscribers who subscribe to the topic receive the same message. This method can enable multiple subscribers to receive the same message at the same time.
- In a message queue, messages are distributed through the queue, and each message is received by only one subscriber. This method ensures that each subscriber can receive all messages, but cannot allow multiple subscribers to receive the same message at the same time.
How to deal with message loss?
- In publish subscription mode, if a subscriber does not receive the message in time, it will not be able to get the message again. This may cause messages to be lost.
- In a message queue, messages are usually persisted to disk to ensure that messages are not lost even if message consumers are down. If a consumer cannot consume a message, the message will be retained in the queue until it can be processed correctly.
Use case scenario
- The publish subscription model is usually suitable for some event-driven application scenarios, such as message push, real-time data processing, etc.
- Message queues are suitable for some scenarios with high traffic and stable performance, such as e-commerce transactions, order processing, log processing, etc.
Summarize
The above is personal experience. I hope you can give you a reference and I hope you can support me more.