Ali Interview: Tell me about MQ used in your project. What is the role of MQ in distributed systems?

In Ali's interview, the interviewer asked several questions about MQ:

I wrote an article about the implementation of distributed locks by rocketMQ before, mainly introducing how to use RocketMQ to implement distributed locks.

Springcloud+RocketMQ for Solving Distributed Transactions

But this function is not the basic function of MQ, nor is it the function of all MQ.

What role does MQ play in the system? In addition to the basic press and publication subscription, there are the following points:

In the distributed system, it is either called by rest or RPC such as dubbo, but some scenarios need decoupling design and cannot be called directly.

For example, in a message-driven system, the message sender completes the local business and sends the message, and the multi-platform message consumer service needs to receive the pushed message and then continue to process other business.

Looking at these two architecture diagrams, the first BC directly depends on the A service, so if the interface in A is modified, the BC has to be modified accordingly, with high coupling.

Secondly, MQ is a middleware to send and receive messages, and BC only depends on the received messages rather than the specific interfaces, so even if a service modifies or adds other services, it only needs to subscribe to MQ.

Taking the business process of user registration as an example,

In the initial system design, this service flow will be processed in series, that is,1-2-3 first; But we can think about it here. If there are many registered users in the case of single service and single machine, can the system resist?

Assuming that the time of each stage is 1 = 50ms, 2 = 50ms and 3 = 50ms, then a request is all =150ms; ;

Let's assume that the CPU of this server is 1, which can only handle single thread, so use the QPS of this single server and single thread to calculate; QPS = 1000/ 150 ≈ 7

Now, I will triple this QPS * 3 and introduce MQ service as middleware at this time.

As can be seen from the figure, I returned directly after the A service user registered. At this time, MQ is used to send asynchronous processing messages, and B and C services handle them separately.

A doesn't have to wait for the return results of B and C, so the user experience only has a waiting time of 50ms. In the stage of e-mail and short message, due to network delay,

Users can accept waiting for a period of time.

Generally speaking, our request to access the system is a direct request. This mode is not a big problem when the number of users is small.

However, if the user's request reaches a certain bottleneck or causes some problems, we need to consider optimizing our architecture design, and MQ middleware is one of the solutions.

Let's take the spike system as an example to analyze the problem.

How to deal with the instantaneous million concurrency of the system? Generally, the spike system will filter requests, invalid and repeated requests will be filtered again, and the rest will really enter the spike service and order service.

But even so, the degree of concurrency is still high. If the gateway forwards all requests to the downstream order service, it will crush the downstream system, resulting in the unavailability of the service or even an avalanche.

The real spike system is more complicated, including Nginx, gateway, registry, redis cache, mysql cluster and message queue cluster.

The solution is to add the tasks with faster processing speed in the upstream to the queue, and the downstream consumes the queues one by one until all the queues are consumed.

If the number of requests handled by the peak service is 1000/ sec,

Downstream order service processing request: 10/s,

In order not to put pressure on the downstream order service, the information after spike is sent to the queue, and the order service can handle ten requests calmly every second instead of directly blocking 1000 requests.

Whether people like it or not.

Here you can summarize the filtering methods of the spike system:

All services send logs to MQ service for log storage.

MQ, as a middleware, persists and forwards logs.

The big data service reads MQ and performs log analysis.

Some people come up with a performance comparison, saying that RabbitMQ is the best MQ in the world. ...

You compare choosing MQ to choosing a wife. When you come up, you need a full set, beautiful in white, protruding forward and backward, sexy and hot, hardworking and capable. . .

What a lack of social education, brother

Can you afford it? A maintenance package, 1W/ month.

Can you hold it? Lao Wang next door often comes to your house for dinner, doesn't he Crazy brain tonic. . .

Is it easy to eat? Jujube+Lycium barbarum+Shenbao tablets, I'm afraid I don't have enough spare capacity.

Anyway, in fact, I think this is a question of thinking. What are the conditions first?

The example log message in the above figure is kafka, why kafka?

Kafka is an open source distributed publishing and subscribing message system of LinkedIn, which belongs to the top Apache project and has an active community.

Kafka's main feature is to handle message consumption based on pull mode and pursue high throughput. Its original purpose is to collect and transmit logs.

Later, the version began to support copying, not transactions, and there was no strict requirement for repetition, loss and error of messages, which was suitable for the data collection business of Internet services that produced a large amount of data.

But Kafka is heavier and needs to rely on city zoo. It is no problem to use it in big companies, and it also needs special maintenance.

RocketMQ is an open source and reliable messaging system in Ali, which donated Apache as the top-level project. At the beginning, its positioning was reliable message transmission except logs, but in fact it performed well in log processing.

At present, the supported clients are java, c++, GO, active community and comprehensive documentation. However, it is still difficult to modify the core. After all, Alibaba Cloud makes money by selling this service.

Therefore, if the company is not confident about its own strength, it is better to choose carefully. If it really doesn't work, you can buy cloud services directly, which saves trouble and effort. Again, it depends on the actual situation.

The following figure is a picture of the source network. Some descriptions are out of date, but they are basically good for reference only:

Let's talk briefly below, followed by a written confession specifically aimed at this issue.

Probably due to some special reasons, such as network reasons, the message consumption is not recorded due to service restart, which may lead to repeated consumption.

The general treatment is to ensure the idempotency of interface design, and the purpose is to judge whether it exists by unique identification.