Ali Interview: Tell me about MQ used in your project. What is the role of MQ in distributed systems?

In Ali's interview, the interviewer asked several questions about MQ:

I wrote an article about how to realize distributed locks with rocketMQ, mainly introducing how to use RocketMQ to realize distributed locks,

Spring Cloud+Rocket MQ to Solve Distributed Transactions

But this function is not the basic function of MQ, nor is it available in all MQ.

what role does MQ play in the system? Aside from the basic message publishing subscription, there are the following points:

In a distributed system, it is either called by rest or RPC such as dubbo, but some scenarios need decoupling design and cannot be called directly.

For example, in a message-driven system, the message sender completes local business and sends messages. The multi-platform message consumer service needs to receive the pushed messages and then continue to process other businesses.

Looking at these two architecture diagrams, it can be seen that the first BC directly depends on the A service, so if the interface in A is modified, the BC will have to modify it accordingly, which has a high coupling degree.

second, MQ is used as a middleware to send and receive messages, and BC only relies on the received messages rather than the specific interfaces, so even if service A modifies or adds other services, it is only necessary to subscribe to MQ.

Take the user registration business process as an example.

In the original system design, this service process will be processed in series, that is, 1-2-3 first; But we can think about it here. If there are a lot of registered users in the case of a single service and a single machine, can the system resist?

assuming that the time of each stage is 1 = 5ms, 2 = 5ms and 3 = 5ms, then a request is all = 15ms；;

let's assume that the CPU of this server is 1, and it can only handle a single thread, so it is calculated by the QPS of this single server and single thread; QPS = 1/15 ≈ 7

Now I want to triple this QPS * 3, and at this time I will introduce MQ service as middleware

As can be seen from the figure, I will return directly after the user registration of service A is completed, and at this time MQ is used to send asynchronous processing messages, and services B and C will handle them separately.

A doesn't have to wait for the return results of b and c, so the user experience is only 5ms waiting time. At the stage of email and SMS,

users can accept waiting for a certain time because of network delay.

For general services, our requests to access the system are all direct requests. This mode is not a big problem when the number of users is small.

However, if the user's request reaches a certain bottleneck or causes some problems, we need to consider optimizing our architecture design, and MQ middleware is one of the solutions.

Let's take the spike system as an example to analyze the problem.

The spike system has millions of concurrent moments. How to deal with it? Generally, the second kill system will filter requests, and invalid and duplicate requests will be filtered again, and the rest will really enter the second kill service and order service.

But even so, the concurrency is still very high. If the gateway forwards all the requests to the downstream order service, it will crush the downstream system, causing the service to be unavailable or even avalanche.

The real spike system is more complicated, including Nginx, gateway, registry, redis cache, mysql cluster and message queue cluster.

The solution is to add the tasks that are processed faster in the upstream to the queue and consume the queues one by one in the downstream until all the queues are consumed.

if the number of requests processed by spike service is 1/s,

the number of requests processed by downstream order service is 1/s,

in order not to put pressure on downstream order service, the information after spike is sent to the queue, so that the order service can handle 1 requests per second calmly, instead of directly blocking 1 requests

regardless of whether people are willing or not.

At this point, we can summarize the filtering method of the spike system:

All services send logs to MQ service for log storage.

MQ, as a middleware, persists and forwards logs

Big data services read MQ and analyze logs

Some people come up with a performance comparison, and then say that RabbitMQ is the best MQ in the world ...

You compare choosing MQ to picking a wife, and when you come up, you need a full set, which is white and beautiful, forward and backward, sexy and hot, hardworking and capable. . .

It's really a lack of social education. Can brothers

afford it? Can you keep a maintenance package at 1W/ month

? Lao Wang next door often comes to your house for dinner, right? Crazy brain tonic. . .

is it easy to eat? Jujube+Lycium barbarum+Shenbao tablets, I'm afraid I have insufficient spare capacity.

Get to the point. Actually, I think this is a thinking question. First of all, what conditions should we look at?

The example log message in the above figure is kafka. Why kafka?

Kafka is an open source distributed publish-subscribe messaging system of LinkedIn, which belongs to the top Apache project and has an active community.

Kafka's main feature is to handle message consumption based on Pull mode and pursue high throughput. Its initial purpose is to collect and transmit logs.

Later versions began to support replication, not transactions, and there was no strict requirement for message repetition, loss and error, which was suitable for data collection services of Internet services that generated a large amount of data.

however, kafka is relatively heavy and needs to rely on zookeeper. It is no problem to use it in large companies, and it also needs special personnel to maintain it.

RocketMQ is an open source reliable messaging system in Ali, and Apache has been donated as a top-level project. At first, it was positioned as a reliable message transmission other than log, but in fact, it performed well in log processing.

currently supported clients include java, c++, GO, and the community is active and the documentation is comprehensive. However, it is still difficult to modify the core. After all, Alibaba Cloud makes money by selling this service.

Therefore, if the company is not confident in its strength, it is better to choose carefully. If it is really not feasible, you can buy cloud services directly, which will save you the trouble and effort. Again, it depends on the actual situation.

The picture below is a picture of the source network. Some descriptions are out of date, but basically not bad, for reference only:

Here is a brief talk, and a written confession will be made specifically for this issue later.

It is probably due to some special reasons, such as network reasons, that message consumption is not recorded due to service restart, which may lead to repeated consumption.

The general processing method is to ensure the idempotency of interface design, and the purpose is to judge whether it exists by unique identification.