About API Gateway (4) - Current Limitation

In layman's terms, flow control is a strategy for controlling user requests, which mainly includes: permissions, current limiting, and traffic scheduling.

Permissions have been discussed in the previous article. This article will talk about current limiting, and the next article will talk about traffic scheduling.

Current limiting refers to limiting the frequency (QPS/QPM) or number of user calls.

From the perspective of users or operators, the most intuitive effect of traffic restrictions is charging

The external APIs of major mainstream open platforms generally have some The free quota can be used for personal testing. Once you want to call it on a large scale, you need to pay for a larger quota (frequency, number of times), and you will be charged according to the number or frequency of calls. Once the limit is exceeded, calls will be restricted.

In fact, this is the biggest use of current limiting, but users or operating students are not aware of it, so it is not well understood by most people.

Behind the gateway are various services, and the interfaces of each service are exposed through the gateway for users to call. Theoretically speaking, user traffic is unpredictable, and there may be a wave at any time. Once the peak traffic exceeds the service's carrying capacity, the service will be down. For example, a certain wave of Weibo when big news happened, such as in the past few years. 12306.

Therefore, the gateway must ensure that the traffic passed to the back-end service must not exceed the upper limit that the service can carry. This upper limit is negotiated between the gateway and each service.

From simple to difficult, current limiting can be divided into single machine current limiting, single cluster current limiting, and full cluster current limiting.

We will not discuss specific current-limiting algorithms such as leaky buckets and token buckets here, only concepts and ideas.

The idea of ??single-machine current limiting is very simple, that is, the current limiting value of each machine x the number of machines = the total current limiting value.

For example, the QPS limit of user A is 100, and the gateway has deployed 10 machines, then each machine can be limited to 10QPS.

Let’s talk about the benefits first. This method is very simple to implement. Each machine can calculate qps in the local memory. If the threshold is exceeded, the flow will be denied.

However, the shortcomings of single-machine current limiting are also very obvious, mainly reflected in two points:

When the number of machines deployed on the gateway changes, the current limiting value of each machine needs to be based on Machine number adjustment. In reality, it is common for the number of machines to change due to expansion, shrinkage, machine downtime and other reasons.

? The premise of single-machine traffic limiting is that the user traffic carried by each gateway is average, but in fact, at certain times, the user traffic is not completely evenly distributed on each machine. .

For example:

There are 10 machines, each with a qps limit of 10, and the actual qps of 3 of them is 15. Because the limit is exceeded, user traffic is rejected. The qps of each of the remaining 7 units is 7. In this way, the user's total qps = 15 * 3 7 * 7 = 94. The user's qps did not exceed the limit, but some traffic was rejected, which is very problematic.

In fact, the threshold of a single device's current limit will also be set slightly larger to offset the problem of uneven traffic.

Because of the above problems, single-machine current limiting is usually used as a backup method, and most of the time cluster current limiting is used.

Let’s look at a schematic diagram first:

Compared with single-machine current limiting, the counting work of cluster current limiting is moved up to the redis cluster, which solves the shortcomings of single-machine current limiting.

However, cluster current limiting is not perfect, because redis is introduced. Then, when the network between the gateway and redis is jittering or redis itself fails, the cluster current limiting will be ineffective. At this time, you still have to Rely on single-machine current limit to save money.

In other words, cluster current limiting and single-machine current limiting are a more reliable solution.

Next, let’s think about this issue: Large gateways are generally deployed in multiple computer rooms and multiple regions. Of course, back-end services are also deployed in multiple computer rooms and multiple regions. In terms of protection services, , cluster current limiting is enough. But for users, there are still some problems:

For example, the upper limit of QPS purchased by users is 30. Our gateways are deployed in three regions of northern, central and southern China. So how to allocate these 30 QPS?

The average is definitely not enough. The user's traffic may be obviously unbalanced. For example, the user's business is mainly concentrated in northern China, then most of the user's traffic will enter the gateway in the north. If the gateway limits QPS to 10 , users will definitely come to complain.

Is it okay to limit it to 30 in each region? No, if the user's traffic is relatively evenly distributed in various regions, then the user purchased 30QPS, but may actually use 90QPS, which is a big loss.

According to the idea of ????solving the uneven flow of single-machine current limit, is it okay to build a public redis cluster to count?

No, limited by the signal propagation speed and the vast territory of China, it is definitely unrealistic to count every traffic. If rt is too high, the current limit will be meaningless, and the bandwidth cost will become extremely expensive. , the specification requirements for redis will also be very high. In short, it’s very expensive but it doesn’t solve the problem.

There is a clever solution: local cluster step count full cluster check.

Still the same example as before:

If the current limiting threshold is 90, then the three regions will count separately. When the value of the local region reaches 30, go to the other two regions to get the current value of the other region. The count value of the three regions is added up. If it exceeds the count value, it tells the other two regions that the count value has exceeded and starts to reject the flow. If it does not exceed, repeat the above action every time the local QPS increases by 10.

This can effectively reduce the number of interactions with redis, and at the same time achieve true cluster current limiting in all regions.

Of course, this kind of all-region cluster current limiting must be inaccurate because of the existence of rt and step counting intervals, but it is still much better than single cluster current limiting.

When a user's traffic is particularly large, redis counting will encounter a typical hot key problem, resulting in excessive pressure on a single node of the redis cluster. There are two ways to solve this problem: break up and sampling.

Breaking up means adding some suffixes to the hotspot key to turn it into multiple keys, so that they can be hashed to unreachable redis nodes to evenly share the pressure.

For example, if the hotspot key is abcd, then after breaking up, the key becomes abcd1, abcd2, abcd3, and abcd4. When using technology, just add the suffixes 1, 2, 3, and 4 in turn.

Sampling means that for hotspot keys, instead of counting every request when it comes, sampling is performed, for example, counting every 10 requests, so that the pressure on redis will be reduced to One tenth.

Now that I have finished talking about traffic scheduling, haha, let’s talk about monitoring in the next article. By the way, I would like to recommend the domestic gateway I am currently using: GOKU, from Eolinker. I think it is easier to use than KONG. Interested students can find out on their own.

www.eolinker.com