What application is usually used for message queue in large PHP applications?

1. Overview of Message Queue\x0d\ Message queue middleware is an important component in distributed systems. It mainly solves problems such as application coupling, asynchronous messages, and traffic shaving. Achieve high performance, high availability, scalability and eventually consistent architecture. It is an indispensable middleware for large-scale distributed systems. \x0d\ At present, in the production environment, the most commonly used message queues include ActiveMQ, RabbitMQ, ZeroMQ, Kafka, MetaMQ, RocketMQ, etc. \x0d\ 2. Message queue application scenarios \x0d\ The following introduces the common usage scenarios of message queue in practical applications. Four scenarios: asynchronous processing, application decoupling, traffic cutting and message communication. \x0d\ 2.1 Asynchronous processing \x0d\ Scenario description: After the user registers, he needs to send a registration email and registration SMS. There are two traditional methods: 1. Serial method; 2. Parallel method. \x0d\　 (1) Serial mode: After successfully writing the registration information into the database, send a registration email, and then send a registration SMS. After all the above three tasks are completed, return to the client. (Architecture KKQ: 466097527, welcome to join)\x0d\ (2) Parallel method: After successfully writing the registration information into the database, the registration email and registration SMS are sent at the same time. After the above three tasks are completed, return to the client. The difference from serial is that the parallel method can improve the processing time. \x0d\ Assuming that each of the three business nodes uses 50 milliseconds, without considering other overheads such as the network, the serial time is 150 milliseconds and the parallel time may be 100 milliseconds. \x0d\ Since the number of requests processed by the CPU in unit time is certain, assume that the CPU throughput is 100 times in 1 second. Then the number of requests that the CPU can handle in 1 second in serial mode is 7 times (1000/150). The number of requests processed in parallel is 10 times (1000/100). \x0d\Summary: As described in the above case, the performance (concurrency, throughput, response time) of the traditional system will have bottlenecks. How to solve this problem? \x0d\　Introduction of message queue will eliminate unnecessary business logic and process it asynchronously. The modified architecture is as follows: \x0d\ According to the above agreement, the user's response time is equivalent to the time it takes for the registration information to be written into the database, which is 50 milliseconds. After registering an email and sending a text message and writing it to the message queue, it returns directly. Therefore, the writing speed to the message queue is very fast and can basically be ignored. Therefore, the user's response time may be 50 milliseconds. Therefore, after the architecture change, the system throughput increased to 20 QPS per second. It is 3 times better than serial and 2 times better than parallel. \x0d\ 2.2 Application decoupling \x0d\ Scenario description: After the user places an order, the order system needs to notify the inventory system. The traditional approach is that the order system calls the interface of the inventory system. As shown below: \x0d\ Disadvantages of the traditional model: \x0d\ 1) If the inventory system cannot be accessed, the order reduction will fail, resulting in order failure; \x0d\ 2) The order system is coupled with the inventory system; \x0d\ How How to solve the above problems? The solution after introducing the application message queue is as shown below: \x0d\ Order system: After the user places an order, the order system completes persistence processing, writes the message to the message queue, and returns the user's order success. \x0d\　Inventory system: Subscribe to order information and use pull/push methods to obtain order information. The inventory system performs inventory operations based on the order information. \x0d\　What if: The inventory system cannot be used normally when placing an order. It does not affect normal order placement, because after the order is placed, the order system writes to the message queue and no longer cares about other subsequent operations. Realize the application decoupling of order system and inventory system. \x0d\ 2.3 Traffic Shaving \x0d\ Traffic Shaving is also a common scenario in message queues, and is generally used extensively in flash sales or group grabs. \x0d\　Application scenario: flash sale activities, usually due to excessive traffic, will lead to a sudden increase in traffic and the application will hang up. To solve this problem, it is generally necessary to add a message queue to the application front-end. \x0d\　It can control the number of people active;\x0d\　It can alleviate the overwhelming application of high traffic in a short period of time;\x0d\　After the user's request is received by the server, it is first written into the message queue.

If the length of the message queue exceeds the maximum number, the user request will be discarded directly or jump to the error page; \x0d\ The flash sale business will perform subsequent processing based on the request information in the message queue. \x0d\　2.4 Log processing\x0d\　Log processing refers to using message queues in log processing, such as Kafka applications, to solve the problem of large number of log transmissions. The architecture is simplified as follows: \x0d\ Log collection client, responsible for log data collection, regular writing and writing to the Kafka queue; \x0d\ Kafka message queue, responsible for receiving, storing and forwarding log data; \x0d\ Log processing application: subscription And consume the log data in the kafka queue; \x0d\ The following are Sina kafka log processing application cases: \x0d\ (1) Kafka: message queue for receiving user logs. \x0d\　(2)Logstash: Do log parsing, unify it into JSON and output it to Elasticsearch. \x0d\　(3) Elasticsearch: The core technology of real-time log analysis service, a schemaless, real-time data storage service, which organizes data through index and has powerful search and statistical functions. \x0d\　(4)Kibana: A data visualization component based on Elasticsearch. Super data visualization capabilities are an important reason why many companies choose ELK stack. \x0d\　2.5 Message Communication\x0d\　Message communication means that message queues generally have built-in efficient communication mechanisms, so they can also be used for pure message communication. For example, implement point-to-point message queues, or chat rooms, etc. \x0d\ Point-to-point communication: \x0d\ Client A and Client B use the same queue for message communication. \x0d\ Chat room communication: \x0d\ Client A, Client B, and Client N subscribe to the same topic to publish and receive messages. Achieve a chat room-like effect. \x0d\　The above are actually two message modes of message queue, point-to-point or publish-subscribe mode. The model is a schematic diagram for reference only. \x0d\ 3. Message middleware example \x0d\ 3.1 E-commerce system \x0d\ The message queue uses highly available and durable message middleware. Such as Active MQ, Rabbit MQ, Rocket Mq. (1) After the application completes the main logic processing, it writes to the message queue. Whether the message is sent successfully or not can enable the confirmation mode of the message. (After the message queue returns the message reception success status, the application returns again, thus ensuring the integrity of the message) \x0d\　 (2) Extend the process (sending text messages, delivery processing) to subscribe to queue messages. Use push or pull methods to obtain messages and process them. \x0d\　(3) While messages decouple applications, they also bring about data consistency issues, which can be solved using eventual consistency. For example, the main data is written to the database, and the extended application implements subsequent processing based on the message queue and combined with the database method. \x0d\ 3.2 Log collection system \x0d\ It is divided into four parts: Zookeeper registration center, log collection client, Kafka cluster and Storm cluster (OtherApp). \x0d\ Zookeeper registration center proposes load balancing and address lookup services; \x0d\ Log collection client is used to collect application system logs and push data to the kafka queue; \x0d\ 4. JMS message service \x0d\ When talking about message queues, we have to mention JMS. JMS (Java Message Service, Java Message Service) API is a standard/specification for message services that allows application components to create, send, receive and read messages based on the JavaEE platform. It makes distributed communication less coupled and message services more reliable and asynchronous. \x0d\　In the EJB architecture, there are message beans that can be seamlessly integrated with the JM message service. In the J2EE architectural model, there is the message server model, which is used to achieve direct decoupling of messages and applications. \x0d\ 4.1 Message Model \x0d\ In the JMS standard, there are two message models: P2P (Point to Point) and Publish/Subscribe (Pub/Sub).

\x0d\ 4.1.1 P2P mode \x0d\ The P2P mode includes three roles: message queue (Queue), sender (Sender), and receiver (Receiver). Each message is sent to a specific queue and the receiver gets the message from the queue. The queue holds messages until they are consumed or time out. \x0d\ Characteristics of P2P \x0d\ Each message has only one consumer (that is, once consumed, the message is no longer in the message queue) \x0d\ There is no time dependence between the sender and the receiver , that is to say, after the sender sends a message, regardless of whether the receiver is running or not, it will not affect the message being sent to the queue\x0d\ If every message sent will be successfully processed, then P2P mode is required. (Architecture KKQ: 466097527, welcome to join) \x0d\ 4.1.2 Pub/sub mode \x0d\ It contains three roles: Topic, Publisher, and Subscriber. Multiple publishers send messages to Topic, and the system delivers these messages to multiple subscribers. \x0d\ Features of Pub/Sub \x0d\ Each message can have multiple consumers \x0d\ There is a time dependency between publishers and subscribers. For a subscriber of a certain topic (Topic), it must create a subscriber before it can consume the publisher's messages. \x0d\ In order to consume messages, the subscriber must remain running. \x0d\ To alleviate such strict time dependencies, JMS allows subscribers to create a durable subscription. This way, even if the subscriber is not activated (running), it can receive messages from the publisher. \x0d\ If you want to send a message without any processing, or only be processed by one message sender, or can be processed by multiple consumers, then you can use the Pub/Sub model. \x0d\ 4.2 Message consumption \x0d\ In JMS, the generation and consumption of messages are asynchronous. For consumption, JMS messengers can consume messages in two ways. \x0d\　 (1) Synchronous \x0d\? Subscribers or receivers receive messages through the receive method. The receive method will block until the message is received (or before timeout); \x0d\? (2) Asynchronous \x0d\? Subscriber Or the receiver can register as a message listener. When the message arrives, the system automatically calls the listener's onMessage method. \x0d\　JNDI: Java Naming and Directory Interface, is a standard Java naming system interface. Services can be found and accessed on the web. By specifying a resource name, the name corresponds to a record in the database or naming service, and returns the information necessary to establish a connection to the resource. \x0d\　JNDI plays the role of finding and accessing the sending target or message source in JMS. (Architecture KKQ: 466097527, welcome to join)\x0d\ 4.3JMS Programming Model\x0d\ (1) ConnectionFactory\x0d\ Create a factory for Connection objects. For two different jms message models, there are two types: QueueConnectionFactory and TopicConnectionFactory. ConnectionFactory objects can be found through JNDI. \x0d\　(2) Destination\x0d\　Destination means the message sending target of the message producer or the message source of the message consumer. For a message producer, its Destination is a certain queue (Queue) or a certain topic (Topic); for a message consumer, its Destination is also a certain queue or topic (that is, the message source). \x0d\ Therefore, Destination is actually two types of objects: Queue and Topic. Destination can be found through JNDI.

\x0d\　(3) Connection\x0d\　Connection represents the link established between the client and the JMS system (packaging of TCP/IP socket). Connection can generate one or more Sessions. Like ConnectionFactory, Connection also has two types: QueueConnection and TopicConnection. \x0d\　(4) Session\x0d\　Session is the interface for operating messages. Producers, consumers, messages, etc. can be created through sessions. Session provides transaction functions. When you need to use a session to send/receive multiple messages, you can put these send/receive actions into a transaction. Similarly, it is also divided into QueueSession and TopicSession. \x0d\　(5) Message producer\x0d\　Message producer is created by Session and used to send messages to Destination. Similarly, there are two types of message producers: QueueSender and TopicPublisher. You can call the message producer's method (send or publish method) to send the message. \x0d\　(6) Message consumer\x0d\　Message consumer is created by Session and is used to receive messages sent to Destination. Two types: QueueReceiver and TopicSubscriber. It can be created through createReceiver(Queue) or createSubscriber(Topic) of session respectively. Of course, you can also create persistent subscribers using the session's creatDurableSubscriber method. \x0d\　(7) MessageListener\x0d\　Message listener. If a message listener is registered, the listener's onMessage method will be automatically called once the message arrives. MDB (Message-Driven Bean) in EJB is a kind of MessageListener. \x0d\　In-depth study of JMS is very helpful for mastering JAVA architecture and EJB architecture. Message middleware is also a necessary component of large-scale distributed systems. This sharing is mainly an overall introduction. The specific in-depth requires everyone to study, practice, summarize and understand. \x0d\ 5. Commonly used message queues \x0d\ General commercial containers, such as WebLogic and JBoss, support the JMS standard, which is very convenient for development. But free ones such as Tomcat, Jetty, etc. require the use of third-party message middleware. This section introduces commonly used message middleware (Active MQ, Rabbit MQ, Zero MQ, Kafka) and their characteristics. \x0d\　5.1 ActiveMQ\x0d\　ActiveMQ is the most popular and powerful open source message bus produced by Apache. ActiveMQ is a JMS Provider implementation that fully supports JMS1.1 and J2EE 1.4 specifications. Although the JMS specification has been released for a long time, JMS still plays a special position in today's J2EE applications. \x0d\ ActiveMQ features are as follows: \x0d\ ⒈ Clients can be written in multiple languages ??and protocols. Language: Java, C, C++, C#, Ruby, Perl, Python, PHP.

Application protocols: OpenWire, Stomp REST, WS Notification, XMPP, AMQP\x0d\　⒉ Fully supports JMS1.1 and J2EE 1.4 specifications (persistence, Embedded into a system using Spring, and also supports Spring 2.0 features\x0d\　⒋ Passed the test of common J2EE servers (such as Geronimo, JBoss 4, GlassFish, WebLogic), and through the configuration of JCA 1.5 resource adapters, it can Allows ActiveMQ to be automatically deployed to any commercial server compatible with J2EE 1.4\x0d\　⒌ Supports multiple transport protocols: in-VM, TCP, SSL, NIO, UDP, JGroups, JXTA\x0d\　⒍ Supports high-speed provision through JDBC and journal Message persistence\x0d\　⒎ Designed to ensure high-performance cluster, client-server, point-to-point\x0d\　⒏ Support Ajax\x0d\　⒐ Support integration with Axis\x0d\　⒑ Can be easily called embedded JMS provider, for testing\x0d\ 5.2 RabbitMQ\x0d\ RabbitMQ is a popular open source message queuing system developed in erlang language. RabbitMQ is a standard implementation of AMQP (Advanced Message Queuing Protocol). Supports a variety of clients, such as: Python, Ruby, .NET, Java, JMS, C, PHP, ActionScript, XMPP, STOMP, etc., supports AJAX, and persistence. It is used to store and forward messages in distributed systems, and performs well in terms of ease of use, scalability, and high availability. \x0d\ Several important concepts: \x0d\ Broker: Simply put, it is the message queue server entity. \x0d\　Exchange: Message switch, which specifies the rules according to which messages are routed to which queue. \x0d\　Queue: Message queue carrier, each message will be put into one or more queues. \x0d\Binding: Binding, its function is to bind exchange and queue according to routing rules. \x0d\　Routing Key: routing keyword, exchange delivers messages based on this keyword. \x0d\　vhost: virtual host. Multiple vhosts can be opened in a broker to separate the permissions of different users. \x0d\　Producer: Message producer is the program that delivers messages. \x0d\　consumer: Message consumer is the program that accepts messages. \x0d\　Channel: Message channel. In each connection of the client, multiple channels can be established, each channel represents a session task. \x0d\ The process of using message queue is as follows: \x0d\ (1) The client connects to the message queue server and opens a channel. \x0d\　(2) The client declares an exchange and sets related attributes. \x0d\　(3) The client declares a queue and sets related attributes. \x0d\　(4) The client uses routing key to establish a binding relationship between exchange and queue. \x0d\　(5) The client delivers the message to exchange. \x0d\ After exchange receives the message, it will route the message based on the key of the message and the binding that has been set, and deliver the message to one or more queues.

\x0d\ 5.3 ZeroMQ\x0d\ Known as the fastest message queue in history, it is actually similar to a series of interfaces of Socket. The difference between it and Socket is: ordinary socket is end-to-end (1:1 relationship), while ZMQ But it can have an N:M relationship. People know more about BSD sockets from point-to-point connections. Point-to-point connections need to explicitly establish connections, destroy connections, select protocols (TCP/UDP), and handle errors, etc., while ZMQ shields these details and makes your network programming simpler. ZMQ is used for communication between nodes. A node can be a host or a process. \x0d\ Quoting the official statement: "ZMQ (hereinafter ZeroMQ is referred to as ZMQ) is a simple and easy-to-use transport layer, a socket library like a framework. It makes Socket programming simpler, more concise and more performant. It is a message processing Queue library that scales elastically across multiple threads, cores, and host boxes. The stated goal of ZMQ is to "become part of the standard network protocol stack and later enter the Linux kernel." However, it has not yet seen success. It is undoubtedly a very promising and more needed layer of packaging on "traditional" BSD sockets. ZMQ makes writing high-performance network applications extremely easy and fun. "\x0d\" Features are: \x0d\ High performance, non-persistent; \x0d\ Cross-platform: supports Linux, Windows, OS X, etc. \x0d\　 Multi-language support; more than 30 development languages ??such as C, C++, Java, .NET, and Python. \x0d\　Can be deployed separately or integrated into applications; \x0d\　Can be used as a Socket communication library. \x0d\ Compared with RabbitMQ, ZMQ is not like a message queue server in the traditional sense. In fact, it is not a server at all, but more like an underlying network communication library, which is a layer above the Socket API. Encapsulation abstracts network communication, process communication and thread communication into a unified API interface. Supports three basic models and extended models: "Request-Reply", "Publisher-Subscriber", and "Parallel Pipeline". \x0d\ ZeroMQ high-performance design points: \x0d\ 1. Lock-free queue model \x0d\ For the data exchange channel pipe between cross-thread interactions (client and session), the lock-free queue algorithm CAS is used; Asynchronous events are registered at both ends of the pipe. When reading or writing messages to the pipe, read and write events will be automatically triggered. \x0d\ 2. Batch processing algorithm \x0d\ For traditional message processing, each message requires system calls when sending and receiving, so for a large number of messages, the system overhead is relatively large, and zeroMQ is suitable for batch Messages have been adaptively optimized and can receive and send messages in batches. \x0d\ 3. Thread binding under multi-core without CPU switching \x0d\ Different from traditional multi-thread concurrency mode, semaphore or critical section, zeroMQ takes full advantage of multi-core, each core is bound to run a worker thread , to avoid the CPU switching overhead between multiple threads. \x0d\　5.4 Kafka\x0d\　Kafka is a high-throughput distributed publish-subscribe messaging system that can handle all action flow data in consumer-scale websites. Such actions (web browsing, searches and other user actions) are a key factor in many social functions on the modern web. This data is typically addressed by processing logs and log aggregation due to throughput requirements. For log data and offline analysis systems like Hadoop, but requiring real-time processing constraints, this is a feasible solution. The purpose of Kafka is to unify online and offline message processing through Hadoop's parallel loading mechanism, and to provide real-time consumption through cluster machines.

\x0d\ Kafka is a high-throughput distributed publish-subscribe messaging system with the following features: \x0d\ Provides message persistence through an O(1) disk data structure, which can handle even terabytes of messages. Storage is also capable of maintaining stable performance over long periods of time. (Data is written by appending files, and expired data is deleted regularly) \x0d\ High throughput: Even very common hardware Kafka can support millions of messages per second. \x0d\　 Supports partitioning messages through Kafka server and consumer machine clusters. \x0d\　 Support Hadoop parallel data loading. \x0d\ 　Kafka related concepts\x0d\ Broker\x0d\ A Kafka cluster contains one or more servers, which are called brokers[5]\x0d\ Category, this category is called Topic. (Physically, messages of different topics are stored separately. Logically, although messages of a topic are stored on one or more brokers, users only need to specify the topic of the message to produce or consume data without caring about where the data is stored.)\x0d \Partition\x0d\ Parition is a physical concept. Each Topic contains one or more Partition.\x0d\ Producer\x0d\ Responsible for publishing messages to Kafka broker\x0d\ Consumer\x0d\ Message consumer, reading from Kafka broker The client that gets the message. \x0d\　Consumer Group\x0d\　Each Consumer belongs to a specific Consumer Group (the group name can be specified for each Consumer, if the group name is not specified, it belongs to the default group). \x0d\　Generally used in big data log processing or scenarios with slightly lower requirements for real-time performance (a small amount of delay) and reliability (a small amount of data loss).