What applications do large PHP applications usually use for message queuing?

Message queuing middleware is an important part of distributed system, which mainly solves the problems of application coupling, asynchronous message and traffic clipping. Achieve high performance, high availability, scalability and ultimate consistency architecture. It is an indispensable middleware for large-scale distributed systems.

At present, in the production environment, message queues are widely used, such as ActiveMQ, RabbitMQ, ZeroMQ, Kafka, MetaMQ, RocketMQ and so on.

Second, message queuing application scenarios

The following describes the common usage scenarios of message queuing in practical applications. Asynchronous processing, application decoupling, traffic clipping and message communication are four scenarios.

2. 1 asynchronous processing

Scenario description: After registration, users need to send registration email and registration SMS. There are two traditional methods: 1. Serialization; 2. Parallel mode.

(1) serial mode: After the registration information is successfully written into the database, send the registration email and then send the registration SMS. After all the above three tasks are completed, they are returned to the client. (Building KKQ:466097527, welcome to join)

(2) Parallel mode: After the registration information is successfully written into the database, the registration email and registration SMS are sent at the same time. After the above three tasks are completed, return to the client. The difference from serial mode is that parallel mode will increase processing time.

Assuming that three service nodes each use 50 milliseconds, regardless of other overhead such as network, the time of serial mode is 150 milliseconds, and the time of parallel mode may be 100 milliseconds.

Because the number of requests processed by the CPU per unit time is certain, it is assumed that the throughput of the CPU in 1 second is 100 times. Then the number of requests that the CPU can handle in the serial mode of 1 sec is 7 times (1000/ 150). The number of requests processed in parallel mode is 10 times (1000/ 100).

Summary: As mentioned in the above case, the performance (concurrency, throughput, response time) of traditional systems will have bottlenecks. How to solve this problem?

For business logic and asynchronous processing, it is unnecessary to introduce message queues. The reconstructed structure is as follows:

According to the above convention, the user's response time is equivalent to the time when the registration information is written into the database, that is, 50 milliseconds. Register a mailbox, send a short message and write it in the message queue, and then reply directly. Therefore, the writing speed of message queue is very fast, which can be basically ignored, so the response time of users may be 50 milliseconds. Therefore, after the architecture change, the throughput of the system is improved to 20 QPS per second. 3 times higher than serial and 2 times higher than parallel.

2.2 Application decoupling

Scenario description: After the user places an order, the order system needs to notify the inventory system. Traditionally, the order system calls the interface of the inventory system. As shown in the figure below:

Disadvantages of the traditional model:

1) If you can't access the inventory system, the order fails to reduce inventory, resulting in the failure of the order;

2) The order system is coupled with the inventory system;

How to solve the above problems? Introduce the scheme after applying message queuing, as shown in the following figure:

Order system: after the user places an order, the order system completes the persistence processing, writes the message into the message queue, and returns that the user placed the order successfully.

Inventory system: Subscribe to the ordering message and get the ordering information by pulling/pushing. The inventory system performs inventory operation according to the ordering information.

If: the inventory system cannot be used normally when placing an order. It will not affect the normal ordering, because after ordering, the ordering system will no longer care about other subsequent operations when writing to the message queue. Realize the application decoupling of order system and inventory system.

2.3 traffic clips

Traffic clipping is also a common scene in message queues, which is generally widely used in spike or group grabbing activities.

Application scenario: Peak activity, usually due to excessive traffic, which leads to a surge in traffic and application suspension. In order to solve this problem, it is generally necessary to join the message queue at the front end of the application.

The number of people who can control the activity;

Can alleviate the application of high-flow crushing in a short time;

After receiving the user's request, the server first writes it into the message queue. If the message queue length exceeds the maximum number, directly abandon the user request or jump to the error page;

The spike service performs subsequent processing according to the request information in the message queue.

2.4 log processing

Log processing refers to the use of message queues in log processing, such as Kafka application, to solve the problem of a large number of log transmission. The architecture is simplified as follows:

The log collection client is responsible for collecting log data and writing and receiving it in Kafka queue regularly;

Kafka message queue, which is responsible for receiving, storing and forwarding log data;

Log processing application: subscribing and consuming log data in kafka queue;

The following is the application case of Sina kafka log processing:

(1)Kafka: Message queue for receiving user logs.

(2)Logstash: Do log parsing, unify it into JSON and output it to Elasticsearch.

(3)Elasticsearch: the core technology of real-time log analysis service, a modeless, real-time data storage service, which organizes data through indexes and has powerful search and statistics functions.

(4)Kibana: a data visualization component based on Elasticsearch. Its superior data visualization capability is an important reason why many companies choose ELK stack.

2.5 Information exchange

Message communication means that message queues generally have built-in efficient communication mechanisms, so they can also be used for pure message communication. For example, implementing a peer-to-peer message queue or chat room.

Point-to-point communication:

Client a and client b use the same queue for message communication.

Chat room communication:

Client a, client b and client n subscribe to the same topic to publish and receive messages. Achieve the effect similar to a chat room.

The above are actually two message modes of message queuing, point-to-point or publish-subscribe mode. The model is a schematic diagram for reference.

Thirdly, an example of message middleware.

3. 1 e-commerce system

The message queue adopts highly available and persistent message middleware. Such as active Mq, rabbit MQ and rocket MQ. (1) The application writes the backbone logic to the message queue after processing. Whether the message is sent successfully can start the confirmation mode of the message. (After the message queue returns the message receiving success status, the application program returns again, thus ensuring the integrity of the message. )

(2) Extend the process (SMS, delivery processing) to subscribe to queue messages. Get information and process it by pushing or pulling.

(3) While the message is decoupled from the application, it brings the problem of data consistency, which can be solved through the final consistency. For example, the main data is written into the database, and the extended application is combined with the database mode to realize the subsequent processing based on the message queue.

3.2 Log Collection System

It consists of Zookeeper Registration Center, Log Collection Client, Kafka Cluster and Storm Cluster (OtherApp).

Zookeeper registration center, providing load balancing and address lookup services;

The log collection client is used for collecting the logs of the application system and pushing the data to the kafka queue;

Fourthly, JMS message service.

When it comes to message queuing, you have to mention JMS. JMS (Java Message Service) API is the standard/specification of message service, which allows application components to create, send, receive and read messages based on JavaEE platform. It reduces the coupling of distributed communication and makes the message service more reliable and asynchronous.

In EJB architecture, message bean can be seamlessly integrated with JM message services. In J2EE architecture mode, there is a message server mode, which is used to realize direct decoupling between messages and applications.

4. 1 message model

In JMS standard, there are two message models, P2P (Peer-to-Peer) and pub/sub.

4. 1. 1 P2P mode

P2P model contains three roles: message queue, sender and receiver. Each message is sent to a specific queue from which the receiver gets the message. Queues hold messages until they are used or time out.

Characteristics of P2P

Each message has only one consumer (that is, once used, the message is no longer in the message queue).

There is no time dependence between the sender and the receiver, that is, after the sender sends the message, it will not affect whether the receiver runs.

The receiver needs to reply to the queue successfully after successfully receiving the message.

If every message you want to send is successfully processed, then P2P mode is needed. (Building KKQ:466097527, welcome to join)

4. 1.2 publish/subscribe mode

There are three role themes: Publisher and Subscriber. Multiple publishers send messages to the topic, and the system delivers these messages to multiple subscribers.

Characteristics of publish/subscribe

Each message can have multiple consumers.

There is a time dependency between the publisher and the subscriber. Subscribers to a topic must create a subscriber before they can use the publisher's message.

In order to use messages, subscribers must keep running.

In order to alleviate this strict time dependence, JMS allows subscribers to create persistent subscriptions. In this way, even if the subscriber is not activated (running), it can receive messages from the publisher.

If the message you want to send can be processed without any processing, or only by one messenger, or by multiple consumers, then the Pub/Sub model can be adopted.

4.2 message consumption

In JMS, the generation and use of messages are asynchronous. For consumption, JMS messengers can consume messages in two ways.

(1) synchronization

The subscriber or receiver receives the message through the receive method, which will block until the message is received (or before timeout);

(2) Asynchronous

Subscribers or recipients can register as message listeners. When the message arrives, the system automatically calls the onMessage method of the listener.

JNDI:Java naming and directory interface, which is a standard Java naming system interface. Services can be found and accessed on the network. By specifying a resource name corresponding to a record in a database or naming service, you can return the information needed to establish a resource connection.

JNDI plays the role of finding and accessing the sending destination or message source in JMS. (Building KKQ:466097527, welcome to join)

4.3JMS programming model

(1) connecting factory

There are two different jms message models: QueueConnectionFactory and TopicConnectionFactory. You can find the ConnectionFactory object through JNDI.

② Destination

Destination refers to the message sending target of the message producer or the message source of the message consumer. For message producers, its destination is a queue or a topic; For message consumers, its destination is also a queue or topic (that is, the message source).

Therefore, destinations are actually two types of objects: queues and topics, which can be found through JNDI.

(3) Connection

Connection represents the link established between the client and JMS system (wrapper of TCP/IP socket). A connection can generate one or more sessions. Like ConnectionFactory, there are two types of connections: QueueConnection and TopicConnection.

(4) Meeting

A session is an interface for manipulating messages. You can create producers, consumers, messages, etc. Through conversation. A session provides the function of a transaction. When you need to send/receive multiple messages using a session, you can put these sending/receiving operations in one transaction. Similarly, it is also divided into QueueSession and TopicSession.

(5) the producer of the message

Message producers are created by sessions and used to send messages to destinations. Similarly, there are two types of message generators: QueueSender and TopicPublisher. You can call the method of the message generator (send or publish method) to send a message.

(6) message consumers

A message consumer is created by a session to receive messages sent to a target. Two types: QueueReceiver and TopicSubscriber. It can be created through the createReceiver(Queue) or createSubscriber(Topic) of the session, respectively. Of course, you can also create a durable subscriber by using the creatDurableSubscriber method of the session.

(7) Message Listener

Message listener. If a message listener is registered, the onMessage method of the listener will be called automatically once the message arrives. The MDB (message-driven Bean) of EJB is a MessageListener.

Studying JMS deeply is very helpful to master JAVA architecture and EJB architecture, and message middleware is also an essential component of large distributed systems. This sharing mainly introduces the overall situation, and the specific depth needs everyone to learn, practice, summarize and understand.

Five, common message queue

General commercial containers, such as WebLogic and JBoss, all support JMS standards, which is very convenient to develop. But free ones like Tomcat and Jetty need to use third-party message middleware. This part introduces the commonly used message middleware (Active MQ, Rabbit MQ, Zero MQ, Kafka) and their characteristics.

5. 1 ActiveMQ

ActiveMQ is the most popular and powerful open source message bus produced by Apache. ActiveMQ is a JMS provider implementation, which fully supports JMS 1. 1 and J2EE 1.4 specifications. Although the JMS specification has been published for a long time, JMS still plays a special role in today's J2EE applications.

ActiveMQ functions are as follows:

1. Write the client in multiple languages and protocols. Languages: Java, C, C++, C #, Ruby, Perl, Python, PHP. Application protocols: openwire, stomrest, wsnotification, xmpp, amqp.

1. fully supports JMS 1. 1 and J2EE 1.4 specifications (persistence, XA message, transaction).

3. Support for spring, ActiveMQ can be easily embedded into systems using Spring, and it also supports the features of Spring2.0.

4. It has passed the test of common J2EE servers (such as Geronimo, JBoss 4, Glassfish, WebLogic), and through the configuration of JCA 1.5 resource adapter, ActiveMQ can be automatically deployed to any commercial server compatible with J2EE 1.4.

5. Support multiple transport protocols: in-VM, TCP, SSL, NIO, UDP, JGroups, JXTA.

Supports high-speed message persistence through JDBC and logging.

7. High-performance cluster, client-server and peer-to-peer network are guaranteed by design.

⒏ support Ajax

(9) Support the integration with Axis.

⒑: You can easily call the embedded JMS provider for testing.

5.2 rabbit q

RabbitMQ is a popular open source message queuing system, which is developed in erlang language. RabbitMQ is the standard implementation of AMQP (Advanced Message Queuing Protocol). Support a variety of clients, such as Python, Ruby,. NET, Java, JMS, c, PHP, ActionScript, XMPP, STOMP, etc. , support AJAX, insist. It is used to store and forward messages in a distributed system, and it performs well in terms of ease of use, scalability and high availability.

Several important concepts:

Broker: Simply put, it is a message queuing server entity.

Exchange: Message switch, which specifies which queues messages are routed to according to what rules.

Queue: Message queue carrier, which puts each message into one or more queues.

Binding: binding, which is used to bind switches and queues according to routing rules.

Routing keyword: the routing keyword according to which exchange delivers mail.

Virtual host. You can open multiple vhost in a proxy to separate the permissions of different users.

Producer: A message producer is a program that delivers messages.

Consumer: A message consumer is a program that accepts messages.

Channel: news channel. In each connection of the client, multiple channels can be established, and each channel represents a session task.

The use process of message queue is as follows:

(1) The client connects to the Message Queuing server and opens a channel.

(2) The client declares exchange and sets related properties.

(3) The client declares the queue and sets related attributes.

(4) The client uses the routing key to establish a good binding relationship between the switch and the queue.

(5) The client sends a message to the switch.

After receiving the mail, exchange routes the mail according to the mail key and the set binding, and sends the mail to one or more queues.

5.3 Zero MQ

Known as the fastest message queue in history, it is actually similar to a series of interfaces of socket. The difference between BSD and Socket is that ordinary Socket is end-to-end (1: 1), while ZMQ is n: m, people know more about BSD socket, and point-to-point connection needs to explicitly establish connection, destroy connection and select protocol (TCP/UDP). ZMQ is used for communication between nodes, which can be hosts or processes.

Quote the official statement: "ZMQ (hereinafter referred to as ZMQ) is a simple and easy-to-use transport layer, a socket library like a framework, which makes socket programming simpler, more concise and higher in performance. Is a message processing queue library, which can be flexibly extended among multithreading, kernel and main chassis. ZMQ's clear goal is to "become a part of the standard network protocol stack and then enter the Linux kernel". We haven't seen their success yet. However, it is undoubtedly a promising and more needed encapsulation layer on top of the "traditional" BSD socket. ZMQ makes it extremely simple and interesting to write high-performance network applications. "

Features are:

High performance, non-persistence;

Cross-platform: Support Linux, Windows, OS X, etc.

Multilingual support; C, C++, Java,. NET and Python.

It can be deployed separately or integrated into an application.

Can be used as a Socket communication library.

Compared with RabbitMQ, ZMQ is not like a traditional message queuing server. In fact, it is not a server at all, but more like an underlying network communication library. It is encapsulated in Socket API, which abstracts network communication, process communication and thread communication into a unified API interface. Three basic models and extended models are supported: request-reply, publisher-subscriber and parallel pipeline.

Key points of high performance design of ZeroMQ;

1, lock-free queue model

For the data exchange channel pipeline between cross-thread interactions (client and session), the lock-free queuing algorithm CAS is adopted. Asynchronous events are registered at both ends of the pipeline. When a message is read or written to the pipeline, the read-write event will be triggered automatically.

2. Batch processing algorithm

For traditional message processing, each message needs to be called by the system when it is sent and received, so for a large number of messages, the overhead of the system is relatively large. ZeroMQ adaptively optimizes batch messages and can send and receive messages in batches.

3. Thread binding without CPU switching under multi-core.

Different from the traditional multi-thread concurrency mode, semaphore or critical section, zeroMQ makes full use of the advantages of multi-cores, and each core is bound to run a worker thread, thus avoiding the overhead of CPU switching between multi-threads.

5.4 Kafka

Kafka is a high-throughput distributed publish-subscribe message system, which can handle all the action stream data in consumer-scale websites. This kind of behavior (web browsing, search and other user behaviors) is the key factor of many social functions on modern networks. Due to throughput requirements, these data are usually solved by processing logs and log aggregation. This is a feasible solution for log data and offline analysis system such as Hadoop, but it needs real-time processing. Kafka aims to unify online and offline message processing through Hadoop's parallel loading mechanism, and also provide real-time consumption through cluster machines.

Kafka is a high-throughput distributed publish-subscribe message system with the following characteristics:

The message persistence is provided by O( 1) disk data structure, which can maintain stable performance for a long time even for several TB of message storage. (Write data by attaching files, and delete expired data periodically)

High throughput: Even very common hardware Kafka can support millions of messages per second.

Support message partition through Kafka server and consumer cluster.

Support Hadoop parallel data loading.

Kafka related concepts

broker

Kafka cluster contains one or more servers, which are called proxies [5].

theme

Every message published to Kafka cluster has a category called Topic. (Physically, messages with different topics are stored separately. Logically, topic messages are stored in one or more agents, but users can produce or consume data only by specifying the topic of the message, without worrying about where the data is stored. )

divide

Parity is a physical concept, and each topic contains one or more parity.

producer

Responsible for publishing messages to Kafka broker.

consumer

Message consumer, the client that reads messages from Kafka broker.

Consumer groups

Each user belongs to a specific user group (you can specify a group name for each user, and if no group name is specified, it belongs to the default group).

It is generally applied to the processing of big data logs or scenes with slightly lower requirements for real-time (a little delay) and reliability (a little lost data).