message queue zookeeper

Distributed coordination technology----zookeeper

Distributed coordination technology is used to solve the synchronization control between multiple processes in a distributed environment so that they can access in an orderly manner Certain critical resources prevent the consequences of "dirty data". Distribution cannot be easily solved by a simple scheduling algorithm. This is a misunderstanding. If all the processes are on one machine, it will indeed take so long to handle, but how to solve it in a distributed environment?

Let me analyze this picture for you. Each of the three machines in the picture runs an application. These are connected through the network diagram to form a system to provide services to users. For users, This is a transparent architecture system, we can call this system a distributed system.

Let’s analyze how this distributed system schedules processes. Assume that a resource is mounted on server1. Three distributed processes have to compete for this resource, but they are not expected to run at the same time. Access, at this time, a "coordinator" is needed to allow them to obtain resources in an orderly manner, and the "coordinator" is called a lock. For example, "Process-1" indicates that the program will first acquire the lock when using the resource. If "Process 1" acquires the lock, it will monopolize the resource. After use, the lock will be released to allow other processes to acquire the lock. This lock is a distributed lock and a distributed coordination technology.

The better distributed ones are Google's Chubby and Apache's Zookeeper. Chubby is non-open source. Later, Yahoo imitated Chubby and developed Zookeeper, which also implemented similar distributed functions and used zookeeper as The open source program is donated to Apache. Zookeeper is used to build our system, which not only has fewer bugs but also saves costs.

The advantage of Zookeeper is that it is a highly available, high-performance and consistent open source coordination tool. It also provides "distributed lock service", which is powerful and can configure and maintain, group services, distributed notifications, distributed message queues, etc. In terms of performance, it adopts the Zab protocol, the Znode used for the data structure, which defines primitives on the data structure, and the Watcher mechanism used for the notification mechanism. It has strict sequence access control and will not crash due to a node error.

(1) Two master nodes are set up for Zookeeper, master node-A master-00001, master node-B master-00002. After startup, they register a node like zookeeper, and master node-A becomes The master node, master node-B, becomes the backup node and schedules the two Master processes.

(2) Master node-A dies, and its registered node is automatically deleted. The sensing node initiates an election. After the election, master node-B becomes the master node, replacing master node-A.

(3) When Master recovers, it will register a node master-00003 with zookeeper. After re-election, master node-B is still the master node, and master node A becomes a backup node.

Installation and configuration steps:

Test machine: 192.168.10.10 hostname:zk1

192.168.10.11? hostname:zk2

192.168. 10.12 hostname:zk3

1. Install jdk

[^_^] ~# tar xf jdk-8u131-linux-x64.tar.gz -C /usr/local/< /p>

[^_^] ~# mkdir /usr/local/java

[^_^] java# mv jdk1.8.0_171 /usr/local/java

< p> [^_^] ~# cat /etc/profile.d/java.sh

JAVA_HOME=/usr/local/java/jdk1.8.0_131

JRE_HOME=/ usr/local/java/jdk1.8.0_131/jre

CLASS_PATH=.:$JAVA_HOME/lib/dt.jar:$JAVA_HOME/lib/tools.jar:$JRE_HOME/lib

< p> PATH=$JAVA_HOME/bin:$JRE_HOME/bin:$PATH

export JAVA_HOME JRE_HOME CLASS_PATH

[^_^] ~# source /etc/profile.d/java .sh

Verification: java -version

2. Install zookeeper

[^_^] ~# cat /etc/hosts

127.0.0.1? localhost localhost.localdomain localhost4 localhost4.localdomain4

::1 localhost localhost.localdomain localhost6 localhost6.localdomain6

192.168.36.10? zk1

192.168.36.11? zk2

192.168.36.12? zk3

[^_^] ~# cat /etc/profile.d/zk.sh

# Set Zookeeper Environment

export ZOOKEEPER_HOME=/usr/local/zookeeper-3.4.11

export PATH=$PATH:$ZOOKEEPER_HOME/bin:$ZOOKEEPER_HOME/conf

< p> [^_^] ~# source /etc/profile.d/zk.sh

[^_^] ~#cp /usr/local/zookeeper/conf/zoo_sample.cfg /usr/ local/zookeeper/conf/zoo.cfg

[^_^] ~#mkdir -p /usr/local/zookeeper/data/log

[^_^] ~# echo "1" >?/usr/local/zookeeper/data/myid ? #The myids of the three zookeepers are different

[^_^] ~# egrep -v "^$|#" /usr/local/zoo

keeper-3.4.11/conf/zoo.cfg

tickTime=2000

initLimit=10

syncLimit=5

dataDir= /tmp/zookeeper

dataLogDir=/tmp/zookeeper/log

clientPort=2181 #Service port

server.1=zk1:2888:3888? # 2888 is the port for follower connection to leader, and 3888 is the port for selecting leader

server.2=zk2:2888:3888

server.3=zk3:2888:3888

< p> [^_^] ~# cd /usr/local/zookeeper-3.4.11/bin/

[^_^] bin# ./zkServer.sh start? &

< p> [^_^] bin#jps #Check whether the startup is successful

1155 Jps

1093 QuorumPeerMain? #QuorumPeerMain is the zookeeper process

[^_^ ] bin# ./zkServer.sh status ? #Check node status

[^_^] bin#zkCli.sh -server zk1:2181,zk2:2181,zk3.:2181 ? #Can create zk Cluster