Starting a ZooKeeper ensemble

Use this procedure to start a ZooKeeper ensemble.

This procedures uses the nc utility to query ensemble hosts. If nc is not available, you can use telnet with interactive ZooKeeper commands.

The window of time for starting a ZooKeeper ensemble is relatively short. The goal of this procedure is to restart Control Center on each ensemble node at about the same time, so that each node can participate in electing the leader.

  1. Log in to the Control Center master host as root, or as a user with superuser privileges.
  2. In a separate window, log in to the second node of the ZooKeeper ensemble (Delegate-A) as root, or as a user with superuser privileges.
  3. In a different window, log in to the third node of the ZooKeeper ensemble (Delegate-B) as root, or as a user with superuser privileges.
  4. On all ensemble hosts, stop and start serviced.
    systemctl stop serviced && systemctl start serviced
  5. On the master host, check the status of the ZooKeeper ensemble.
    1. Attach to the container of the ZooKeeper service.
      docker exec -it serviced-isvcs_zookeeper /bin/bash
    2. Define IP address variables for each node in the ZooKeeper ensemble.
      Replace Master with the IP address or hostname of the Control Center master host, and replace Delegate-A and Delegate-B with the IP addresses or hostnames of the delegate hosts in the ensemble:
      node1=Master
      node2=Delegate-A
      node3=Delegate-B
    3. Query the master host and identify its role in the ensemble.
      { echo stats; sleep 1; } | nc $node1 2181 | grep Mode
      The result includes leader or follower.
    4. Query delegate host A and identify its role in the ensemble.
      { echo stats; sleep 1; } | nc $node2 2181 | grep Mode
    5. Query delegate host B and identify its role in the ensemble.
      { echo stats; sleep 1; } | nc $node3 2181 | grep Mode
    6. Detach from the container of the ZooKeeper service.
      exit
    If none of the hosts reports that it is the ensemble leader within a few minutes of starting serviced, reboot the hosts.