Starting a ZooKeeper ensemble

Use this procedure to start a ZooKeeper ensemble.

This procedures uses the nc utility to query ensemble hosts. If nc is not available, you can use telnet with interactive ZooKeeper commands.

The window of time for starting a ZooKeeper ensemble is relatively short. The goal of this procedure is to restart Control Center on each ensemble node at about the same time, so that each node can participate in electing the leader.

  1. Log in to the Control Center master host as root, or as a user with superuser privileges.
  2. In a separate window, log in to the second node of the ZooKeeper ensemble (Delegate-A) as root, or as a user with superuser privileges.
  3. In a different window, log in to the third node of the ZooKeeper ensemble (Delegate-B) as root, or as a user with superuser privileges.
  4. On all ensemble hosts, stop and start serviced.
    systemctl stop serviced && systemctl start serviced
  5. On the master host, check the status of the ZooKeeper ensemble.
    1. Attach to the container of the ZooKeeper service.
      docker exec -it serviced-isvcs_zookeeper /bin/bash
    2. Query the master host and identify its role in the ensemble.
      Replace Master with the hostname or IP address of the master host:
      { echo stats; sleep 1; } | nc Master 2181 | grep Mode
      The result includes leader or follower.
    3. Query delegate host A and identify its role in the ensemble.
      Replace Delegate-A with the hostname or IP address of delegate host A:
      { echo stats; sleep 1; } | nc Delegate-A 2181 | grep Mode
    4. Query delegate host B and identify its role in the ensemble.
      Replace Delegate-B with the hostname or IP address of delegate host B:
      { echo stats; sleep 1; } | nc Delegate-B 2181 | grep Mode
    5. Detach from the container of the ZooKeeper service.
      exit
    If none of the hosts reports that it is the ensemble leader within a few minutes of starting serviced, reboot the hosts.