Defining resources

This procedure defines the following logical resources required for the cluster:

  • DRBD Master/Secondary DFS set
  • Two mirrored filesystems running on top of DRBD:
    • /opt/serviced/var/isvcs
    • /opt/serviced/var/volumes
  • serviced logical volume group running on top of DRBD
  • Manage serviced storage
  • The floating virtual IP address of the cluster (HA-Virtual-IP), which the management software assigns to the active node
  • Docker
  • NFS
  • Control Center
  1. Log in to the primary node as root, or as a user with superuser privileges.
  2. In a separate window, log in to the secondary node as root, or as a user with superuser privileges.
  3. Define a resource for the DRBD device, and a clone of that resource to act as the master.
    1. On the primary node, define a resource for the DRBD device.
      pcs resource create DFS ocf:linbit:drbd \
        drbd_resource=serviced-dfs \
        op monitor interval=30s role=Master \
        op monitor interval=60s role=Slave
    2. On the primary node, define a clone of that resource to act as the master.
      pcs resource master DFSMaster DFS \
        master-max=1 master-node-max=1 \
        clone-max=2 clone-node-max=1 notify=true
    For a master/slave resource, Pacemaker requires separate monitoring intervals for the different roles. In this case, Pacemaker checks the master every 30 seconds and the slave every 60 seconds.
  4. Define the filesystems that are mounted on the DRBD devices.
    1. On the primary node, define a resource for Control Center internal services data.
      pcs resource create serviced-isvcs Filesystem \
        device=/dev/drbd/by-res/serviced-dfs/0 \
        directory=/opt/serviced/var/isvcs fstype=xfs
    2. On the primary node, define a resource for Control Center metadata.
      pcs resource create serviced-volumes Filesystem \
        device=/dev/drbd/by-res/serviced-dfs/1 \
        directory=/opt/serviced/var/volumes fstype=xfs
    In the preceding definitions, serviced-dfs is the name of the DRBD resource defined previously, in /etc/drbd.d/serviced-dfs.res.
  5. On the primary node, define the logical volume for serviced that is backed by a DRBD device.
    pcs resource create serviced-lvm ocf:heartbeat:LVM volgrpname=serviced
  6. On the primary node, define the storage resource for serviced, to ensure that the device mapper device is deactivated and unmounted properly.
    pcs resource create serviced-storage ocf:zenoss:serviced-storage
  7. On the primary node, define the resource that represents the floating virtual IP address of the cluster.
    For dual-NIC deployments, the definition includes the nic key-value pair, which specifies the name of the network interface that is used for all traffic except the private DRBD traffic between the primary and seconday nodes. For single-NIC deployments, omit nic key-value pair.

    For dual-NIC deployments, replace HA-Virtual-IP with the floating virtual IP address of the cluster, and replace HA-Virtual-IP-NIC with the name of the network interface that is bound to HA-Virtual-IP:

    pcs resource create VirtualIP ocf:heartbeat:IPaddr2 \
      ip=HA-Virtual-IP nic=HA-Virtual-IP-NIC \
      cidr_netmask=32 op monitor interval=30s

    For single-NIC deployments, replace HA-Virtual-IP with the floating virtual IP address of the cluster:

    pcs resource create VirtualIP ocf:heartbeat:IPaddr2 \
      ip=HA-Virtual-IP cidr_netmask=32 op monitor interval=30s
  8. Define the Docker resource.
    1. On the primary node, define the resource.
      pcs resource create docker systemd:docker
    2. On both nodes, ensure that the automatic startup of Docker by systemd is disabled.
      systemctl stop docker && systemctl disable docker
  9. Define the NFS resource.
    Control Center uses NFS to share configuration in a multi-host deployment, and failover will not work properly if NFS is not stopped on the failed node.
    1. On the primary node, define the resource.
      pcs resource create nfs systemd:nfs
    2. On the primary node, disable Pacemaker monitoring of NFS health.
      During normal operations, Control Center occasionally stops and restarts NFS, which could be misinterpreted by Pacemaker and trigger an unwanted failover.
      pcs resource op remove nfs monitor interval=60s
      pcs resource op add nfs monitor interval=0s
    3. On both nodes, ensure that the automatic startup of NFS by systemd is disabled.
      systemctl stop nfs && systemctl disable nfs
  10. Define the Control Center resource.
    1. On the primary node, define the resource.
      pcs resource create serviced ocf:zenoss:serviced
    2. On both nodes, ensure that the automatic startup of serviced by systemd is disabled.
      systemctl stop serviced && systemctl disable serviced

    Pacemaker uses the default timeouts defined by the Pacemaker resource agent for Control Center to decide if serviced is unable to start or shutdown correctly. Starting with version 0.0.5 of the Pacemaker resource agent for Control Center, the default values for the start and stop timeouts are 360 and 130 seconds respectively.

    The default startup and shutdown timeouts are based on the worst case scenario. In practice, Control Center typically starts and stops in much less time. However, this does not mean that you should decrease these timeouts. There are potential edge cases, especially for startup, where Control Center may take longer than usual to start or stop. If the start/stop timeouts for Pacemaker are set too low, and Control Center encounters one of those edge cases, then Pacemaker takes unnecessary or incorrect actions. For example, if the startup timeout is artificially set too low, 2.5 minutes for example, and Control Center startup encounters an unusual case where it requires at least 3 minutes to start, then Pacemaker initiates failover prematurely.