ZenPacks

Distributed Collector ZenPack

Distributed Collector allows you to deploy additional performance collection and event monitoring daemons to the Resource Manager server or other servers.

Support

This ZenPack is included with commercial versions of Zenoss and enterprise support for this ZenPack is provided to Zenoss customers with an active subscription.

Background

The ZenPacks.zenoss.DistributedCollector ZenPack allows you to deploy additional performance collection and event monitoring daemons, on the Zenoss platform master host, and on other hosts.


This feature allows you to:

  • Distribute processor, disk, and network load across multiple servers.
  • Collect performance and events from networks that cannot be reached by the Zenoss platform server.
  • Configure more than one set of monitoring settings, such as different cycle times, for the zenperfsnmp daemon.

When you first install Distributed Collector, Zenoss platform is configured with one hub and one collector. You cannot delete the initial hub and collector (each named localhost) that are set up by Distributed Collector.

About Collectors

A collector is a set of collection daemons, on the Zenoss platform server or another server, that shares a common configuration. That configuration contains values, such as:

  • Number of seconds between SNMP collections cycles
  • Default discovery networks
  • Maximum number of zenprocess parallel jobs

Each collector has its own copy of each of the Zenoss platform collection daemons. For example, Zenoss platform initially contains collection daemons with names like zenperfsnmp, zenprocess, and zenping. If you create a new collector named My2ndCollector, then the system creates new daemons named My2ndCollector_zenperfsnmp, My2ndCollector_zenprocess, and My2ndCollector_zenping.

About Hubs

Distributed Collector also allows you to set up new hubs. A hub represents an instance of the zenhub daemon, through which all collector daemons communicate with the object and event databases.

All collectors must belong to exactly one hub; however, a hub can have many collectors associated with it. All hubs (and indirectly all collectors) refer to the same object and event databases. Typically, only very large systems with more than five collectors (or more than 1,500 devices) benefit from multiple hubs.

Hubs manage configuration data and pass it to the collectors. Hubs also take data from the collectors and pass it to the Zenoss DataStore. Multiple hubs can be a more efficient way to manage larger deployments, as they help distribute the computing resources when configuration changes are made. They further remove the potential for configuration changes to be a bottleneck to gathering and processing data.

Typical Usage Scenarios for Distributed Monitoring

The correct distributed strategy for your environment depends on network security restrictions, as well as scale. Contact Zenoss Support if you are unsure which option best suits your enterprise.

Typical setup scenarios for using multiple hubs and collectors are shown in the following table:

Distributed Monitoring Use Scenarios
HubCollectorDescription
Local hubLocal collectorThis setup requires only a single server, and is the most common Zenoss platform deployment type. You would most likely use this configuration if you need to monitor fewer than 1000 devices, and your master Zenoss platform server has direct network access to all of the monitored devices.
Local hubRemote collectorThis setup requires two servers, and is the most basic distributed setup. The primary benefit of this configuration over the local hub/local collector configuration is that the master server does no collection. This frees resources, optimizing the server's ability to perform its central role of database server and Web interface.
Local hubMultiple remote collectorsThis is the most common distributed Zenoss platform configuration. You might use this configuration to scale Zenoss platform to monitor more than 1000 devices. Depending on the hardware of the collectors, it is possible to monitor up to 1000 devices for each collector using this configuration. You might also use this configuration to handle differing network security policies. Often, your master Zenoss platform server will not have access to all of the devices you need to monitor. In this case you can set up a remote collector with the required network access.
Multiple remote hubsMultiple remote collectorsThis configuration is for large installations only. For cases in which you have more than five collectors, you should consider deploying one or more hub servers to handle them.

Prerequisites

PrerequisiteRestriction
ProductZenoss platform 4.x, Zenoss 2.4 or higher
Required ZenPacksZenPacks.zenoss.DistributedCollector

Navigating Collectors and Hubs

To view and manage collectors and hubs:

  1. Log in as the Zenoss platform user.
  2. From the navigation menu, select Advanced > Collectors. The Collectors page appears. Collectors Page
    (DistributedCollector)DistributedCollector.hub.png

The page lists existing hubs and collectors in hierarchical form. Hubs are listed at the top level; collectors are nested below the hub to which they belong.

From this page, you can:

  • Add a hub
  • Delete a hub (which also deletes its associated collectors)
  • View and edit hub settings
  • Configure associated monitoring templates

Select a hub to display details and configuration options. From here, you can:

  • Update a hub
  • Add and delete collectors


Collectors Page - Overview


(DistributedCollector)DistributedCollector.hub configuration.png


In the left panel, select Daemons to display details about the zenhub daemon that belongs to the collector. Links adjacent to the daemon name allow you to view the daemon log, and view and edit its configuration. Use the buttons to the right of the daemon name to stop, start, and restart it.


Collectors Page - Daemons


(DistributedCollector)DistributedCollector.hub daemon.png

Updating Collectors

You must update all collectors after you:

  • Update your version of Zenoss platform
  • Install patches
  • Install, update, or remove ZenPacks
  • Change the zenhub port of an associated hub

To update a collector:

  1. From the navigation menu, select Advanced > Collectors.
  2. Select the collector to display its Overview page.
  3. Select Update Collector from the Action menu. Zenoss platform copies the most recent code and ZenPacks to the server, and restarts the daemons running there.

Using nginx as a Reverse Proxy

Note: This section is only relevant for users upgrading from Zenoss Core.


After installing Zenoss platform, existing collectors must be configured to use the nginx reverse proxy. Otherwise, the host name of the collector cannot be resolved from the master, and zenwebserver (nginx) will not start.

To configure a collector to use the reverse proxy, set the render url property to:

/remote-collector/CollectorID

where CollectorID is the ID of the collector.

Backing Up Remote Collector Performance Data

Zenoss platform does not automatically back up remote collector performance data (RRD files). To back up this data, set up a cron job on the remote collector. The cron job should invoke zenbackup with these options:

zenbackup --no-eventsdb --no-zodb

Old backup data is not automatically deleted; therefore, the backup solution you use to save the data should remove the backup file when it is no longer needed.

Configuring Collector Data Storage

You can configure the amount of data stored by RRDcached for each collector. Edit one or more options in the zenrrdcached.conf file; options are:

  • write_threads Value - Specifies the number of threads for writing files. By default, this value is 4.
  • write_timeout Value - Specifies the frequency at which data is written to disk. By default, this value is 300 seconds.
  • write_delay Value - Specifies the delay for writing. By default, this value is 0 seconds.
  • flush_timeout Value - Specifies the timeout value for flushing old data. By default, this value is 3600 seconds.

Deleting Collectors

When you delete a collector, its devices are left without an assigned collector. Zenoss recommends that you reassign assigned devices prior to deleting a collector.

To delete a collector, click the name of the hub where the collector exists from the main collectors page. The Hub overview page appears. From the list of Collectors, select the collector you want to delete. From the Action menu, select Delete Collector.

When you delete collectors using this Zenoss platform instance, they are not removed or "uninstalled" in any way from the collector device. They continue to exist on the device until manually removed through the file system.

Adding Devices to Collectors

Adding devices to collectors occurs when you add the device to Zenoss platform.

  1. Log in to the Zenoss platform user interface as a user with ZenManager or Manager privileges.
  2. Navigate to INFRASTRUCTURE > Devices.
  3. From the Add Device menu, select Add a Single Device.
  4. In the Add a Single Device dialog, use the Collector field to select a local or remote collector for the device to add.
  5. Populate the remaining fields as required, and then click Add.


The device appears in the Devices list, located at the bottom of the collector overview page.

Moving Devices Between Collectors

You can move devices from one collector to another.

  1. Select one or more device rows in the device list.
  2. Select Set Collector from the Actions list of options.
  3. Select a collector, and then click OK. Zenoss platform moves the devices to the selected collector.

Note: If you move devices between collectors, you also can select an option choose to move their associated performance data.

Managing Collector Daemons

Collector daemons appear on the Daemons page for each collector, and can be started, stopped and restarted from there.

Only one zentrap instance can be run on a server, as it must bind to the SNMP trap port (162). If you install multiple collectors on the same server, you must assign different port numbers to additional zentrap daemons. Attempting to run additional zentrap daemons using the same port will cause them to fail at startup.

Each collector installs the zenrender daemon with the rest of the collector package. If multiple collectors are installed on the same host, then the zenrender daemon will attempt to run for each collector. Since each zenrender daemon attempts to bind to the same port, all but the first daemon will fail to start. This is a problem for HA/failover environments, since failure detection systems will detect these stopped zenrender daemons and assume they are down. (ZEN-2967)

The remedy for this is to assign each zenrender daemon (beyond the first) its own unique port. This is accomplished by adding the following line to each Collector_zenrender.conf on your collector's host (in $ZENHOME/etc), where Collector is the collector name:

http-port 809X

X is a number greater than 1 and unique among multiple collectors.


Note: The ports you choose for the additional collectors are arbitrary, as they are not used. However, you must leave at least one zenrender daemon using the default port (8091).

Specifying Daemons for Collectors

You may specify the collector daemons to start for all collectors, and for individual collectors.

  • If you do not specify the daemons to start for all collectors, Zenoss platform starts all collector daemons for all collectors, on the master host and on all remote collector hosts.
  • If you do specify the daemons to start for all collectors, only the specified daemons are started.
  • You may specify the daemons to start for all collectors, and override the global setting for individual collectors.

To specify the daemons to start for all collectors, follow these steps:

  1. Log in to the Zenoss platform master host as zenoss.
  2. Open $ZENHOME/etc/collectordaemons.txt in a text editor.
  3. List the names of collector daemons to start, one per line. The following list provides the collector daemon names. Daemons may be entered in any order.
    • zencommand
    • zeneventlog
    • zenjmx
    • zenmailtx
    • zenmodeler
    • zenperfsnmp
    • zenping
    • zenprocess
    • zenpython
    • zensyslog
    • zentrap
    • zenucsevents
    • zenvcloud
    • zenvmwareevents
    • zenvmwaremodeler
    • zenvmwareperf
    • zenvsphere
    • zenwebtx
    • zenwin
    • zenwinperf
  4. Save and close the file.
  5. Use the Zenoss platform user interface to update each collector.

To specify the daemons to start for an individual collector, follow these steps:

  1. Log in to the Zenoss platform master host as zenoss.
  2. Open $ZENHOME/etc/collectordaemons.Collector-ID.txt in a text editor. Substitute the ID of the collector for Collector-ID.
  3. List the names of collector daemons to start, one per line. The preceding table provides the collector daemon names. Daemons may be listed in any order.
  4. Save and close the file.
  5. Use the Zenoss platform user interface to update the remote collector.

SSH security information

The Distributed Collector ZenPack uses openSSH and an RSA public/private key pair for secure communications between the master host and remote hub and collector hosts.

During installation, the Distributed Collector ZenPack invokes the OpenSSH ssh-keygen command to generate a new, unique RSA key pair for user zenoss on the master host. The command generates an RSA key pair with an empty passphrase, and places the key pair in the zenoss user's $HOME/.ssh directory.


Note: The Distributed Collector ZenPack does not support SSH key pairs that require a passphrase.


You may use the generated key pair to deploy a remote hub or collector, or replace the pair with a new key pair. However, you must use the same key pair for all hub and collector hosts. Therefore, if you choose to replace the key pair, Zenoss recommends doing so before deploying any remote hub or collector.

When a remote hub or collector is created, the Distributed Collector ZenPack copies the key pair of the zenoss user on the master host to the authorized keys file of the zenoss user on the remote hub or collector host, if the entry does not already exist.

When a remote hub or collector is deleted, entries for the zenoss user (on the master host) in the authorized keys file of the root and zenoss user on the remote hub or collector are not removed. Likewise, the known hosts file of the zenoss user (on the master host) is not edited to remove the entry for the remote hub or collector. You must remove these manually.

Installation

Normal Installation (packaged egg)

  1. Download the appropriate egg file for the version of Zenoss you are running.
  2. Ensure you are logged in as the zenoss user:
    $ sudo su - zenoss
  3. Install the ZenPack:
    $ zenpack --install ZenPacks.zenoss.DistributedCollector-*.egg
  4. Restart these services:
    $ zenoss restart

Commercial

This ZenPack is developed and supported by Zenoss Inc. Commercial ZenPacks are available to Zenoss commercial customers only. Contact Zenoss to request more information regarding this or any other ZenPacks. Click here to view all available Zenoss Commercial ZenPacks.

randomness