Performing initial performance data ETL configuration

Since monitoring templates are a function of the ZenPacks that provide them, the default set of aliases provided for any particular ZenPack is a function of what is provided by that ZenPack. Details of default aliases provided by a particular ZenPack are provided in the "Zenoss Analytics" section of the webpage for that ZenPack on https://www.zenoss.com/product/zenpacks.

Where it exists, such default supplied configuration should be thought of as definitive documentation of a fully comprehensive set of ETL configuration for all the data points monitored by that ZenPack and this ETL configuration will evolve with future ZenPack development. Thus, it is likely significant overkill to ETL all such data points to your Analytics data warehouse without any actual requirement for reporting on them. Doing so is likely to add more ETL load and use a lot of disk space on the Analytics server, which makes managing Analytics much harder than it would otherwise be.

The following procedure should be run immediately after initial install and repeated after every ZenPack upgrade or Resource Manager upgrade to re-audit the ETL configuration so that it is not inadvertently changed as a result of such activities.

If you are on Resource Manager 4.x, enter the following on the Resource Manager master:
su - zenoss
If you are on Resource Manager 5.x, enter the following on the Control Center master:
serviced service attach zope/0
su - zenoss
Then, regardless of your Resource Manager version, change directory to the ZenETL ZenPack directory. In the following command, replace <version> with the current ZenETL ZenPack version:
cd /opt/zenoss/ZenPacks
cd ZenPacks.zenoss.ZenETL-<version>-py2.7.egg
cd ZenPacks/zenoss/ZenETL/bin
In this directory, there are two relevant python scripts:
  • dumpAliases.py
    • This script can be used to traverse your entire monitoring template hierarchy and dump out details of data points and aliases.
    • Run dumpAliases.py --help to read the detailed help.
    • The script takes one required parameter --aliases= with a value of either with or without. The former dumps details of all aliases on data points, the later dumps details of all data points with no aliases.
    • The optional parameter -f or --outputFile can use to specify a filename to write the information to. If you exclude this file, the default filename of aliases.txt is used. The output format of this file is in the exact required input format for manageAliases.py.
  • manageAliases.py
    • This script can be used to bulk or remove aliases from data points in your monitoring template hierarchy. Run manageAliases.py --help to read the detailed help on options. Run manageAliases.py --printhelp to read the detailed help on input file format and output results.
    • The script takes one required parameter --action= with a value of either add or remove. The former adds aliases to data points, the later removes existing aliases from data points.
    • By default it is “read-only”, that is it only reports what it could/would do regarding changing aliases on monitoring templates. You must call it with the --commit option to actually make changes effective on your monitoring templates.
Use these scripts as follows to get your ETL configuration to an initial known state with metrics for device-level CPU and memory usage, and component-level filesystem and IP interface metrics ETL to Analytics. This will allow out-of-the-box sample reports to execute successfully. In the following steps, replace <YYYYMMDD> with today's date.
  1. Dump out the current state of aliases in your install:
    ./dumpAliases.py --aliases=with --outputFile=original_aliases_<YYYYMMDD>.txt
  2. Make a copy of this file and keep it somewhere safe for future reference:
    scp original_aliases_<YYYYMMDD>.txt myuser@myserver:/pathtosafeplace/
  3. Use this file to remove ALL aliases currently in place:
    ./manageAliases.py --action=remove --inputFile=original_aliases_<YYYYMMDD>.txt 2>&1 | tee removal_test.log
    Check that this would remove all aliases successfully. The following grep should return no alias lines:
    grep -iv "^-" removal_test.log | grep -v ControlCenter

    If not, run manageAliases.py –-printhelp to get information of what the error codes at the start of each line indicate.

    If successful, repeat with --commit to actually make the changes:
    ./manageAliases.py --action=remove --inputFile=original_aliases_<YYYYMMDD>.txt --commit 2>&1 | tee aliases_removed_<YYYYMMDD>.log
  4. Confirm this was successful, that is you now have no aliases at all:
    ./dumpAliases.py --aliases=with
    A file named aliases.txt with no contents should have been created. To check that no contents were created, execute the following grep command:
    grep -v ControlCenter aliases.txt
  5. Create a desired subset of aliases from this file for the above mentioned device- and component-level metrics:
    head -1 original_aliases_<YYYYMMDD>.txt > current_aliases_<YYYYMMDD>.txt
    	egrep "\|(cpu__pct|mem__pct|fs__pct|in__pct|out__pct)\|" original_aliases_<YYYYMMDD>.txt >> current_aliases_<YYYYMMDD>.txt
  6. Keep a copy of this file safe for future reference. It documents what your ETL configuration is always supposed to be. Zenoss highly recommends you put it under configuration management, and incrementally add to it over time as you add more data to the data warehouse to meet Analytics reporting needs.
  7. Add these aliases to your monitoring templates:
    ./manageAliases.py --action=add --inputFile=current_aliases_<YYYYMMDD>.txt  2>&1 | tee add_test.log
    Check that this would add all aliases successfully. The following grep should return no alias lines:
    grep -iv "^+" add_test.log | grep -v ControlCenter

    If not, run manageAliases.py –-printhelp to get information of what the error codes at the start of each line indicate.

    If successful, repeat with --commit to actually make the changes:
    ./manageAliases.py --action=add --inputFile=current_aliases_<YYYYMMDD>.txt  --commit 2>&1 | tee aliases_added_<YYYYMMDD>.log
  8. Confirm this was successful, that is you now only have exactly the aliases you added:
    ./dumpAliases.py --aliases=with --outputFile=aliases.txt
               sort -u aliases.txt > a
               sort -u current_aliases_<YYYYMMDD>.txt > b
               diff a b

    The diff should show no differences between the two files.