ZenPacks

AutoTune ZenPack

The AutoTune ZenPack enables the ZenTune "tuning advisor" feature in Resource Manager. ZenTune analyzes your system configuration and makes recommendations for better performance.

Support

This ZenPack is included with commercial versions of Zenoss and enterprise support for this ZenPack is provided to Zenoss customers with an active subscription.

Background

This ZenPack provides the ZenTune "tuning advisor" feature of Zenoss platform.


This ZenPack analyzes your system configuration and makes recommendations for better performance.

Prerequisites

PrerequisiteRestriction
ProductZenoss platform 4.2
Required ZenPacksZenPacks.zenoss.AutoTune

Configuring ZenTune

You can set values for several options in the zentune.conf configuration file (or when running ZenTune from the command line) to configure behavior.

When setting up ZenTune, you can define options to send a test event through the lifecycle to make sure that Zenoss is processing events before the timeout. If it fails to process in time, an email can be sent out.

  • testevent-enable -- When testevent-enable is present in the zentune.conf file, a test event will be sent. If it is not present or commented out, no test event is sent.
  • testevent-email ValidEmailAddress -- If testevent-enable is present and the test event times out, an email will be sent to the defined email address.

ZenTune can perform an analysis one or more times each day, depending on the values of these two options:

  • tune-offset Value -- Sets the number of minutes after midnight when the ZenTune will first run. By default, the value is 0.
  • tune-interval Value -- Sets the number of minutes to wait before running ZenTune again. By default, the value is 0, which is equivalent to 1440 (24 hours).

So, for example, if you want ZenTune to run twice each day, set the value of tune-offset to 0 and the value of tune-interval to 720.

Configuring ZenTune for Remote Databases

If you have installed the Zenoss DataStore on a server other than your master server, then you must set additional configuration options. Set the following options in the $ZENHOME/etc/zentune.conf file of the master server:

Remote Database Configuration Options
OptionDescription
mysqltuner-zodb-forcemem MegaBytesSets the amount of memory available on the server running the ZODB database server.
mysqltuner-zep-forcemem MegaBytesSets the amount of memory available on the server running the ZEP database server.

Using ZenTune

To access ZenTune, select Advanced > Tuning from the Zenoss platform interface.


ZenTune


(AutoTune)ZenTune.tuning advisor.png


To run ZenTune, click Update (located at the bottom left of the page). ZenTune may require several minutes to run.


Note: To check the update status, refresh the browser page and then check the "Update at" value for any watched items.


ZenTune returns information about current and optimal values for several configuration parameters. Click + to the left of each item to display recommendations, if any, for configuration changes.


ZenTune Issue Detail


(AutoTune)ZenTune.tuning advisor detail.png


To refresh the view, click Refresh. (This does not run ZenTune again.)

To filter the list of displayed items, select Not Acknowledged, Acknowledged, or both in the Acknowledge column. To acknowledge one or more items, select the option in the Acknowledge column.

You also can filter the display by severity, host, and description.

Running ZenTune from the Command Line

You can run ZenTune from the command line. On the master server, use the command:

zentune run

To run ZenTune on a remote hub or collector, prefix the command with the name of the hub or collector, followed by an underscore. For example, if the remote collector name is centos6-coll, then run the command as:

centos6-coll_zentune run

This generates a report to the console output. If you additionally specify the --events option, events are instead issued (the same events issued by the zentune daemon). The results appear on the Tuning page of the Zenoss platform interface.

Tuneable Items

The following table lists included tuneable items.

NameGroupDescription
zeneventd workersEventsZenTune monitors the number of incoming events to the zeneventd daemon. If the number of incoming events exceeds the configured threshold per worker, then a tuning event is generated. If the threshold is exceeded only during peak load times, a WARNING severity tuning event is generated. If it is exceeded more often, an ERROR severity tuning event is generated.

The threshold checked is controlled by these configuration options on the zentune daemon:

  • zeneventd-worker-events-per-second
  • zeneventd-worker-count-max-recommended

For detailed information about each of these options, run this command on any Zenoss platform server:

zentune run --help

zeneventserverEvents
zeneventserver age-eligible eventsEvents
zeneventserver archive-eligible eventsEvents
zeneventserver processed eventsEvents
zeneventserver deduped eventsEvents
zeneventserver dropped eventsEvents
zeneventserver cleared eventsEvents
zeneventserver archived eventsEvents
zeneventserver aged eventsEvents
zeneventserver summary queue lengthEvents
zeneventserver archive queue lengthEvents
zeneventserver summary index sizeEvents
zeneventserver archive index sizeEvents
zeneventserver summary index doc countEvents
zeneventserver archive index doc countEvents
age eligible event countEvents
archive eligible event countEvents
zeneventserver summary queue lengthEvents
zeneventserver archive queue lengthEvents
zeneventd object cacheGlobalThe zeneventd zodb-cachesize configuration setting controls the number of objects that zeneventd will store locally to avoid querying ZODB. Zenoss platform expects this value to be between certain thresholds, proportional to the size of the global catalog. If any of these thresholds are violated, then a WARNING or ERROR severity tuning event is generated, depending on the configured thresholds.

The thresholds checked are controlled by the following configuration options on the zentune daemon:

  • zeneventd-obj-cache-bad
  • zeneventd-obj-cache-warn

For detailed information about each of these options, run this command on any Zenoss platform server:

zentune run --help

ZODB cache serversGlobalZenoss platform expects at least one memcached server to be configured for its use as a ZODB cache. If this is not the case, then an ERROR severity tuning event is generated.

Memcached is a third-party object caching system used by Zenoss platform to improve performance for daemons that connect to Zope and ZODB. It is not required for Zenoss platform to function, but it is highly recommended.

More information on memcached can be found here:

http://memcached.org/

globalConfigGlobal
Global config sip sizeGlobalThe configsipsize global configuration setting controls the number of device configuration objects that a collector daemon will receive in a single request. A setting of 0 implies that all device configurations will be requested at once. Combined with the configsipdelay option, this effectively controls the traffic to collector daemons.
Global config sip delayGlobalThe configsipdelay global configuration setting controls the number of seconds between requests for device configuration objects that a collector daemon will make. Combined with the configsipsize option, this effectively controls the traffic to collector daemons.
Event flush chunk sizeGlobalThe eventflushchunksize global configuration setting controls the number of events a collector daemon will send to zenhub at one time. Each collector daemon will periodically flush its outgoing event queue and send events to zenhub until the queue is empty.
Maximum queue lengthGlobalThe maxqueuelen global configuration setting controls the number of events a collector daemon can store in its outgoing event queue before it must start dropping events.
IO CPU Wait TimeIOZenTune monitors iostat statistics, and if any device causes wait times exceeding the configured threshold in more than 5% of cases, then an ERROR severity tuning event is generated. The threshold checked is controlled by the following configuration option on the zentune daemon:
  • iostat-wait-threshold

For detailed information about each of these options, run this command on any Zenoss platform server:

zentune run --help

The iostat utility is a third-party program that provides statistics on the time the CPU spends waiting for I/O requests from various devices. More information on iostat can be found here:

http://en.wikipedia.org/wiki/Iostat

Cache miss percentageMemcachedMemcached is a third-party object caching system used by Zenoss platform to improve performance for daemons that connect to Zope and ZODB. It is not required for Zenoss platform to function, but it is highly recommended.

More information on memcached can be found here:

http://memcached.org/

Maximum sizeMemcachedMemcached is a third-party object caching system used by Zenoss platform to improve performance for daemons that connect to Zope and ZODB. It is not required for Zenoss platform to function, but it is highly recommended.

More information on memcached can be found here:

http://memcached.org/

Cache eviction rateMemcachedMemcached is a third-party object caching system used by Zenoss platform to improve performance for daemons that connect to Zope and ZODB. It is not required for Zenoss platform to function, but it is highly recommended.

More information on memcached can be found here:

http://memcached.org/

Cache serversMemcachedZenoss platform expects at least one memcached server to be configured for its use. If this is not the case, then an INFO severity tuning event is generated. Zenoss platform also expects all configured memcached servers to be available and responding to connection attempts using the standard memcached client. If this is not the case, then an ERROR severity tuning event is generated.

Memcached is a third-party object caching system used by Zenoss platform to improve performance for daemons that connect to Zope and ZODB. It is not required for Zenoss platform to function, but it is highly recommended.

More information on memcached can be found here:

http://memcached.org/

Cache sizeMemcachedZenoss platform expects the use of each memcached server to conform to certain performance thresholds. Specifically, the utilization level and eviction rate of each memcached server are checked. If any of these thresholds are violated, then a WARNING or ERROR severity tuning event is generated, depending on the configured thresholds. The thresholds checked are controlled by the following configuration options on the zentune daemon:
  • memcache-size-high-warn
  • memcache-size-high-bad
  • memcache-size-low-warn
  • memcache-size-low-bad
  • memcache-size-evictions-warn
  • memcache-size-evictions-bad

For detailed information about each of these options, run this command on any Zenoss platform server:

zentune run --help

Memcached is a third-party object caching system used by Zenoss platform to improve performance for daemons that connect to Zope and ZODB. It is not required for Zenoss platform to function, but it is highly recommended.

More information on memcached can be found here:

http://memcached.org/

MySqlTuner scriptResourcesZenTune expects the mysqltuner.pl utility to be installed and available for its use to enable more detailed tuning advice. The mysqltuner.pl utility is a third-party tuning script that provides advanced statistics on MySQL. More information on mysqltuner.pl can be found here:

http://mysqltuner.pl/help

MySQL is a third-party, open-source relational database. Zenoss platform uses MySQL as the backing data store for ZODB, as well as directly to store events and user sessions. More information on MySQL can be found here:

http://www.mysql.com/

iostat UtilityResourcesZenTune expects the iostat utility to be installed and available for its use to enable more detailed tuning advice. The iostat utility is a third-party program that provides statistics on the time the CPU spends waiting for I/O requests from various devices. More information on iostat can be found here:

http://en.wikipedia.org/wiki/Iostat

MemoryResourcesThis tuning item provides information about the total amount of RAM installed in the Zenoss platform master server.
ProcessesResourcesThis tuning item provides advice on the distribution of CPU-intensive Zenoss platform processes according to the number of cores available on the Zenoss platform server. If any of the thresholds are violated, a WARNING or ERROR severity tuning event will be generated, depending on the configured thresholds. The thresholds checked are controlled by the following configuration options on the zentune daemon:
  • resources-available-cores-warn
  • resources-available-cores-bad

For detailed information about each of these options, run this command on any Zenoss platform server:

zentune run --help

MySQL VersionMySQL DatabaseZenoss platform expects that at least a specific, minimum version of MySQL is installed and available for its use. Earlier versions may not support all the features that Zenoss platform requires, or may have hidden incompatibilities. If this minimum threshold is violated, then an ERROR severity tuning event is generated, depending on the configured threshold. The threshold checked is controlled by the following configuration option on the zentune daemon:
  • mysql-recommended-version

For detailed information about this option, run this command on any Zenoss platform server:

zentune run --help

MySQL is a third-party, open-source relational database. Zenoss platform uses MySQL as the backing data store for ZODB, as well as directly to store events and user sessions. More information on MySQL can be found here:

http://www.mysql.com/

InnoDB buffer pool sizeMySQL DatabaseIf the mysqltuner.pl script recommends increasing the amount of memory available to InnoDB, then an ERROR level tuning event is generated. More information on the innodb_buffer_pool_size configuration setting can be found here:

http://dev.mysql.com/doc/refman/5.5/en/innodb-parameters.html\#sysvar_innodb_buffer_pool_size

MySQL is a third-party, open-source relational database. Zenoss platform uses MySQL as the backing data store for ZODB, as well as directly to store events and user sessions. More information on MySQL can be found here:

http://www.mysql.com/

Table sizesMySQL DatabaseThis tuning item provides information about the total size and number of tables in each MySQL instance configured for Zenoss platform. MySQL is a third-party open-source relational database. Zenoss uses MySQL as the backing data store for ZODB, as well as directly to store events and user sessions. More information on MySQL can be found here: http://www.mysql.com/

MySQL is a third-party, open-source relational database. Zenoss platform uses MySQL as the backing data store for ZODB, as well as directly to store events and user sessions. More information on MySQL can be found here:

http://www.mysql.com/

Table fragmentationMySQL DatabaseIf the mysqltuner.pl script recommends de-fragmenting the tables in a MySQL instance, then an ERROR level tuning event is generated.

MySQL is a third-party, open-source relational database. Zenoss platform uses MySQL as the backing data store for ZODB, as well as directly to store events and user sessions. More information on MySQL can be found here:

http://www.mysql.com/

Thread cacheMySQL DatabaseIf the mysqltuner.pl script recommends increasing the number of threads cached for reuse by MySQL, then an ERROR level tuning event will be generated. More information on the thread_cache_size configuration setting can be found here: http://dev.mysql.com/doc/refman/5.5/en/server-system-variables.html#sysvar_thread_cache_size

MySQL is a third-party, open-source relational database. Zenoss platform uses MySQL as the backing data store for ZODB, as well as directly to store events and user sessions. More information on MySQL can be found here:

http://www.mysql.com/

Versionrabbitmq
HubHubs
zenhub workersHubs
Check intervalZope
Object cacheZope
RelStorage cacheZope
Pool sizeZope
Cache serversZope
Maximum number of session objectsZope
Debug modeZope
Application server processesZope
Application server threadsZope
Request latencyZope

Daemons

TypeName
Performance Collectorzentune

Installation

Normal Installation (packaged egg)

  1. Download the appropriate egg file for the version of Zenoss you are running.
  2. Ensure you are logged in as the zenoss user:
    $ sudo su - zenoss
  3. Install the ZenPack:
    $ zenpack --install ZenPacks.zenoss.AutoTune-*.egg
  4. Restart these services:
    $ zenoss restart

Commercial

This ZenPack is developed and supported by Zenoss Inc. Commercial ZenPacks are available to Zenoss commercial customers only. Contact Zenoss to request more information regarding this or any other ZenPacks. Click here to view all available Zenoss Commercial ZenPacks.