Notes

The following section provide additional information or procedures regarding Analytics issues.

ANA-223: Wide performance ETL batches have ability to blow out the Heap in CentralQuery service.

If you create too wide a perf ETL batch, you may blow out the Heap and receive error messages similar to the following in zenoss_analytics.out:
2017-03-07 13:59:07,748 ERROR perfextractjob-8 com.zenoss.reporting.extract.service.PerfExtractionTask - Failed to execute request: Unexpected end of ZLIB input stream task="fs_inodes__pct-4052", batch=8746, exception=EOFException
2017-03-07 13:59:07,748 ERROR perfextractjob-6 com.zenoss.reporting.extract.service.PerfExtractionTask - Failed to execute request: Unexpected end of ZLIB input stream task="fs_inodes_used-4050", batch=8746, exception=EOFException
2017-03-07 13:59:07,748 WARN  quartzScheduler_Worker-6 com.zenoss.reporting.schedule.ExtractPerfJob - [ExecutionException] java.io.EOFException: Unexpected end of ZLIB input stream
2017-03-07 13:59:07,748 WARN  quartzScheduler_Worker-6 com.zenoss.reporting.schedule.ExtractPerfJob - [ExecutionException] java.io.EOFException: Unexpected end of ZLIB input stream
2017-03-07 13:59:07,748 ERROR perfextractjob-7 com.zenoss.reporting.extract.service.PerfExtractionTask - Failed to execute request: Bad Gateway   task="fs_used__pct-4051", batch=8746, exception=HttpResponseException
2017-03-07 13:59:07,749 ERROR perfextractjob-4 com.zenoss.reporting.extract.service.PerfExtractionTask - Failed to execute request: Bad Gateway   task="fs_used__pct-4048", batch=8746, exception=HttpResponseException
2017-03-07 13:59:07,748 ERROR perfextractjob-1 com.zenoss.reporting.extract.service.PerfExtractionTask - Failed to execute request: Bad Gateway   task="fs_used__pct-4045", batch=8746, exception=HttpResponseException
2017-03-07 13:59:07,750 ERROR quartzScheduler_Worker-6 com.zenoss.reporting.schedule.ExtractPerfJob - [HttpResponseException] Bad Gateway
2017-03-07 13:59:07,750 ERROR quartzScheduler_Worker-6 com.zenoss.reporting.schedule.ExtractPerfJob - [HttpResponseException] Bad Gateway

To remedy this situation, you need to adjust the QueryService RAM commitment from its current value to 4G, and retry the batch.

To increase the QueryService RAM commitment, perform the following:
  1. In Control Center, navigate to the CentralQuery service of your deployed Zenoss application Zenoss.resmgr > Zenoss > Metrics > CentralQuery.
  2. Click the Edit Service link near the top of the page.
  3. In the dialog box, locate the field labeled RAM Requested and change the value in that field to 4G. If the value is already 4G or has a larger value, then make no changes.
  4. Click Save Changes.

ZPS-757: RPN syntax error in several aliases

In the LinuxMonitor ZenPack, there are syntax errors that need to be manually fixed in order for Analytics to perform data extraction. After you have successfully installed or upgraded Analytics to version 5.0.4, perform the following commands on the Resource Manager master as root or a user with superuser permissions:
serviced service attach zope/0

su - zenoss

cd /opt/zenoss/ZenPacks/ZenPacks.zenoss.ZenETL-5.0.4-py2.7.egg/ZenPacks/zenoss/ZenETL/bin

./dumpAliases.py --aliases=with

egrep "(mem_buffers__pct|mem_cached__pct|mem_free__pct|mem_swap_free__pct|lvm_pv_free__pct|lvm_vg_free__pct)" aliases.txt > aliases_changes.txt

sed -i 's/,\*,100/,100,\*/' aliases_changes.txt

./manageAliases.py --action=add --inputFile=aliases_changes.txt --commit

exit
exit

# Remove the disk cache which does not get cleared on restart

serviced service stop zenperfetl

rm -rf /opt/serviced/var/volumes/<tenantid>/etl-analytics/zenperfetl/perf-config-cache

serviced service start zenperfetl

# Wait until the new PERFORMANCE extractor has registered with Analytics

select * from meta_extractor where extractor_name = 'PERFORMANCE';

# The above command must return one row before you proceed

# Get the key of the new extractor

select extractor_key into @extractor_key from meta_extractor where extractor_name = 'PERFORMANCE';

# Update all existing PERF batches to have the new key

update meta_batch set extractor_key = @extractor_key where extractor_key != @extractor_key and extractor_key in (select extractor_key from meta_extractor where extractor_type = 'PERFORMANCE');

# Remove all the old performance extractors

delete from meta_extractor where extractor_name like '%/_%';

ANA-44: Provide visibility into when Analytics actually begins to extract perf data

Prior to version 5.0.4, the analysis and prep of batches was merged into the same phase. Now the process is more granular to identify the key phases, so you will see progression through the following states: UNSTARTED, CONFIGURING, READY, EXTRACTING, STAGING, FAILED, COMPLETED. A CANCELLED state indicates that the collector went offline for a long period of time.