ZenPacks

Hadoop ZenPack

Monitoring for the Apache Hadoop framework.

Support

This is an Open Source ZenPack developed by Zenoss, Inc. Enterprise support for this ZenPack is available to commercial customers with an active subscription.

Releases

Version 1.0.0- Download
Released on 2014/09/05
Requires PythonCollector ZenPack
Compatible with Zenoss Core 4.2.x, Zenoss Resource Manager 4.1.x, Zenoss Resource Manager 4.2.x, Zenoss Resource Manager 5.0.x

Background

The ZenPacks.zenoss.Hadoop ZenPack monitors Hadoop clusters via JMX web interface (JMXJsonServlet) and Python twisted.web.client asynchronous framework.

Features

The features added by this ZenPack can be summarized as follows. They are each detailed further below.

  • Discovery and periodic remodeling of relevant components.
  • Performance monitoring.
  • Event monitoring.
  • Optional service impact with addition of Zenoss Service Dynamics product.
  • Optional HBase auto discovery.

Discovery

The following components will be automatically discovered through zProperties you provide:

Hadoop Data Node
Attributes: Name, HBase Device, Last Contacted, Health State
Hadoop Secondary Name Node
Attributes: Name, Health State
Hadoop Job Tracker
Attributes: Name, Health State
Hadoop Task Tracker
Attributes: Name, Health State
Hadoop Resource Manager
Attributes: Name, Health State
Hadoop Node Manager
Attributes: Name, Health State
Hadoop Job History
Attributes: Name, Health State

Note: Yarn components (Resource Manager, Node Manager, Job History) will be modeled for 2.x version of Hadoop and MapReduce components (Job Tracker, Task Tracker) for previous versions.

Performance Monitoring

The following metrics will be collected and graphed every 5 minutes by default:

Device (Hadoop Name Node)
Heap/Non Heap Memory: Heap Memory Capacity Bytes, Heap Memory Used Bytes, Non Heap Memory Capacity Bytes, Non Heap Memory Used Bytes
Nodes: Dead Nodes Count, Live Nodes Count
Performance metrics: Threads, Total Files, HDFS: Configured Capacity, DFS Remaining, DFS Used
Hadoop Data Node
Blocks: Blocks Written, Blocks Read, Blocks Removed
Heap/Non Heap Memory: Heap Memory Capacity Bytes, Heap Memory Used Bytes, Non Heap Memory Capacity Bytes, Non Heap Memory Used Bytes
Hadoop Secondary Name Node
Heap/Non Heap Memory: Heap Memory Capacity Bytes, Heap Memory Used Bytes, Non Heap Memory Capacity Bytes, Non Heap Memory Used Bytes
Hadoop Task Tracker
Tasks: Tasks Completed, Tasks Failed Timeout, Tasks Failed Ping
Performance metrics: Reduces Running, Maps Running
Hadoop Job Tracker
Heap/Non Heap Memory: Heap Memory Capacity Bytes, Heap Memory Used Bytes, Non Heap Memory Capacity Bytes, Non Heap Memory Used Bytes
Jobs: Jobs Running, Jobs Failed, Jobs Completed
Performance metrics: Reduces Running, Maps Running
Hadoop Resource Manager
Heap/Non Heap Memory: Heap Memory Capacity Bytes, Heap Memory Used Bytes, Non Heap Memory Capacity Bytes, Non Heap Memory Used Bytes
Nodes: Active Nodes, Decommissioned Nodes, Lost Nodes, Unhealthy Nodes, Rebooted Nodes
Apps: Apps Submitted, Apps Pending, Apps Running, Apps Completed
Hadoop Node Manager
Heap/Non Heap Memory: Heap Memory Capacity Bytes, Heap Memory Used Bytes, Non Heap Memory Capacity Bytes, Non Heap Memory Used Bytes
Containers: Containers Running, Containers Launched, Containers Completed, Containers Failed, Containers Initing, Containers Killed
Hadoop Job History
Heap/Non Heap Memory: Heap Memory Capacity Bytes, Heap Memory Used Bytes, Non Heap Memory Capacity Bytes, Non Heap Memory Used Bytes

Event monitoring

The following events will be triggered:

  • Error: Any of the components is down or not available
  • Info: HBase was discovered on one of the Data Nodes or Name Node

Service Impact

When combined with the Zenoss Service Dynamics product, this ZenPack adds built-in service impact capability for Hadoop. The following service impact relationships are automatically added. These will be included in any services containing one or more of the explicitly mentioned entities.

Service Impact Relationships
  • Secondary Name Node, Job Tracker, Task Tracker, Resource Manager, Node Manager or Job History failure affects associated Data Node.
  • Secondary Name Node, Job Tracker, Task Tracker, Resource Manager, Node Manager and Job History are affected by hosting Device failure.
External Impact Relationships
  • Data Node failure affects associated HBase Region Servers.
  • Secondary Name Node, Job Tracker, Task Tracker, Resource Manager, Node Manager and Job History are affected by ZooKeeper failure.

HBase Autodiscover

With a fully deployed Hadoop environment, the HBase Master server can move around to different Hadoop Nodes. This ZenPack provides optional discovery of HBase Master Device within Hadoop Data Nodes, which may be configured with the help of the following configuration properties:

  • zHBaseAutodiscover
  • zHBaseDeviceClass

If you checkmark zHBaseAutodiscover property, every 5 minutes by default each Hadoop Data Node will be probed for hosting HBase Master. In case HBase is found, the HBase device will be created in the Device Class set in zHBaseDeviceClass (usually /Server/Linux or /Server/Microsoft/Windows) with HBase modeler plugins bound. If the HBase device was already discovered, but HBase Master moves to a different Node, the device's IP address will be changed to a new one.

Usage

Use the following steps to start monitoring Hadoop cluster:

  1. Select Infrastructure from the navigation bar.
  2. Click the device name in the device list. The device overview page appears.
  3. Select Configuration Properties from the left panel.
  4. Set zHadoopUsername, zHadoopPassword and select https for zHadoopScheme if you have Basic access authentication configured on your Hadoop UI instances (otherwise leave zHadoopUsername and zHadoopPassword blank).
  5. Set zHadoopNameNodePort if the Name Node UI instance port differs from the default one.
  6. Optionally checkmark zHBaseAutodiscover to setup discovering HBase on Hadoop data nodes.
  7. Optionally set the device class for discovered HBase device in zHBaseDeviceClass if it differs from the default one.
  8. Navigate to the Modeler plugins page of the device containing your Hadoop server, add the HadoopDataNode and HadoopServiceNode modeler plugins.
  9. Select Model device from the gear menu.


Alternatively you can use zenbatchload to add Hadoop monitoring servers from the command line. To do this, you must create a text file with hostname, username and password of all the servers you want to add. The devices should be added under the target machine device class (e.g. '/Devices/Server/Linux' in the example below).

/Devices/Server/Linux <HOST NAME or IP>
zCollectorPlugins=['HadoopDataNode', 'HadoopServiceNode'], zHadoopNameNodePort='50070' zHadoopPassword='', zHadoopUsername='', zHadoopScheme='<http(s)>'

You can then load the Hadoop monitoring servers into Zenoss Core or Resource Manager as devices with the following command.

$ zenbatchload <filename>

Prerequisites

This ZenPack uses the NameNode WebUI port for Hadoop modeling and performance monitoring, therefore please make sure that the UI instance is running. For more information refer to HDFS Ports.

Troubleshooting

Please refer to the Zenoss Service Dynamics documentation if you run into any of the following problems:

  • ZenPack will not install
  • Adding a device fails
  • Don't understand how to add a device
  • Don't understand how to model a device

If you cannot find the answer in the documentation, then Resource Manager (Service Dynamics) users should contact Zenoss Customer Support. Core users can use the #zenoss IRC channel or the community.zenoss.org forums.

Installed Items

Installing this ZenPack will add the following items to your Zenoss system.

Modeler Plugins
  • HadoopDataNode
  • HadoopServiceNode
Monitoring Templates
  • Hadoop (in /Device)
  • HadoopDataNode (in /Device)
  • HadoopSecondaryNameNode (in /Device)
  • HadoopJobTracker (in /Device)
  • HadoopTaskTracker (in /Device)
  • HadoopResourceManager (in /Device)
  • HadoopNodeManager (in /Device)
  • HadoopJobHistory (in /Device)
  • HBaseDiscoverMonitor (in /Device)
Component Types
  • HadoopDataNode (on related device)
  • HadoopSecondaryNameNode (on related device)
  • HadoopJobTracker (on related device)
  • HadoopTaskTracker (on related device)
  • HadoopResourceManager (on related device)
  • HadoopNodeManager (on related device)
  • HadoopJobHistory (on related device)

Daemons

TypeName
Performance Collectorzenpython

Changes

1.0.0
  • Initial release

Installation

Normal Installation (packaged egg)

  1. Download the appropriate egg file for the version of Zenoss you are running.
  2. Ensure you are logged in as the zenoss user:
    $ sudo su - zenoss
  3. Install the ZenPack:
    $ zenpack --install ZenPacks.zenoss.Hadoop-*.egg
  4. Restart these services:
    $ zenoss restart

Developer Mode Installation

In order to do a development mode installation you will want to clone the existing git repository, and then use the --link flag with the zenpack command:

  1. Ensure you are logged in as the zenoss user:
    $ sudo su - zenoss
  2. Start by cloning the upstream repository:
    $ git clone https://github.com/zenoss/ZenPacks.zenoss.Hadoop.git
  3. Next, perform the installation:
    $ zenpack --link --install ZenPacks.zenoss.Hadoop
  4. Finally, restart these serivices:
    $ zenoss restart

Open Source

This ZenPack is developed and supported by Zenoss Inc. Contact Zenoss to request more information regarding this or any other ZenPacks. Click here to view all available Zenoss Open Source ZenPacks.

randomness