ZenPacks

Ceph Zenpack

Ceph is a distributed object store and file system.

Releases

Download

Version 2.0.1
Released on 2016/11/03
Requires PythonCollector ZenPack
Compatible with Zenoss Resource Manager 5.0.x, Zenoss Resource Manager 5.1.x
  • Improved SSH connectivity
  • Reduced log noise in modeler
Version 2.0.0
Released on 2016/06/20
Requires PythonCollector ZenPack
Compatible with Zenoss Resource Manager 5.0.x, Zenoss Resource Manager 5.1.x
  • Added OpenStack integration for volumes and snapshots
  • Added SSH type API connectivity to clusters
  • Updated DynamicView and Impact relationships
Version-1.0.0
Released on 2016/01/22
Requires PythonCollector ZenPack
Compatible with Zenoss Resource Manager 5.0.x

Background

This ZenPack monitors the Ceph storage clustering system.

We support two APIs for connecting to Ceph: Calamari and SSH.

  • Calamari is beneficial when you already have Calamari montoring setup and don't want to talk directly to your cluster. Calamari caches data and will put less load on your server while maintaining existing security.
  • The SSH API is appropriate when Calamari monitoring is not desired. This choice is easier to setup if no existing Calamari system exists. The SSH API make direct contact to each host and pulls data from each and so is more direct and reactive. This API choice requires you setup SSH keys or provide SSH password authentication.

Prerequisites

  • Ceph (Hammer or newer)
  • Calamari (Version 1.3.1 or newer)
  • Calamari credentials
  • Administrative credentials for your Ceph cluster

Features

Ceph features include:

  • Overall Cluster Health Monitoring
  • Health Monitoring for Monitors, OSDs, MDSs and Hosts
  • Graphs for key Ceph settings and performance metrics
  • Service Impact and root cause analysis. (Requires Zenoss Service Dynamics)
  • Event Management

Ceph Structure and Discovery

Objects are automatically discovered via Calamari API. The ZenPack class structure can be visualized in the following Diagram:

Ceph ZP Class Structure

Ceph ZP Class Structure

Hosts (CephHost)

  • Description: Hosts that various Ceph services run on
  • Attributes: fqdn, ceph_hostname, ceph_version, managed, frontend_addr, frontend_iface, backend_addr, backend_iface: Cluster facing interface, services: Services managed on host
  • Relationships: cephEndpoint, cephServices
  • Datasources:
    • Events:
      • Late Reporting
      • Rebooted
      • Regained Contact
    • Datapoints:
      • cpu.total.idle
      • cpu.total.system
      • cpu.total.iowait
      • diskspace.root.byte_used
      • diskspace.root.byte_free
      • diskspace.root.byte_avail
      • memory.Active
      • memory.MemFree
      • memory.MemTotal
  • Graphs:
    • Idle Cpu
    • CPU IO Wait
    • Disk Space
    • Memory Utilization

Ceph Clusters (CephCluster)

  • Description: The cluster component. Ceph can host multiple clusters.
  • Attributes: Status: Health Status, FSID: The cluster ID, Cluster Name
  • Relationships: cephEndpoint, cephClusterComponents
  • Datasources:
    • Events:
      • Health Warn
      • Health OK
      • Regained Contact
      • Late Reporting
    • Datapoints:
      • Cluster:
        • total_bytes
        • total_avail_bytes
        • total_used_bytes
      • PG Data:
        • active, backfill_toofull, backfilling, clean, creating, degraded, down, inactive, incomplete, inconsistent, peering, recovering, recovery_wait, remapped, repair, replay, scrubbing, splitting, stale, undersized, wait_backfill
  • Graphs:
    • Disk Usage
    • OSDs In and Up
    • PG States (Only in Calamari)
    • OSD Disk Space (Only in Calamari)
    • Utilization (Only in SSH)

Ceph Monitors (CephMon)

  • Description: Creates and monitors all Ceph services and health
  • Attributes: rank in_quorum, public_ipaddr, public_port, running, status
  • Relationships: cephCluster, cephHost
  • Datasources:
    • Events:
      • Joined Quorum
      • Left Quorum
    • Datapoints:
      • cluster.num_mon
      • cluster.num_mon_quorum
      • mon.election_call
      • mon.election_win
      • mon.election_lose
      • mon.num_elections
  • Graphs:
    • Monitors in Quorum
    • Elections

Ceph Object Storage Daemon (CephOSD)

  • Description: Service that provides raw storage to Ceph
  • Attributes: status, ceph_uuid, up, in, reweight, public_ipaddr, public_port, cluster_ipaddr, cluster_port
  • Relationships: cephCluster, cephHost, cephPools
  • Datasources:
    • Events: OSD Down, OSD Up, OSD Reweight, Running

Pools (CephPool)

  • Description: The basic Ceph logical areas that data is organized into
  • Attributes: status
  • Relationships: cephCluster, cephOSDs, cephMDSs, cephRBDs, cephSnaps, cephFS
  • Datasources:
    • Datapoints: num_objects, num_objects_degraded, num_objects_dirty, num_objects_misplaced, num_objects_missing_on_primary, num_objects_recovered, num_objects_clones, num_read_kb, num_write_kb, num_scrub_errors
  • Graphs:
    • Pool Storage
    • Pool IO
    • Dirty Objects
    • Misplaced Objects
    • Objects Recovered

Ceph Filesystem (CephFS)

  • Description: Global filesystem
  • Attributes: mds_pool, data_pools
  • Relationships:
    • cephCluster
    • cephMDSs
    • cephPools
  • Datasources: None

Ceph Metadata Servers (CephMDS)

  • Description: Service provides all meta-data to CephFS files and objects. It is currently modeled as a single service which requires at least max_mds active servers and one standby server.
  • Attributes: created, enabled, epoch, flags, last_failure, last_failure_osd_epoch, max_file_size, max_mds, modified, root, session_autoclose, session_timeout, tableserver
  • Relationships: cephCluster, cephPools, cephFS
  • Datasources:
    • Events:
      • Active Daemons
      • Standby Daemons
    • Datapoints:
      • total_count, up_count, in_count, failed_count, stopped_count, max_mds, active_count, standby_count, other_count, datapool_count, epoch, first_committed, last_committed
  • Graphs:
    • MDS Daemons States: Note: Totals don't sum to the subgroups
    • MDS Daemon Secondary States

Ceph RADOS Block Device (CephRBD)

  • Description: Block devices exported to systems as disk images
  • Attributes: parent: (If COW image), size
  • Relationships: cephCluster, cephPool, cephSnaps
  • Datasources: None

Ceph Snapshots (CephSnap)

  • Description: Snapshot block devices for backups and copy-on-write (COW) images
  • Attributes: size, protected
  • Relationships: cephCluster, cephPool, cephRBD
  • Datasources: None

Usage

Recall that we have two types of API connectivity to Ceph: Calamari and SSH.

ZenPack setup with Calamari/Graphite

The Calamari API is outlined here.

Calamari/Graphite setup is covered here. You must have access to both the Calamari and Graphite APIs which are bundled into the Calamari packages. Calamari has other dependencies that must be installed on the Ceph cluster.

Installation consists of the following steps:

  • If possible, ensure Linux hosts are not configure: See Host Discovery below
  • Ensure all prerequisite ZenPacks are installed
  • Install the Ceph ZenPack on Zenoss
  • Restart all Zenoss Services
  • Install the target device in Zenoss:
    • Go to Infrastructure tab
    • Add Ceph Endpoint, set properties as follows:
      • Device Name: Use a unique name for the endpoint
      • Calamari Host IP
      • Calamari User Name: See zCephCalamariUsername
      • Calamari Password: See zCephCalamariPassword
      • Collector: Choose the appropriate Collector

Calamari/Graphite Batch Configuration

You can also add your devices in batch for convenience and automation.

  • Attach to the Zope container:

    serviced service attach zope
    
  • Create a text file (filename: /tmp/ceph.txt) as follows:

    /Devices/Storage/Ceph
    ceph zCephCalamariUrl='http://calamari.fqdn/api/v2', \
         zCephCalamariUsername='username', \
         zCephCalamariPassword='password', \
         zCephGraphiteUrl='http://calamari.fqdn/render'
    
  • Now run the zenbatchload command:

    zenbatchload  /tmp/ceph.txt
    
  • The Ceph device should be modeled automatically

ZenPack setup with SSH

For SSH access you must enable an SSH account on each Ceph cluster server. Make sure all your Ceph cluster servers have the Zenoss ssh key install as per the SSH section of Prepare Remote Device Document. We recommend setting up a dedicated user with restricted access.

Ssh authentication can take two forms:

  • An SSH user/password combination
  • An SSH public/private key pair

SSH setup on your Ceph servers is beyond the scope of this document. Please reference this document for more information.

We do recommend that you setup sudo your Ceph servers so that the following commands can be used by your ssh user:

  • zCephRBDCommand: sudo /usr/bin/rbd --format json
  • zCephCommand: sudo /usr/bin/ceph -f json-pretty

Installation consists of the following steps:

  • If possible, ensure Linux hosts are not configure: See Host Discovery below
  • Install the Ceph ZenPack on Zenoss
  • Restart all Zenoss Services
  • Set the zCommandUsername and zCommandPassword if doing SSH password authentication
  • Set zCephHosts as the list of your Ceph Hosts. Set this on the device after installation, one FQDN or IP per line.
    • Go to Infrastructure tab
    • Add Single Device: set properties as follows:
      • Hostname or IP address: Use a unique name for the endpoint: Don't use an IP or DNS name!
      • Device Class: /Storage/Ceph/SSH
      • Collector: Choose the appropriate Collector

SSH Batch Configuration

Installation is easiest and simplest with zenbatchload as follows:

  • Attach to the Zope container:

    serviced service attach zope

  • Create a text file (filename: /tmp/ceph.txt) as follows:

/Devices/Storage/Ceph/SSH
ceph.ssh  zCommandUsername='username', \
          zCommandPassword='password', \
          zCephHosts=['ceph1.url','ceph2.url','ceph3.url', ...]
  • Note: if you are using SSH key-pair authentication, you may not need zCommandUsername and zCommandPassword as above.

  • Ensure to include ALL your ceph servers in the zCephHosts list.

  • Run the zenbatchload command:

    zenbatchload /tmp/ceph.txt

  • The Ceph device should be modeled automatically

Host Discovery

If you have set zCephDiscoverHosts to true, then the first time you visit the Hosts components, each host listed there will be created as a device of type zCephHostDeviceClass.

These hosts will not be modeled until you provide the appropriate zProperties in each host so that the modeler can have proper access.

If your hosts already exist, they should have the same hostname that Ceph believes them to be, either from Ceph's DNS servers for from /etc/hosts files. To avoid having multiple versions of the hosts, you should first remove the hosts from Zenoss prior to installing your first Ceph cluster.

Linux Host Setup Tips

Warning

We recommend against manually configuring Linux host devices!

Linux Host device names should use the same FQDN hostname that Ceph ZenPack is expecting whenever possible. Choosing different device names will complicate this process. Therefore it is highly recommended that you do not manually pre-configure those Linux devices, but rather allow the Ceph ZenPack to automatically discover and configure those devices at model time.

If this is not possible, you may have to re-name and re-configure those hosts and place them into the desired device classes manually.

Installed Items

Installing this ZenPack will add the following items to your Zenoss system:

Configuration and zProperties

The zProperties and example format settings for the Ceph are as follows:

Under the Calamari/Graphite API:

Under the SSH API:

  • zCommandUsername: The SSH username for your servers
  • zCommandPassword: The SSH password for your servers
  • zCephHosts: All Ceph Monitors and OSD server IP or FQDN, one host per line

For all API:

  • zCephDiscoverHosts: Set true to create Linux devices. These devices will only monitor with correct setting of their credentials.
  • zCephHostDeviceClass: Hosts with zCephDiscoverHosts set true have device class of either /Server/Linux or /Server/SSH/Linux

Modeler Plugin

  • zenoss.Ceph

Zenoss Analytics

This ZenPack provides additional support for Zenoss Analytics. Perform the following steps to install extra reporting resources into Zenoss Analytics after installing the ZenPack.

  1. Copy analytics-bundle.zip from $ZENHOME/ZenPacks/ZenPacks.zenoss.Ceph*/ZenPacks/zenoss/Ceph/analytics/ on your Zenoss server.
  2. Navigate to Zenoss Analytics in your browser.
  3. Login as superuser.
  4. Remove any existing Ceph_ZenPack folder.
  1. Choose Repository from the View menu at the top of the page.
  2. Expand Public in the list of folders.
  3. Right-click on Ceph_ZenPack folder and choose Delete.
  4. Confirm deletion by clicking OK.
  1. Add the new Ceph_ZenPack folder.
  1. Choose Server Settings from the ''Manage' menu at the top of the page.
  2. Choose Import in the left page.
  3. Remove checks from all check boxes.
  4. Click Choose File to import a data file.
  5. Choose the analytics-bundle.zip file copied from your Zenoss server.
  6. Click Import.

You can now navigate back to the Ceph_ZenPack folder in the repository to see the following resources added by the bundle.

Domains

  • Ceph Domain

Domains can be used to create ad hoc views using the following steps.

  1. Choose Ad Hoc View from the Create menu.
  2. Click Domains at the top of the data chooser dialog.
  3. Expand Public then Ceph ZenPack.
  4. Choose the Ceph Domain domain

Service Impact and Root Cause Analysis

When combined with the Zenoss Service Dynamics product, this ZenPack adds built-in service impact and root cause analysis capabilities for Ceph. The service impact relationships shown in the diagram (right) and described below are automatically added and maintained. These will be included in any services that contain one or more of the explicitly mentioned components.

The following Ceph objects types would typically be added to Impact services.

  • CephFS
  • RBD
  • Pool
Impact Relationship Diagram

Impact Relationship Diagram

Impact Relationships between Ceph Components

  • Ceph Endpoints impact all managed Ceph Hosts.
  • Ceph Hosts impact the Mon, and OSD processes they're running.
  • Ceph Mons impact their associated cluster.
  • Ceph Clusters impact all associated Ceph Pools, and MDS if CephFS is used.
  • Ceph OSDs impact their associated pools.
  • Ceph Pools impact all RBDs, Snapshots, and CephFSs that use them.
  • Ceph RBDs impact any associated snapshots.
  • Ceph MDS impacts all associated CephFSs.

OpenStack Integration

The Ceph ZenPack has support for OpenStack integration of RBD images to OpenStack volumes and snapshots.

OpenStack Integration requires:

  • Your OpenStack cluster is fully integrated with your Ceph cluster
  • The Ceph ZenPack version 2.0.0+ is installed and modeled
  • The OpenStackInfrastructure ZenPack version 2.2.0+ is installed and modeled
  • The OpenStack (/Server/SSH/Linux/NovaHost) block devices are fully modeled

Once integration is achieved, you should expect to see corresponding volume links in the respective component details relating Ceph RBDs to OpenStack volumes and snapshots.

Changes

2.0.1

  • Upgrade txsshclient to fix critical change in twisted.conch (ZEN-25737)
  • Reduce INFO logs in modeler to reduce noise (ZEN-25738)

2.0.0

  • OpenStack integration for volumes and snapshots
  • Added SSH type API connectivity to clusters
  • Updated DynamicView and Impact relationships
  • Add eventsClassKey mappings
  • Improved error handling of API communications
  • Improved Data Collection
  • New Graphs: The following new graphs were added:
    • Clusters: Utilization (SSH only)
    • Pools: Pool Storage
    • Pools: Pool IO
  • Most Calamari Datapoints Renamed: All prior data besides Cluster PG States will be lost
  • Moved Graphs: OSD Graphs were moved to Cluster:
    • OSDs: OSD's In and Up
    • OSDs: Disk Space

1.0.0

  • Initial Release
Commercial

This ZenPack is developed and supported by Zenoss Inc. Commercial ZenPacks are available to Zenoss commercial customers only. Contact Zenoss to request more information regarding this or any other ZenPacks. Click here to view all available Zenoss Commercial ZenPacks.

randomness