Use Process Sets to Monitor More Efficiently and Reduce Monitoring Costs

If you are like most IT Operations folks, you always have too much to do with too little time. Change management moves at the speed of light. New applications get deployed by someone on the business apps team, but this information never makes it over to the IT Operations team. This means that the application is partially or totally off the grid and unmonitored. The first time you even realize it exists is when there is a performance or availability problem and your end users start calling in and complaining. It seems like there just aren’t enough hours in the day to track all of the new applications coming up, let alone effectively monitor them.

You know that in order to fully understand the load on your system resources, you need to know about the performance of each critical process associated with an application or service running on the server. However, knowing what processes are important for different applications and configuring the monitoring processes individually is very labor-intensive. Process monitoring consumes even more of your already limited cycles ─ unless you have a dynamic, easy-to-maintain, rules-based approach to process monitoring.

Automate Process Monitoring with Zenoss

New process monitoring capabilities available in Zenoss Core 4.2.5, the open-source Zenoss product, as well as in Zenoss Service Dynamics Resource Manager 4.2.5, the enterprise version of Zenoss, help you monitor and manage processes that are running on discovered devices. Zenoss uses SNMP and SSH to monitor the availability of processes running on Windows and Linux servers. Zenoss can also use WinRM when the Microsoft Windows ZenPack is installed and the target operating system is Windows 2003 or later.

When you enable process monitoring in Zenoss, Zenoss OSprocess class definition filters use both inclusion and exclusion regular expression comparisons to match a list of OS process status command lines. You can also apply a regular expression-based substitution specification to group the matching list of processes into one or more “process sets” to track each set separately using an optional mnemonic, or easy-to-remember, name for each set.

Process monitoring enhancements include the following new capabilities:

• Modeling dynamic lists of “process sets” by grouping processes that match class filters
• Automatic matching of processes to process sets
• Dynamic creation of new process sets
• Alerting on process restarts
• Alerting on changed process counts within a set
• Algorithmic class sequence processing, set assignment, and count monitoring
• Ensured monitoring of transient processes
• Automatic and consistent enforcement of process monitoring best practices
• Iterative class regex testing and sequence development testing

 

Modeling Dynamics Lists of “Process Sets” by Grouping Processes That Match Class Filters

During Zenoss modeling intervals on each monitored operating system, process management defines a list of “process sets” related to every process class definition.

Zenoss provides advanced class definition configurations allowing you to both filter a list of interesting processes and group them into sets, as well as define one or more variables (based on substrings in the process command line) you want to use for organizing and grouping your processes. Using the process set-based approach and class definitions in Zenoss, you can now filter and group similar processes into a dynamic number of process sets, which simplifies your process monitoring process and makes it easier to track and understand the process sets, as well as the individual processes, that you are monitoring.

 

Automatic Matching of Processes to Process Sets

When Zenoss detects a process that matches the criteria you specified, Zenoss automatically adds the process to the appropriate process set. Zenoss can notify you so that you know a new process has been added to the group. At the same time, Zenoss also immediately begins monitoring the process. You are no longer dependent on change management notification and configuration steps that may or may not occur before automatically monitoring important processes matching key application types.

With Zenoss, you simply need to create and maintain only a single process set class definition. When Zenoss detects a new process that matches the process set class definition, Zenoss automatically adds the process into the appropriate process set. The number and content of process sets can be dynamic, which eliminates the need to repeatedly recreate rules for monitoring individual processes.

 

Dynamic Creation of New Process Sets

You can use Zenoss to not only automatically add new processes to the appropriate existing process set for monitoring, but during modeling Zenoss also creates new process sets as needed based on the class configuration and the processes running at that time. When reviewing details for a process set, you can identify the list of processes that resulted in creation of the set. If necessary, you can adjust the filtering and the set grouping and naming criteria, as well as minimum and maximum set count thresholds.

 

Alerting on Process Restarts

It is often important to know if key application processes are crashing and restarting. Simply monitoring the count alone of those processes during each monitoring cycle may not detect that an application is thrashing. In order to detect and alert on this situation, Zenoss can now accurately detect when processes within a process set have restarted since the last monitoring cycle and notify you of this situation.

 

Alerting on Changed Process Counts Within a Set

Zenoss gives you the ability to specify that you want to be notified if there are changes in the number of processes in a process set. Zenoss also identifies the number of restarted processes vs. net new (or net deleted) processes in the set. Notifications can be delivered on restart or net additional or net reduced number of processes. Zenoss can also generate events whenever counts fall below the minimum or maximum configured thresholds.

Having a precise handle on changes to processes within a set allows detection and isolation of problems due to an abnormal number of processes.

 

Algorithmic Class Sequence Processing, Set Assignment, and Count Monitoring

Each monitoring cycle, each process running on a target operating system is compared to see if it should be counted against at most one process class and one of its process sets according to the following sequence:

1. A list of processes is retrieved from the operating system being monitored.
2. The list of processes is compared against each process class definition one at a time, in class sequential order.
3. Once a process matches the inclusion, but not the exclusion, filters of a class, that process is:
   1. Removed from the list of processes to compare against subsequent class definitions
   2. Added to the appropriate matching “process set” for that matching class
4. After all processes remaining in the list have been compared to a class and organized into process sets, each process set is evaluated against the prior monitor cycle to identify:
   1. Count of restarted processes
   2. Total count of processes
5. Notifications for restarted processes and total counts increasing or decreasing are processed.
6. Events are generated if the total count is below the minimum or above the maximum threshold configured for the class.

 

Ensured Monitoring of Transient Processes

With Zenoss, you can ensure that even transient process sets are monitored appropriately. When you create a process class definition, you can specify that you want to lock prior modeled process sets. When you specify that you want to lock process sets in place, if Zenoss does not find any processes falling into an existing process set during device remodeling, which is typically done once or twice a day, Zenoss will not eliminate that process set. Subsequent monitoring cycles will continue to count, event, and notify as configured against the number of processes that are detected for the set.

 

Automatic and Consistent Enforcement of Process Monitoring Best Practices

With Zenoss process set monitoring, you finally have a way to automatically enforce process monitoring best practices. When Zenoss detects a new process matching the signature of an OSprocesses class configuration, it can automatically create new process sets or include processes as part of existing sets and apply the appropriate monitoring rules. You no longer have to configure process monitoring on a manual, process-by-process basis. If an application is deployed to a host without notifying IT Operations, the signature of the processes will be automatically detected, organized and monitored.

Example: Monitoring Oracle Database Instances Using Process Sets

To see how you can use the new dynamic process set capability in Zenoss in your environment, first assume that you want to monitor all of the processes related to an Oracle database instance. Next assume your company is in the process of consolidating multiple Oracle instances onto a single very large host to reduce operating and licensing costs. Finally, assume that after consolidation from other hosts is complete, the number of instances you want to monitor may change due to new applications or the creation of temporary databases created for testing.

Grouping multiple database processes performing the same function (for example, transaction log writing processes, file I/O caching processes, etc.) and tracking them as a set is very useful. You don’t want to group all of the functional processes for all Oracle instances into one set because you want to separately detect the health of each instance. Otherwise, if one database instance shuts down, it might appear that all of your instances are impacted, and not detect which one actually crashed. Also, if another Oracle instance is added to the same host in the future, you don’t want to create another process class definition each time.

In this scenario, you decide to group the processes related to all Oracle database instances (existing now or in the future) into process sets distinct from each other based on two variables that you can identify from the process command line - the instance name and process type.

You could create a class definition similar to the following image.

The inclusion regular expression matches any process with “ORA” followed by 1 or more numbers and an underscore. The exclusion eliminates any editing or greping processes that might be working on files matching the inclusion filter. The replace command line text isolates 4 key elements of the command line using regular expression group definitions each identified by parentheses ( … ) as variables to reference in the With: field. In this case, only the 4th and 2nd variable is used to create and name process sets as identified by the contents of the With field Ora\4_\2*.

Some of the resulting process sets generated are identified in the Display area. From the portion of the list of sets showing, you can see that there are at least two Oracle instances for a financial (FINC) and data warehouse (WHSE). You can see four separate types of process sets for the FINC instance and one for the WHSE that are displayed and database writer (DBW) is shown for both.

Now assume that an Oracle instance has 15 types of processes (with one or more processes for each type). In the past, you would monitor each of the 15 types of processes by creating a class definition for each process type.

Also now assume that you have a particular host running four Oracle instances, with each Oracle instance running each of the 15 types of processes. In this scenario, using the new dynamic process set capabilities in Zenoss, you only have to create and maintain a single class definition to separately monitor 60 different sets. In the past you would have had to either:

• Define 60 separate classes, one per instance and type
• Define 4 classes for each instance, but then be unable to isolate the types of processes affected
• Define 15 classes to monitor by type, but then be unable to detect which instance
• Define one class that groups all Oracle processes into one group, but then not be able to detect which instance or process type is having a problem
• Manually update class definitions each time a new instance is added or removed

Now, let’s extend this example event further. Instead of having one host running four Oracle instances, you have a total of 10 hosts in your environment, each running four Oracle instances for a total of 80 instances and 600 potential classes instead of the single class definition monitoring on each host!

 

Iterative Class Regex Testing and Sequence Development Testing

Working with regular expressions and precisely monitoring different processes can be a challenge. In the past, you would have to remote into a host and try out various regular expressions on the processes there. When you thought you had the configuration correct, you configured a class and deployed it. If certain process strings didn’t exist and you ended up monitoring with an incorrect expression configuration, you didn’t want to find that out after deploying the class to your production environment.

Zenoss has two utilities that make it significantly easier to define and test process class configuration, as well as test the class definition to make sure it is defined in the proper position in the class sequence (for example, another earlier class didn’t capture processes you intended to match with the new class).

The class definition edit and test interface also allows you to create a list of processes by retrieving and appending process lists from numerous devices, as well as manually crafting or editing process command lines to test obscure processes that you can’t retrieve from existing devices. These lists can be saved to files and reused in future sessions. The following image shows an example of the tester for the previous Oracle example.

Configuring Process Set Monitoring

Now that you understand the power of Zenoss process set monitoring capabilities, go test them out in you environment using our step-by-step instructions!

• If you use Zenoss Core, the free, open source version of Zenoss, see the Zenoss Core documentation.
• If you use Zenoss Service Dynamics Resource Manager, the enterprise version of Zenoss, see the Zenoss Resource Manager Administration documentation.

New to Zenoss? Learn More!

Want to learn more about how you can use Zenoss to more effectively manage and monitor your environment?

Watch a demo that gives you an overview of Zenoss and how Zenoss can help your more efficiently and cost-effectively monitor your environment.

Read the Zenoss Service Dynamics: 4 Profiles in Unified Monitoring Successs white paper to learn more about how some of our customers use Zenoss to improve their monitoring efficiency and productivity and avoid outages.

Request a free trial! See how you can use Zenoss more in your environment to more effectively and cost-effectively monitor and manage your environment using a single, unified monitoring view and unified monitoring processes.

Try Zenoss!