Zenoss Filesystems Monitoring

Have came across many zenoss users asking about changing the default thresholds for filesystem, disk utilisation. In many cases, the issue is about the fact that the default template calculates percentage utilisation and and a threshold like (95%) may not be appropriate for large (hundreds of terabytes) filesystems.

There are a couple of things that can be done.
1. Modify the default template
2. Use a transform in /Events/Perf/Filesystem that calculate actual figures, then decide the fate of events based on these figures (you can obtain such a transform here)

If the first approach above sounds good, I may be able to upload a zenpack which adds a second graph on the template, and also contains a modified copy of FileSystem.py

GPFS Highly Available (HA) SNMP monitoring configuration

The available/documented GPFS SNMP implementation (By IBM) is not designed to be highly available.

Here is the link

There is only one SNMP collector node at a time. If that node fails the SNMP monitoring does not fail over to any other node and thus complete loss of cluster monitoring/reporting.

This blog post offers a simple implementation of gpfs snmp monitoring failover. The failover scheme uses a callback mechanism triggered by a quorumNodeLeave event and the eventNode is its only parameter.

First create a folder to contain your callbacks (if you already have a location for your callbacks, use that instead)

Download/copy the following script into the callbacks location and make it executable.

gpfs_snmp_failover

Modify the script to indicate your available collector nodes by substituting "quorum" and "quorum_node_2" with the hostnames of your gpfs quorum nodes.

Note that in this case the callbacks location is /callback so you may have to modify the script accordingly.

Copy the modified script to all quorum nodes

Add the callback (run once from any quorum node);

mmaddcallback NodeDownCallback --command  /callback/snmp_collector_failover.sh --event quorumNodeLeave --parms %eventNode

if you want monitoring to be reverted back to the default prefered collector node after it comes back online, you may consider adding a node join callback;

mmaddcallback NodeJoinCallback --command  /callback/snmp_collector_failover.sh --event quorumNodeJoin --parms %eventNode

[zenoss] employ ping down dependency to suppress other events

sum = str(evt.summary)
if getattr(evt, 'severity', 0) > 0 \
and getattr(evt, 'eventClass', '/Unknown') != '/Status/Ping' \
and device and device.getPingStatus() > 0:
       evt._action = "drop"

if (sum.find("Command timed out")>=0) \
or (sum.find("Socket timeout")>=0) \
or (sum.find("Unable to read")>=0) \
or (sum.find("Process not running")>=0) \
or (sum.find("SNMP agent down")>=0) \
or (sum.find("No response from")>=0):
     if device.getPingStatus() <= 0:
           time.sleep(8)
           if device.getPingStatus() > 0:
                  evt._action = "drop"
     else:
           evt._action = "drop"

   if getattr(evt, 'prodState', 99)<400:
      evt._action = "drop"

Zenoss Audio Alerts

I have published a blog in the Zenoss community on HowTo setup audible alerts for Zenoss.
Check this out here: http://community.zenoss.org/people/chitambira/blog/2010/01/19/audible-alerts

Zenoss: Setting IP interface speed using a script

Zen modeler sometimes returns wrong interface speeds for some interfaces. This is usually not straight forward to rectify if the monitored device is a hardware appliance like environment monitors etc. Zenoss have as a matter of policy or best practice, not allowed modification of interfaces for automatically discovered devices. However, they have suggested that this can be done via ZenDMD. Punching the commands again and again everytime you want to set interface speed could be monotonous. The script below could be used as a one liner to set interface speed for a specific device.

#!/usr/bin/env python
import sys
import Globals
from Products.ZenUtils.ZenScriptBase import ZenScriptBase
from transaction import commit

dmd = ZenScriptBase(connect=True).dmd

for dev in dmd.Devices.getSubDevices():
  if dev.id.startswith(sys.argv[1]):
   for interface in dev.os.interfaces():
    if interface.id.startswith(sys.argv[2]):
     interface.speed = float(sys.argv[3])
     interface.lockFromUpdates()
commit()


save the script as script.py
[zenoss@server ~]chmod 755 script.py
You can then use this script on the commandline as zenoss user, usage:

./script.py
e. g.[zenoss@server ~]./script.py cisco Fast 100000000

##for setting speed to 100Mbps on all FastEthernet interfaces on ALL devices that begin with cisco (like cisco-002.net.myorg.com, cisco-003. ... ...)
###NOTE: use appropriate indention in the script

Summer of community ZenPack Contest

My MGE-UPS Zenpack won the runner up award for the Summer of Community ZenPack Contest 2009. My prize is the Chumby Media Player (http://www.chumby.com)

Check out the Zenoss announcement/blog here

The MGE UPS ZenPack provides monitoring for MGE UPS devices and MIBs for translating traps. It provides performance graphs for:
* Battery Levels
* Battery Times
* UPS Currents
* UPS Voltages