Nagios on BAFM

Nagios: Service Check Timed Out

Fri, 07 Dec 2018 21:24:54 +0000

Since I got the pleasure of watching some Windows boxen with Nagios, I took the Windows Update plugin from Michal Jankowski and implemented it. It took me some time, to initially set up the nsclient++ correctly so it just works, but up till now the check plugin sometimes reported the usual “Service Check Timed Out”.

Usually I ended up increasing the cscript timeout, or the nsclient++ socket timeout, but it still kept showing up. Since I rely heavily on my surveillance tools, I have the demand, that as few as possible false positives show up. So I ended up chasing down this error today, and after that I have to say it was quite simple.

Nagios: Integrating Cisco switches

Wed, 20 Aug 2014 08:20:53 +0000

Well, as I wrote recently, we received a new BladeCenter a few weeks back. Now, as we slowly take it into service I was interested in watching the utilization of the back planes as well as the CPU utilization of the Cisco Catalyst 3012 network switches.

The first mistake I made, was to trust Cisco with their guide about how to get the utilization from the device using SNMP. They stated some OID’s, which I tried with snmpwalk and got a result from.

1
2


snmpwalk -v1 -c public -O n 10.0.0.35 .1.3.6.1.4.1.9.5.1.1.8
.1.3.6.1.4.1.9.5.1.1.8.0 = INTEGER: 0

Now, as I tried retrieving the SNMP data by means of the check_snmp plugin, I got some flaky results:

1
2
3
4


/usr/lib/nagios/plugins/check_snmp -H 10.0.0.35 -C public
 .1.3.6.1.4.1.9.5.1.1.8
SNMP problem - No data received from host
CMD: /usr/bin/snmpget -t 1 -r 5 -m '' -v 1 [authpriv] 10.0.0.35:161

Those of you, who read the excerpts carefully will notice the difference between snmpwalk and the OID I passed on to check_snmp.

The point being, the OID’s Cisco gave in their Design tech notes are either old, or just not accurate at all. After passing on the .0 to each value given by Cisco, the check_snmp is all honky dory and integrated into Nagios.

As usual, the Nagios definitions are further down, for those interested.

Nagios virtualization

Sat, 16 Aug 2014 09:49:36 +0000

As virtualization seems to be a trendy thing to do, I went ahead and virtualized our nagios (while reinstalling the whole thing …).

Now as I went into work today and started my email client, I received 4 nagios warnings about a LOAD service reaching critical state. Looked at the nagios box itself, opened up the VM console, looked into the syslog. Nothing.

Yet over 3/4 of the services were flapping, some ping checks were critical (for whatever reason). So I opened the nagios webinterface again, and noticed it dropping the connection over and over again (had to reauthentificate me again and again).

Nagios Hostgroup Inheritance

Sat, 16 Aug 2014 09:48:08 +0000

As I wrote earlier, I recently virtualized our nagios. Along with that came a complete " redesign" of how checks are applied. Up till now, I defined checks for each and every single server, thus ending up with ~25 files, each holding roughly 6 checks which are in the same file just sorted by hostname.

As you can imagine, it gets quite confusing with that amount of checks (~150). So the last two days I spent on reorganizing (with Visio), on which object/hostgroup placing a check would make sense. Now, this is my first result of two days planning, reorganizing, reordering and moving hosts into different hostgroups.

MessPC Ethernetbox 2 and Nagios

Fri, 08 Aug 2014 09:46:05 +0000

As I talked to Tobi yesterday, we came to talk about our Ethernet Box thermometer. It’s a neat device, which works pretty much out of the box. Integrating it with Nagios is a bit of a bummer.

That’s what the ~300 EUR box looks like. It’s basically a small black box with a RJ45 jack, and four RJ11 jacks for attached external devices. The box itself only functions as a " management station" and doesn’t come with a sensor. Normally, you can attach up till four RJ11 sensors to it. But, MessPC also has RJ11 port splitters, which enables you to attach up to eight RJ11 sensors to the MessPC.

As you can see, the box has a RJ45 jack on the other side, which you basically hook up to your network and then configure an IP address (or if you fancy DHCP for those things, it’s possible too).

On the opposite site, are the RJ11 jacks for the sensors. As you can see, we currently do have 4 splitters attachted to the box, enabling up till 8 sensors to be measured. Once you have it up and running, you can look at the web interface and you’ll be able to see the state of the sensors right on the first page.

Monitoring Brocade FC switches with SNMP/Nagios

Fri, 08 Aug 2014 08:39:27 +0000

I looked into the mess a bit more, and as it turns out, the weird crap I was talking about only happens if you have a port with LossofSynchronization, LossofSignal or LinkFailures value with the base of ten (i.e. 10, 101 or 10.000).

Additionally, the OID’s for those three failure elements seem to be dependent on the firmware version, as with 6.3.x they appear as different OIDs. So I may need to introduce another command-line switch, which selects the firmware version and depending on that, the OID.

Monitoring Brocade FC switches with Nagios

Fri, 08 Aug 2014 08:37:46 +0000

The last four days I spent looking for ways on monitoring a Brocade Fibrechannel switch (in my case IBM 2145 B32/F40). The first thing I came up with, is using SNMP. As it was already configured for the previous monitoring with Munin, getting information should be quite easy. After looking through Google for a bit, there is already one script that worked for me.

Only trouble I had with that script, is that it crams every single port into one result. As I wanted something, that a) could watch a single port and b) return performance data, I went ahead an used the script to do a basic rewrite. But after a short while, I grew antsy and started writing a script from scratch, using the OIDs I got from that script and a Cacti template.

NetApp: Monitoring of SnapVault/SnapMirror/LUN/Snapshot information with Nagios

Sun, 15 Dec 2013 15:51:03 +0000

As I wrote before, we have a bunch of filers (and a ton of volumes w/ luns on them), that I need to monitor. At first, I tried the existing NetApp Nagios-Plugin(s), but they all use SNMP and with that I can either watch all volumes or none. And that didn’t satisfy me.

Don’t get me wrong, the existing plugins are okay and I still use them for stuff (like GLOBALSTATUS or FAN/CPU/POWER) which isn’t present in the API or real hard to get at, however I wanted more. So I ended up looking at the NetApp API, and ended up writing a “short” plugin for Nagios using Perl.

Maybe if I’m ever bored, I’ll rewrite it using C, but for now the Perl plugin has to suffice.

So far the plugin supports the following things:

Monitoring FlexVolumes (simply watching the free space)
Monitoring LUN space (the allocated space inside a FlexVolume for iSCSI/FC LUNs)
Monitoring Snapshot space (the allocated space inside a FlexVolume for Snapshots)
Monitoring SnapVault relations (and their age)
Monitoring SnapMirror relations (and their age)

The plugin will return performance data for most (if not all) of those classes. It needs a user on the filer you wish to monitor - which sadly needs to have the admin role.

Generate Nagios config for check_netapp-api-pl

Tue, 05 Feb 2013 19:46:24 +0000

As so often, I wanted a script, that’ll crawl my filers and regenerate the configuration if there are any new volumes/snapvaults/snapmirrors or if one of them has been removed.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96


#!/bin/bash

FAS_HOSTS="$( ls /etc/nagios/objects/hosts/san/fas*{a,b}.cfg | cut -d/ -f7 | cut -d. -f1 )"

for host in $FAS_HOSTS; do
 OUTPUT_FILE=/etc/nagios/objects/hosts/san/$host-vol.cfg

 # Clear the output file
 echo "" > $OUTPUT_FILE

 # Get the volume list
 for volume in `ssh $host vol status | awk '{ print $1 }' | grep ^vol | sort -u | grep -v vol0$`; do
 user="$( grep "USER=" /etc/netapp-sdk/$host | cut -d= -f2 )"
 pass="$( grep "PASS=" /etc/netapp-sdk/$host | cut -d= -f2 )"
# echo "define service {"
# echo " use generic-service"
# echo ""
# echo " check_command check_netapp-volfree!$user!$pass!${volume}!92!98"
# echo " check_interval 5"
# echo " host_name ${host}"
# echo " notifications_enabled 0"
# echo " notification_interval 720"
# echo " service_description VOLSPACE ${volume}"
# echo "}"
 echo
 echo "define service {"
 echo " use generic-service-san-perfdata"
 echo ""
 echo " check_command check_netapp-lunspace!$user!$pass!${volume}"
 echo " check_interval 5"
 echo " host_name ${host}"
 echo " notifications_enabled 0"
 echo " notification_interval 720"
 echo " service_description LUNSPACE ${volume}"
 echo "}"
 echo

 SR="$( ssh $host snap reserve $volume | cut -d -f7 )"
 if [ "$SR" != "0%" ] ; then
 echo "define service {"
 echo " use generic-service-san-perfdata"
 echo ""
 echo " check_command check_netapp-snapreserve!$user!$pass!${volume}"
 echo " check_interval 10"
 echo " host_name ${host}"
 echo " notifications_enabled 0"
 echo " notification_interval 720"
 echo " # SR: $SR"
 echo " service_description SNAPRESERVE ${volume}"
 echo "}"
 echo
 fi
 done | tee -a $OUTPUT_FILE

 # Check snapvault foo
 for sv in `ssh $host snapvault status -l 2>/dev/null | awk '{ print $2 }' | grep vol`; do
 # only do the checks on sv_secondary
 if [ "$( echo $sv | grep $host | cut -d: -f1 )" == "${host}" ]; then
 vol="$( echo $sv | cut -d/ -f3 )"
 user="$( grep "USER=" /etc/netapp-sdk/$host | cut -d= -f2 )"
 pass="$( grep "PASS=" /etc/netapp-sdk/$host | cut -d= -f2 )"
 echo "define service {"
 echo " use generic-service-san-perfdata"
 echo ""
 echo " check_command check_netapp-snapvault!$user!$pass!$vol!38!42!"
 echo " check_interval 60"
 echo " host_name ${host}"
 echo " notifications_enabled 0"
 echo " notification_interval 720"
 echo " service_description SNAPVAULT ${vol}"
 echo "}"
 echo
 fi
 done | tee -a $OUTPUT_FILE

 # Check snapmirror foo
 for sm in `ssh $host snapmirror status 2>/dev/null | awk '{ print $2 }' | grep vol | grep $host`; do
 # only do the checks on sm_secondary
 if [ "$( echo $sm | grep $host | cut -d: -f1 )" == "${host}" ]; then
 vol="$( echo $sm | cut -d/ -f3 | cut -d: -f2 )"
 user="$( grep "USER=" /etc/netapp-sdk/$host | cut -d= -f2 )"
 pass="$( grep "PASS=" /etc/netapp-sdk/$host | cut -d= -f2 )"
 echo "define service {"
 echo " use generic-service-san-perfdata"
 echo ""
 echo " check_command check_netapp-snapmirror!$user!$pass!$vol!38!42!"
 echo " check_interval 60"
 echo " host_name ${host}"
 echo " notifications_enabled 0"
 echo " notification_interval 720"
 echo " service_description SNAPMIRROR ${vol}"
 echo "}"
 echo
 fi
 done | tee -a $OUTPUT_FILE
done

Generate Nagios config for NetApp filers

Tue, 01 Jan 2013 09:46:39 +0000

At some point in the last few weeks, I repeatedly had to recreate my Nagios config for currently six filers. After doing that a few times, I ended up (like sooo often) writing a short Bash script, that’ll do this for me - without any fuss.

The only thing the script needs, is that the filers and the filers are registered in DNS … Here’s an example:

1
2
3
4


fas3240a IN A 172.31.76.150
fas3240a-sp IN A 172.31.74.150
fas3240b IN A 172.31.76.151
fas3240b-sp IN A 172.31.74.151

With that done, the script will create the necessary Nagios config for those filers.

Configuring nagios-plugins-zypper

Thu, 12 Nov 2009 17:45:43 +0000

Since I’m running check_zypper via nrpe (which in turn runs as nobody), I need to set up sudo. In order for the plugin to work, we need to add the following line to /etc/sudoers (by means of visudo):

`1`	`nobody ALL = NOPASSWD: /usr/bin/zypper sl, /usr/bin/zypper --non-interactive --no-gpg-checks --terse list-updates`

(Keep in mind this needs to be a single line …)

Praxisbuch Nagios by Tobias Scherbaum

Sat, 25 Jul 2009 11:58:20 +0000

Tobi recently finished writing yet another book, which he also talked about in a blog post. Shortly after, I asked him a rather curious question. What exactly is the plant or animal on the cover of the book ? He was kind enough to send a voucher copy of the book my way.

Praxisbuch Nagios

He actually mentions it in the credits at the beginning of the book. Turns out it is an animal, a sea pen or sea feather (I’m guessing at Pennatula aculeata).

Nagios: SNMP OID's for IBM's RSA II adapter

Wed, 01 Apr 2009 10:10:15 +0000

Well, after some poking around I finally found some OID’s for the RSA’s (only through these two links: check_rsa_fan and check_rsa_temp).

For Nagios, I dismissed the fans, since the fan speed is only passed on in percent values. So I only added this:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28


define hostgroup{
 hostgroup_name rsa-snmp
 alias Remote Supervisor Adapter (allowing SNMP connections)
}

define service{
 use generic-perfdata

 check_command check_rsa_snmpv1_public!.1.3.6.1.4.1.2.3.51.1.2.1.2.1.1!45!60!°C!Temperature CPU0!
 hostgroup_name rsa-snmp
 service_description TEMP CPU0
}

define service{
 use generic-perfdata

 check_command check_rsa_snmpv1_public!.1.3.6.1.4.1.2.3.51.1.2.1.2.2.1!45!60!°C!Temperature CPU1!
 hostgroup_name rsa-snmp
 service_description TEMP CPU1
}

define service{
 use generic-perfdata

 check_command check_rsa_snmpv1_public!.1.3.6.1.4.1.2.3.51.1.2.1.5.1.0!29!35!°C!Temperature Ambient!
 hostgroup_name rsa-snmp
 service_description TEMP AMBIENT
}

Oh, and if anyone else is curious like me, here’s the list with the OID’s, courtesy of Gerhard Gschlad and Leonardo Calamai.

Nagios: check_snmp again

Fri, 27 Feb 2009 10:59:25 +0000

Well, today I had to grind my head again, regarding the way check_snmp handles WARNING and CRITICAL events. From my point of view, check_snmp is really just retarded sometimes.

As you know, all the other plugins accept WARNING and CRITICAL-thresholds based on the calculation, if the return integer is above this threshold it reached WARNING/CRITICAL state. But check_snmp doesn’t play that way.

It expects only ranges, which are NOT gonna result in warning or critical events. Which is kinda stupid, since you gotta rethink twice about the thresholds 😛

Nagios: NSclient++ in a clustered Environment

Thu, 26 Feb 2009 10:50:36 +0000

Well, most of you already know that I’m a Nagios fanatic. I like to watch as many aspects as I possibly can. So, yesterday I started figuring out ways to watch our different cluster groups (housing a bunch – try above 20.000 – of file shares).

Now, my first tries failed horribly. I brought down a complete cluster group, resulting in a major annoyance. Now, today I went at it a bit smarter 😛 I cloned myself two VM’s off my Windows Server 2003 Enterprise R2 template, created a new cluster.

Monitoring the IBM BladeCenter chassis with Nagios

Tue, 10 Feb 2009 11:40:00 +0000

Today I ended up working out the details on what we want to monitor regarding our BladeCenter. The most interesting details (for us that is) are these:

Fan speeds for Chassis Cooling/Power Module Cooling Bay(s)
Temperature
Power Domain utilization

It wasn’t * that* hard to implement. Only trouble(s) I ran into, were ( 1) IBM did a real shitty job with the MIB’s. If you look closely into the mmblade.mib, you’re gonna notice, that not a single OID is specified for the events. ( 2) As the MIB’s weren’t documented anywhere, I had to look them up via snmpwalk (which I had never used before). So as a reminder (to myself), here’s how it is done:

`1`	`snmpwalk -v1 -c public -O n 10.0.0.35 .1.3.6.1.4.1.2.3.51.2.2`

This will get you a list, with a lot of output (5154 lines to be exact). Lucky me, the web interface of the management module/ssh interface is rather verbose, so all you need to do is compare those values with what you are looking for.

So for myself (and anyone interested) read ahead for the list of checks we are currently running on the management module.

Opsview installation reviewed

Sat, 24 Jan 2009 11:33:50 +0000

Well, I recently (well, yesterday) built the opsview RPM’s for SLES10, and started fiddeling about with it today. Alex " recommended" I should rather look at Opsview instead of Centreon, but boy was there a surprise waiting for me …

Opsview has the advantage that it at least lets you use the package manager. But, it also needs a lot of handy work (just like Centreon, which I really dislike since it’s real error prone).

Nagios and check_ram yet again

Fri, 14 Nov 2008 18:08:30 +0000

As some people know, I previously " created" (mostly modified the check_swap plug-in to print RAM usage) check_ram in C. Now one of my problems for the past few months was putting the C plug-in as well as " supported" environment under the same hat. Today I had another look at the amount of available plug-ins in NagiosExchange. There are quite a few plug-ins available, but as I do have some experience with Python, I used the one written in Python.

It was rather easy hacking in support for performance data into it, as the below shows. Someone else already posted a non-unified diff for performance data support, but that ain’t quite right according to the Nagios plug-in development guidelines.

Suspected NRPE weirdness

Sun, 10 Aug 2008 07:55:28 +0000

Well, I just noticed a really weird thing, when you have command line arguments enabled.

Here’s a snippet from my nrpe.cfg:

1
2


dont_blame_nrpe=1
command[check_disk]=/usr/lib/nagios/plugins/check_disk -E -w $ARG1$ -c $ARG2$ -p $ARG3$

Now, if you’d check the free space for the root, it ain’t gonna show any inode percentage (that one isn’t what I’m talking about). But if you have to use bind mounts like I do (Tivoli needs a separate " domain" -- that is a separate mount point for each domain), you might wanna check the free space on the real device, rather than the free space on the bind mount (which is gonna show you the free space of the parent file system - in my case the root fs).

Nagios 3 and hostgroup inheritance

Fri, 08 Aug 2008 12:24:01 +0000

As I wrote some time ago, I was trying to utilize Nagios 3.x’s neat feature of " nested" hostgroups. Well, as it turned out I thought it worked differently; basically like this:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24


define hostgroup {
 hostgroup_name a-parent-hostgroup
 alias Our toplevel parent hostgroup
}

define service {
 use generic-service
 check_command check_dummy!0!
 service_description SSH
 hostgroup_name a-parent-hostgroup
}

define hostgroup {
 hostgroup_name a-child-hostgroup
 hostgroup_members a-parent-hostgroup
 alias Our child hostgroup
}

define service {
 use generic-service
 check_command check_dummy!0!
 service_description LOAD
 hostgroup_name a-child-hostgroup
}

As you can cleary see on line 14, I thought you define the relation between two hostgroups in the child hostgroup. The problem with it was basically (as I said in the earlier posts), that all the services defined for the child hostgroups are handed on upwards to the parent hostgroup(s).

Nagios Hostgroup Inheritance (continued)

Sat, 19 Jul 2008 09:41:21 +0000

Well, it turns out that my thought was ultimativly flawed. When defining the hostgroup_members in the lower tiers, nagios is association the checks from the lower tier with the upper tiers. Thus propagandating all checks upwards, and me ending up with ~250 checks instead of ~150.

Gonna have to try to define the dependency backwards, maybe that’ll help. But that’s a topic for Monday. Guess I’ll finish viewing Ghost in the Shell - Stand Alone Complex first.

Nagios3 with Active Directory authorization on SLES10

Mon, 14 Jul 2008 17:32:04 +0000

Well, it seems to be getting a “trend” for me, to integrate stuff into our Active Directory. Now that I know why, and how easy that is, I expect to add more stuff. The good thing about the integration is, that you only need to maintain a single source for authorization.

The bad thing about that is, that stuff becomes dependent on the Active Directory (we do have four domain controllers, so that should be fine).

Nagios undamp; plugins

Sat, 06 Oct 2007 15:12:43 +0000

Since we started utilizing Nagios’s power two months ago, I finally came up with a C-based ram-plugin for nagios. The biggest problem I had with the python and perl based plugins, that some distributions (yes, SLES and Debian) don’t install either Python or Perl.

Since I wanted a manageable setup (as in unified code base across all distributions), I wanted it to work without installing too much. So I took the swap plugin and basically removed what wasn’t necessary and voila!