Work

Well, another day - another UCS error. Out of the blue, one of our chassis started displaying that one PSU had failed, however the UCS was showing no PSU had failed 🤷 Well, as it turns out - this is yet another known bug in 2.0.2(r). You’ll either have to unplug and plug all the power cables (that’s four) in a maintainance window - or simply change the Equipment Power Policy (found in the Root of your UCS, tab Policy) ...

As I wrote before, we have a bunch of filers (and a ton of volumes w/ luns on them), that I need to monitor. At first, I tried the existing NetApp Nagios-Plugin(s), but they all use SNMP and with that I can either watch all volumes or none. And that didn’t satisfy me. Don’t get me wrong, the existing plugins are okay and I still use them for stuff (like GLOBALSTATUS or FAN/CPU/POWER) which isn’t present in the API or real hard to get at, however I wanted more. So I ended up looking at the NetApp API, and ended up writing a “short” plugin for Nagios using Perl. Maybe if I’m ever bored, I’ll rewrite it using C, but for now the Perl plugin has to suffice. So far the plugin supports the following things: Monitoring FlexVolumes (simply watching the free space) Monitoring LUN space (the allocated space inside a FlexVolume for iSCSI/FC LUNs) Monitoring Snapshot space (the allocated space inside a FlexVolume for Snapshots) Monitoring SnapVault relations (and their age) Monitoring SnapMirror relations (and their age) The plugin will return performance data for most (if not all) of those classes. It needs a user on the filer you wish to monitor - which sadly needs to have the admin role.

One of my co-workers recently approached me, that he needed a simple shell script which would generate a simple report about a Vserver’s current connections. After ironing out a few things with him (he had the intention of it being on a CIFS share on our file-server - which I changed to a simple HTML page) I went to work. Out came two scripts. One is the collection instance, and the other is the processing instance. First the collection script runs, finds the current HA master node and then collects the Vserver’s current connections. After that script has dumped the information (date, time, current connections) into a file, the processing script will go and create a simple HTML page that’ll show exactly those informations. ...

Well, I’ve been fiddling around with SLES and openSUSE VMware templates. I know it’s a stupid idea when you have a PXE server from which you could install this in a matter of minutes (seriously the SLES PXE installation takes about 5 minutes). However, when dealing with DMZ’s (yeah, they exist!) you usually don’t have any PXE servers there. So I decided to go with simple VMware templates (like we do with Windows already), but had to iron out a few kinks. ...

Well, I went to work today … yeah, I know it’s Sunday right ? I ended up updating two MDS9148 switches and I didn’t want to figure out everything all over again. So I put the system image and kickstart onto one of our FTP servers and ran a short bash loop on it: 1 2 3 4 root:(ftp.daheim.heimdaheim.de) PWD:~ Sun Oct 20, 08:57:24 [0] > for file in `ls /srv/ftp/firmware/mds9148/*6.2.3*`; do echo "copy ftp://10.0.0.55/`echo $file | sed 's,/srv/ftp/,,'` bootflash:/`echo $file | sed 's,/srv/ftp/firmware/mds9148/,,'`" done Now that’ll generate me two lines, which in turn I can use on the MDS’n: ...

Well, what annoyed me in the past was that I had to patch each XenServer patch by patch (no bulk applying) and when used in combination with UCS blades (especially if those have >250GB RAM), it takes ages to keep a pool up-to-date. So I ended up writing yet another script (I know why I hate Citrix XenServer … the XenCenter GUI is lacking sooooo much) which will download new patches from a directory on a HTTP server and then print the lines necessary to apply the patches to all hosts in a pool. ...

Well, today I ended up writing a short script that’ll give me a list of VMPPs with the VMs that are associated to it. 1 2 3 4 5 6 7 8 9 10 11 12 #!/bin/bash # Get a list of VMPPs for vmpp in `xe vmpp-list params=uuid --minimal | sed "s/,/ /g"`; do VMPP_NAME=`xe vmpp-list params=name-label uuid=$vmpp --minimal` for vm in `xe vmpp-list params=VMs --minimal uuid=$vmpp | sed -e "s/;//g" -e "s/,//g"`; do VM=`xe vm-list params=name-label uuid=$vm --minimal` echo "$VMPP_NAME: $VM" done done

Well, today I encountered a old problem (or so I thought). Basically a specific udevadm version causes the vmware-config-tools.pl script to error out like this: 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 The vmblock enables dragging or copying files between host and guest in a Fusion or Workstation virtual environment. Do you wish to enable this feature? [no] !!! [EXPERIMENTAL] !!! VMware automatic kernel modules enables automatic building and installation of VMware kernel modules at boot that are not already present. By selecting yes, you will be enabling this experimental feature. You can always disable this feature by re-running vmware-config-tools.pl. Would you like to enable VMware automatic kernel modules? [no] No X install found. Could not find Parent Node... I’ve had encountered this before in the past, and before there was some explanation on the VMware forums, which I couldn’t locate. Lucky me, the VMware Tools updater keeps modified versions of vmware-tools-config.pl around. So I ended up creating this short diff, so that I may find it in the future - if I still need it: ...

As the title pretty much tells, I’ve been working on fixing the Root-Disk-Multipathing feature of our XenServer installations. Our XenServer boot from a HA-enabled NetApp controller, however we recently noticed that during a controller fail-over some, if not all, paths would go offline and never come back. If you do a cf takeover and cf giveback in short succession, you’ll end up with a XenServer host that is unusable, as the Root-Disk would be pretty much non-responsive. ...

As so often, I wanted a script, that’ll crawl my filers and regenerate the configuration if there are any new volumes/snapvaults/snapmirrors or if one of them has been removed. 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 #!/bin/bash FAS_HOSTS="$( ls /etc/nagios/objects/hosts/san/fas*{a,b}.cfg | cut -d/ -f7 | cut -d. -f1 )" for host in $FAS_HOSTS; do OUTPUT_FILE=/etc/nagios/objects/hosts/san/$host-vol.cfg # Clear the output file echo "" > $OUTPUT_FILE # Get the volume list for volume in `ssh $host vol status | awk '{ print $1 }' | grep ^vol | sort -u | grep -v vol0$`; do user="$( grep "USER=" /etc/netapp-sdk/$host | cut -d= -f2 )" pass="$( grep "PASS=" /etc/netapp-sdk/$host | cut -d= -f2 )" # echo "define service {" # echo " use generic-service" # echo "" # echo " check_command check_netapp-volfree!$user!$pass!${volume}!92!98" # echo " check_interval 5" # echo " host_name ${host}" # echo " notifications_enabled 0" # echo " notification_interval 720" # echo " service_description VOLSPACE ${volume}" # echo "}" echo echo "define service {" echo " use generic-service-san-perfdata" echo "" echo " check_command check_netapp-lunspace!$user!$pass!${volume}" echo " check_interval 5" echo " host_name ${host}" echo " notifications_enabled 0" echo " notification_interval 720" echo " service_description LUNSPACE ${volume}" echo "}" echo SR="$( ssh $host snap reserve $volume | cut -d -f7 )" if [ "$SR" != "0%" ] ; then echo "define service {" echo " use generic-service-san-perfdata" echo "" echo " check_command check_netapp-snapreserve!$user!$pass!${volume}" echo " check_interval 10" echo " host_name ${host}" echo " notifications_enabled 0" echo " notification_interval 720" echo " # SR: $SR" echo " service_description SNAPRESERVE ${volume}" echo "}" echo fi done | tee -a $OUTPUT_FILE # Check snapvault foo for sv in `ssh $host snapvault status -l 2>/dev/null | awk '{ print $2 }' | grep vol`; do # only do the checks on sv_secondary if [ "$( echo $sv | grep $host | cut -d: -f1 )" == "${host}" ]; then vol="$( echo $sv | cut -d/ -f3 )" user="$( grep "USER=" /etc/netapp-sdk/$host | cut -d= -f2 )" pass="$( grep "PASS=" /etc/netapp-sdk/$host | cut -d= -f2 )" echo "define service {" echo " use generic-service-san-perfdata" echo "" echo " check_command check_netapp-snapvault!$user!$pass!$vol!38!42!" echo " check_interval 60" echo " host_name ${host}" echo " notifications_enabled 0" echo " notification_interval 720" echo " service_description SNAPVAULT ${vol}" echo "}" echo fi done | tee -a $OUTPUT_FILE # Check snapmirror foo for sm in `ssh $host snapmirror status 2>/dev/null | awk '{ print $2 }' | grep vol | grep $host`; do # only do the checks on sm_secondary if [ "$( echo $sm | grep $host | cut -d: -f1 )" == "${host}" ]; then vol="$( echo $sm | cut -d/ -f3 | cut -d: -f2 )" user="$( grep "USER=" /etc/netapp-sdk/$host | cut -d= -f2 )" pass="$( grep "PASS=" /etc/netapp-sdk/$host | cut -d= -f2 )" echo "define service {" echo " use generic-service-san-perfdata" echo "" echo " check_command check_netapp-snapmirror!$user!$pass!$vol!38!42!" echo " check_interval 60" echo " host_name ${host}" echo " notifications_enabled 0" echo " notification_interval 720" echo " service_description SNAPMIRROR ${vol}" echo "}" echo fi done | tee -a $OUTPUT_FILE done