Netapp

Well as the title says, sadly we bought our FAS6210 without CIFS/NFS license. Thus, in order to create the folder structure/add the authorized_keys file, you’ll have to work for your money a little bit. First, you need to run cifs setup / cifs passwd somewhere. I did it on our Data ONTAP simulator, which comes in handy for things like that. You’ll get a cryptic looking password (no clue which format that is), looking like this: _OnWddr)xa. ...

Well, today I had another idea (basically like the one I wrote for SVC’s VDisk mappings a while back) for a script: 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 #!/bin/bash KEY_FILE="/root/.ssh/netapp.dsa" SSH_OPTS="/root/.ssh/netapp-ssh_config" if [ $# -ne 3 ] ; then echo "fas-copy-lunmap: FAS_CONTROLLER SOURCE_IGROUP TARGET_IGROUP" echo echo "Copy the LUN map from one igroup to another, ie for ESX reinstallation" echo echo " Usage:" echo " - FAS_CONTROLLER: Hostname/IP-adress of the DATA ONTAP controller" echo " - SOURCE_IGROUP: igroup that is used as a reference for the copy process" echo " - TARGET_IGROUP: igroup that is actually modified" echo exit 1 fi FAS_CTRL=$1 SOURCE=$2 TARGET=$3 ssh_fas() { # $@: commands for Data ONTAP COMMANDS="$@" /usr/bin/ssh -i $KEY_FILE -l root -F $SSH_OPTS $COMMANDS } #set -x # Get the lun list. for lun in $( ssh_fas $FAS_CTRL lun show -g $SOURCE | awk '{ print $1 }' | sort -u ); do # Get the LUN number its mapped LUN_ID="$( ssh_fas $FAS_CTRL lun show -m $lun | grep "^/vol" | awk '{ print $3 }' )" # If the LUN id is 0, skip otherwise we would copy the boot LUN if [ "$LUN_ID" != "0" ] ; then # Actually map the lun to our host echo "Mapping $lun to $TARGET as LUN_ID $LUN_ID" ssh_fas $FAS_CTRL lun map $lun $TARGET $LUN_ID fi done #set +x I’ll post the counterpart of the script (to remove the LUNs) in a second post later on.

As I wrote before, we’re using SnapManager (for SQL/Oracle) to create consistent snapshots. However my database guys don’t want to name their snapshots daily. (which I can understand), as once you archive those snapshots to a secondary (and tertiary) system, the names become junk. So, they’re naming the snapshots like snap__vcsrv_29_12_2012-10.00.01. Sadly, when it comes to SnapVault, it expects the names in form of daily. otherwise you won’t be able to transfer the snapshots with the CLI (none that I have found anyway). But we didn’t want to move away from naming the snapshots the way they are, so I ended up writing a PowerShell script, that once triggered archives the Snapshots needed for a set of databases. It took me a while to figure a bunch of stuff out, but in the end I think I have a working way of archiving custom-named snapshots.

Well, I recently had the pleasant task of implementing SnapVault backups, that are being shipped to an offsite location with SnapMirror. That in itself isn’t the bad thing, however we decided against Protection Manager (since it was a charged product back when we decided on this). So I basically had the three tasks: Actually implement the SnapVault stuff (and learn my way around it and also document it) Write a bunch of scripts, that help us in creating scheduled backups of our databases Create a monitoring script, that’ll fit into our Nagios environment already in place Well, two months later (sadly it still has some kinks - I can’t figure out this one bug though for the life of it) and a few hundred hours of working on/with it and out came four things: ...

I’ve been spending a lot of my time the last week on getting SnapVault with out FAS-filers to work. Out came a script, which does this for a given volume (and of course SnapVault Primary and Secondary). The script expects, that SSH public key authentification has been set up.

As I wrote before, we have a bunch of filers (and a ton of volumes w/ luns on them), that I need to monitor. At first, I tried the existing NetApp Nagios-Plugin(s), but they all use SNMP and with that I can either watch all volumes or none. And that didn’t satisfy me. Don’t get me wrong, the existing plugins are okay and I still use them for stuff (like GLOBALSTATUS or FAN/CPU/POWER) which isn’t present in the API or real hard to get at, however I wanted more. So I ended up looking at the NetApp API, and ended up writing a “short” plugin for Nagios using Perl. Maybe if I’m ever bored, I’ll rewrite it using C, but for now the Perl plugin has to suffice. So far the plugin supports the following things: Monitoring FlexVolumes (simply watching the free space) Monitoring LUN space (the allocated space inside a FlexVolume for iSCSI/FC LUNs) Monitoring Snapshot space (the allocated space inside a FlexVolume for Snapshots) Monitoring SnapVault relations (and their age) Monitoring SnapMirror relations (and their age) The plugin will return performance data for most (if not all) of those classes. It needs a user on the filer you wish to monitor - which sadly needs to have the admin role.

As the title pretty much tells, I’ve been working on fixing the Root-Disk-Multipathing feature of our XenServer installations. Our XenServer boot from a HA-enabled NetApp controller, however we recently noticed that during a controller fail-over some, if not all, paths would go offline and never come back. If you do a cf takeover and cf giveback in short succession, you’ll end up with a XenServer host that is unusable, as the Root-Disk would be pretty much non-responsive. ...

Well, for the past two months I had a case open with NetApp to figure out this SnapVault replication issue we were seeing. The initial transfer of the SnapVault relation would complete with a hick up, manual snapshot transfers also work - just the scheduled, auto-created Snapshots won’t replicate. At first I (and the NetApp support) thought this was an issue with SnapVault itself, however after being away for the last four weeks I looked at the issue with fresh eyes. After a short peek into the logs, I found what I had found back when I first looked into this. ...

As so often, I wanted a script, that’ll crawl my filers and regenerate the configuration if there are any new volumes/snapvaults/snapmirrors or if one of them has been removed. 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 #!/bin/bash FAS_HOSTS="$( ls /etc/nagios/objects/hosts/san/fas*{a,b}.cfg | cut -d/ -f7 | cut -d. -f1 )" for host in $FAS_HOSTS; do OUTPUT_FILE=/etc/nagios/objects/hosts/san/$host-vol.cfg # Clear the output file echo "" > $OUTPUT_FILE # Get the volume list for volume in `ssh $host vol status | awk '{ print $1 }' | grep ^vol | sort -u | grep -v vol0$`; do user="$( grep "USER=" /etc/netapp-sdk/$host | cut -d= -f2 )" pass="$( grep "PASS=" /etc/netapp-sdk/$host | cut -d= -f2 )" # echo "define service {" # echo " use generic-service" # echo "" # echo " check_command check_netapp-volfree!$user!$pass!${volume}!92!98" # echo " check_interval 5" # echo " host_name ${host}" # echo " notifications_enabled 0" # echo " notification_interval 720" # echo " service_description VOLSPACE ${volume}" # echo "}" echo echo "define service {" echo " use generic-service-san-perfdata" echo "" echo " check_command check_netapp-lunspace!$user!$pass!${volume}" echo " check_interval 5" echo " host_name ${host}" echo " notifications_enabled 0" echo " notification_interval 720" echo " service_description LUNSPACE ${volume}" echo "}" echo SR="$( ssh $host snap reserve $volume | cut -d -f7 )" if [ "$SR" != "0%" ] ; then echo "define service {" echo " use generic-service-san-perfdata" echo "" echo " check_command check_netapp-snapreserve!$user!$pass!${volume}" echo " check_interval 10" echo " host_name ${host}" echo " notifications_enabled 0" echo " notification_interval 720" echo " # SR: $SR" echo " service_description SNAPRESERVE ${volume}" echo "}" echo fi done | tee -a $OUTPUT_FILE # Check snapvault foo for sv in `ssh $host snapvault status -l 2>/dev/null | awk '{ print $2 }' | grep vol`; do # only do the checks on sv_secondary if [ "$( echo $sv | grep $host | cut -d: -f1 )" == "${host}" ]; then vol="$( echo $sv | cut -d/ -f3 )" user="$( grep "USER=" /etc/netapp-sdk/$host | cut -d= -f2 )" pass="$( grep "PASS=" /etc/netapp-sdk/$host | cut -d= -f2 )" echo "define service {" echo " use generic-service-san-perfdata" echo "" echo " check_command check_netapp-snapvault!$user!$pass!$vol!38!42!" echo " check_interval 60" echo " host_name ${host}" echo " notifications_enabled 0" echo " notification_interval 720" echo " service_description SNAPVAULT ${vol}" echo "}" echo fi done | tee -a $OUTPUT_FILE # Check snapmirror foo for sm in `ssh $host snapmirror status 2>/dev/null | awk '{ print $2 }' | grep vol | grep $host`; do # only do the checks on sm_secondary if [ "$( echo $sm | grep $host | cut -d: -f1 )" == "${host}" ]; then vol="$( echo $sm | cut -d/ -f3 | cut -d: -f2 )" user="$( grep "USER=" /etc/netapp-sdk/$host | cut -d= -f2 )" pass="$( grep "PASS=" /etc/netapp-sdk/$host | cut -d= -f2 )" echo "define service {" echo " use generic-service-san-perfdata" echo "" echo " check_command check_netapp-snapmirror!$user!$pass!$vol!38!42!" echo " check_interval 60" echo " host_name ${host}" echo " notifications_enabled 0" echo " notification_interval 720" echo " service_description SNAPMIRROR ${vol}" echo "}" echo fi done | tee -a $OUTPUT_FILE done

At some point in the last few weeks, I repeatedly had to recreate my Nagios config for currently six filers. After doing that a few times, I ended up (like sooo often) writing a short Bash script, that’ll do this for me - without any fuss. The only thing the script needs, is that the filers and the filers are registered in DNS … Here’s an example: 1 2 3 4 fas3240a IN A 172.31.76.150 fas3240a-sp IN A 172.31.74.150 fas3240b IN A 172.31.76.151 fas3240b-sp IN A 172.31.74.151 With that done, the script will create the necessary Nagios config for those filers.