Life

As might have noticed, I haven’t posted anything in the last week. Well, as you might have figured, (oh I don’t know - maybe from the blog title) I’m enjoying a three week long vacation. I’m staying with my parents, doing day trips and getting a tan (I’m working pretty hard on that!). I still have two weeks left (well, actually only 11 days - makes it one week and change), as I’m leaving for Stuttgart next Friday. ...

As promised in the earlier post, for completeness sake, here’s the counterpart for removing the LUNs in the first place. 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 #!/bin/bash KEY_FILE="/root/.ssh/netapp.dsa" SSH_OPTS="/root/.ssh/netapp-ssh_config" if [ $# -ne 2 ] ; then echo "fas-remove-lunmap: FAS_CONTROLLER TARGET_IGROUP" echo echo "Remove every mapped lun from the target igroup (for example reinstallation)" echo echo " Usage:" echo " - FAS_CONTROLLER: Hostname/IP-adress of the DATA ONTAP controller" echo " - TARGET_IGROUP: igroup that is actually modified" echo exit 1 fi FAS_CTRL=$1 TARGET=$2 ssh_fas() { # $@: commands for Data ONTAP COMMANDS="$@" /usr/bin/ssh -i $KEY_FILE -l root -F $SSH_OPTS $COMMANDS } # Get the hostname of the controller, necessary for the reporting CTRL_HOSTNAME="$( ssh_fas $FAS_CTRL rdfile /etc/rc | grep ^hostname | cut -d -f2 | tr 'a-z' 'A-Z' )" #set -x # Get the lun list. for lun in $( ssh_fas $FAS_CTRL lun show -g $SOURCE | awk '{ print $1 }' | sort -u ); do # If the LUN id is 0, skip otherwise we would remove the boot LUN if [ "$LUN_ID" != "0" ] ; then # Actually remove the mapping to our host echo "Removing $lun from $TARGET" ssh_fas $FAS_CTRL lun unmap $lun $TARGET fi done #set +x With that, you can simply run it against a NetApp controller and remove every LUN map except the one with LUN ID 0 (which is pretty handy when installing/reinstalling ESX servers).

Well, after figuring things out (and realizing that if you create a LUN in the same size as the volume it’ll break), I decided to write yet another script to figure out which LUNs needed fixing. 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 #!/bin/bash KEY_FILE="/root/.ssh/netapp.dsa" SSH_OPTS="/root/.ssh/netapp-ssh_config" FAS_CTRL=$1 ssh_fas() { # $@: commands for Data ONTAP COMMANDS="$@" /usr/bin/ssh -i $KEY_FILE -l root -F $SSH_OPTS $COMMANDS } for volume in $( ssh_fas $FAS_CTRL vol status | grep "vol.* online" | awk '{ print $1 }' ); do VOL_SIZE="$( ssh_fas $FAS_CTRL vol size $volume | awk '{ print $8 }' | sed "s,.,," )" VOL_UNIT_PARAM="$( echo "${#VOL_SIZE} - 1" | bc )" VOL_UNIT="${VOL_SIZE:$VOL_UNIT_PARAM:1}" case $VOL_UNIT in k) CONV="1024";; m) CONV="1024 * 1024";; g) CONV="1024 * 1024 * 1024";; esac VOL_SIZE="$( echo "scale=0; ${VOL_SIZE/$VOL_UNIT/} * $CONV" | bc )" # Subtract the snap reserve (if any) SNAP="$( ssh_fas $FAS_CTRL snap reserve $volume | cut -d -f7 | sed "s,%,," )" if [ $SNAP -gt 0 ] ; then SNAP_SIZE="$( echo "$VOL_SIZE * $SNAP / 100" | bc )" REMAINING_VOL_SIZE="$( echo "$VOL_SIZE - $SNAP_SIZE" | bc )" else SNAP_SIZE=0 REMAINING_VOL_SIZE=$VOL_SIZE fi # Get the luns created inside the volume LUN_SIZE_TOTAL=0 for lunsize in $( ssh_fas $FAS_CTRL lun show -l ${volume} | awk '{ print $3 }' | sed -e "s,(,," -e "s,),," ); do LUN_SIZE_TOTAL="$( echo "$LUN_SIZE_TOTAL + $lunsize" | bc )" REMAINING_VOL_SIZE="$( echo "$REMAINING_VOL_SIZE - $lunsize" | bc )" done # Get the estimated space necessary. VOL_RESERVE="$( echo "$VOL_SIZE * 3 / 100" | bc )" echo " VOL_NAME: $volume" echo " VOLUME_SIZE: $( echo "scale=2; $VOL_SIZE / 1024 / 1024 / 1024" | bc )G" echo " - SNAP_SIZE: $( echo "scale=2; $SNAP_SIZE / 1024 / 1024 / 1024" | bc )G" echo " SUB_TOTAL: $( echo "scale=2; ($VOL_SIZE - $SNAP_SIZE) / 1024 / 1024 / 1024" | bc )G" echo echo " - TOTAL_LUN_SIZE: $( echo "scale=2; $LUN_SIZE_TOTAL / 1024 / 1024 / 1024" | bc )G" echo " SUB_TOTAL: $( echo "scale=2; ($VOL_SIZE - $SNAP_SIZE - $LUN_SIZE_TOTAL) / 1024 / 1024 / 1024" | bc )G" echo echo " MISSING_SPACE:" DIFFERENCE_VOL_LUN="$( echo "scale=2; ($VOL_SIZE - $SNAP_SIZE - $LUN_SIZE_TOTAL)" | bc | sed "s,-,," )" echo " DIFFERENCE_VOL_LUN: $( echo "scale=2; $DIFFERENCE_VOL_LUN / 1024 / 1024 / 1024" | bc )G" echo " + VOL_RESERVE: $( echo "scale=2; $VOL_RESERVE / 1024 / 1024 / 1024" | bc )G" echo " SUB_TOTAL: $( echo "scale=2; ($DIFFERENCE_VOL_LUN + $VOL_RESERVE) / 1024 / 1024 / 1024" | bc )G" echo echo echo done And with that you have a list of volumes, with the amount of space they need to resized in order to accomodate the contained LUNs and the snapshots.

Well, we had some issues with XenServers “automated” metadata backup, so I decided - with the help of one of our consultants - to automate it on our own to an external target. 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 #!/bin/bash # Crontab entry for each server: # 02 5 * * * root /usr/local/sbin/xen-pool-backup.sh # Get the pool name POOL_NAME="$( xe pool-list | grep name-label | awk '{ print $4 }' )" HOST_UUID="$( xe host-list hostname=`hostname` | grep "uuid ( RO)" | awk '{ print $5 }' )" DAILY_GENERATIONS=7 WEEKLY_GENERATIONS=4 NFS_MOUNT="nfs.home.barfoo.org:/srv/xenbackup" NFS_LOCAL="/tmp/backup-mount/$POOL_NAME" # Figure out if we're the pool master POOL_MASTER="$( xe pool-list | grep master | awk '{ print $4 }' )" if [ "$POOL_MASTER" == "$HOST_UUID" ] ; then # Only the pool master should backup the pool database, as this is the only # one who has a authoritive pool database # Create the necessary directory and mount the NFS volume mkdir -p ${NFS_LOCAL%/*} mount -t nfs $NFS_MOUNT ${NFS_LOCAL%/*} mkdir -p $NFS_LOCAL if [ -f $NFS_LOCAL/daily.$DAILY_GENERATIONS.gz ]; then rm -f $NFS_LOCAL/daily.$DAILY_GENERATIONS.gz fi OLD_DAILY="$( echo "scale=0; $DAILY_GENERATIONS - 1" | bc )" for OLD in $( seq $OLD_DAILY -1 1 ); do if [ -f $NFS_LOCAL/daily.$OLD.gz ] ; then NEW="$( echo "scale=0; $OLD+1" | bc )" # Save the time stamp somewhere touch $NFS_LOCAL/.timestamp -r $NFS_LOCAL/daily.$OLD.gz mv $NFS_LOCAL/daily.$OLD.gz $NFS_LOCAL/daily.$NEW.gz # Restore the date touch $NFS_LOCAL/daily.$NEW.gz -r $NFS_LOCAL/.timestamp fi done [ -f $NFS_LOCAL/daily.0.gz ] && mv $NFS_LOCAL/daily.0.gz $NFS_LOCAL/daily.1.gz xe pool-dump-database file-name=$NFS_LOCAL/daily.0 gzip -9 $NFS_LOCAL/daily.0 [ -f $NFS_LOCAL/.timestamp ] && rm $NFS_LOCAL/.timestamp umount ${NFS_LOCAL%/*} rm -rf ${NFS_LOCAL%/*} fi With that, I have at least a daily backup - and in combination with our daily TSM backup, I have at least month long history of metadata backups.

Well, I had another task for today … I have an amount of FlexVolumes (sixty currently per controller), and I didn’t know if we had any, that didn’t have any LUNs on them. Now I thought there was a command for that since my co-worker mentioned something like that. However, once again … there isn’t. 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 #!/bin/bash MAILTO="san@barfoo.org" KEY_FILE="/root/.ssh/netapp.dsa" SSH_OPTS="/root/.ssh/netapp-ssh_config" FAS_CTRL=$1 TMPDIR="$( mktemp -d )" ssh_fas() { # $@: commands for Data ONTAP COMMANDS="$@" /usr/bin/ssh -i $KEY_FILE -l root -F $SSH_OPTS $COMMANDS } # Get the hostname of the controller, necessary for the reporting CTRL_HOSTNAME="$( ssh_fas $FAS_CTRL rdfile /etc/rc | grep ^hostname | cut -d -f2 | tr 'a-z' 'A-Z' )" # Get a list of all volumes / luns VOL_LIST="$( ssh_fas $FAS_CTRL vol status | grep ^vol | egrep -v '(nfs|cifs)' | awk '{ print $1 }' )" LUN_LIST="$( ssh_fas $FAS_CTRL lun show | grep '/vol' | awk '{ print $1 }' )" for lun in $LUN_LIST; do VOL_EXTRACT="$( echo $lun | cut -d/ -f3 )" VOL_LIST=${VOL_LIST/${VOL_EXTRACT}/} done for vol in $VOL_LIST; do echo "Empty Flex Volume: $vol." done > $TMPDIR/mailcontent if [ "$( grep Flex $TMPDIR/mailcontent )" ] ; then cat $TMPDIR/mailcontent | mailx -r $MAILTO -s "$CTRL_HOSTNAME: Empty volume check" $MAILTO fi rm -r $TMPDIR

Well, once again I was presented with a nice AutoSupport warning once I logged into my NOW account. Since we don’t have CIFS and/or NFS licensed on our filers, I wrote a cute little script that does the whole work for me. 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 #!/bin/bash TMPDIR="$( mktemp -d )" KEY_FILE="/root/.ssh/netapp.dsa" SSH_OPTS="/root/.ssh/netapp-ssh_config" FAS_CTRL=$1 QUALDEVICES=$2 ssh_fas() { # $@: commands for Data ONTAP COMMANDS="$@" /usr/bin/ssh -i $KEY_FILE -l root -F $SSH_OPTS $COMMANDS } #set -x echo "Updating qual_devices on $FAS_CTRL" # Enable ftpd if it isn't enabled already echo -n "Checking FTP subsystem" FTPD_INITIAL_STATE="$( ssh_fas $FAS_CTRL options ftpd.enable | awk '{ print $2 }' )" if [ $FTPD_INITIAL_STATE == "off" ] ; then ssh_fas $FAS_CTRL options ftpd.enable on echo " .... ok" else echo " .... ok" fi echo read -s -p "Please supply the root password for $FAS_CTRL: " ROOT_PASSWD=$REPLY echo echo "Checking qual_devices:" echo -n " Running: " mkdir $TMPDIR/fas-version # Check the old version. ftp -n $FAS_CTRL >/dev/null <<END_SCRIPT prompt user root $ROOT_PASSWD lcd $TMPDIR/fas-version cd /etc mget qual_devices* quit END_SCRIPT FAS_VERSION="$( grep Datecode $TMPDIR/fas-version/qual_devices_v3 | head -n1 | cut -d -f3 )" echo "$FAS_VERSION" # Unzip the qual_devices.zip file and compare it. ORIGPWD=$PWD mkdir $TMPDIR/new-version cd $TMPDIR/new-version unzip $QUALDEVICES >/dev/null echo -n " New: " NEW_VERSION="$( grep Datecode $TMPDIR/new-version/qual_devices_v3 | head -n1 | cut -d -f3 )" echo "$NEW_VERSION" echo # Upload the supplied version if the new file doesn't match the one running # on the controller if [ $NEW_VERSION != $FAS_VERSION ] ; then echo "Uploading qual_devices to $FAS_CTRL" ftp -n $FAS_CTRL >/dev/null <<END_SCRIPT prompt user root $ROOT_PASSWD lcd $TMPDIR/new-version cd /etc mput qual_devices* quit END_SCRIPT # Send an asup message, that the issue is corrected echo "Generating AutoSupport message" ssh_fas $FAS_CTRL options autosupport.doit qual_devices_fixed_$NEW_VERSION fi #set +x # Enable ftpd if [ $FTPD_INITIAL_STATE == "off" ] ; then ssh_fas $FAS_CTRL options ftpd.enable off fi rm -r $TMPDIR The whole thing is surly based, that FTP is configured ( as I described previously).

Well, I recently started backing up my not-copyrighted DVDs to hard drive; however the language is sometimes mixed up. In some English is the first, in others it’s German. So, I’ve come up with this little script which fixes it - and puts each file into order. 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 #!/bin/bash [ -x /usr/bin/bc ] || exit 1 [ -x /usr/bin/mkvmerge ] || exit 1 [ -x /usr/bin/mediainfo ] || exit 1 for file in $1/*.mkv; do # First, run mediainfo on the file. TRACK_ORDER="$( mediainfo '--Output=Audio;%ID/String% %Language%x' $file | grep -v '^$' )" TRACK1_ID="$( echo $TRACK_ORDER | cut -dx -f1 | cut -d&nbsp; -f1 )" TRACK1_LANG="$( echo $TRACK_ORDER | cut -dx -f1 | cut -d&nbsp; -f2 )" TRACK2_ID="$( echo $TRACK_ORDER | cut -dx -f2 | cut -d&nbsp; -f1 )" TRACK2_LANG="$( echo $TRACK_ORDER | cut -dx -f2 | cut -d&nbsp; -f2 )" case $TRACK1_LANG in "de") NEEDS_FIX=1 ;; *) NEEDS_FIX=0 ;; esac if [ $NEEDS_FIX -eq 1 ] ; then OPTS_1="--language $TRACK1_ID:ger --track-name $TRACK1_ID:German" OPTS_2="--default-track $TRACK1_ID:no --forced-track $TRACK1_ID:no" OPTS_3="--language $TRACK2_ID:eng --track-name $TRACK2_ID:English" OPTS_4="--default-track $TRACK2_ID:yes --forced-track $TRACK2_ID:no" OPTS_5="--track-order 0:1,0:$TRACK2_ID,0:$TRACK1_ID" OPTS="$OPTS_1 $OPTS_2 $OPTS_3 $OPTS_4 $OPTS_5" else OPTS_1="--language $TRACK2_ID:ger --track-name $TRACK2_ID:German" OPTS_2="--default-track $TRACK2_ID:no --forced-track $TRACK2_ID:no" OPTS_3="--language $TRACK1_ID:eng --track-name $TRACK1_ID:English" OPTS_4="--default-track $TRACK1_ID:yes --forced-track $TRACK1_ID:no" OPTS="$OPTS_1 $OPTS_2 $OPTS_3 $OPTS_4" fi mkvmerge -o $file.new --language "1:eng" --default-track "1:yes" --forced-track "1:no" --audio-tracks "$TRACK1_ID,$TRACK2_ID" --video-tracks "1" --no-subtitles --no-track-tags --no-global-tags $OPTS $file done

Well, we’ve been trying to come up with a decent way to backup NetApp snapshots to tape (SnapMirror To Tape), so we evaluated all the available methods of using NDMP backups. There’s Image Backup in two different variants - FULL and DIFFerntial There’s SnapMirror To Tape So the Image Backup is one of the ways. However the DIFFerntial backup only works for CIFS and NFS shares (which we don’t use). We only have FC luns (or rather FCoE luns), so there’s only a single (or in case of the boot luns more than one) file in each volume. With that however, each run of the Image Backup with the DIFFerential option, it’s gonna backup the full size of the volume (plus the deduplicated amount). ...

Well, on Friday I had a short chat with someone from one of our application departments, stating he wanted a backup copy of a VM (ain’t to hard), but a) they don’t want any downtime and b) it has to be identical to the original. So I sat down today, googled for a bit and actually found something that pretty much does what I want, though I had to fix it up a bit … So find attached a script, which creates a hot-clone from a snapshot and then only if the latest clone was successful deletes the old one. ...

Well, to save everyone else the trouble (since it isn’t documented anywhere - and I just spent about an hour finding the cause for this), if you need to configure NDMP on your NetApp Filer, make sure you also configure an interface other than e0M. Apparently the necessary controlport for NDMP (10000) is being blocked on e0M, thus ndmp may be configured and running, however TSM is gonna complain that it is unable to connect to the specified data mover. ...