Work

Well, I have yet another weird UCS problem. I have a single blade, that has trouble with it’s primary fabric attachment. The problem get’s even more weird, if you look at the details. After looking at the IO modules, the error doesn’t become any clearer: So far, I have tried nearly everything. I’ve tried resetting the active and passive Connectivity of the vNIC, I tried resetting the DCE adapter for the vNIC, but nothing. I even tried resetting the vHBA that’s associated with this fabric, but that didn’t result to anything. Not even the usual flogi (fibre channel login) errors, that you get when either booting/resetting the blade. ...

Well, I recently had yet another UCS display/I2C communication problem. Somehow one of my chassis’ started to think, that the power redundancy was lost. After looking at it a bit deeper, it seems only the GUI or the chassis did notice this power glitch: As you can see, all PSU’s still have power. Now, since I had a big maintainance window the last weekend anyhow (and I spent ~14 hours at work), I decided to restart the IO modules in that chassis. And guess what: The error is gone! Another weird I2C communication issue with the firmware release 2.0.2 …

Well, I’ve been having issues with our SMT. Basically after the release of SLES11 SP2 in February, I’ve been waiting for the repositories to turn “mirrorable”, which they haven’t yet. So I wrote a mail to the EMEA customer support. And yesterday I got a reply, stating that my mirror credentials had been regenerated. And guess what ? I can mirror stuff again. So if you ever have any issues like the ones I described (or any of the people in the forum threads experienced), write a mail to your Novell customer support. 😛

Well, we recently upgraded the SnapManager version on our test box to 6.4.1. Now however, after restarting the box the SnapManager service failed to start … The error was something like this: Now, first I stumbled upon this NetApp Community post, which only contained the “solution” to increase the global! wait time for services. That didn’t sit well with me. So after looking through NOW! for a bit, I actually found the correct way. The fix is described in KB2010835. Yet again, another certificate error. Why do vendors deploy SSL certificates, when they use untrusted ones, which defeats the purpose of SSL certs or at least “brings up” users to ignore any error message they get concerning SSL certificates ? ...

In a Total Commander file list press Shift+Alt+Enter to calculate the directory size of all sub-directories.

Here’s an old script, that also uses the magic provided by pxe-menu-generation (the script I posted before), but for VMware ESX/ESXi. 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 #!/bin/bash TFTP_DIR=/srv/tftp INSTSRC_DIR=/srv/instsrc if [ -z $@ ] ; then echo "Usage: register-vmware <path to iso-file> labelname" exit 1 fi #set -x ISO_FILE=$1 mkdir /mnt/loop mount -o loop $ISO_FILE /mnt/loop if [ -f /mnt/loop/packages.xml ] ; then # Determine ISO information OS_LABEL="$( grep "<name>" /mnt/loop/packages.xml | sed -e "s,<name>,," -e "s,</name>,," )" OS_NAME="$( grep "<ProductLineId>" /mnt/loop/packages.xml | sed -e "s,<ProductLineId>,," -e "s,</ProductLineId>,," )" OS_RELEASE="$( grep "<version>" /mnt/loop/packages.xml | sed -e "s,<version>,," -e "s,</version>,," )" OS_BUILD="$( grep "<release>" /mnt/loop/packages.xml | sed -e "s,<release>,," -e "s,</release>,," )" shopt -s extglob OS_LABEL=${OS_LABEL##*( )} OS_NAME=${OS_NAME##*( )} OS_RELEASE=${OS_RELEASE##*( )} OS_BUILD=${OS_BUILD##*( )} elif [ -f /mnt/loop/README.txt ] ; then OS_LABEL="" OS_NAME="" OS_RELEASE="" OS_BUILD="" fi # Create directories mkdir -p $TFTP_DIR/boot/$OS_NAME/$OS_RELEASE-$OS_BUILD mkdir -p $TFTP_DIR/pxelinux.cfg/$OS_NAME mkdir -p $INSTSRC_DIR/$OS_NAME/$OS_RELEASE-$OS_BUILD #set +x # Copy CD/DVD content rsync -av /mnt/loop/* $INSTSRC_DIR/$OS_NAME/$OS_RELEASE-$OS_BUILD # Copy the PXE boot file (initrd/kernel) cp $INSTSRC_DIR/$OS_NAME/$OS_RELEASE-$OS_BUILD/isolinux/initrd.img $TFTP_DIR/boot/$OS_NAME/$OS_RELEASE-$OS_BUILD/initrd.img cp $INSTSRC_DIR/$OS_NAME/$OS_RELEASE-$OS_BUILD/isolinux/vmlinuz $TFTP_DIR/boot/$OS_NAME/$OS_RELEASE-$OS_BUILD/vmlinuz # Create the PXE boot menu cat > $TFTP_DIR/pxelinux.cfg/$OS_NAME/$OS_NAME-$OS_RELEASE-$OS_BUILD.menu << EOF #LABEL $OS_LABEL $OS_RELEASE - build $OS_BUILD #OSNAME $OS_NAME #OSARCH x86_64 MENU TITLE VMware Installservice MENU BACKGROUND addons/background-vmware.png MENU COLOR screen 37;40 #80ffffff #00000000 MENU COLOR border 0 #ffffffff #ee000000 std MENU COLOR title 0 #ffffffff #ee000000 std MENU COLOR unsel 0 #ffffffff #ee000000 std MENU COLOR sel 0 #ffffffff #85000000 std MENU COLOR scrollbar 30;44 #40000000 #00000000 MENU COLOR tabmsg 0 #ee000000 #ffffffff std MENU COLOR cmdmark 0 #ff00ff00 #00000000 std MENU COLOR cmdline 0 #ee000000 #ffffffff std MENU COLOR timeout_msg 0 #ee000000 #ffffffff std MENU COLOR timeout 0 #ee000000 #ffffffff std MENU COLOR disabled 0 #ffffffff #ee000000 std MENU COLOR pwdheader 0 #ff000000 #99ffffff rev MENU COLOR pwdborder 0 #ff000000 #99ffffff rev MENU COLOR pwdentry 0 #ff000000 #99ffffff rev MENU COLOR hotkey 0 #ff00ff00 #ee000000 std MENU COLOR hotsel 0 #ffffffff #85000000 std LABEL DESC MENU LABEL $OS_LABEL $OS_RELEASE MENU DISABLE MENU SEPARATOR LABEL kickstart MENU LABEL ^Kickstart Installation KERNEL boot/$OS_NAME/$OS_RELEASE-$OS_BUILD/vmlinuz APPEND initrd=boot/$OS_NAME/$OS_RELEASE-$OS_BUILD/initrd.img vmkopts=debugLogToSerial:1 mem=512M ks=http://install.home.barfoo.org/kickstart/$OS_NAME-$OS_RELEASE-$OS_BUILD.cfg HOSTNAME= IAPPEND 1 TEXT HELP Es ist zu beachten, das mit TAB der Hostname angepasst werden muss! ENDTEXT LABEL netinst MENU LABEL ^Netzwerkbasierte Installation KERNEL boot/$OS_NAME/$OS_RELEASE-$OS_BUILD/vmlinuz APPEND initrd=boot/$OS_NAME/$OS_RELEASE-$OS_BUILD/initrd.img vmkopts=debugLogToSerial:1 mem=512M url=http://install.home.barfoo.org/instsrc/$OS_NAME/$OS_RELEASE-$OS_BUILD IAPPEND 1 MENU SEPARATOR LABEL back MENU LABEL <-- ^Vorherige Ansicht KERNEL addons/vesamenu.c32 APPEND pxelinux.cfg/$OS_NAME.menu EOF umount /mnt/loop rmdir /mnt/loop

Well, I’ve been tinkering with a way to “unify” the way we deploy new OpenSUSE/SLES ISOs onto our installation server, once they are released. So here, I’ll show you the script I came up with: 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 #!/bin/bash TFTP_DIR=/srv/tftpboot INSTSRC_DIR=/srv/instsrc if [ -z $@ ] ; then echo "Usage: register-suse <path to iso-file> labelname" exit 1 fi ISO_FILE=$1 mkdir /mnt/loop mount -o loop $ISO_FILE /mnt/loop # Determine ISO information OS_LABEL="$( grep '^LABEL' /mnt/loop/content | cut -d -f10- )" if [ -z "$OS_LABEL" ] ; then OS_LABEL="$( grep '^LABEL' /mnt/loop/content | cut -d -f2- | head -n1 )" fi OS_NAME="$( grep '^DISTRIBUTION' /mnt/loop/content | cut -d -f3- | tr '[:upper:]' '[:lower:]' )" if [ -z "$OS_NAME" ] ; then OS_NAME="$( grep '^PRODUCT' /mnt/loop/content | cut -d -f2- | cut -d_ -f1-2 | tr '[:upper:]' '[:lower:]' )" fi case "$OS_NAME" in suse_sle|suse_sles) OS_NAME=sles;; esac OS_RELEASE="$( grep ^VERSION /mnt/loop/content | awk '{ print $2 }' | cut -d- -f1 )" if [ "$( echo $OS_LABEL | grep -i vmware )" != "" ] ; then OS_RELEASE="$OS_RELEASE-vmware" fi OS_ARCH="$( grep ^BASEARCHS /mnt/loop/content | awk '{ print $2 }' )" if [ -z "$OS_ARCH" ] ; then OS_ARCH="$( grep ^DEFAULTBASE /mnt/loop/content | awk '{ print $2 }' )" fi OS_SHORTLABEL="$( grep ^SHORTLABEL /mnt/loop/content | awk '{ print $2 }' | cut -d( -f1 | sed -e "s,SLES-,SLES," -e "s,-, ,g" )" case "$OS_ARCH" in i*86) OS_ARCH=x86;; x86_64) OS_ARCH=x64;; esac # Create directories mkdir -p $TFTP_DIR/boot/$OS_NAME/$OS_RELEASE/$OS_ARCH mkdir -p $TFTP_DIR/pxelinux.cfg/$OS_NAME mkdir -p $INSTSRC_DIR/$OS_NAME/$OS_RELEASE/$OS_ARCH # Copy CD/DVD content rsync -av /mnt/loop/* $INSTSRC_DIR/$OS_NAME/$OS_RELEASE/$OS_ARCH/ # Create the info file cat > $INSTSRC_DIR/$OS_NAME/$OS_RELEASE/$OS_ARCH/info <<EOF Language: en_US Keytable: de-lat1-nd Install: http://install.home.barfoo.org/{INSTSRC_DIR//srv/}/$OS_NAME/$OS_RELEASE/$OS_ARCH/ Autoyast: http://install.home.barfoo.org/autoyast/ Textmode: 1 EOF # Copy the PXE boot file (initrd/kernel) cp $INSTSRC_DIR/$OS_NAME/$OS_RELEASE/$OS_ARCH/boot/*/loader/initrd $TFTP_DIR/boot/$OS_NAME/$OS_RELEASE/$OS_ARCH/initrd cp $INSTSRC_DIR/$OS_NAME/$OS_RELEASE/$OS_ARCH/boot/*/loader/linux $TFTP_DIR/boot/$OS_NAME/$OS_RELEASE/$OS_ARCH/linux # Create the PXE boot menu cat > $TFTP_DIR/pxelinux.cfg/$OS_NAME/$OS_NAME-$OS_RELEASE-$OS_ARCH.menu << EOF #LABEL $OS_SHORTLABEL #OSNAME $OS_NAME #OSARCH $OS_ARCH MENU TITLE Linux Installationservices MENU BACKGROUND addons/background-suse.png MENU COLOR screen 37;40 #80ffffff #00000000 MENU COLOR border 0 #ffffffff #ee000000 std MENU COLOR title 0 #ffffffff #ee000000 std MENU COLOR unsel 0 #ffffffff #ee000000 std MENU COLOR sel 0 #ffffffff #85000000 std MENU COLOR scrollbar 30;44 #40000000 #00000000 MENU COLOR tabmsg 0 #ee000000 #ffffffff std MENU COLOR cmdmark 0 #ff00ff00 #00000000 std MENU COLOR cmdline 0 #ee000000 #ffffffff std MENU COLOR timeout_msg 0 #ee000000 #ffffffff std MENU COLOR timeout 0 #ee000000 #ffffffff std MENU COLOR disabled 0 #ffffffff #ee000000 std MENU COLOR pwdheader 0 #ff000000 #99ffffff rev MENU COLOR pwdborder 0 #ff000000 #99ffffff rev MENU COLOR pwdentry 0 #ff000000 #99ffffff rev MENU COLOR hotkey 0 #ff00ff00 #ee000000 std MENU COLOR hotsel 0 #ffffffff #85000000 std LABEL DESC MENU LABEL $OS_LABEL ($OS_ARCH) MENU DISABLE MENU SEPARATOR LABEL autoyast MENU LABEL ^Automatisierte Installation KERNEL boot/$OS_NAME/$OS_RELEASE/$OS_ARCH/linux APPEND initrd=boot/$OS_NAME/$OS_RELEASE/$OS_ARCH/initrd info=http://install.home.barfoo.org/${INSTSRC_DIR//srv/}/$OS_NAME/$OS_RELEASE/$OS_ARCH/info splash=native ramdisk_size=65535 vga=791 barrier=off LABEL netinst MENU LABEL ^Netzwerkbasierte Installation KERNEL boot/$OS_NAME/$OS_RELEASE/$OS_ARCH/linux APPEND initrd=boot/$OS_NAME/$OS_RELEASE/$OS_ARCH/initrd install=http://install.home.barfoo.org/${INSTSRC_DIR//srv/}/$OS_NAME/$OS_RELEASE/$OS_ARCH/ splash=native ramdisk_size=65535 vga=791 barrier=off MENU SEPARATOR LABEL back MENU LABEL <-- ^Vorherige Ansicht KERNEL addons/vesamenu.c32 APPEND pxelinux.cfg/$OS_NAME.menu EOF umount /mnt/loop rmdir /mnt/loop This also needs a second script, as this will only create a menu entry, not the menu itself. You’ll find the script in the next post!

Well, I’m back at work after three weeks of vacation (some pictures to follow) and the provider hosting our disaster datacenter had their annual (or is it monthly now?) SAN maintainance, so we shut down everything over there by 9:00 am. After things were back up around 5pm, I booted the ESX hosts, however the VMs we’re all displaying the alert state - as if either the VMs had an HA event or we’re using to much CPU time. It didn’t matter whether or not the VM was running or not, the state persisted. ...

As promised in the earlier post, for completeness sake, here’s the counterpart for removing the LUNs in the first place. 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 #!/bin/bash KEY_FILE="/root/.ssh/netapp.dsa" SSH_OPTS="/root/.ssh/netapp-ssh_config" if [ $# -ne 2 ] ; then echo "fas-remove-lunmap: FAS_CONTROLLER TARGET_IGROUP" echo echo "Remove every mapped lun from the target igroup (for example reinstallation)" echo echo " Usage:" echo " - FAS_CONTROLLER: Hostname/IP-adress of the DATA ONTAP controller" echo " - TARGET_IGROUP: igroup that is actually modified" echo exit 1 fi FAS_CTRL=$1 TARGET=$2 ssh_fas() { # $@: commands for Data ONTAP COMMANDS="$@" /usr/bin/ssh -i $KEY_FILE -l root -F $SSH_OPTS $COMMANDS } # Get the hostname of the controller, necessary for the reporting CTRL_HOSTNAME="$( ssh_fas $FAS_CTRL rdfile /etc/rc | grep ^hostname | cut -d -f2 | tr 'a-z' 'A-Z' )" #set -x # Get the lun list. for lun in $( ssh_fas $FAS_CTRL lun show -g $SOURCE | awk '{ print $1 }' | sort -u ); do # If the LUN id is 0, skip otherwise we would remove the boot LUN if [ "$LUN_ID" != "0" ] ; then # Actually remove the mapping to our host echo "Removing $lun from $TARGET" ssh_fas $FAS_CTRL lun unmap $lun $TARGET fi done #set +x With that, you can simply run it against a NetApp controller and remove every LUN map except the one with LUN ID 0 (which is pretty handy when installing/reinstalling ESX servers).

Well, after figuring things out (and realizing that if you create a LUN in the same size as the volume it’ll break), I decided to write yet another script to figure out which LUNs needed fixing. 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 #!/bin/bash KEY_FILE="/root/.ssh/netapp.dsa" SSH_OPTS="/root/.ssh/netapp-ssh_config" FAS_CTRL=$1 ssh_fas() { # $@: commands for Data ONTAP COMMANDS="$@" /usr/bin/ssh -i $KEY_FILE -l root -F $SSH_OPTS $COMMANDS } for volume in $( ssh_fas $FAS_CTRL vol status | grep "vol.* online" | awk '{ print $1 }' ); do VOL_SIZE="$( ssh_fas $FAS_CTRL vol size $volume | awk '{ print $8 }' | sed "s,.,," )" VOL_UNIT_PARAM="$( echo "${#VOL_SIZE} - 1" | bc )" VOL_UNIT="${VOL_SIZE:$VOL_UNIT_PARAM:1}" case $VOL_UNIT in k) CONV="1024";; m) CONV="1024 * 1024";; g) CONV="1024 * 1024 * 1024";; esac VOL_SIZE="$( echo "scale=0; ${VOL_SIZE/$VOL_UNIT/} * $CONV" | bc )" # Subtract the snap reserve (if any) SNAP="$( ssh_fas $FAS_CTRL snap reserve $volume | cut -d -f7 | sed "s,%,," )" if [ $SNAP -gt 0 ] ; then SNAP_SIZE="$( echo "$VOL_SIZE * $SNAP / 100" | bc )" REMAINING_VOL_SIZE="$( echo "$VOL_SIZE - $SNAP_SIZE" | bc )" else SNAP_SIZE=0 REMAINING_VOL_SIZE=$VOL_SIZE fi # Get the luns created inside the volume LUN_SIZE_TOTAL=0 for lunsize in $( ssh_fas $FAS_CTRL lun show -l ${volume} | awk '{ print $3 }' | sed -e "s,(,," -e "s,),," ); do LUN_SIZE_TOTAL="$( echo "$LUN_SIZE_TOTAL + $lunsize" | bc )" REMAINING_VOL_SIZE="$( echo "$REMAINING_VOL_SIZE - $lunsize" | bc )" done # Get the estimated space necessary. VOL_RESERVE="$( echo "$VOL_SIZE * 3 / 100" | bc )" echo " VOL_NAME: $volume" echo " VOLUME_SIZE: $( echo "scale=2; $VOL_SIZE / 1024 / 1024 / 1024" | bc )G" echo " - SNAP_SIZE: $( echo "scale=2; $SNAP_SIZE / 1024 / 1024 / 1024" | bc )G" echo " SUB_TOTAL: $( echo "scale=2; ($VOL_SIZE - $SNAP_SIZE) / 1024 / 1024 / 1024" | bc )G" echo echo " - TOTAL_LUN_SIZE: $( echo "scale=2; $LUN_SIZE_TOTAL / 1024 / 1024 / 1024" | bc )G" echo " SUB_TOTAL: $( echo "scale=2; ($VOL_SIZE - $SNAP_SIZE - $LUN_SIZE_TOTAL) / 1024 / 1024 / 1024" | bc )G" echo echo " MISSING_SPACE:" DIFFERENCE_VOL_LUN="$( echo "scale=2; ($VOL_SIZE - $SNAP_SIZE - $LUN_SIZE_TOTAL)" | bc | sed "s,-,," )" echo " DIFFERENCE_VOL_LUN: $( echo "scale=2; $DIFFERENCE_VOL_LUN / 1024 / 1024 / 1024" | bc )G" echo " + VOL_RESERVE: $( echo "scale=2; $VOL_RESERVE / 1024 / 1024 / 1024" | bc )G" echo " SUB_TOTAL: $( echo "scale=2; ($DIFFERENCE_VOL_LUN + $VOL_RESERVE) / 1024 / 1024 / 1024" | bc )G" echo echo echo done And with that you have a list of volumes, with the amount of space they need to resized in order to accomodate the contained LUNs and the snapshots.