Work on BAFM

Create a Proxmox VM via CLI

Tue, 21 Jun 2022 23:09:44 +0000

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23


# Clone the VM
qm clone 9000 4140 --name nzbget.ka.beiheimdaheim.de --full=true --storage vmstorage

# Set the nameserver
qm set 4140 --nameserver=10.76.41.1

# Set the ipconfiguration
qm set 4140 --ipconfig0 ip=10.76.41.40/24,gw=10.76.41.1

# Configure the virtual network adapter.
qm set 4140 --net0 virtio,bridge=vmbr0,tag=41

# Enable autoboot
qm set 4140 --onboot 1

# Set the RAM of the VM
qm set 4140 --memory 2048

# Set the vCPU count for the VM
qm set 4140 --cores 2

# Start the VM
qm start 4140

Create a Proxmox Ubuntu Cloud-Init Template

Tue, 21 Jun 2022 19:02:19 +0000

Step 1: Download the Ubuntu image

1
2
3
4
5
6


SRC_IMG="https://cloud-images.ubuntu.com/jammy/current/jammy-server-cloudimg-amd64-disk-kvm.img"
#VMSTORAGE="/vmstorage"
VMSTORAGE="/var/lib/vz/images"
IMG_NAME="jammy-server-cloudimg-amd64-disk-kvm.qcow2"
IMG_PATH="$VMSTORAGE/$IMG_NAME"
wget -O $IMG_PATH $SRC_IMG

Step 2: Add necessary packages

1
2


sudo apt install -y libguestfs-tools
sudo virt-customize -a $IMG_PATH --install cloud-initramfs-growroot,atop,htop,nano,vim,qemu-guest-agent,curl,wget,unattended-upgrades,git

Step 3: Modify /etc/ssh/sshd_config

1

sudo virt-customize -a $IMG_PATH --run-command "sed -i 's/.*PasswordAuthentication.*/PasswordAuthentication yes/g' /etc/ssh/sshd_config"

Step 4: Modify `/etc/apt/apt.conf.d/50unattended-upgrades`

1
2


export EDITOR=nano
sudo -E virt-edit -a $IMG_PATH /etc/apt/apt.conf.d/50unattended-upgrades

Step 5: Modify `/etc/default/grub`

1
2
3


export EDITOR=nano
sudo -E virt-edit -a $IMG_PATH /etc/default/grub
sudo virt-customize -a $IMG_PATH --run-command "update-grub"

Step 6: Install `ohmybash`

1
2

sudo virt-customize -a $IMG_PATH --run-command "git clone https://github.com/ohmybash/oh-my-bash.git /opt/ohmybash"

Step 7: Fix timezone

1

sudo virt-customize -a $IMG_PATH --timezone "Europe/Berlin"

Step 8: Create the Proxmox template

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19


TEMPL_NAME="ubuntu2204-cloud"
VMID="9000"
MEM="512"
DISK_SIZE="10G"
DISK_STOR="vmstorage"
NET_BRIDGE="vmbr0"
qm create $VMID --name $TEMPL_NAME --memory $MEM --net0 virtio,bridge=$NET_BRIDGE
qm importdisk $VMID $IMG_NAME $DISK_STOR
qm set $VMID --agent enabled=1
qm set $VMID --scsihw virtio-scsi-pci --scsi0 $DISK_STOR:vm-$VMID-disk-0
qm set $VMID --ide2 $DISK_STOR:cloudinit
qm set $VMID --boot c --bootdisk scsi0
qm set $VMID --serial0 socket --vga serial0
qm set $VMID --ipconfig0 ip=dhcp
qm set $VMID --ciuser=christian
qm set $VMID --cipassword="bla"
qm set $VMID --searchdomain=ka.beiheimdaheim.de
qm resize $VMID scsi0 $DISK_SIZE
qm template $VMID

Nagios: Service Check Timed Out

Fri, 07 Dec 2018 21:24:54 +0000

Since I got the pleasure of watching some Windows boxen with Nagios, I took the Windows Update plugin from Michal Jankowski and implemented it. It took me some time, to initially set up the nsclient++ correctly so it just works, but up till now the check plugin sometimes reported the usual “Service Check Timed Out”.

Usually I ended up increasing the cscript timeout, or the nsclient++ socket timeout, but it still kept showing up. Since I rely heavily on my surveillance tools, I have the demand, that as few as possible false positives show up. So I ended up chasing down this error today, and after that I have to say it was quite simple.

Mass-updating Tivoli Storage Manager drive status

Wed, 11 Oct 2017 19:48:40 +0000

I was fighting with our VTL again, and TSM was thinking all the drives were offline. In order to update the drive status, you’d need to go into the ISC and select each drive and set them to ONLINE. Since I’m a bit click-lazy, I wrote a simple nested for-loop, which gives me the output to update all the drives at once:

1
2
3
4
5


for i in 1 2; do
 for k in $( seq -w 1 32 ); do
 echo "UPDATE DRIVE VTL$i VTL${i}_DR${k} ONLINE=YES"
 done
done

Result is a list like this:

VMware Auto Deploy already registered / RuleEngine

Sun, 03 Sep 2017 19:19:28 +0000

I had this weird plugin error the other day which bothered me on Friday. I decided to go fixing it. So after poking around in the vCenter installed software list. I couldn’t find the Auto Deploy in the list, so I figured due to my recent vCenter reinstallation while keeping the database, I forgot to reinstall Auto Deploy.

I went ahead and started the Auto Deploy setup from the DVD again, until I received this weird looking error. Apparently the setup thought (and decided correctly) that Auto Deploy was already installed in my vCenter.

UCS blades w/ Boot-from-SAN and AutoYaST

Thu, 30 Jun 2016 14:49:01 +0000

As I wrote before about enabling multipathing for the AutoYaST installation it’s about time I write this one here.

Sadly AutoYaST needs a little push in the right direction (as to where to actually put the root device), so here’s part of my AutoYaST profile for such a Cisco blade:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54


<profile xmlns="http://www.suse.com/1.0/yast2ns" xmlns:config="http://www.suse.com/1.0/configns">

 <bootloader>
 <device_map config:type="list">
 <device_map_entry>
 <firmware>hd0</firmware>
 <linux>/dev/sda</linux>
 </device_map_entry>
 </device_map>
 </bootloader>

 <partitioning config:type="list">
 <drive>
 <device>/dev/sda</device>
 </drive>
 </partitioning>

 <scripts>
 <pre-scripts config:type="list">

 <script>
 <debug config:type="boolean">false</debug>
 <feedback config:type="boolean">false</feedback>
 <filename>config-ucs.sh</filename>
 <interpreter>shell</interpreter>
 <source><![CDATA[
cat /tmp/profile/autoinst.xml | sed "s,/dev/sda,/dev/mapper/`/sbin/multipath -ll | grep dm-0 | cut -d -f1`," > /tmp/profile/modified.xml
]]>
 </source>
 </script>

 </pre-scripts>

 <chroot-scripts config:type="list">

 <script>
 <chrooted config:type="boolean">true</chrooted>
 <debug config:type="boolean">true</debug>
 <feedback config:type="boolean">true</feedback>
 <filename>config-ucs-chroot.sh</filename>
 <interpreter>shell</interpreter>
 <location>http://install.home.barfoo.org/autoyast/scripts/config-ucs-chroot.sh</location>
 </script>

 </chroot-scripts>
 </scripts>

 <software>
 <packages config:type="list">
 <package>multipath-tools</package>
 </packages>
 </software>

</profile>

Now, the profile addition takes care of the placement of the root-device now (simply parses multipath -ll) and adjusts the pulled profile accordingly ( /tmp/profile/modified.xml), which AutoYaST then re-reads.

Windows *: mscorsvw-exe high CPU usage

Tue, 01 Sep 2015 17:14:06 +0000

Another .NET update later, an hour spent looking this up. Why is mscorsvw.exe using 25% CPU for >30 minutes?

Here’s a short outliner on how to speed it up:

The scripts we’ve provided are a convenience for people who don’t want to deal with a command prompt. If you prefer to use the command prompt, you can use the commands below instead. These commands depend on the version of the .NET Framework you have installed and the version of Windows that you have and whether it’s 32-bit or 64-bit.

Migrating from XenServer to ESXi

Thu, 19 Feb 2015 09:17:05 +0000

For the past two months we’ve been trying to migrate a bunch (90 or so) VMs from XenServer to ESXi … However for some reason on some of them, the Converter Service would crash.

VMware Converter crashing due to rsintcor32.dll

Up till Monday, I had no idea why. I decided to look into the error once again, and this time decided just to Google the failing module… And guess what ? Out came this Citrix forum post regarding the failing module. So, after knowing that rsintcor32.dll belongs to the Citrix System Monitoring Agent service (well, I could have guessed that from the DLLs path 😛) I decided to simply stop the service.

PowerShell - and how to add new DNS servers to remote systems

Wed, 01 Oct 2014 17:14:23 +0000

Well, I’ve been playing around with PowerShell today. I had the task to move (as in change the IP address) two domain controllers into another VLAN. I could have done it the easy way and added the DNS servers by hand (by RDP’ing to each system having these particular DNS servers configured) - which wouldn’t have been very hard considering the domain only has 7 members at this point …

Rename a Standard Port Group on all hosts in a cluster

Wed, 24 Sep 2014 10:59:27 +0000

Well, I recently decided to rename a bunch of my Standard Port Groups, since they did no longer reflect the network they were providing. Since I’m a lazy bastard (well lazy as in click lazy), I wrote this little PowerCLI script:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37


param(
 [string] $vcenter,
 [string] $cluster,
 [string] $oldpg,
 [string] $newpg,
 [string] $vlan )

# Add the VI-Snapin if it isn't loaded already
if ( (Get-PSSnapin -Name "VMware.VimAutomation.Core" -ErrorAction SilentlyContinue) -eq $null )
{
 Add-PSSnapin -Name "VMware.VimAutomation.Core"
}

If ( !($vcenter) -or !($cluster) -or !($oldpg) -or !($newpg) -or !($vlan) )
{
 Write-Host `n "pg-cluster-rename: <vcenter-server> <cluster> <oldpg> <newpg>" `n
 Write-Host "This script renames each port group with the name <oldpg> to <newpg>" `n
 Write-Host " <vcenter-server> - DNS name of your vCenter server." `n
 Write-Host " <cluster> - Display-Name of the vCenter cluster, on which we are"
 Write-Host " gonna create the new portgroup." `n
 Write-Host " <oldpg> - Name of the old Port Group that is to be replaced (ie VLAN2)." `n
 Write-Host " <newpg> - Name of the new Port Group (ie PG-VLAN2-Produktion)." `n
 Write-Host " <vlan> - VLAN-ID for of the new port group." `n
 exit 1
}

Connect-VIServer -Server $vcenter

Get-Cluster $cluster | Get-VMHost | Get-VirtualSwitch -Name $vswitch | `
 New-VirtualPortGroup -Name $pg -VLanId $vlan
Get-Cluster $cluster | Get-VM | Get-NetworkAdapter | Where { $_.NetworkName -eq "$oldpg" } | `
 Set-NetworkAdapter -NetworkName $newpg -Confirm:$false
Get-Cluster $cluster | Get-VMHost | %{Get-View (Get-View $_.ID).configmanager.networkSystem} | %{
 $_.RemovePortGroup($oldpg)
}

Disconnect-VIServer -server $vcenter -Confirm:$false

This script basically takes a vCenter instance and a single cluster, then creates a new Port Group on each host, after which it reconfigures all VMs possessing a virtual NIC with that Port Group and then deletes the old Port Group.

Nagios: Integrating Cisco switches

Wed, 20 Aug 2014 08:20:53 +0000

Well, as I wrote recently, we received a new BladeCenter a few weeks back. Now, as we slowly take it into service I was interested in watching the utilization of the back planes as well as the CPU utilization of the Cisco Catalyst 3012 network switches.

The first mistake I made, was to trust Cisco with their guide about how to get the utilization from the device using SNMP. They stated some OID’s, which I tried with snmpwalk and got a result from.

1
2


snmpwalk -v1 -c public -O n 10.0.0.35 .1.3.6.1.4.1.9.5.1.1.8
.1.3.6.1.4.1.9.5.1.1.8.0 = INTEGER: 0

Now, as I tried retrieving the SNMP data by means of the check_snmp plugin, I got some flaky results:

1
2
3
4


/usr/lib/nagios/plugins/check_snmp -H 10.0.0.35 -C public
 .1.3.6.1.4.1.9.5.1.1.8
SNMP problem - No data received from host
CMD: /usr/bin/snmpget -t 1 -r 5 -m '' -v 1 [authpriv] 10.0.0.35:161

Those of you, who read the excerpts carefully will notice the difference between snmpwalk and the OID I passed on to check_snmp.

The point being, the OID’s Cisco gave in their Design tech notes are either old, or just not accurate at all. After passing on the .0 to each value given by Cisco, the check_snmp is all honky dory and integrated into Nagios.

As usual, the Nagios definitions are further down, for those interested.

SLES-9-2

Sat, 16 Aug 2014 10:23:06 +0000

Hrm, today I tried to install some extra programs I need for devel-stuff (quilt, git, subversion) on an ancient (not sooo ancient) SuSE Linux Enterprise Server 9.2.

Did I already mention I hate rpm-based distros ? Ah and I missed to tell you, that I really love USE-flags …

Ok, so it took me half the afternoon, to rebuild half of all installed packages (heh, kde* depends upon expat*; but who need X or even KDE on a server-machine?) and figured that everything was for nothing.

Adapter teaming on SLES10

Sat, 16 Aug 2014 10:18:13 +0000

Since one of the requirements for my current project is having NIC redundancy, I didn’t get around looking at the available " adapter teaming" (or adapter bonding) solutions available for Linux/SLES.

First I tried to dig into the Broadcom solution (since the Blade I first implemented the stuff uses a Broadcom NetXtreme II card) , but found out pretty soon that the basp configuration tool, which is only available on the Broadcom driver CD’s shipped with the Blade itself, pretty much doesn’t work.

Bloody cluster solutions

Sat, 16 Aug 2014 10:17:48 +0000

In preparation to get our website (and all those other websites - like www.fh-neubrandenburg.de or www.hmt-rostock.de) clustered, someone bought the cluster version of the PacketPro 450. These things are nice, especially considering you don’t need to fiddle around with LVS yourself (which is a real pain in the ass).

The only problem I have currently with them is that they scan the database and web nodes every 30 seconds, and since we have an active node and a hot-standby both do this and producing this:

VBscript undamp; Active Directory and printers

Sat, 16 Aug 2014 10:15:10 +0000

Well, since our current solution for mapping printers is an ugly batch file, which needs to be put into Startup, I today poked at doing it in VBscript (I know, but it’s less ugly than the batch script, trust me).

As some of you know, printers are only applicable to users (as in you can’t put a startup script onto an OU, which is going to map the printers). So as we store users and the computes in different OU’s in our Active Directory (we do have about 15.000 students), I can’t apply the printer.vbs to the users OU directly either, unless I implement some intelligence into the script itself.

Bloody cluster solutions (continued)

Sat, 16 Aug 2014 10:10:22 +0000

So, as the previous try on getting the teamix people to fix the bloody LoadBalancer (as in sending at least an identification string for the SSH check) didn’t work so well (they told me, I should configure MASQ uerading/ ROUTE ing on the PacketPro(which is kinda icky), I went on today and looked at what SLES10 installs as default logger.

Surprisingly they install a rather new syslog-ng (well, syslog-ng-1.6.8 is what they ship) so it was rather easy to workaround the situation.

Research project

Sat, 16 Aug 2014 10:07:08 +0000

OK, as I wrote earlier today I went onto a research mission for today, looking at the alternatives for the 2x stuff. Looks like Citrix Presentation Server is just the software I’m looking for. Watched the demo’s on their website, which are quite impressing, but sadly don’t tell me everything I’d like to know.

Maybe I’ll ask some people in Greifswald and in the vicinity, how stuff works with Citrix. Maybe I should even get in contact with Citrix itself and ask for a test version, or some other sort of demonstration.

OCFS2 fun yet again

Sat, 16 Aug 2014 10:05:13 +0000

I’m coming back today from a six day vacation in the warm south (that is Stuttgart), back at work and find three sheets of paper on my desk. Two tell me something I haven’t done yet, the other one tells me something I haven’t seen yet.

One of my colleagues had to restart one of our web nodes and now the thing can’t mount the logging volume (and thus, logrotate / awstats failed to do it’s job). OCFS2 ain’t spitting any error messages, when trying to mount the volume you see it joining the domain the volume belongs to on the other nodes, so from a first glance at things .. nothing is wrong ?

Windows XP Embedded and GPO settings

Sat, 16 Aug 2014 09:57:51 +0000

We’re currently having a weird issue (which we had before); the Windows XP Embedded powering our Wyse V90’s isn’t applying any GPO settings if you log on with a user that has a configured profile.

Googling (is that a valid word yet ?!) for it, only resulted in one useful link, which is apparently a guy with the exact same problem … 🤷 I’m completely out of ideas by now, as I don’t even have a place to start (as in where the reason might be located).

IBM RDAC and Windows Cluster Service

Sat, 16 Aug 2014 09:56:43 +0000

Okay, so we received a brand new x3650 the other day entitled to replace one (or better two) of our NAS frontend servers. We installed Windows on it the other day (had to create a custom Windows Server 2003 CD first, since the default one doesn’t recognize the integrated ServeRAID), and we prepped the box during the week with the usual things.

On Monday I started installing the " IBM StorageManager RDAC" MultiPath driver (since the box got two single port PCIe FC-HBA’s) and figured I’d be nice if we had this. I asked a IBM Systems Engineer of one of our partners, which told me generally there wouldn’t be a problem with Microsoft Cluster Services (MSCS) and the IBM MPIO driver. Only requirement would be that I’d install the new storport.sys driver (version 5.2.3790.4021) first (as in Microsoft KB932755).

Windows XP Embedded and GPO settings (continued)

Sat, 16 Aug 2014 09:56:20 +0000

Well, as I said in my previous post, I do have some weird things happening. Apparently adding the domain user to the local group “Administrators” makes everything just works fine, yet he can’t do administrator like stuff (like turning off the write protection, changing local user accounts, …).

Also, if you’re looking for a smart way of how to add a certain global group (as in Active Directory group) to a local group, try this:

VMware design rules

Sat, 16 Aug 2014 09:52:13 +0000

I’m just got back from four days in Rostock over at S&N, where I was attending a VMware design course and here’s a list of questions I did ask the trainer:

What’s the disadvantage of having a 1016 ported vSwitch ?
Any clues on how to exchange the default certificate of the Virtual Center ?
Are there any tools to stress test the virtual system ?
Are there any performance impacts of having more than 10 users in Virtual Center ?
Any clues and/or guides on how to do time synchronization in VMware guests, especially Linux guests ?
What’s the preferred NIC type for Linux guests ?
Any clues to using Raw Device Mappings with VMotion ?
Is there a way of defining CPU masks on a global level ?

Answers:

Building RPMs on SLES10SP2-ppc64

Sat, 16 Aug 2014 09:50:36 +0000

Well, it turns out that building stuff on ppc64 is a real pain in the ass, at least on anything SUSE related. I do have to tweak every damn spec to include this:

1
2
3


%ifarch ppc64
export LDFLAGS="$LDFLAGS -m64"
%endif

Otherwise, ld is gonna fail when linking, as it’s gonna try linking the generated 64bit code ( -m64 is passed on via RPM_OPT_FLAGS to CFLAGS) as 32bit code, which ain’t gonna work at all …

Extending vMotion compatiblity

Sat, 16 Aug 2014 09:49:58 +0000

Today I did something horrible. I yet again noticed that I bought the wrong CPU’s (basically I bought Xeon DP’s with four cores). Those have apparently a feature called SSSE3, which makes vMotion with our old Xeon DP’s (dual cores) fail before even trying.

But as we had a cooling outage today (basically ‘cause it broke), I needed to turn off some ESX servers. Thus leaving me with the new ones and one of the old ones. * yuck*

Nagios virtualization

Sat, 16 Aug 2014 09:49:36 +0000

As virtualization seems to be a trendy thing to do, I went ahead and virtualized our nagios (while reinstalling the whole thing …).

Now as I went into work today and started my email client, I received 4 nagios warnings about a LOAD service reaching critical state. Looked at the nagios box itself, opened up the VM console, looked into the syslog. Nothing.

Yet over 3/4 of the services were flapping, some ping checks were critical (for whatever reason). So I opened the nagios webinterface again, and noticed it dropping the connection over and over again (had to reauthentificate me again and again).

Nagios Hostgroup Inheritance

Sat, 16 Aug 2014 09:48:08 +0000

As I wrote earlier, I recently virtualized our nagios. Along with that came a complete " redesign" of how checks are applied. Up till now, I defined checks for each and every single server, thus ending up with ~25 files, each holding roughly 6 checks which are in the same file just sorted by hostname.

As you can imagine, it gets quite confusing with that amount of checks (~150). So the last two days I spent on reorganizing (with Visio), on which object/hostgroup placing a check would make sense. Now, this is my first result of two days planning, reorganizing, reordering and moving hosts into different hostgroups.

Windows Cluster Service (continued)

Sat, 16 Aug 2014 09:47:05 +0000

Well, guess my " solution" didn’t work sooo good. Lemme tell you what’s happening. I successfully added the node to the cluster group, but I can’t get any resources online.

The node tries bringing it online, then shows a failure and immidiately moves them over to the next node. There the resource is being successfully moved online .. So again, I’m out of ideas ..

Already tried reinstalling the box, after that I could get the third node successfully into the cluster, without the " Advanced (minimum)" trick … 🤷 still ain’t bringing any resources online.

SUSE Linux Enterprise Server 10 on VMware ESX

Sat, 16 Aug 2014 09:46:16 +0000

We’re currently having a really weird problem with our VM’s. Sometime last week, SUSE released a kernel update. Now, once you install it and you reboot the selected VM with a DVD/CD image present, you’re gonna see this:

msg.vmxaiomgr.retrycontabort.unkown

The only workaround so far has been to unmount any cleanse any CD-Drives attached to the VM. And yes, this is reproduceable, even reinstalling from scratch doesn’t change the fact, that after installing the patch the VM quits working.

SUSE Linux Enterprise Server 10 on VMware ESX (continued)

Sat, 16 Aug 2014 09:45:36 +0000

Well, after some searching today (we applied the VMware Update 2 today, thus the VMware Tools update too), I finally found out what is causing that problem.

Though the problem seems to be not limited to virtual systems alone, I just browsed through this Novell Forum thread which pretty much describes my problem. I found the same error in the VM’s I tried to mount a CD image.

More VirtualCenter troubles

Sat, 16 Aug 2014 09:44:54 +0000

Well, after my co-worker switched the VirtualCenter certificates with one produced by our RA a few days ago, I can’t clone anything using a customization specification anymore.

Unable to decrypt passwords in customization specification

Guess, we’re shit outa luck. At least both of those linked VMTN discussions don’t contain any (that is for us) workable solution (well besides storing the password in cleartext in the spec – which ain’t sooo good). Gonna bug him tomorrow to open up a VMware support request, maybe that’ll help somewhat. I sure hope so.

patch2mail for SLES10

Fri, 08 Aug 2014 09:51:16 +0000

Well, there is this “nifty” tool called patch2mail, which basically converts the XML for the updates to a more readable format. But you’re screwed if you want to do the same on SLES10. Since it ain’t shipping with the zypper xml wrapper thing, you need to do it a bit different.

So I ended up writing a small (and yet, ugly) shell script to generate me a mail of my liking ..

MessPC Ethernetbox 2 and Nagios

Fri, 08 Aug 2014 09:46:05 +0000

As I talked to Tobi yesterday, we came to talk about our Ethernet Box thermometer. It’s a neat device, which works pretty much out of the box. Integrating it with Nagios is a bit of a bummer.

That’s what the ~300 EUR box looks like. It’s basically a small black box with a RJ45 jack, and four RJ11 jacks for attached external devices. The box itself only functions as a " management station" and doesn’t come with a sensor. Normally, you can attach up till four RJ11 sensors to it. But, MessPC also has RJ11 port splitters, which enables you to attach up to eight RJ11 sensors to the MessPC.

As you can see, the box has a RJ45 jack on the other side, which you basically hook up to your network and then configure an IP address (or if you fancy DHCP for those things, it’s possible too).

On the opposite site, are the RJ11 jacks for the sensors. As you can see, we currently do have 4 splitters attachted to the box, enabling up till 8 sensors to be measured. Once you have it up and running, you can look at the web interface and you’ll be able to see the state of the sensors right on the first page.

Linux-HA and Tivoli Storage Manager

Fri, 08 Aug 2014 09:44:38 +0000

Well, since we received part of our shipment on Wednesday, I finally looked at how we’re gonna deploy our active/active Tivoli Storage Manager configuration. Right now, we do have a single pSeries box hosting ~100 client nodes which we’re looking to split by two (since we do have two x366 for that purpose now).

Now, as there ain’t no solution for this scenario yet (neither from International Business Machines nor someone out of the open source community), I sat down and started writing an OCF Resource agent for dsmserv (that is the Tivoli Storage Manager server).

VMware vCenter: is not connected

Fri, 08 Aug 2014 09:08:15 +0000

Well, today I once again had the case where a virtual machine (in my case a Virtual Machine Template) was kinda stuck. You couldn’t remove the template (as in the entries for “Remove from inventory” was grayed out) and you couldn’t re-add the Virtual Machine’s VMX from the datastore browser either.

VI Client - Disconnected templates

Though, a simple putting the host into maintenance mode and rebooting helped that problem. Maybe there is a simpler solution for this, I just don’t know about it.

Tivoli Storage Manager Client and Microsoft Cluster Services

Fri, 08 Aug 2014 09:06:34 +0000

Well, I just had another look at our client scheduler services on our Microsoft Cluster. A while back we noticed that those scheduler services were going nuts after some time. Well, as it turns out, I can tell why. Microsoft Cluster Services have a feature called registration replication, which replicates a given key, if changed when the resource is online, to all connected cluster nodes.

Now, we added the obvious registry key to the settings of our cluster resources for the scheduler services ( SOFTWAREIBMADSMCurrentVersionBackupClientNodes) and the scheduler service would use the same registry key to store it’s passwords. But it seems we were far off with that assumption.

IBM RDAC: Installing the driver for a (not yet) running version

Fri, 08 Aug 2014 09:04:36 +0000

Well, kernel updates on our Linux servers running IBM’s RDAC driver (developed by LSI) is a real pest .. especially if you have to reboot the box two times in order to install the drivers/initrd correctly.

So I sat down and looked at the Makefile. Turns out, it just needs four tweaks in order to be working with a different kernel version (which you have to pass using environment variables to make).

Novell KMP: KMP'ing IBM's RDAC driver

Fri, 08 Aug 2014 09:03:51 +0000

Well, after yesterday’s lesson about getting the IBM RDAC to install for a not-yet-running kernel, I decided to take it a step further. Novell does have some documentation about KMP’s, which is actually rather good, especially the guide written by Andreas Grünbacher.

After a short tinkering, I got it actually working. I was kinda surprised, at how easily it actually is. One problem I still have to deal with, is modifying the %post, to generate the mpp-initrd image. For now, the KMP only contains the default %post, which updates the modules.* stuff.

Novell KMP: Useable version of ibm-rdac-ds4000

Fri, 08 Aug 2014 09:03:03 +0000

After some more tinkering, a lot more looking at the macros in /usr/lib/rpm/rpm-suse-kernel-module-subpackage and /usr/lib/rpm/suse_macros, I think I finally have a usable RPM’ified version of IBM’s Multipathing driver ready for use.

There is still one major annoyance left: each time you install a new ibm-rdac-ds4000-kmp RPM, you also need to reinstall the corresponding ibm-rdac-ds4000-initrd package, as the macros in /usr/lib/rpm don’t allow for custom %post or %postun.

As mentioned before, I’m gonna send them to LSI/IBM for review, and maybe, MAYBE they are actually gonna make use of that.

Weird TS3500 problem

Fri, 08 Aug 2014 09:00:33 +0000

Well, today we had a rather weird problem with our TS3500. TSM running on AIX basically went bonko and spit out weird media sense errors, all stating that there is a hardware or media error of unknown nature:

1
2
3
4
5
6
7
8
9


ANR8943E Hardware or media error on library LIB3584 (OP=00006C03, CC=-1,
KEY=04, ASC=44, ASCQ=00,
SENSE=70.00.04.00.00.00.00.46.00.00.00.00.44.00.00.00.00-
 .00.40.82.00.00.00.40.00.00.02.00.48.01.A1.00.00.00.00.0-
 0.06.1B.00.01.09.00.00.00.00.00.00.00.00.00.00.00.00.00.-
 00.00.00.00.00.00.00.00.00.00.00.00.00.00.00.00.00.00.00-
 .00.00.00.00.00., Description=An undetermined error has occurred).
Refer to Appendix C in the 'Messages' manual for recommended action.
ANR8381E LTO volume HG4480L4 could not be mounted in drive DR9 (/dev/rmt8).

After restarting the TSM server (as in the service, not the whole box) five times, which didn’t resolve squat we decided to take a look at the TS3500 itself. We opened up the Management interface and tried moving a tape into a drive. That didn’t work. Hrmmmmm.

Linux-HA and Tivoli Storage Manager (Finito!)

Fri, 08 Aug 2014 08:59:07 +0000

As I previously said, I was writing my own OCF resource agent for IBM’s Tivoli Storage Manager Server. And I just finished it yesterday evening (it took me about two hours to write this post).

Only took me about four work days (that is roughly four hours each, which weren’t recorded in that subversion repository) plus most of this week at home (which is 10 hours a day) and about one hundred subversion revisions. The good part about it is, that it actually just works :-D (I was amazed on how good actually). Now you’re gonna say, “but Christian, why didn’t you use the included Init-Script and just fix it up, so it is actually compilant to the LSB Standard ?”

The answer is rather simple: Yeah I could have done that, but you also know that wouldn’t have been fun. Life is all about learning, and learn something I did (even if I hit the head against the wall from time to time ;-) during those few days) … There’s still one or two things I might want to add/change in the future (that is maybe next week), like

adding support for monitor depth by querying the dsmserv instance via dsmadmc (if you read through the resource agent, I already use it for the shutdown/pre-shutdown stuff)
I still have to properly test it (like Alan Robertson mentioned in his one hour thirty talk on Linux-HA 2.0 and on his slides, Page 100-102) in a pre-production environment
I’m probably configure the IBM RSA to act as a stonith device ( s hoot t he o ther n ode i n t he h ead) - just for the case one of them ever gets stuck in a case, where the box is still up, but doesn’t react to any requests anymore

VMware: New VirtualCenter 2-5 Update 4

Fri, 08 Aug 2014 08:47:01 +0000

As many people on the VM-Planet already blogged about this, I ain’t gonna write just about it. Let’s turn the clock back a few months, to January 2008.

As the institution I work for, is part of the DFN we took the opportunity to be a part of the " I want you to run our RA"-gang. In January 2008 we thought about changing the vCenter certificate. Now, apparently there’s a slight difference between the DFN-PCA and what VMware considers common practice.

Xen dom0 failing with kernel panic

Fri, 08 Aug 2014 08:45:02 +0000

I’m building a 6-node cluster, using Xen at the moment. For the last few days, I tried my setup in a virtual machine, simply because VM’s boot much faster than the real hardware. However, certain things you can only replicate on the real hardware (for example, the InfiniBand interfaces, as well as certain nfs-stuff).

So I spent most of the day to replicate my configurations onto the hardware. After getting all done, the moment of the first boot … kaput! Doesn’t boot, just keeps hanging before booting the real kernel. Now what ? I removed the Xen vga parameters and rebooted (waited ~2 minutes in the process) until I finally saw the root cause for my trouble:

IBM RSA II adapter and Java RE

Fri, 08 Aug 2014 08:43:14 +0000

Today, after a short break (you can call it break, I think), I sat down and looked at the IBM RSA II adapter’s remote management GUI and it’s trouble with JRE versions. Ever since the last Java updates, I was unable to access the RSA console because Java would throw an error like this:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21


Initializing RemoteDisk v2.2
MCS v.3.6 initialized
Established connection to rsa.home.barfoo.org:1045
Connected via socket: Socket[addr=rsa.home.barfoo.org/10.0.0.150,port=2000,localport=4292]
Closing socket
java.lang.NullPointerException
 at mcsClient.Row.isValid(Unknown Source)
 at java.awt.Component.invalidateIfValid(Unknown Source)
 at java.awt.Component.setLocale(Unknown Source)
 at javax.swing.JComponent.(Unknown Source)
 at javax.swing.JPanel.(Unknown Source)
 at javax.swing.JPanel.(Unknown Source)
 at javax.swing.JPanel.(Unknown Source)
 at mcsClient.Row.(Unknown Source)
 at mcsClient.Options.(Unknown Source)
 at mcsClient.McsToolBar.(Unknown Source)
 at mcsClient.McsClient.begin(Unknown Source)
 at mcsClient.McsClient.init(Unknown Source)
 at sun.plugin2.applet.Plugin2Manager$AppletExecutionRunnable.run(Unknown Source)
 at java.lang.Thread.run(Unknown Source)
Ausnahme: java.lang.NullPointerException

In the end, I downloaded every version since JRE 1.5.0.11 (that is 20 different versions ❗), as wittnessed by Michael Ellerbeck that the last working version for him was JRE 1.5.0.11, and gave each one a try (since I want to report the issue to IBM, so that they gonna release a fix sometime soon).

Loooong time

Fri, 08 Aug 2014 08:42:36 +0000

It’s been very quiet around here, I’ve been rather busy with my real life. During that busy time, a lot of things happened. I switched jobs starting on October 1st, I’m now working in Karlsruhe (as compared to the 870km northern Greifswald). It may sound far, but it’s actually quite pleasant. You know, I was born down here (well not exactly here – 70 kilometers afar) and I still had the feeling that this is my home.

SLES11 and AutoYaST

Fri, 08 Aug 2014 08:41:46 +0000

After the first week passed awfully quick, the last week I worked on refining the way on how we are doing openSuSE / SUSE Linux Enterprise Server installations. Up till now, they were done by hand (without a predefined schema) and were getting ugly to maintain. Working my way through the Novell documentation on AutoYaST was pretty straight forward, but the little details were getting hairy. So I decided to write them down, in case someone was gonna end up in the same situation like me.

Run a command on a list of Fibre-Channel switches (fc-switch-commands-pl)

Fri, 08 Aug 2014 08:40:32 +0000

Up till now, we did have a bunch of shell and perl scripts doing this work. Today, as I was looking for some stuff to do, I found them and decided rewriting it, so you wouldn’t need a shell script to call the perl worker script … This is pretty much the result!

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63


#!/usr/bin/perl

# LICENSE: GNU General Public License v2. (see LICENSE.txt)
# COPYRIGHT: Copyright 2010 Christian Heim <christian.heim@barfoo.org>

# This script runs a batch of commands on a list of FC-switches. For example:
# fc-switch-commands.pl statsclear | configupload | supportsave

use strict;
use warnings;
use Net::Telnet;

our( @san_fabric1_core, @san_fabric1_edge_1, @san_fabric1_edge_2 );
our( @san_fabric2_core, @san_fabric2_edge_1, @san_fabric2_edge_2 );
our( @switches );
our( $ftp_host, $ftp_username, $ftp_password );
our( $telnet );
our( $i, $version, @debug );

@san_fabric1_core = ( "10.144.20.50", "admin", "JuNxJFSAS!", 'san_fabric1_core' );
@san_fabric1_edge_1 = ( "10.144.20.51", "admin", "JuNxJFSAS!", 'san_fabric1_edge_1' );
@san_fabric1_edge_2 = ( "10.144.20.52", "admin", "JuNxJFSAS!", 'san_fabric1_edge_2' );
@san_fabric2_core = ( "10.144.20.60", "admin", "JuNxJFSAS!", 'san_fabric2_core' );
@san_fabric2_edge_1 = ( "10.144.20.61", "admin", "JuNxJFSAS!", 'san_fabric2_edge_1' );
@san_fabric2_edge_2 = ( "10.144.20.62", "admin", "JuNxJFSAS!", 'san_fabric2_edge_2' );

$ftp_host = '10.144.20.45';
$ftp_username = 'brocade_cfg';
$ftp_password = 'JuNxJPFC!';

@switches = ( @san_fabric1_core, @san_fabric2_core,
 @san_fabric1_edge_1, @san_fabric1_edge_2,
 @san_fabric2_edge_1, @san_fabric2_edge_2 );

@debug = ( Dump_Log => 'dump.log', Output_Log => 'out.log', Input_Log => 'in.log' );

$telnet = new Net::Telnet(Timeout=>240, Errmode=>'die',
 Prompt => '/.*:admin> $/is', @debug);

for ($i = 0; $i < $#switches + 1; $i++) {
 $telnet->open($switches[$i][0]);
 $telnet->login($switches[$i][1], $switches[$i][2]);

 if ( $ARGV[0] eq "statsclear" ) {
 $telnet->cmd("statsclear");
 } elsif ( $ARGV[0] eq "configupload" ) {
 # Check the FabricOS version, as Brocade decided to break compatiblity with
 # earlier firmware versions w/ v6 (at least configupload)
 $telnet->cmd("firmwareshow");
 $version = $telnet->lastline;
 $version =~ s/s+|s+$//g;

 if ( $version =~ "v6" ) {
 $telnet->cmd("configupload -all -ftp "$ftp_host","$ftp_username","$switches[$i][3].cfg","$ftp_password"");
 } else {
 $telnet->cmd("configupload "$ftp_host","$ftp_username","$switches[$i][3].cfg","$ftp_password"");
 }
 } elsif ( $ARGV[0] eq "supportsave" ) {
 $telnet->cmd("supportsave -n -u "$ftp_username" -p "$ftp_password" -h "$ftp_host" -d "supportsave/$switches[$i][3]" -l ftp");
 }

 $telnet->close();
}

Monitoring Brocade FC switches with SNMP/Nagios

Fri, 08 Aug 2014 08:39:27 +0000

I looked into the mess a bit more, and as it turns out, the weird crap I was talking about only happens if you have a port with LossofSynchronization, LossofSignal or LinkFailures value with the base of ten (i.e. 10, 101 or 10.000).

Additionally, the OID’s for those three failure elements seem to be dependent on the firmware version, as with 6.3.x they appear as different OIDs. So I may need to introduce another command-line switch, which selects the firmware version and depending on that, the OID.

Monitoring Brocade FC switches with Nagios

Fri, 08 Aug 2014 08:37:46 +0000

The last four days I spent looking for ways on monitoring a Brocade Fibrechannel switch (in my case IBM 2145 B32/F40). The first thing I came up with, is using SNMP. As it was already configured for the previous monitoring with Munin, getting information should be quite easy. After looking through Google for a bit, there is already one script that worked for me.

Only trouble I had with that script, is that it crams every single port into one result. As I wanted something, that a) could watch a single port and b) return performance data, I went ahead an used the script to do a basic rewrite. But after a short while, I grew antsy and started writing a script from scratch, using the OIDs I got from that script and a Cacti template.

Windows Server 2003 SP1, WSUS and Security Updates

Fri, 08 Aug 2014 08:35:23 +0000

Recently, we found some systems (sadly, customer systems) that weren’t getting any Security Updates anymore. Much more sadly, them is running Windows Server 2003, and as you know Security Updates are pretty important for Windows Systems.

At the time of finding this, I had no clue as to why the were not getting any updates. At first we thought it had something to do with the WSUS server, so I upgraded the WSUS 3.0 SP1 to SP2. Since that didn’t solve nothing, I went searching for a internal VM, that showed the same symptoms and I quickly found one.

VBscript: Query remote OS and SP info

Fri, 08 Aug 2014 08:34:49 +0000

As I wrote on Thursday, I am battling with Windows Server 2003. Now I got a list out of our change management database, which sadly ain’t that accurate. So in order to get reliable information about the target systems (in order to do some accurate planning), I ended up writing a small vbscript which simply takes the hostname on the command line (cscript //NoLogo win_sp_level.vbs 10.0.0.5) and returns a csv-like element.

Reset Master Boot Record (MBR)

Fri, 08 Aug 2014 08:34:04 +0000

Since I’ve been playing with my AutoYAST setup for the last few days, working out the kinks (for example SLES10 not being able to install into the MBR), I needed a way to zap the MBR (as in remove grub to see whether or not the installation would install a new loader). So after quickly googling, I found this:

`1`	`dd if=/dev/zero of=/dev/hda bs=512 count=1`

That actually does the trick. The loader as well as the partition table are gone after wards!

Enabling multipathing in autoyast installations

Fri, 08 Aug 2014 08:30:46 +0000

As I mentioned before, we’re starting to utilize Boot-from-SAN as a means to strip the blades of their local disk. As the title says, after trying a manual installation of SLES 11.1 via CD/HTTP I wanted to automate the process, in order to get a reproducible, consistent installation method. As you might have figured, AutoYaST doesn’t have any built in support for configuring multipathing (hey, that’s what Novell says here). Now, they also provide a comprehensive how-to on how to “add” this to your AutoYaST, using a DUD (or D river U pdate D isk).

NetApp FAS/Data ONTAP public key authentification with CIFS/NFS license

Fri, 08 Aug 2014 08:29:59 +0000

Well as the title says, sadly we bought our FAS6210 without CIFS/NFS license. Thus, in order to create the folder structure/add the authorized_keys file, you’ll have to work for your money a little bit.

First, you need to run cifs setup / cifs passwd somewhere. I did it on our Data ONTAP simulator, which comes in handy for things like that.

You’ll get a cryptic looking password (no clue which format that is), looking like this: _OnWddr)xa.

IBM SVC: Copy VDisk Host-Mapping from one host to another

Fri, 08 Aug 2014 08:28:32 +0000

As I wrote a few days ago, I started a new job. One of my first (voluntary) tasks was writing a shell script which would copy a VDisk Host-Mapping from a given host to another. This is useful, if you do have a lot of ESX servers for example and a few roaming ones.

Now, if say, you need to do some ESX-Updates and you would like to add the roaming one to a given farm, you would be in a dark an deary place. You would be required to either click through the GUI a dozen times (in my case, it might have needed ~200 clicks) or type svcinfo lshostvdiskmap and svctask mkhostvdiskmap -force (these are incomplete command references) a few times.

NetApp - Copy LUN mappings

Fri, 08 Aug 2014 08:27:40 +0000

Well, today I had another idea (basically like the one I wrote for SVC’s VDisk mappings a while back) for a script:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44


#!/bin/bash

KEY_FILE="/root/.ssh/netapp.dsa"
SSH_OPTS="/root/.ssh/netapp-ssh_config"

if [ $# -ne 3 ] ; then
 echo "fas-copy-lunmap: FAS_CONTROLLER SOURCE_IGROUP TARGET_IGROUP"
 echo
 echo "Copy the LUN map from one igroup to another, ie for ESX reinstallation"
 echo
 echo " Usage:"
 echo " - FAS_CONTROLLER: Hostname/IP-adress of the DATA ONTAP controller"
 echo " - SOURCE_IGROUP: igroup that is used as a reference for the copy process"
 echo " - TARGET_IGROUP: igroup that is actually modified"
 echo
 exit 1
fi

FAS_CTRL=$1
SOURCE=$2
TARGET=$3

ssh_fas() {
 # $@: commands for Data ONTAP
 COMMANDS="$@"
 /usr/bin/ssh -i $KEY_FILE -l root -F $SSH_OPTS $COMMANDS
}

#set -x

# Get the lun list.
for lun in $( ssh_fas $FAS_CTRL lun show -g $SOURCE | awk '{ print $1 }' | sort -u ); do

 # Get the LUN number its mapped
 LUN_ID="$( ssh_fas $FAS_CTRL lun show -m $lun | grep "^/vol" | awk '{ print $3 }' )"

 # If the LUN id is 0, skip otherwise we would copy the boot LUN
 if [ "$LUN_ID" != "0" ] ; then
 # Actually map the lun to our host
 echo "Mapping $lun to $TARGET as LUN_ID $LUN_ID"
 ssh_fas $FAS_CTRL lun map $lun $TARGET $LUN_ID
 fi
done
#set +x

I’ll post the counterpart of the script (to remove the LUNs) in a second post later on.

Installserver: Regenerate PXE base menu

Fri, 08 Aug 2014 08:27:12 +0000

Well, here’s the promised script to regenerate the main PXE menu based on the menu entries created by register-suse and register-vmware.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52


#!/bin/bash

TFTP_DIR=/srv/tftpboot/pxelinux.cfg

mkdir -p $TFTP_DIR &>/dev/null

# Iterate through the pxelinux.cfg directory and recreate the index menues.
for dir in $( find $TFTP_DIR -mindepth 1 -maxdepth 1 -type d ); do
 echo "Generating summary menu for: $dir"
 count=1

 cat > $dir.menu << EOF

 MENU TITLE Linux Installationservices
 MENU BACKGROUND addons/background.png

 MENU COLOR screen 37;40 #80ffffff #00000000
 MENU COLOR border 0 #ffffffff #ee000000 std
 MENU COLOR title 0 #ffffffff #ee000000 std
 MENU COLOR unsel 0 #ffffffff #ee000000 std
 MENU COLOR sel 0 #ffffffff #85000000 std
 MENU COLOR scrollbar 30;44 #40000000 #00000000
 MENU COLOR tabmsg 0 #ee000000 #ffffffff std
 MENU COLOR cmdmark 0 #ff00ff00 #00000000 std
 MENU COLOR cmdline 0 #ee000000 #ffffffff std
 MENU COLOR timeout_msg 0 #ee000000 #ffffffff std
 MENU COLOR timeout 0 #ee000000 #ffffffff std
 MENU COLOR disabled 0 #ffffffff #ee000000 std
 MENU COLOR pwdheader 0 #ff000000 #99ffffff rev
 MENU COLOR pwdborder 0 #ff000000 #99ffffff rev
 MENU COLOR pwdentry 0 #ff000000 #99ffffff rev
 MENU COLOR hotkey 0 #ff00ff00 #ee000000 std
 MENU COLOR hotsel 0 #ffffffff #85000000 std
EOF

 for file in $( find $dir -mindepth 1 -name "*.menu" | sort -r ); do
 FILE_LABEL="$( grep ^#LABEL $file | cut -d -f2- )"
 FILE_OSARCH="$( grep ^#OSARCH $file | cut -d -f2- )"

 cat >> $dir.menu << EOF

 LABEL $count
 MENU LABEL $FILE_LABEL ($FILE_OSARCH)
 KERNEL addons/vesamenu.c32

 APPEND pxelinux.cfg/${dir##*/}/${file##*/}
EOF

 count=$(( $count + 1 ))
 done

done

 1
 2
 3
 4
 5
 6
 7
 8
 9
 10
 11
 12
 13
 14
 15
 16
 17
 18
 19
 20
 21
 22
 23
 24
 25
 26
 27
 28
 29
 30
 31
 32
 33
 34
 35
 36
 37
 38
 39
 40
 41
 42
 43
 44
 45
 46
 47
 48
 49
 50
 51
 52
 53
 54
 55
 56
 57
 58
 59
 60
 61
 62
 63
 64
 65
 66
 67
 68
 69
 70
 71
 72
 73
 74
 75
 76
 77
 78
 79
 80
 81
 82
 83
 84
 85
 86
 87
 88
 89
 90
 91
 92
 93
 94
 95
 96
 97
 98
 99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229


Tue Jul 24, 17:18:35 [0] > ll -R /srv/tftpboot/
/srv/tftpboot/:
total 28K
drwxr-xr-x 2 root root 4.0K Jun 20 2011 addons/
drwxr-xr-x 7 root root 4.0K Jun 24 2011 boot/
drwxr-xr-x 6 root root 4.0K Jul 20 13:14 pxelinux.cfg/
-rw-r--r-- 1 root root 15K Oct 7 2009 pxelinux.0
/srv/tftpboot/addons:
total 280K
-rw-r--r-- 1 root root 16K Dec 28 2009 background.png
-rw-r--r-- 1 root root 23K Dec 28 2009 background-sles.png
-rw-r--r-- 1 root root 24K Dec 28 2009 background-suse.png
-rw-r--r-- 1 root root 20K Dec 28 2009 background-vmware.png
-rw-r--r-- 1 root root 48K Oct 7 2009 mboot.c32
-rw-r--r-- 1 root root 142K Oct 7 2009 vesamenu.c32
/srv/tftpboot/boot:
total 20K
drwxr-xr-x 3 root root 4.0K Jun 24 2011 esx/
drwxr-xr-x 2 root root 4.0K Oct 14 2009 firmware/
drwxr-xr-x 5 root root 4.0K Dec 14 2011 opensuse/
drwxr-xr-x 10 root root 4.0K Mar 9 17:28 sles/
drwxr-xr-x 3 root root 4.0K Oct 7 2009 tests/
/srv/tftpboot/boot/esx:
total 4.0K
drwxr-xr-x 2 root root 4.0K Jun 24 2011 4.1-348481/
/srv/tftpboot/boot/esx/4.1-348481:
total 81M
-r--r--r-- 1 root root 79M Jun 24 2011 initrd.img
-r--r--r-- 1 root root 1.5M Jun 24 2011 vmlinuz
/srv/tftpboot/boot/firmware:
total 4.3M
-rw-r--r-- 1 root root 1.5M Oct 14 2009 fdboot.img
-rw-r--r-- 1 root root 20K Oct 14 2009 memdisk
-rw-r--r-- 1 root root 1.5M Oct 14 2009 qmh2462_1.76.img
-rw-r--r-- 1 root root 1.5M Oct 14 2009 qmh2462_1.81.img
/srv/tftpboot/boot/opensuse:
total 12K
drwxr-xr-x 4 root root 4.0K Jun 20 2011 11.3/
drwxr-xr-x 4 root root 4.0K Jun 20 2011 11.4/
drwxr-xr-x 4 root root 4.0K Dec 14 2011 12.1/
/srv/tftpboot/boot/opensuse/11.3:
total 8.0K
drwxr-xr-x 2 root root 4.0K Jun 20 2011 x64/
drwxr-xr-x 2 root root 4.0K Jun 20 2011 x86/
/srv/tftpboot/boot/opensuse/11.3/x64:
total 32M
-r--r--r-- 1 root root 28M Jun 20 2011 initrd
-r--r--r-- 1 root root 3.8M Jun 20 2011 linux
/srv/tftpboot/boot/opensuse/11.3/x86:
total 32M
-r--r--r-- 1 root root 28M Jun 20 2011 initrd
-r--r--r-- 1 root root 3.7M Jun 20 2011 linux
/srv/tftpboot/boot/opensuse/11.4:
total 8.0K
drwxr-xr-x 2 root root 4.0K Jun 20 2011 x64/
drwxr-xr-x 2 root root 4.0K Jun 20 2011 x86/
/srv/tftpboot/boot/opensuse/11.4/x64:
total 37M
-r--r--r-- 1 root root 33M Jun 20 2011 initrd
-r--r--r-- 1 root root 4.1M Jun 20 2011 linux
/srv/tftpboot/boot/opensuse/11.4/x86:
total 37M
-r--r--r-- 1 root root 33M Jun 20 2011 initrd
-r--r--r-- 1 root root 3.9M Jun 20 2011 linux
/srv/tftpboot/boot/opensuse/12.1:
total 8.0K
drwxr-xr-x 2 root root 4.0K Dec 14 2011 x64/
drwxr-xr-x 2 root root 4.0K Dec 14 2011 x86/
/srv/tftpboot/boot/opensuse/12.1/x64:
total 43M
-r--r--r-- 1 root root 39M Dec 14 2011 initrd
-r--r--r-- 1 root root 4.5M Dec 14 2011 linux
/srv/tftpboot/boot/opensuse/12.1/x86:
total 43M
-r--r--r-- 1 root root 38M Dec 14 2011 initrd
-r--r--r-- 1 root root 4.4M Dec 14 2011 linux
/srv/tftpboot/boot/sles:
total 32K
drwxr-xr-x 4 root root 4.0K Jun 24 2011 10.2/
drwxr-xr-x 4 root root 4.0K Sep 28 2011 10.3/
drwxr-xr-x 4 root root 4.0K Jun 17 2011 10.4/
drwxr-xr-x 4 root root 4.0K Jun 17 2011 11/
drwxr-xr-x 4 root root 4.0K Jun 17 2011 11.1/
drwxr-xr-x 4 root root 4.0K Jun 17 2011 11.1-vmware/
drwxr-xr-x 3 root root 4.0K Mar 9 17:28 11.2/
drwxr-xr-x 4 root root 4.0K Jul 24 17:04 11.2-vmware/
/srv/tftpboot/boot/sles/10.2:
total 8.0K
drwxr-xr-x 2 root root 4.0K Jun 24 2011 x64/
drwxr-xr-x 2 root root 4.0K Jun 24 2011 x86/
/srv/tftpboot/boot/sles/10.2/x64:
total 11M
-r--r--r-- 1 root root 9.0M Jun 24 2011 initrd
-r--r--r-- 1 root root 1.4M Jun 24 2011 linux
/srv/tftpboot/boot/sles/10.2/x86:
total 11M
-r--r--r-- 1 root root 8.9M Jun 24 2011 initrd
-r--r--r-- 1 root root 1.3M Jun 24 2011 linux
/srv/tftpboot/boot/sles/10.3:
total 8.0K
drwxr-xr-x 2 root root 4.0K Sep 28 2011 x64/
drwxr-xr-x 2 root root 4.0K Sep 28 2011 x86/
/srv/tftpboot/boot/sles/10.3/x64:
total 12M
-r--r--r-- 1 root root 11M Sep 28 2011 initrd
-r--r--r-- 1 root root 1.4M Sep 28 2011 linux
/srv/tftpboot/boot/sles/10.3/x86:
total 12M
-r--r--r-- 1 root root 11M Sep 28 2011 initrd
-r--r--r-- 1 root root 1.3M Sep 28 2011 linux
/srv/tftpboot/boot/sles/10.4:
total 8.0K
drwxr-xr-x 2 root root 4.0K Jun 17 2011 x64/
drwxr-xr-x 2 root root 4.0K Jun 17 2011 x86/
/srv/tftpboot/boot/sles/10.4/x64:
total 13M
-r--r--r-- 1 root root 12M Jun 17 2011 initrd
-r--r--r-- 1 root root 1.4M Jun 17 2011 linux
/srv/tftpboot/boot/sles/10.4/x86:
total 13M
-r--r--r-- 1 root root 12M Jun 17 2011 initrd
-r--r--r-- 1 root root 1.3M Jun 17 2011 linux
/srv/tftpboot/boot/sles/11:
total 8.0K
drwxr-xr-x 2 root root 4.0K Jun 17 2011 x64/
drwxr-xr-x 2 root root 4.0K Jun 17 2011 x86/
/srv/tftpboot/boot/sles/11/x64:
total 24M
-r--r--r-- 1 root root 21M Jun 17 2011 initrd
-r--r--r-- 1 root root 2.5M Jun 17 2011 linux
/srv/tftpboot/boot/sles/11/x86:
total 23M
-r--r--r-- 1 root root 21M Jun 17 2011 initrd
-r--r--r-- 1 root root 2.4M Jun 17 2011 linux
/srv/tftpboot/boot/sles/11.1:
total 8.0K
drwxr-xr-x 2 root root 4.0K Jun 17 2011 x64/
drwxr-xr-x 2 root root 4.0K Jun 17 2011 x86/
/srv/tftpboot/boot/sles/11.1/x64:
total 27M
-r--r--r-- 1 root root 23M Jun 17 2011 initrd
-r--r--r-- 1 root root 3.1M Jun 17 2011 linux
/srv/tftpboot/boot/sles/11.1/x86:
total 26M
-r--r--r-- 1 root root 23M Jun 17 2011 initrd
-r--r--r-- 1 root root 3.1M Jun 17 2011 linux
/srv/tftpboot/boot/sles/11.1-vmware:
total 8.0K
drwxr-xr-x 2 root root 4.0K Jun 17 2011 x64/
drwxr-xr-x 2 root root 4.0K Jun 17 2011 x86/
/srv/tftpboot/boot/sles/11.1-vmware/x64:
total 27M
-r--r--r-- 1 root root 23M Jun 17 2011 initrd
-r--r--r-- 1 root root 3.1M Jun 17 2011 linux
/srv/tftpboot/boot/sles/11.1-vmware/x86:
total 26M
-r--r--r-- 1 root root 23M Jun 17 2011 initrd
-r--r--r-- 1 root root 3.1M Jun 17 2011 linux
/srv/tftpboot/boot/sles/11.2:
total 4.0k
drwxr-xr-x 2 root root 4.0K Mar 9 17:29 x64/
/srv/tftpboot/boot/sles/11.2/x64:
total 36M
-r--r--r-- 1 root root 33M Mar 9 17:29 initrd
-r--r--r-- 1 root root 3.7M Mar 9 17:29 linux
/srv/tftpboot/boot/sles/11.2-vmware:
total 8.0K
drwxr-xr-x 2 root root 4.0K Mar 9 16:52 x64/
drwxr-xr-x 2 root root 4.0K Jul 24 17:05 x86/
/srv/tftpboot/boot/sles/11.2-vmware/x64:
total 36M
-r--r--r-- 1 root root 33M Jul 20 12:46 initrd
-r--r--r-- 1 root root 3.7M Jul 20 12:46 linux
/srv/tftpboot/boot/sles/11.2-vmware/x86:
total 35M
-r--r--r-- 1 root root 32M Jul 24 17:17 initrd
-r--r--r-- 1 root root 3.6M Jul 24 17:17 linux
/srv/tftpboot/boot/tests:
total 4.0K
drwxr-xr-x 2 root root 4.0K Oct 7 2009 memtest/
/srv/tftpboot/boot/tests/memtest:
total 164K
-rw-r--r-- 1 root root 157K Oct 7 2009 memtest86+-4.00
/srv/tftpboot/pxelinux.cfg:
total 48K
-rw-r--r-- 1 root root 2.0K Jun 20 2011 default
drwxr-xr-x 2 root root 4.0K Jun 24 2011 esx/
-rw-r--r-- 1 root root 1.3K Jul 24 17:18 esx.menu
drwxr-xr-x 2 root root 4.0K Jun 20 2011 esxi/
-rw-r--r-- 1 root root 1.1K Jul 24 17:18 esxi.menu
-rw-r--r-- 1 root root 423 Oct 20 2009 firmware.menu
drwxr-xr-x 2 root root 4.0K Dec 14 2011 opensuse/
-rw-r--r-- 1 root root 1.8K Jul 24 17:18 opensuse.menu
-rw-r--r-- 1 root root 2.6K Jun 4 2010 rescue.menu
drwxr-xr-x 2 root root 4.0K Jul 24 17:05 sles/
-rw-r--r-- 1 root root 2.7K Jul 24 17:18 sles.menu
-rw-r--r-- 1 root root 329 Oct 20 2009 tests.menu
/srv/tftpboot/pxelinux.cfg/esx:
total 8.0K
-rw-r--r-- 1 root root 1.9K Jun 24 2011 esx-4.1-260247.menu
-rw-r--r-- 1 root root 1.9K Jun 24 2011 esx-4.1-348481.menu
/srv/tftpboot/pxelinux.cfg/esxi:
total 4.0K
-rw-r--r-- 1 root root 1.7K Jun 20 2011 esxi41.menu
/srv/tftpboot/pxelinux.cfg/opensuse:
total 24K
-rw-r--r-- 1 root root 1.8K Jun 20 2011 opensuse-11.3-x64.menu
-rw-r--r-- 1 root root 1.8K Jun 20 2011 opensuse-11.3-x86.menu
-rw-r--r-- 1 root root 1.8K Jun 20 2011 opensuse-11.4-x64.menu
-rw-r--r-- 1 root root 1.8K Jun 20 2011 opensuse-11.4-x86.menu
-rw-r--r-- 1 root root 1.8K Dec 14 2011 opensuse-12.1-x64.menu
-rw-r--r-- 1 root root 1.8K Dec 14 2011 opensuse-12.1-x86.menu
/srv/tftpboot/pxelinux.cfg/sles:
total 60K
-rw-r--r-- 1 root root 1.8K Jul 24 16:58 sles-10.2-x64.menu
-rw-r--r-- 1 root root 1.8K Jul 24 16:57 sles-10.2-x86.menu
-rw-r--r-- 1 root root 1.8K Jul 24 16:57 sles-10.3-x64.menu
-rw-r--r-- 1 root root 1.8K Jul 24 16:57 sles-10.3-x86.menu
-rw-r--r-- 1 root root 1.8K Jul 24 16:57 sles-10.4-x64.menu
-rw-r--r-- 1 root root 1.8K Jul 24 16:57 sles-10.4-x86.menu
-rw-r--r-- 1 root root 1.8K Jul 24 16:58 sles-11-x64.menu
-rw-r--r-- 1 root root 1.8K Jul 24 16:58 sles-11-x86.menu
-rw-r--r-- 1 root root 1.8K Jul 24 16:59 sles-11.1-vmware-x64.menu
-rw-r--r-- 1 root root 1.8K Jul 24 16:59 sles-11.1-vmware-x86.menu
-rw-r--r-- 1 root root 1.8K Jul 24 16:59 sles-11.1-x64.menu
-rw-r--r-- 1 root root 1.8K Jul 24 17:00 sles-11.1-x86.menu
-rw-r--r-- 1 root root 1.8K Jul 24 17:00 sles-11.2-vmware-x64.menu
-rw-r--r-- 1 root root 1.8K Jul 24 17:17 sles-11.2-vmware-x86.menu
-rw-r--r-- 1 root root 1.8K Jul 24 17:00 sles-11.2-x64.menu

NetApp: Archive SnapManager SQL snapshots

Fri, 08 Aug 2014 08:25:56 +0000

As I wrote before, we’re using SnapManager (for SQL/Oracle) to create consistent snapshots. However my database guys don’t want to name their snapshots daily. (which I can understand), as once you archive those snapshots to a secondary (and tertiary) system, the names become junk.

So, they’re naming the snapshots like snap__vcsrv_29_12_2012-10.00.01. Sadly, when it comes to SnapVault, it expects the names in form of daily. otherwise you won’t be able to transfer the snapshots with the CLI (none that I have found anyway).

But we didn’t want to move away from naming the snapshots the way they are, so I ended up writing a PowerShell script, that once triggered archives the Snapshots needed for a set of databases. It took me a while to figure a bunch of stuff out, but in the end I think I have a working way of archiving custom-named snapshots.

Implementing SnapVault backups - the hard way

Fri, 08 Aug 2014 08:25:10 +0000

Well, I recently had the pleasant task of implementing SnapVault backups, that are being shipped to an offsite location with SnapMirror.

That in itself isn’t the bad thing, however we decided against Protection Manager (since it was a charged product back when we decided on this). So I basically had the three tasks:

Actually implement the SnapVault stuff (and learn my way around it and also document it)
Write a bunch of scripts, that help us in creating scheduled backups of our databases
Create a monitoring script, that’ll fit into our Nagios environment already in place

Well, two months later (sadly it still has some kinks - I can’t figure out this one bug though for the life of it) and a few hundred hours of working on/with it and out came four things:

NetApp: Establishing SnapVault relations

Fri, 08 Aug 2014 08:22:46 +0000

I’ve been spending a lot of my time the last week on getting SnapVault with out FAS-filers to work. Out came a script, which does this for a given volume (and of course SnapVault Primary and Secondary).

The script expects, that SSH public key authentification has been set up.

Nexus 5000: Configure CEST

Tue, 24 Jun 2014 21:07:52 +0000

Since I’m living and working in Germany, most of my hardware is using timezone configuration for CEST (or Europe/Berlin).

This here is the simple configuration for our Nexus 5000’s:

1
2
3
4
5


conf t
clock timezone CET 1 0
clock summer-time CEST 5 Sun Mar 02:00 5 Sun Oct 03:00 60
exit
copy running-config startup-config

UCS 5108 power redundancy lost

Sun, 22 Jun 2014 10:41:22 +0000

Well, another day - another UCS error. Out of the blue, one of our chassis started displaying that one PSU had failed, however the UCS was showing no PSU had failed 🤷

Well, as it turns out - this is yet another known bug in 2.0.2(r). You’ll either have to unplug and plug all the power cables (that’s four) in a maintainance window - or simply change the Equipment Power Policy (found in the Root of your UCS, tab Policy)

NetApp: Monitoring of SnapVault/SnapMirror/LUN/Snapshot information with Nagios

Sun, 15 Dec 2013 15:51:03 +0000

As I wrote before, we have a bunch of filers (and a ton of volumes w/ luns on them), that I need to monitor. At first, I tried the existing NetApp Nagios-Plugin(s), but they all use SNMP and with that I can either watch all volumes or none. And that didn’t satisfy me.

Don’t get me wrong, the existing plugins are okay and I still use them for stuff (like GLOBALSTATUS or FAN/CPU/POWER) which isn’t present in the API or real hard to get at, however I wanted more. So I ended up looking at the NetApp API, and ended up writing a “short” plugin for Nagios using Perl.

Maybe if I’m ever bored, I’ll rewrite it using C, but for now the Perl plugin has to suffice.

So far the plugin supports the following things:

Monitoring FlexVolumes (simply watching the free space)
Monitoring LUN space (the allocated space inside a FlexVolume for iSCSI/FC LUNs)
Monitoring Snapshot space (the allocated space inside a FlexVolume for Snapshots)
Monitoring SnapVault relations (and their age)
Monitoring SnapMirror relations (and their age)

The plugin will return performance data for most (if not all) of those classes. It needs a user on the filer you wish to monitor - which sadly needs to have the admin role.

NetScaler: Generate a simple usage statistic per Vserver

Thu, 21 Nov 2013 16:39:08 +0000

One of my co-workers recently approached me, that he needed a simple shell script which would generate a simple report about a Vserver’s current connections. After ironing out a few things with him (he had the intention of it being on a CIFS share on our file-server - which I changed to a simple HTML page) I went to work.

Out came two scripts. One is the collection instance, and the other is the processing instance. First the collection script runs, finds the current HA master node and then collects the Vserver’s current connections. After that script has dumped the information (date, time, current connections) into a file, the processing script will go and create a simple HTML page that’ll show exactly those informations.

VMware templates: post-processing for SUSE Linux Enterprise Server and openSUSE

Thu, 21 Nov 2013 16:31:40 +0000

Well, I’ve been fiddling around with SLES and openSUSE VMware templates. I know it’s a stupid idea when you have a PXE server from which you could install this in a matter of minutes (seriously the SLES PXE installation takes about 5 minutes).

However, when dealing with DMZ’s (yeah, they exist!) you usually don’t have any PXE servers there. So I decided to go with simple VMware templates (like we do with Windows already), but had to iron out a few kinks.

MDS9100 firmware updates - generating copy commands

Sun, 20 Oct 2013 11:20:02 +0000

Well, I went to work today … yeah, I know it’s Sunday right ? I ended up updating two MDS9148 switches and I didn’t want to figure out everything all over again. So I put the system image and kickstart onto one of our FTP servers and ran a short bash loop on it:

1
2
3
4


root:(ftp.daheim.heimdaheim.de) PWD:~
Sun Oct 20, 08:57:24 [0] > for file in `ls /srv/ftp/firmware/mds9148/*6.2.3*`; do
echo "copy ftp://10.0.0.55/`echo $file | sed 's,/srv/ftp/,,'` bootflash:/`echo $file | sed 's,/srv/ftp/firmware/mds9148/,,'`"
done

Now that’ll generate me two lines, which in turn I can use on the MDS’n:

XenServer 6-x: Update all hosts in pool

Sun, 15 Sep 2013 16:23:51 +0000

Well, what annoyed me in the past was that I had to patch each XenServer patch by patch (no bulk applying) and when used in combination with UCS blades (especially if those have >250GB RAM), it takes ages to keep a pool up-to-date. So I ended up writing yet another script (I know why I hate Citrix XenServer … the XenCenter GUI is lacking sooooo much) which will download new patches from a directory on a HTTP server and then print the lines necessary to apply the patches to all hosts in a pool.

XenServer 6-x: Quick VM Protection Policy to VM name-label script

Sun, 15 Sep 2013 16:14:04 +0000

Well, today I ended up writing a short script that’ll give me a list of VMPPs with the VMs that are associated to it.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12


#!/bin/bash

# Get a list of VMPPs
for vmpp in `xe vmpp-list params=uuid --minimal | sed "s/,/ /g"`; do
 VMPP_NAME=`xe vmpp-list params=name-label uuid=$vmpp --minimal`

 for vm in `xe vmpp-list params=VMs --minimal uuid=$vmpp | sed -e "s/;//g" -e "s/,//g"`; do
 VM=`xe vm-list params=name-label uuid=$vm --minimal`

 echo "$VMPP_NAME: $VM"
 done
done

vmware-config-tools-pl finished with Could not find Parent Node

Mon, 09 Sep 2013 19:15:53 +0000

Well, today I encountered a old problem (or so I thought). Basically a specific udevadm version causes the vmware-config-tools.pl script to error out like this:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16


The vmblock enables dragging or copying files between host and guest in a
Fusion or Workstation virtual environment. Do you wish to enable this feature?
[no]

!!! [EXPERIMENTAL] !!!
VMware automatic kernel modules enables automatic building and installation of
VMware kernel modules at boot that are not already present. By selecting yes,
you will be enabling this experimental feature. You can always disable this
feature by re-running vmware-config-tools.pl.

Would you like to enable VMware automatic kernel modules?
[no]

No X install found.

Could not find Parent Node...

I’ve had encountered this before in the past, and before there was some explanation on the VMware forums, which I couldn’t locate. Lucky me, the VMware Tools updater keeps modified versions of vmware-tools-config.pl around. So I ended up creating this short diff, so that I may find it in the future - if I still need it:

XenServer 6-0-2: Fixing Root-Disk-Multipathing with Boot-from-SAN

Tue, 16 Jul 2013 16:27:12 +0000

As the title pretty much tells, I’ve been working on fixing the Root-Disk-Multipathing feature of our XenServer installations. Our XenServer boot from a HA-enabled NetApp controller, however we recently noticed that during a controller fail-over some, if not all, paths would go offline and never come back. If you do a cf takeover and cf giveback in short succession, you’ll end up with a XenServer host that is unusable, as the Root-Disk would be pretty much non-responsive.

Generate Nagios config for check_netapp-api-pl

Tue, 05 Feb 2013 19:46:24 +0000

As so often, I wanted a script, that’ll crawl my filers and regenerate the configuration if there are any new volumes/snapvaults/snapmirrors or if one of them has been removed.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96


#!/bin/bash

FAS_HOSTS="$( ls /etc/nagios/objects/hosts/san/fas*{a,b}.cfg | cut -d/ -f7 | cut -d. -f1 )"

for host in $FAS_HOSTS; do
 OUTPUT_FILE=/etc/nagios/objects/hosts/san/$host-vol.cfg

 # Clear the output file
 echo "" > $OUTPUT_FILE

 # Get the volume list
 for volume in `ssh $host vol status | awk '{ print $1 }' | grep ^vol | sort -u | grep -v vol0$`; do
 user="$( grep "USER=" /etc/netapp-sdk/$host | cut -d= -f2 )"
 pass="$( grep "PASS=" /etc/netapp-sdk/$host | cut -d= -f2 )"
# echo "define service {"
# echo " use generic-service"
# echo ""
# echo " check_command check_netapp-volfree!$user!$pass!${volume}!92!98"
# echo " check_interval 5"
# echo " host_name ${host}"
# echo " notifications_enabled 0"
# echo " notification_interval 720"
# echo " service_description VOLSPACE ${volume}"
# echo "}"
 echo
 echo "define service {"
 echo " use generic-service-san-perfdata"
 echo ""
 echo " check_command check_netapp-lunspace!$user!$pass!${volume}"
 echo " check_interval 5"
 echo " host_name ${host}"
 echo " notifications_enabled 0"
 echo " notification_interval 720"
 echo " service_description LUNSPACE ${volume}"
 echo "}"
 echo

 SR="$( ssh $host snap reserve $volume | cut -d -f7 )"
 if [ "$SR" != "0%" ] ; then
 echo "define service {"
 echo " use generic-service-san-perfdata"
 echo ""
 echo " check_command check_netapp-snapreserve!$user!$pass!${volume}"
 echo " check_interval 10"
 echo " host_name ${host}"
 echo " notifications_enabled 0"
 echo " notification_interval 720"
 echo " # SR: $SR"
 echo " service_description SNAPRESERVE ${volume}"
 echo "}"
 echo
 fi
 done | tee -a $OUTPUT_FILE

 # Check snapvault foo
 for sv in `ssh $host snapvault status -l 2>/dev/null | awk '{ print $2 }' | grep vol`; do
 # only do the checks on sv_secondary
 if [ "$( echo $sv | grep $host | cut -d: -f1 )" == "${host}" ]; then
 vol="$( echo $sv | cut -d/ -f3 )"
 user="$( grep "USER=" /etc/netapp-sdk/$host | cut -d= -f2 )"
 pass="$( grep "PASS=" /etc/netapp-sdk/$host | cut -d= -f2 )"
 echo "define service {"
 echo " use generic-service-san-perfdata"
 echo ""
 echo " check_command check_netapp-snapvault!$user!$pass!$vol!38!42!"
 echo " check_interval 60"
 echo " host_name ${host}"
 echo " notifications_enabled 0"
 echo " notification_interval 720"
 echo " service_description SNAPVAULT ${vol}"
 echo "}"
 echo
 fi
 done | tee -a $OUTPUT_FILE

 # Check snapmirror foo
 for sm in `ssh $host snapmirror status 2>/dev/null | awk '{ print $2 }' | grep vol | grep $host`; do
 # only do the checks on sm_secondary
 if [ "$( echo $sm | grep $host | cut -d: -f1 )" == "${host}" ]; then
 vol="$( echo $sm | cut -d/ -f3 | cut -d: -f2 )"
 user="$( grep "USER=" /etc/netapp-sdk/$host | cut -d= -f2 )"
 pass="$( grep "PASS=" /etc/netapp-sdk/$host | cut -d= -f2 )"
 echo "define service {"
 echo " use generic-service-san-perfdata"
 echo ""
 echo " check_command check_netapp-snapmirror!$user!$pass!$vol!38!42!"
 echo " check_interval 60"
 echo " host_name ${host}"
 echo " notifications_enabled 0"
 echo " notification_interval 720"
 echo " service_description SNAPMIRROR ${vol}"
 echo "}"
 echo
 fi
 done | tee -a $OUTPUT_FILE
done

Generate Nagios config for NetApp filers

Tue, 01 Jan 2013 09:46:39 +0000

At some point in the last few weeks, I repeatedly had to recreate my Nagios config for currently six filers. After doing that a few times, I ended up (like sooo often) writing a short Bash script, that’ll do this for me - without any fuss.

The only thing the script needs, is that the filers and the filers are registered in DNS … Here’s an example:

1
2
3
4


fas3240a IN A 172.31.76.150
fas3240a-sp IN A 172.31.74.150
fas3240b IN A 172.31.76.151
fas3240b-sp IN A 172.31.74.151

With that done, the script will create the necessary Nagios config for those filers.

NetApp: Establishing SnapMirror relationships

Mon, 31 Dec 2012 09:44:19 +0000

After figuring out the SnapVault stuff, I needed to implement a whole bunch of SnapMirror relations. As I am lazy (as in click-lazy), I ended up writing a somewhat short Bash script, that’ll either establish a bunch of SnapMirror relations (for a single host) or just for a single volume.

The script expects, that SSH public key authentification has been set up, and that the source for the SnapMirror exists and is online/not-restricted.

NetApp: SnapVault snapshot retention for non-standard snapshot names

Sat, 29 Dec 2012 09:37:48 +0000

Well, the name says it pretty much. Once you rename the snapshot on the SnapVault destination from daily.0 to something else, the whole builtin SnapVault snapshot retention isn’t gonna work anymore.

Back when I started all the code-writing, I wasn’t aware of this. One of my co-worker complained to me about it on Wednesday that there are an assfull of snapshots on the SnapVault destination (one snapshot each day since the end of October, meaning more than 50 snapshots per volume, in a total of 12 or so FlexVolumes, making the total about 500 snapshots).

So I took the time to write this little Bash script (yeah, I know I’m mixing a bunch of languages - I really like the KISS principle), which will get the necessary information from the filer (snapvault snap sched needs to be set) and then deletes the over-aged snapshots.

NetApp: Archive SnapManager SQL Snapinfo

Sat, 29 Dec 2012 09:16:45 +0000

The MSSQL admins decided to dump the SMSQL Snapinfo stuff on a separate volume, that SMSQL also snapshots. Same as before, I need a PowerShell script that’ll archive the snapshot and rename it.

NetApp: Archive SnapManager Oracle Snapshots

Sat, 29 Dec 2012 09:12:57 +0000

And here’s the script for SMO. However, since different people administrate the Oracle Databases, they don’t want me to tinker with the database like the MSSQL admins. They give me a CSV-list of volumes, that should be backed up and I work with that.

MDS9000: Setting summer time for CET

Wed, 17 Oct 2012 20:37:36 +0000

After rebuilding two MDS9148, I wanted them to correctly switch the summer/winter time for my time zone. Currently I’m in CET (or CEST during the summer), so I googled for that. The search came up with Cisco-FAQ, however that needed a slight adjustment.

Apparently the NXOS doesn’t support the feature “recurring” in the clock configuration. So I had to slightly adapt the configuration line:

1
2


clock timezone MET 2 0
clock summer-time MET 5 Sun Mar 02:00 5 Sun Oct 03:00 60

vCenter: Removing VSC custom attributes

Wed, 17 Oct 2012 20:10:01 +0000

Well, yesterday I got pissed of those Virtual Storage Console custom attributes.

Currently we don’t use the Provisioning & Cloning feature of the VSC, thus we don’t need the custom attributes. After poking around, I decided to write a short PowerCLI script to do the task.

It’s really rather simple, so here goes:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23


param( [string] $vCenter )

# Add the VI-Snapin if it isn't loaded already
if ( (Get-PSSnapin -Name "VMware.VimAutomation.Core" -ErrorAction SilentlyContinue) -eq $null )
{
 Add-PSSnapin -Name "VMware.VimAutomation.Core"
}

If ( !($vCenter) )
{
 Write-Host
 Write-Host "vcenter-remove-vsc-attributes: <vcenter-server>"
 Write-Host
 Write-Host " <vcenter-server> - DNS name of your vCenter server."
 Write-Host
 exit 1
}

Connect-VIServer -Server $vCenter

Remove-CustomAttribute -CustomAttribute "PnC.CustSpec", "PnC.Deployed", "PnC.GroupID", "PnC.Source" -Confirm:$false

Disconnect-VIServer -server $vCenter -Confirm:$false

NetApp: Changing DS4243 shelf ID

Thu, 11 Oct 2012 21:17:39 +0000

I’m working on a project right now, providing a SnapVault target for our “big” NetApp. So we moved our 3240 to it’s target location, I spent most of my time yesterday doing the cabling (SAS and ACP, as well as power).

I’m still not finished, I still need to " beautify" the power cables, need to fix the network cables (currently I don’t have none ….) and some other minor stuff. But lemme skip back a bit.

NetApp: Migrating FCP luns with ndmpcopy to another controller

Sat, 06 Oct 2012 06:31:10 +0000

Well, I’m in a situation, where I need to move all volumes from one controller to two others. So I looked at the ways I had available:

Freshly implementing everything: No option at all!
vol copy: Is rather slow, thus no option
ndmpcopy: That’s exactly what I needed!

ndmpcopy is a great way to copy over a whole volume including it’s files (thus FCP luns) to another volume/controller.

First I threw in a crossover cable, since at around 6 PM our backup system starts it’s daily run, and everything else running via IP in between 6 PM and 6 AM is seriously impaired by this. Configured the additional ports on all three controllers (picked a private, not-routed range just in case) and then kicked of a simple bash script that ran the following:

NetApp LUN creation/vol sizing

Thu, 20 Sep 2012 04:49:43 +0000

Well, as you might know I’ve been tinkering with a NetApp FAS at work. The last few months, I’ve been trying to figure out a few things, which I actually did.

One “error” I ran into with creating the lun’s and volumes by hand was that the volumes were running out of space. Even if the volume was a bit larger than the LUN. After that happened a few times, I decided to see how to fix that. As it turns out, the GUI “fixes” that already in a way I wouldn’t have expected.

UCS 5108: VIF down

Mon, 17 Sep 2012 18:20:26 +0000

Well, I have yet another weird UCS problem. I have a single blade, that has trouble with it’s primary fabric attachment.

The problem get’s even more weird, if you look at the details.

After looking at the IO modules, the error doesn’t become any clearer:

So far, I have tried nearly everything. I’ve tried resetting the active and passive Connectivity of the vNIC, I tried resetting the DCE adapter for the vNIC, but nothing. I even tried resetting the vHBA that’s associated with this fabric, but that didn’t result to anything. Not even the usual flogi (fibre channel login) errors, that you get when either booting/resetting the blade.

UCS 5108: Power problem

Tue, 04 Sep 2012 17:09:12 +0000

Well, I recently had yet another UCS display/I2C communication problem. Somehow one of my chassis’ started to think, that the power redundancy was lost.

After looking at it a bit deeper, it seems only the GUI or the chassis did notice this power glitch:

As you can see, all PSU’s still have power. Now, since I had a big maintainance window the last weekend anyhow (and I spent ~14 hours at work), I decided to restart the IO modules in that chassis. And guess what: The error is gone! Another weird I2C communication issue with the firmware release 2.0.2 …

SMT mirroring troubles

Thu, 30 Aug 2012 17:46:27 +0000

Well, I’ve been having issues with our SMT. Basically after the release of SLES11 SP2 in February, I’ve been waiting for the repositories to turn “mirrorable”, which they haven’t yet. So I wrote a mail to the EMEA customer support.

And yesterday I got a reply, stating that my mirror credentials had been regenerated. And guess what ? I can mirror stuff again. So if you ever have any issues like the ones I described (or any of the people in the forum threads experienced), write a mail to your Novell customer support. 😛

SnapManager for SQL Server: Service fails to respond in a timely fashin

Thu, 30 Aug 2012 17:34:58 +0000

Well, we recently upgraded the SnapManager version on our test box to 6.4.1. Now however, after restarting the box the SnapManager service failed to start … The error was something like this:

Now, first I stumbled upon this NetApp Community post, which only contained the “solution” to increase the global! wait time for services. That didn’t sit well with me.

So after looking through NOW! for a bit, I actually found the correct way. The fix is described in KB2010835. Yet again, another certificate error. Why do vendors deploy SSL certificates, when they use untrusted ones, which defeats the purpose of SSL certs or at least “brings up” users to ignore any error message they get concerning SSL certificates ?

Total Commander: Generate the sum of all directories

Tue, 07 Aug 2012 07:39:58 +0000

In a Total Commander file list press Shift+Alt+Enter to calculate the directory size of all sub-directories.

Installserver: Publish a VMware ESX ISO - register-vmware

Tue, 24 Jul 2012 16:21:12 +0000

Here’s an old script, that also uses the magic provided by pxe-menu-generation (the script I posted before), but for VMware ESX/ESXi.

 1
 2
 3
 4
 5
 6
 7
 8
 9
 10
 11
 12
 13
 14
 15
 16
 17
 18
 19
 20
 21
 22
 23
 24
 25
 26
 27
 28
 29
 30
 31
 32
 33
 34
 35
 36
 37
 38
 39
 40
 41
 42
 43
 44
 45
 46
 47
 48
 49
 50
 51
 52
 53
 54
 55
 56
 57
 58
 59
 60
 61
 62
 63
 64
 65
 66
 67
 68
 69
 70
 71
 72
 73
 74
 75
 76
 77
 78
 79
 80
 81
 82
 83
 84
 85
 86
 87
 88
 89
 90
 91
 92
 93
 94
 95
 96
 97
 98
 99
100
101
102
103
104
105
106
107
108
109
110
111


#!/bin/bash

TFTP_DIR=/srv/tftp
INSTSRC_DIR=/srv/instsrc

if [ -z $@ ] ; then
 echo "Usage: register-vmware <path to iso-file> labelname"
 exit 1
fi

#set -x

ISO_FILE=$1
mkdir /mnt/loop
mount -o loop $ISO_FILE /mnt/loop

if [ -f /mnt/loop/packages.xml ] ; then
 # Determine ISO information
 OS_LABEL="$( grep "<name>" /mnt/loop/packages.xml | sed -e "s,<name>,," -e "s,</name>,," )"
 OS_NAME="$( grep "<ProductLineId>" /mnt/loop/packages.xml | sed -e "s,<ProductLineId>,," -e "s,</ProductLineId>,," )"
 OS_RELEASE="$( grep "<version>" /mnt/loop/packages.xml | sed -e "s,<version>,," -e "s,</version>,," )"
 OS_BUILD="$( grep "<release>" /mnt/loop/packages.xml | sed -e "s,<release>,," -e "s,</release>,," )"

 shopt -s extglob
 OS_LABEL=${OS_LABEL##*( )}
 OS_NAME=${OS_NAME##*( )}
 OS_RELEASE=${OS_RELEASE##*( )}
 OS_BUILD=${OS_BUILD##*( )}
elif [ -f /mnt/loop/README.txt ] ; then
 OS_LABEL=""
 OS_NAME=""
 OS_RELEASE=""
 OS_BUILD=""
fi

# Create directories
mkdir -p $TFTP_DIR/boot/$OS_NAME/$OS_RELEASE-$OS_BUILD
mkdir -p $TFTP_DIR/pxelinux.cfg/$OS_NAME
mkdir -p $INSTSRC_DIR/$OS_NAME/$OS_RELEASE-$OS_BUILD

#set +x

# Copy CD/DVD content
rsync -av /mnt/loop/* $INSTSRC_DIR/$OS_NAME/$OS_RELEASE-$OS_BUILD

# Copy the PXE boot file (initrd/kernel)
cp $INSTSRC_DIR/$OS_NAME/$OS_RELEASE-$OS_BUILD/isolinux/initrd.img
 $TFTP_DIR/boot/$OS_NAME/$OS_RELEASE-$OS_BUILD/initrd.img

cp $INSTSRC_DIR/$OS_NAME/$OS_RELEASE-$OS_BUILD/isolinux/vmlinuz
 $TFTP_DIR/boot/$OS_NAME/$OS_RELEASE-$OS_BUILD/vmlinuz

# Create the PXE boot menu
cat > $TFTP_DIR/pxelinux.cfg/$OS_NAME/$OS_NAME-$OS_RELEASE-$OS_BUILD.menu << EOF
#LABEL $OS_LABEL $OS_RELEASE - build $OS_BUILD
#OSNAME $OS_NAME
#OSARCH x86_64

MENU TITLE VMware Installservice
MENU BACKGROUND addons/background-vmware.png

MENU COLOR screen 37;40 #80ffffff #00000000
MENU COLOR border 0 #ffffffff #ee000000 std
MENU COLOR title 0 #ffffffff #ee000000 std
MENU COLOR unsel 0 #ffffffff #ee000000 std
MENU COLOR sel 0 #ffffffff #85000000 std
MENU COLOR scrollbar 30;44 #40000000 #00000000
MENU COLOR tabmsg 0 #ee000000 #ffffffff std
MENU COLOR cmdmark 0 #ff00ff00 #00000000 std
MENU COLOR cmdline 0 #ee000000 #ffffffff std
MENU COLOR timeout_msg 0 #ee000000 #ffffffff std
MENU COLOR timeout 0 #ee000000 #ffffffff std
MENU COLOR disabled 0 #ffffffff #ee000000 std
MENU COLOR pwdheader 0 #ff000000 #99ffffff rev
MENU COLOR pwdborder 0 #ff000000 #99ffffff rev
MENU COLOR pwdentry 0 #ff000000 #99ffffff rev
MENU COLOR hotkey 0 #ff00ff00 #ee000000 std
MENU COLOR hotsel 0 #ffffffff #85000000 std

LABEL DESC
MENU LABEL $OS_LABEL $OS_RELEASE
MENU DISABLE

MENU SEPARATOR

LABEL kickstart
MENU LABEL ^Kickstart Installation
KERNEL boot/$OS_NAME/$OS_RELEASE-$OS_BUILD/vmlinuz
APPEND initrd=boot/$OS_NAME/$OS_RELEASE-$OS_BUILD/initrd.img vmkopts=debugLogToSerial:1 mem=512M ks=http://install.home.barfoo.org/kickstart/$OS_NAME-$OS_RELEASE-$OS_BUILD.cfg HOSTNAME=
IAPPEND 1

TEXT HELP
Es ist zu beachten, das mit TAB der Hostname angepasst werden muss!
ENDTEXT

LABEL netinst
MENU LABEL ^Netzwerkbasierte Installation
KERNEL boot/$OS_NAME/$OS_RELEASE-$OS_BUILD/vmlinuz
APPEND initrd=boot/$OS_NAME/$OS_RELEASE-$OS_BUILD/initrd.img vmkopts=debugLogToSerial:1 mem=512M url=http://install.home.barfoo.org/instsrc/$OS_NAME/$OS_RELEASE-$OS_BUILD
IAPPEND 1

MENU SEPARATOR

LABEL back
MENU LABEL <-- ^Vorherige Ansicht
KERNEL addons/vesamenu.c32
APPEND pxelinux.cfg/$OS_NAME.menu
EOF

umount /mnt/loop
rmdir /mnt/loop

Installserver: Publish a SLES/OpenSUSE ISO - register-suse

Tue, 24 Jul 2012 13:20:34 +0000

Well, I’ve been tinkering with a way to “unify” the way we deploy new OpenSUSE/SLES ISOs onto our installation server, once they are released. So here, I’ll show you the script I came up with:

 1
 2
 3
 4
 5
 6
 7
 8
 9
 10
 11
 12
 13
 14
 15
 16
 17
 18
 19
 20
 21
 22
 23
 24
 25
 26
 27
 28
 29
 30
 31
 32
 33
 34
 35
 36
 37
 38
 39
 40
 41
 42
 43
 44
 45
 46
 47
 48
 49
 50
 51
 52
 53
 54
 55
 56
 57
 58
 59
 60
 61
 62
 63
 64
 65
 66
 67
 68
 69
 70
 71
 72
 73
 74
 75
 76
 77
 78
 79
 80
 81
 82
 83
 84
 85
 86
 87
 88
 89
 90
 91
 92
 93
 94
 95
 96
 97
 98
 99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128


#!/bin/bash

TFTP_DIR=/srv/tftpboot
INSTSRC_DIR=/srv/instsrc

if [ -z $@ ] ; then
 echo "Usage: register-suse <path to iso-file> labelname"
 exit 1
fi

ISO_FILE=$1

mkdir /mnt/loop
mount -o loop $ISO_FILE /mnt/loop

# Determine ISO information
OS_LABEL="$( grep '^LABEL' /mnt/loop/content | cut -d -f10- )"
if [ -z "$OS_LABEL" ] ; then
 OS_LABEL="$( grep '^LABEL' /mnt/loop/content | cut -d -f2- | head -n1 )"
fi

OS_NAME="$( grep '^DISTRIBUTION' /mnt/loop/content | cut -d -f3- | tr '[:upper:]' '[:lower:]' )"

if [ -z "$OS_NAME" ] ; then
 OS_NAME="$( grep '^PRODUCT' /mnt/loop/content | cut -d -f2- | cut -d_ -f1-2 | tr '[:upper:]' '[:lower:]' )"
fi

case "$OS_NAME" in
 suse_sle|suse_sles) OS_NAME=sles;;
esac

OS_RELEASE="$( grep ^VERSION /mnt/loop/content | awk '{ print $2 }' | cut -d- -f1 )"

if [ "$( echo $OS_LABEL | grep -i vmware )" != "" ] ; then
 OS_RELEASE="$OS_RELEASE-vmware"
fi

OS_ARCH="$( grep ^BASEARCHS /mnt/loop/content | awk '{ print $2 }' )"

if [ -z "$OS_ARCH" ] ; then
 OS_ARCH="$( grep ^DEFAULTBASE /mnt/loop/content | awk '{ print $2 }' )"
fi

OS_SHORTLABEL="$( grep ^SHORTLABEL /mnt/loop/content | awk '{ print $2 }' | cut -d( -f1 | sed -e "s,SLES-,SLES," -e "s,-, ,g" )"

case "$OS_ARCH" in
 i*86) OS_ARCH=x86;;
 x86_64) OS_ARCH=x64;;
esac

# Create directories
mkdir -p $TFTP_DIR/boot/$OS_NAME/$OS_RELEASE/$OS_ARCH
mkdir -p $TFTP_DIR/pxelinux.cfg/$OS_NAME
mkdir -p $INSTSRC_DIR/$OS_NAME/$OS_RELEASE/$OS_ARCH

# Copy CD/DVD content
rsync -av /mnt/loop/* $INSTSRC_DIR/$OS_NAME/$OS_RELEASE/$OS_ARCH/

# Create the info file
cat > $INSTSRC_DIR/$OS_NAME/$OS_RELEASE/$OS_ARCH/info <<EOF
Language: en_US
Keytable: de-lat1-nd
Install: http://install.home.barfoo.org/{INSTSRC_DIR//srv/}/$OS_NAME/$OS_RELEASE/$OS_ARCH/
Autoyast: http://install.home.barfoo.org/autoyast/
Textmode: 1
EOF

# Copy the PXE boot file (initrd/kernel)
cp $INSTSRC_DIR/$OS_NAME/$OS_RELEASE/$OS_ARCH/boot/*/loader/initrd
 $TFTP_DIR/boot/$OS_NAME/$OS_RELEASE/$OS_ARCH/initrd

cp $INSTSRC_DIR/$OS_NAME/$OS_RELEASE/$OS_ARCH/boot/*/loader/linux
 $TFTP_DIR/boot/$OS_NAME/$OS_RELEASE/$OS_ARCH/linux

# Create the PXE boot menu
cat > $TFTP_DIR/pxelinux.cfg/$OS_NAME/$OS_NAME-$OS_RELEASE-$OS_ARCH.menu << EOF

#LABEL $OS_SHORTLABEL
#OSNAME $OS_NAME
#OSARCH $OS_ARCH

MENU TITLE Linux Installationservices
MENU BACKGROUND addons/background-suse.png

MENU COLOR screen 37;40 #80ffffff #00000000
MENU COLOR border 0 #ffffffff #ee000000 std
MENU COLOR title 0 #ffffffff #ee000000 std
MENU COLOR unsel 0 #ffffffff #ee000000 std
MENU COLOR sel 0 #ffffffff #85000000 std
MENU COLOR scrollbar 30;44 #40000000 #00000000
MENU COLOR tabmsg 0 #ee000000 #ffffffff std
MENU COLOR cmdmark 0 #ff00ff00 #00000000 std
MENU COLOR cmdline 0 #ee000000 #ffffffff std
MENU COLOR timeout_msg 0 #ee000000 #ffffffff std
MENU COLOR timeout 0 #ee000000 #ffffffff std
MENU COLOR disabled 0 #ffffffff #ee000000 std
MENU COLOR pwdheader 0 #ff000000 #99ffffff rev
MENU COLOR pwdborder 0 #ff000000 #99ffffff rev
MENU COLOR pwdentry 0 #ff000000 #99ffffff rev
MENU COLOR hotkey 0 #ff00ff00 #ee000000 std
MENU COLOR hotsel 0 #ffffffff #85000000 std

LABEL DESC
 MENU LABEL $OS_LABEL ($OS_ARCH)
 MENU DISABLE

MENU SEPARATOR

LABEL autoyast
 MENU LABEL ^Automatisierte Installation
 KERNEL boot/$OS_NAME/$OS_RELEASE/$OS_ARCH/linux
 APPEND initrd=boot/$OS_NAME/$OS_RELEASE/$OS_ARCH/initrd info=http://install.home.barfoo.org/${INSTSRC_DIR//srv/}/$OS_NAME/$OS_RELEASE/$OS_ARCH/info splash=native ramdisk_size=65535 vga=791 barrier=off

LABEL netinst
 MENU LABEL ^Netzwerkbasierte Installation
 KERNEL boot/$OS_NAME/$OS_RELEASE/$OS_ARCH/linux
 APPEND initrd=boot/$OS_NAME/$OS_RELEASE/$OS_ARCH/initrd install=http://install.home.barfoo.org/${INSTSRC_DIR//srv/}/$OS_NAME/$OS_RELEASE/$OS_ARCH/ splash=native ramdisk_size=65535 vga=791 barrier=off

MENU SEPARATOR

LABEL back
 MENU LABEL <-- ^Vorherige Ansicht
 KERNEL addons/vesamenu.c32
 APPEND pxelinux.cfg/$OS_NAME.menu
EOF

umount /mnt/loop
rmdir /mnt/loop

This also needs a second script, as this will only create a menu entry, not the menu itself. You’ll find the script in the next post!

VMs in Alarm state after scheduled maintainance

Sun, 22 Jul 2012 10:38:04 +0000

Well, I’m back at work after three weeks of vacation (some pictures to follow) and the provider hosting our disaster datacenter had their annual (or is it monthly now?) SAN maintainance, so we shut down everything over there by 9:00 am.

After things were back up around 5pm, I booted the ESX hosts, however the VMs we’re all displaying the alert state - as if either the VMs had an HA event or we’re using to much CPU time. It didn’t matter whether or not the VM was running or not, the state persisted.

NetApp - Remove LUN mappings

Fri, 15 Jun 2012 14:30:58 +0000

As promised in the earlier post, for completeness sake, here’s the counterpart for removing the LUNs in the first place.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41


 #!/bin/bash

KEY_FILE="/root/.ssh/netapp.dsa"
SSH_OPTS="/root/.ssh/netapp-ssh_config"

if [ $# -ne 2 ] ; then
 echo "fas-remove-lunmap: FAS_CONTROLLER TARGET_IGROUP"
 echo
 echo "Remove every mapped lun from the target igroup (for example reinstallation)"
 echo
 echo " Usage:"
 echo " - FAS_CONTROLLER: Hostname/IP-adress of the DATA ONTAP controller"
 echo " - TARGET_IGROUP: igroup that is actually modified"
 echo
 exit 1
fi

FAS_CTRL=$1
TARGET=$2

ssh_fas() {
 # $@: commands for Data ONTAP
 COMMANDS="$@"
 /usr/bin/ssh -i $KEY_FILE -l root -F $SSH_OPTS $COMMANDS
}

# Get the hostname of the controller, necessary for the reporting
CTRL_HOSTNAME="$( ssh_fas $FAS_CTRL rdfile /etc/rc | grep ^hostname | cut -d -f2 | tr 'a-z' 'A-Z' )"

#set -x
# Get the lun list.
for lun in $( ssh_fas $FAS_CTRL lun show -g $SOURCE | awk '{ print $1 }' | sort -u ); do

 # If the LUN id is 0, skip otherwise we would remove the boot LUN
 if [ "$LUN_ID" != "0" ] ; then
 # Actually remove the mapping to our host
 echo "Removing $lun from $TARGET"
 ssh_fas $FAS_CTRL lun unmap $lun $TARGET
 fi
done
#set +x

With that, you can simply run it against a NetApp controller and remove every LUN map except the one with LUN ID 0 (which is pretty handy when installing/reinstalling ESX servers).

NetApp - Get a list of volumes containing too much LUNs

Fri, 08 Jun 2012 19:04:53 +0000

Well, after figuring things out (and realizing that if you create a LUN in the same size as the volume it’ll break), I decided to write yet another script to figure out which LUNs needed fixing.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62


#!/bin/bash

KEY_FILE="/root/.ssh/netapp.dsa"
SSH_OPTS="/root/.ssh/netapp-ssh_config"
FAS_CTRL=$1

ssh_fas() {
 # $@: commands for Data ONTAP
 COMMANDS="$@"
 /usr/bin/ssh -i $KEY_FILE -l root -F $SSH_OPTS $COMMANDS
}

for volume in $( ssh_fas $FAS_CTRL vol status | grep "vol.* online" | awk '{ print $1 }' ); do
 VOL_SIZE="$( ssh_fas $FAS_CTRL vol size $volume | awk '{ print $8 }' | sed "s,.,," )"
 VOL_UNIT_PARAM="$( echo "${#VOL_SIZE} - 1" | bc )"
 VOL_UNIT="${VOL_SIZE:$VOL_UNIT_PARAM:1}"

 case $VOL_UNIT in
 k) CONV="1024";;
 m) CONV="1024 * 1024";;
 g) CONV="1024 * 1024 * 1024";;
 esac

 VOL_SIZE="$( echo "scale=0; ${VOL_SIZE/$VOL_UNIT/} * $CONV" | bc )"

 # Subtract the snap reserve (if any)
 SNAP="$( ssh_fas $FAS_CTRL snap reserve $volume | cut -d -f7 | sed "s,%,," )"

 if [ $SNAP -gt 0 ] ; then
 SNAP_SIZE="$( echo "$VOL_SIZE * $SNAP / 100" | bc )"
 REMAINING_VOL_SIZE="$( echo "$VOL_SIZE - $SNAP_SIZE" | bc )"
 else
 SNAP_SIZE=0
 REMAINING_VOL_SIZE=$VOL_SIZE
 fi

 # Get the luns created inside the volume
 LUN_SIZE_TOTAL=0
 for lunsize in $( ssh_fas $FAS_CTRL lun show -l ${volume} | awk '{ print $3 }' | sed -e "s,(,," -e "s,),," ); do
 LUN_SIZE_TOTAL="$( echo "$LUN_SIZE_TOTAL + $lunsize" | bc )"
 REMAINING_VOL_SIZE="$( echo "$REMAINING_VOL_SIZE - $lunsize" | bc )"
 done

 # Get the estimated space necessary.
 VOL_RESERVE="$( echo "$VOL_SIZE * 3 / 100" | bc )"
 echo " VOL_NAME: $volume"
 echo " VOLUME_SIZE: $( echo "scale=2; $VOL_SIZE / 1024 / 1024 / 1024" | bc )G"
 echo " - SNAP_SIZE: $( echo "scale=2; $SNAP_SIZE / 1024 / 1024 / 1024" | bc )G"
 echo " SUB_TOTAL: $( echo "scale=2; ($VOL_SIZE - $SNAP_SIZE) / 1024 / 1024 / 1024" | bc )G"
 echo
 echo " - TOTAL_LUN_SIZE: $( echo "scale=2; $LUN_SIZE_TOTAL / 1024 / 1024 / 1024" | bc )G"
 echo " SUB_TOTAL: $( echo "scale=2; ($VOL_SIZE - $SNAP_SIZE - $LUN_SIZE_TOTAL) / 1024 / 1024 / 1024" | bc )G"
 echo
 echo " MISSING_SPACE:"
 DIFFERENCE_VOL_LUN="$( echo "scale=2; ($VOL_SIZE - $SNAP_SIZE - $LUN_SIZE_TOTAL)" | bc | sed "s,-,," )"
 echo " DIFFERENCE_VOL_LUN: $( echo "scale=2; $DIFFERENCE_VOL_LUN / 1024 / 1024 / 1024" | bc )G"
 echo " + VOL_RESERVE: $( echo "scale=2; $VOL_RESERVE / 1024 / 1024 / 1024" | bc )G"
 echo " SUB_TOTAL: $( echo "scale=2; ($DIFFERENCE_VOL_LUN + $VOL_RESERVE) / 1024 / 1024 / 1024" | bc )G"
 echo
 echo
 echo
done

And with that you have a list of volumes, with the amount of space they need to resized in order to accomodate the contained LUNs and the snapshots.

XenServer - Automation Metdata backups

Tue, 05 Jun 2012 18:59:48 +0000

Well, we had some issues with XenServers “automated” metadata backup, so I decided - with the help of one of our consultants - to automate it on our own to an external target.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57


#!/bin/bash

# Crontab entry for each server:
# 02 5 * * * root /usr/local/sbin/xen-pool-backup.sh

# Get the pool name
POOL_NAME="$( xe pool-list | grep name-label | awk '{ print $4 }' )"
HOST_UUID="$( xe host-list hostname=`hostname` | grep "uuid ( RO)" |
 awk '{ print $5 }' )"

DAILY_GENERATIONS=7
WEEKLY_GENERATIONS=4

NFS_MOUNT="nfs.home.barfoo.org:/srv/xenbackup"
NFS_LOCAL="/tmp/backup-mount/$POOL_NAME"

# Figure out if we're the pool master
POOL_MASTER="$( xe pool-list | grep master | awk '{ print $4 }' )"

if [ "$POOL_MASTER" == "$HOST_UUID" ] ; then
 # Only the pool master should backup the pool database, as this is the only
 # one who has a authoritive pool database

 # Create the necessary directory and mount the NFS volume
 mkdir -p ${NFS_LOCAL%/*}
 mount -t nfs $NFS_MOUNT ${NFS_LOCAL%/*}
 mkdir -p $NFS_LOCAL

 if [ -f $NFS_LOCAL/daily.$DAILY_GENERATIONS.gz ]; then
 rm -f $NFS_LOCAL/daily.$DAILY_GENERATIONS.gz
 fi

 OLD_DAILY="$( echo "scale=0; $DAILY_GENERATIONS - 1" | bc )"

 for OLD in $( seq $OLD_DAILY -1 1 ); do
 if [ -f $NFS_LOCAL/daily.$OLD.gz ] ; then
 NEW="$( echo "scale=0; $OLD+1" | bc )"

 # Save the time stamp somewhere
 touch $NFS_LOCAL/.timestamp -r $NFS_LOCAL/daily.$OLD.gz
 mv $NFS_LOCAL/daily.$OLD.gz $NFS_LOCAL/daily.$NEW.gz

 # Restore the date
 touch $NFS_LOCAL/daily.$NEW.gz -r $NFS_LOCAL/.timestamp
 fi
 done

 [ -f $NFS_LOCAL/daily.0.gz ] && mv $NFS_LOCAL/daily.0.gz
 $NFS_LOCAL/daily.1.gz
 xe pool-dump-database file-name=$NFS_LOCAL/daily.0
 gzip -9 $NFS_LOCAL/daily.0

 [ -f $NFS_LOCAL/.timestamp ] && rm $NFS_LOCAL/.timestamp
 umount ${NFS_LOCAL%/*}

 rm -rf ${NFS_LOCAL%/*}
fi

With that, I have at least a daily backup - and in combination with our daily TSM backup, I have at least month long history of metadata backups.

NetApp - Get a list of all volumes not being used

Wed, 30 May 2012 17:40:12 +0000

Well, I had another task for today … I have an amount of FlexVolumes (sixty currently per controller), and I didn’t know if we had any, that didn’t have any LUNs on them. Now I thought there was a command for that since my co-worker mentioned something like that. However, once again … there isn’t.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39


#!/bin/bash

MAILTO="san@barfoo.org"
KEY_FILE="/root/.ssh/netapp.dsa"
SSH_OPTS="/root/.ssh/netapp-ssh_config"
FAS_CTRL=$1
TMPDIR="$( mktemp -d )"

ssh_fas() {
 # $@: commands for Data ONTAP
 COMMANDS="$@"
 /usr/bin/ssh -i $KEY_FILE -l root -F $SSH_OPTS $COMMANDS
}

# Get the hostname of the controller, necessary for the reporting
CTRL_HOSTNAME="$( ssh_fas $FAS_CTRL rdfile /etc/rc | grep ^hostname |
 cut -d -f2 | tr 'a-z' 'A-Z' )"

# Get a list of all volumes / luns
VOL_LIST="$( ssh_fas $FAS_CTRL vol status | grep ^vol |
 egrep -v '(nfs|cifs)' | awk '{ print $1 }' )"
LUN_LIST="$( ssh_fas $FAS_CTRL lun show | grep '/vol' |
 awk '{ print $1 }' )"

for lun in $LUN_LIST; do
 VOL_EXTRACT="$( echo $lun | cut -d/ -f3 )"
 VOL_LIST=${VOL_LIST/${VOL_EXTRACT}/}
done

for vol in $VOL_LIST; do
 echo "Empty Flex Volume: $vol."
done > $TMPDIR/mailcontent

if [ "$( grep Flex $TMPDIR/mailcontent )" ] ; then
 cat $TMPDIR/mailcontent | mailx -r $MAILTO
 -s "$CTRL_HOSTNAME: Empty volume check" $MAILTO
fi

rm -r $TMPDIR

Automating qual_devices updates

Wed, 30 May 2012 17:23:58 +0000

Well, once again I was presented with a nice AutoSupport warning once I logged into my NOW account. Since we don’t have CIFS and/or NFS licensed on our filers, I wrote a cute little script that does the whole work for me.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88


#!/bin/bash

TMPDIR="$( mktemp -d )"
KEY_FILE="/root/.ssh/netapp.dsa"
SSH_OPTS="/root/.ssh/netapp-ssh_config"
FAS_CTRL=$1
QUALDEVICES=$2

ssh_fas() {
 # $@: commands for Data ONTAP
 COMMANDS="$@"
 /usr/bin/ssh -i $KEY_FILE -l root -F $SSH_OPTS $COMMANDS
}
#set -x

echo "Updating qual_devices on $FAS_CTRL"

# Enable ftpd if it isn't enabled already
echo -n "Checking FTP subsystem"

FTPD_INITIAL_STATE="$( ssh_fas $FAS_CTRL options ftpd.enable |
 awk '{ print $2 }' )"
if [ $FTPD_INITIAL_STATE == "off" ] ; then
 ssh_fas $FAS_CTRL options ftpd.enable on
 echo " .... ok"
else
 echo " .... ok"
fi

echo
read -s -p "Please supply the root password for $FAS_CTRL: "
ROOT_PASSWD=$REPLY

echo
echo "Checking qual_devices:"
echo -n " Running: "
mkdir $TMPDIR/fas-version
# Check the old version.
ftp -n $FAS_CTRL >/dev/null <<END_SCRIPT
prompt
user root $ROOT_PASSWD
lcd $TMPDIR/fas-version
cd /etc
mget qual_devices*
quit
END_SCRIPT

FAS_VERSION="$( grep Datecode $TMPDIR/fas-version/qual_devices_v3 |
 head -n1 | cut -d -f3 )"

echo "$FAS_VERSION"
# Unzip the qual_devices.zip file and compare it.
ORIGPWD=$PWD
mkdir $TMPDIR/new-version
cd $TMPDIR/new-version
unzip $QUALDEVICES >/dev/null
echo -n " New: "
NEW_VERSION="$( grep Datecode $TMPDIR/new-version/qual_devices_v3 |
 head -n1 | cut -d -f3 )"
echo "$NEW_VERSION"
echo

# Upload the supplied version if the new file doesn't match the one running
# on the controller
if [ $NEW_VERSION != $FAS_VERSION ] ; then
 echo "Uploading qual_devices to $FAS_CTRL"
 ftp -n $FAS_CTRL >/dev/null <<END_SCRIPT
prompt
user root $ROOT_PASSWD
lcd $TMPDIR/new-version
cd /etc
mput qual_devices*
quit
END_SCRIPT

 # Send an asup message, that the issue is corrected
 echo "Generating AutoSupport message"
 ssh_fas $FAS_CTRL options autosupport.doit qual_devices_fixed_$NEW_VERSION
fi

#set +x

# Enable ftpd
if [ $FTPD_INITIAL_STATE == "off" ] ; then
 ssh_fas $FAS_CTRL options ftpd.enable off
fi

rm -r $TMPDIR

The whole thing is surly based, that FTP is configured ( as I described previously).

TSM and NetApp - Another Quick Hint

Sun, 27 May 2012 13:45:22 +0000

Well, we’ve been trying to come up with a decent way to backup NetApp snapshots to tape (SnapMirror To Tape), so we evaluated all the available methods of using NDMP backups.

There’s Image Backup in two different variants - FULL and DIFFerntial
There’s SnapMirror To Tape

So the Image Backup is one of the ways. However the DIFFerntial backup only works for CIFS and NFS shares (which we don’t use). We only have FC luns (or rather FCoE luns), so there’s only a single (or in case of the boot luns more than one) file in each volume. With that however, each run of the Image Backup with the DIFFerential option, it’s gonna backup the full size of the volume (plus the deduplicated amount).

vm-online-backup - Another day, another PowerCLI script

Mon, 30 Apr 2012 16:38:47 +0000

Well, on Friday I had a short chat with someone from one of our application departments, stating he wanted a backup copy of a VM (ain’t to hard), but a) they don’t want any downtime and b) it has to be identical to the original.

So I sat down today, googled for a bit and actually found something that pretty much does what I want, though I had to fix it up a bit … So find attached a script, which creates a hot-clone from a snapshot and then only if the latest clone was successful deletes the old one.

TSM and NetApp - Quick Hint

Wed, 25 Apr 2012 20:09:24 +0000

Well, to save everyone else the trouble (since it isn’t documented anywhere - and I just spent about an hour finding the cause for this), if you need to configure NDMP on your NetApp Filer, make sure you also configure an interface other than e0M.

Apparently the necessary controlport for NDMP (10000) is being blocked on e0M, thus ndmp may be configured and running, however TSM is gonna complain that it is unable to connect to the specified data mover.

SLES11-1 and updated multipath-tools

Sat, 07 Apr 2012 10:15:21 +0000

Well, after I scripted the installation the other day, I tried installing SLES11.1-Updates to the freshly installed systems. Guess what ? The thing broke. Initially (it was late Friday afternoon - like 6 PM - before my one week vacation) I didn’t have much time to debug the issue, so I sat down last week and looked at the issue.

During the installation, when first starting multipath via command line, the scsi-mpatha device appears, and each and every occurance of this is subsequentially being used (and other stuff replaced by this actually) during the whole installation phase.

VMware Update Manager issues

Sat, 31 Mar 2012 21:14:28 +0000

Well, I recently (last Wednesday) had a lot of trouble with Update Manager.

First I thought, upgrading vCenter and modules to 5.0U1 would solve my troubles, however it did not. Update Manager was still complaining about something. Since neither in the vCenter Update Manager nor the vCenter log itself were having any useful information I enabled SSHd and the ESXi Shell via the vCenter client:

SSH’ed into the ESX host and looked at /var/log/esxupdate.log, and found this particular log:

WDS and multi-architecture boot images

Fri, 24 Feb 2012 17:31:26 +0000

Well, I recently stumbled upon another cute bug/feature with Windows Deployment Services. When you already have 32bit boot images (as we do) and then add an 64bit boot image (which we needed, since the drivers for UCS firmware v2.0 only support Windows Server 2008 R2) you still only see the 32bit images.

Why ? Because apparently the client (in my case a UCS blade) isn’t reporting it’s architecture correctly in the PXE phase. Microsoft actually has a KB article for this. You only need to enable architecture discovery.

Microsoft Cluster on VMware and Devices

Sat, 11 Feb 2012 08:40:44 +0000

Well, once again the Microsoft Cluster on VMware bit my ass … As you might know, MSCS on VMware is a particular kind of pain, with each upgrade you end up with the same problem over and over again (SCSI reservations on the RDM-LUNs being one, and the passive node not booting being the other).

So I opened up another support case with VMware, and the responded like this:

Please see this kb entry: http://kb.vmware.com/kb/1016106

Reconfiguring NTP settings vCenter-wide

Sat, 04 Feb 2012 19:55:40 +0000

I recently started reinstalling all my ESX hosts, so I wrote up a short script that is reconfiguring all hosts and sets the NTP configuration according to my wish:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30


param( [string] $vcenter, [string] $ntpserver1, [string] $ntpserver2 )

# Add the VI-Snapin if it isn't loaded already
if ( (Get-PSSnapin -Name "VMware.VimAutomation.Core" -ErrorAction SilentlyContinue) -eq $null )
{
 Add-PSSnapin -Name "VMware.VimAutomation.Core"
}

If ( !($vcenter) -or !($ntpserver1) -or !($ntpserver2) )
{
 Write-Host `n "vcenter-ntp-reconfigure: <vcenter-server> <ntpserver1> <ntpserver2>" `n
 Write-Host "This script clears the NTP servers currently configured and" `n
 Write-Hsot "adds the ones supplied on the command line." `n
 Write-Host " <vcenter-server> - DNS name of your vCenter server." `n
 Write-Host " <ntpserver1> - NTP server #1" `n
 Write-Host " <ntpserver2> - NTP server #2" `n
 exit 1
}

Connect-VIServer -Server $vcenter

foreach ($esxhost in Get-VMHost)
{
 Get-VMHost $esxhost | Remove-VMHostNtpServer -Confirm:$false -NtpServer (Get-VMHost $esxhost | `
 Get-VMHostNtpServer)
 Get-VMHost $esxhost | Add-VMHostNtpServer -NtpServer $ntpserver1
 Get-VMHost $esxhost | Add-VMHostNtpServer -NtpServer $ntpserver2
}

Disconnect-VIServer -server $vcenter -Confirm:$false

As you can see, the script takes the vCenter hostname and two NTP servers and basically applies it to each host in your vCenter environment.

Using HPs vibdeposit with VMware Update Manager

Sat, 04 Feb 2012 17:34:28 +0000

As we’re finally at the point, where I only need to bother with HP hardware (which in itself is troublesome enough), I wanted to use HPs vibdeposit with our Update Manager. The whole purpose of the repository is the integration of HPs custom vibs (download able on each hardware under VMware ESXi 5.0) into the VMware Update Manager. That makes it easy to integrate, say the nmi-sourcing driver, into the VMware built ESXi images.

Emptying a VMFS datastore with PowerCLI

Thu, 26 Jan 2012 21:14:41 +0000

Well, once again I hacked at the Powershell/PowerCLI the other day. Since we don’t yet have a Enterprise Plus license at work (which would support Datastore Maintaince and Storage DRS), I needed a way to empty one datastore and move all the content into another one, while enabling Thin-Provisioning.

So I googled for a bit, and actually found a few hints … So without further yada-yada, here’s the script I came up with:

Doing TSM's job on Windows Server 2008

Thu, 26 Jan 2012 21:02:09 +0000

Ran into another weird problem the other day … Had a few Windows boxens running out of space. Why ? Well, because TSM includes a System-State backup when creating the daily incremental. Now, apparently (as stated by the IBM support) it isn’t TSM’s job to keep track of the VSS snapshots but rather Windows’. Now by default, if you don’t click on the VSS properties of a Windows drive, there is no limit on the volume. Thus, VSS is slowly eating up all your space.

Empty Port SSL after ADAM installation

Wed, 18 Jan 2012 20:43:56 +0000

I’ve been meaning to post this, but never actually got around to doing that. When installing vCenter 5.0, an instance of ADAM is installed, which stores all the configration data for Linked Mode.

It basically boils down to running this script and rebooting the box:

1
2


reg DELETE "HKLMSYSTEMCurrentControlSetservicesADAM_VMwareVCMSDSParameters" /v "Port SSL" /f
reg ADD "HKLMSYSTEMCurrentControlSetservicesADAM_VMwareVCMSDSParameters" /v "Port SSL" /t REG_DWORD /d 0000027c /f

This is no new invention of myself, just writing it down for myself from here or here.

Rebooting a virtual machine via Task scheduler

Mon, 16 Jan 2012 09:03:06 +0000

Since the Scheduled Tasks in vCenter ain’t exportable, I went ahead and wrote a rather simple script, which lets me do this in Windows own Task Scheduler. What this script does, is initiate a graceful shutdown and if the VM isn’t shutdown within 60 seconds (12 * 5 seconds) it simply powers the VM off and immediately after that powers it back on.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48


param( [string] $vCenter, [string] $VMname )

# Add the VI-Snapin if it isn't loaded already
if ( (Get-PSSnapin -Name "VMware.VimAutomation.Core" -ErrorAction SilentlyContinue) -eq $null )
{
 Add-PSSnapin -Name "VMware.VimAutomation.Core"
}

If ( !($vCenter) -or !($VMname) )
{
 Write-Host
 Write-Host "vm-reboot: <vcenter-server> <VMname>"
 Write-Host
 Write-Host " <vcenter-server> - DNS name of your vCenter server."
 Write-Host " <VMname> - Display name of the VM in this vCenter server."
 Write-Host
 exit 1
}

Connect-VIServer -Server $vCenter

$VM = Get-VM $VMname

# First, try shutting down the virtual machine gracefully
Write-Host "Stopping VM $( $VM )"
Write-Host " - Graceful shutdown"
Shutdown-VMGuest -VM $VM -Confirm:$false
$VM = Get-VM $VMname

$i = 0
While ($VM.PowerState -ne "PoweredOff") {
 # If that doesn't work, break out the hammer and just kill it
 if ( $i -gt 12 ) {
 Write-Host " - Forced shutdown"
 Stop-VM -VM $VMname -Confirm:$false
 }

 Start-Sleep -Seconds 5
 $VM = Get-VM $VMname
 $i++
}

If ($VM.PowerState -eq "PoweredOff") {
 Write-Host "Starting VM $( $VM )"
 Start-VM -VM $VM -Confirm:$false >$NULL
}

Disconnect-VIServer -server $vCenter -Confirm:$false

Before this implementation in PowerCLI, I needed three tasks for each VM that was to be scheduled. And when migrating vCenters (and I usually do an empty install) vCenter’s scheduled tasks are not exportable, thus you need to re-create the tasks on the new vCenter by yourself again, which for more than four virtual machines is really a pain in the ass …

Reoccurring memory limits in vCenter

Fri, 13 Jan 2012 15:00:20 +0000

We recently had, after we migrated from vSphere 4 to vSphere 5, a memory limit in size of the configured memory on each and every VM. Since memory limits on VM level pretty much destroy performance, I went ahead an wrote this simple script to remove all memory limits on all VMs that don’t have “Unlimited” configured:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24


param( [string] $vCenter )

# Add the VI-Snapin if it isn't loaded already
if ( (Get-PSSnapin -Name "VMware.VimAutomation.Core" -ErrorAction SilentlyContinue) -eq $null )
{
 Add-PSSnapin -Name "VMware.VimAutomation.Core"
}

If ( !($vCenter) )
{
 Write-Host
 Write-Host "cluster-remove-mem-limits: <vcenter-server>"
 Write-Host
 Write-Host " <vcenter-server> - DNS name of your vCenter server."
 Write-Host
 exit 1
}

Connect-VIServer -Server $vCenter

Get-VM | Get-VMResourceConfiguration | Where-Object { $_.memlimitmb -ne '-1' } |`
 Set-VMResourceConfiguration -memlimitmb $null

Disconnect-VIServer -server $vCenter -Confirm:$false

This script is basically what the guy over at get-admin.com did, just only for memory limits.

PowerCLI undamp; Windows jump list for recent items set to zero

Tue, 05 Jul 2011 17:22:27 +0000

Today I had to install PowerCLI on my workstation. When I tried launching it for the first time, it simply opened and closed again instantly. After browsing the VMware community for this error and not finding a useful solution, I ended up googling the error. As it turns out, this only happens when you did set the " Number of recent items to display in jump lists" to something less than 4.

Changing the vCenter hostname

Wed, 01 Jun 2011 17:56:10 +0000

I recently reinstalled the vCenter Server at work, and in my never ending wisdom cough, I decided to do that on new hardware. That entitled using the same host name plus the appendix _NEW. Now, I know this isn’t conforming with DNS naming schemes (iirc underscore isn’t a valid DNS char), however it worked …

So once I had everything installed on the new hardware, I switched the ESX servers from the old vCenter to the freshly installed one. Once that was finished, I shut down the old vCenter server, changed IP address and host name of the new one and rebooted. That basically worked, even though suddenly every ESX in my inventory was disconnected. After reconnecting all ESX servers everything was back online.

WDS and DL580 G7

Sat, 15 Jan 2011 17:14:56 +0000

We recently received a shipment of Hewlett Packards all-new DL580 G7. While I’m impressed with what they did with the iLO3, I’m quite disappointed with what they did to the PXE-ROM.

Sure, gPXE may be the future and is offering more possibilites than “normal” PXE, however breaking customers deployment option(s) – at least for Windows that is – really wouldn’t be an option.

Now for the long story, we needed to install a temporary Windows on this DL580 (one with testing purposes). That said, we tried for three days to actually make this work (trying different things with the boot image), but it kept ending with the same result.

SVC: Find WWPN

Mon, 15 Nov 2010 17:50:56 +0000

Today we had (once again) hardware troubles. We ended up replacing a lot of things, but in the end it was a) the HBA and b) apparently some memory DIMMs. Now, that isn’t SVC related. However, we built in another HBA (from our Standby hardware), which apparently already had been assigned to a host.

Now, since you can’t search for a WWPN (at least not that I know of), I ended up writing a little script (yup, AGAIN) in order to do that for me!

TSM Client: Service Script for Solaris 10

Wed, 27 Oct 2010 15:31:05 +0000

Today I’ve been fighting with Solaris 10 and the SMF Manifest (others would call it init-script …). Since I wanted to do it the proper way (I could have used a “old-style” init-script, but I didn’t wanna ..), I ended up combing the interweb for examples .. As it turns out, not even IBM has documented a way, on how to do this.

In the end this is what I’ve come up with:

Install issues with Proliant BL460c G6 and Windows Deployment Services

Fri, 22 Oct 2010 17:56:53 +0000

We’ve been dealing with authentification issues on newly delivered HP Proliant BL460c G6 blade servers. Most threads on HPs customer forum, suggests changing the NIC driver, embedded within the WDS boot image.

We tried that, but still were getting the following error:

WDS: installation issues

As it turns out, it ain’t really so damn hard .. we tried several times changing various things within the boot image, but it still didn’t change anything. Somehow it was rather easy.

Create an offline snapshot of a VM

Fri, 22 Oct 2010 16:11:25 +0000

We’re currently thinking about automating Windows Updates and the involved disaster snapshot-copy to a degree, where we don’t need to intervene anymore.

Right now, we already have a rudimentary scheduler in place, which does the reboots for some (200 ..) systems already. Now, we’d like to extend it to also cover the bi-weekly Windows Update spree.

Since PowerShell (and PowerCLI) work quite well with vSphere automation, I cooked up the below script to first shutdown a virtual machine (for snapshot consistency reasons), then take a snapshot and power on the virtual machine again afterwards.

Locking down Firefox

Tue, 12 Oct 2010 17:52:37 +0000

Once again, I had the task of locking down Firefox, so users couldn’t use it to do any harm on a terminal server. Thankfully there’s the guide over at the Faculty of Engineering of the University of Waterloo (by David Collie), who shows which parts to modify.

However, finding the particular part in the Javascript is rather hindersome, so here a short Unix-Diff (for thos who’re able to read unified diffs) as well as the whole file.

Modified SnapReminder

Fri, 08 Oct 2010 16:06:57 +0000

Well, PowerCLI makes my life a little bit easier. Believe it or not, each of us vCenter infrastructure admins has one of these: a Windows admin, thinking a snapshot is also a backup. Thankfully, Alan Renouf over at virtu-al.net wrote the SnapReminder, which already helped me a lot! However, occasionally the script isn’t finding the snapshot author (for whatever reason).

Since I want a notification in that case, I modified the script a little bit to suit my needs.

Fix Path Selection Policy for a whole vCenter Cluster

Thu, 07 Oct 2010 18:38:49 +0000

These last few weeks, I’ve been toying with PowerCLI (and PowerShell for that matter). One thing I do have to say, is that Microsoft finally did it right! It’s a useable, program-able command line interface for Windows after all! Thanks to Ivo Beerens and his post " Best practices for HP EVA, vSphere 4 and Round Robin multi-pathing", I was able to come up with the below:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47


# LICENSE: GNU General Public License v2. (see LICENSE.txt)
# COPYRIGHT: Copyright 2010 Christian Heim <christian.heim@barfoo.org>

# Add the VI-Snapin if it isn't loaded already
if ( (Get-PSSnapin -Name "VMware.VimAutomation.Core" -ErrorAction SilentlyContinue) -eq $null )
{
 Add-PSSnapin -Name "VMware.VimAutomation.Core"
}

if ($args.length -lt 2) {
 Write-Host
 Write-Host "fix_multipathing: "
 Write-Host " - Display-Name of the vCenter cluster, we should check all LUNs"
 Write-Host " on all available Hosts, and set the Path Selection Policy"
 Write-Host " to the selected one."
 Write-Host ""
 Write-Host " - Path Selection Policy"
 Write-Host " Possible:"
 Write-Host " - RoundRobin (VMW_PSP_RR, Round Robin)"
 Write-Host " - Fixed (VMW_PSP_FIXED, Fixed)"
 Write-Host " - MostRecentlyUsed (VMW_PSP_MRU , Most Recently Used)"
 Write-Host
 exit 1
}

$vcserver = "vcenter.home.barfoo.org"
$cluster = $args[0]
$target_policy = $args[1]

# Since I do have only SVC-Disks connected to my hosts, I limit the search to those
$canonical_name = "naa.6005076801808021*"

Write-Host "Target vCenter Cluster: " $cluster
Write-Host "Target PSP: " $target_policy
Write-Host

switch ($target_policy) {
 RoundRobin { $display_policy = "VMW_PSP_RR"; }
 MostRecentlyUsed { $display_policy = "VMW_PSP_MRU"; }
 Fixed { $display_policy = "VMW_PSP_FIXED"; }
 default { Write-Warning "Unknown PSP selected! Please consult the help and try again."; exit }
}

Connect-VIServer $vcserver >/dev/null 2>&1

Write-Host "Found "@(Get-VMHost -location ( get-cluster $cluster ) | Get-ScsiLun -CanonicalName $canonical_name -LunType "disk" | where {$_.MultipathPolicy -ne $target_policy }).Count" LUNs in "$cluster" not using Path Selection Policy "$display_policy
Get-VMHost -location ( get-cluster $cluster ) | Get-ScsiLun -CanonicalName $canonical_name -LunType "disk" | where {$_.MultipathPolicy -ne $target_policy } | Set-ScsiLun -MultipathPolicy $target_policy >/dev/null 2>&1

This works great, however you could make it work on the whole vCenter inventory, which I don’t want. We usually add LUNs to a single cluster at one time. Only thing you might need to change, is the canonical name. Mine simply says “find all SVC LUNs” and you might need to change it, if you’re using a different storage.

SAN reporting

Fri, 20 Aug 2010 14:34:52 +0000

We do have some customers, who get charged on a monthly basis for their SAN usage. We already had “reporting” in place, but that wasn’t very flexible. So I went ahead and rewrote the current reporting script from scratch, and this is what I’ve come up with:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51


#/bin/bash

# LICENSE: GNU General Public License v2. (see LICENSE.txt)
# COPYRIGHT: Copyright 2010 Christian Heim <christian.heim@barfoo.org>

svc_sshkey="~/.ssh/svc-id_dsa"
svc_addr="10.144.0.150"
svc_user="admin"

if [ -z $1 ] ; then
 echo "Please rerun this script with some kind of filter value"
 echo "(for example '$0 NAS')"
 exit 1
else
 filter=$1
fi

IFS="
"

DISK="$( ssh -i $svc_sshkey -l $svc_user $svc_addr svcinfo lsvdisk -nohdr -bytes -delim :
 | egrep -i "V.*$filter" | cut -d: -f2,8 | sort )"

# Get a unique list of systems
SYSTEMS="$( echo $DISK | sed "s, ,n,g" | cut -d -f1 | sed 's,^V,,' | cut -d_ -f1 | sort -u )"

for system in $SYSTEMS ; do
 VDISKS="$( echo $DISK | sed "s, ,n,g" | grep $system | sed "s,:,: ," )"
 SYSTEM_TOTAL="$( echo $DISK | sed "s, ,n,g" | grep $system | cut -d: -f2 | awk '{SUM += $1} END { printf "%.2f", SUM }' )"
 for vdisk in $VDISKS ; do
 NAME="$( echo $vdisk | cut -d: -f1 )"
 SIZE="$( echo $vdisk | cut -d: -f2 | sed "s,^ ,," )"
 GB_SIZE="$( echo "$SIZE / 1024 / 1024 / 1024" | bc )"
 if [ $GB_SIZE -eq 0 ] ; then
 GB_SIZE="$( echo "scale=2; $SIZE / 1024 / 1024 / 1024" | bc )"
 GB_SIZE="${GB_SIZE/./0.}"
 fi
 echo "$NAME: ${GB_SIZE/./,} GB"
 done
 SUB_TOTAL_SYSTEM="$( echo "scale=2; $SYSTEM_TOTAL / 1024 / 1024 / 1024" | bc )"
 echo "SUB TOTAL: ${SUB_TOTAL_SYSTEM/./,} GB"
 echo
done

TOTAL="$( echo $DISK | sed "s, ,n,g" | cut -d: -f2 | awk '{SUM += $1} END { printf "%.2f", SUM }' )"
TOTAL="$( echo "scale=2; $TOTAL / 1024 / 1024 / 1024" | bc ) GB"

echo "------------------------------------------------"
echo $TOTAL

exit 0

I gotta say, once again I learned a lot … two new things about awk!

Displaying Windows Architecture with bginfo

Fri, 18 Jun 2010 15:22:08 +0000

On all our servers in the basement, we do have bginfo installed, in order to quickly get certain information. Now as I was struggling with a big Service Pack roll out, I looked into making bginfo also display the OS architecture. But apparently it isn’t that easy … At least bginfo doesn’t provide it by default.

After (yet another hour in front of Google), I finally found what I was looking for. At first I didn’t limit the query on a specific CPU, but that turned out to be shitty (x32 being displayed twice, once for each CPU). But after limiting it to DeviceID=‘CPU0’ it works like a charm 😉

AutoYAST and custom swap partitioning

Thu, 03 Jun 2010 18:34:24 +0000

Well, we’ve been discussing our swap partitioning the last few days at work, and I finally got around to implementing it. Again, it proved to be kinda hard, basically because AutoYAST decides to do things differently.

SLES10 not installing boot loader in MBR

Thu, 03 Jun 2010 18:15:23 +0000

Well, as I mentioned in my earlier post, I had some trouble during the week. I was having issues with SLES10 installations not finishing during the bootloader installation phase. After trying out different flavors (as in 10SP2 x64/x86, …), and not having any luck with this, I went searching on Google as a last effort try. Guess what, yet again Google helped me out!

It was pretty simple. Putting /dev/cciss/c0d0 into /boot/grub/device.map as (hd0) made the grub-installer finish. Now, figuring out how to transfer those information during the installation proved difficult. I was just about to give up, while reading through the AutoYAST documentation, when it struck me. There is even an extra chapter for this stuff, so simply putting the following into my profile solved my issues:

VMware Consolidated Backup and TRANSPORT_MODE=hotadd

Thu, 18 Mar 2010 17:03:38 +0000

As the title says, I’ve been playing with vCB (inside a VM) and the TSM integration with newer (>6.0) clients for work. Result of all this work should be a feasibility study. We’re currently thinking about replacing our VMware server(s) with ESXi. But as most of you know, if you install ESXi, you simply can’t install anything (well, you can .. on ~100KB of disk space, which is compared to a TSM client weighing roughly 120MB nothing!). As we would like the possibility to backup VMs on image-level, I went looking at solutions.

Windows Server 2003: taskmgr giving Logon failure

Thu, 25 Feb 2010 17:58:22 +0000

I had myself a lot of fun today. I ended up patching a Windows Server 2003 x64 SP1, where the Task Manager wouldn’t start anymore. It simply failed (or in case of right clicking on the task bar wouldn’t even appear), so I went downstairs and pulled a hard disk out of the RAID1 array, just to be sure.

Really weird Windows error

I went ahead, installed SP2 (as you can see on the above picture) while having the jitters. Also installed the VirusScan I was scheduled to install, and the system came back online. Phewww.

VMware vSphere: Safely remove network controller

Tue, 23 Feb 2010 18:24:01 +0000

Well, it’s another day another fight. As we started migrating our VM’s from the old VMware ESX farms to the new environment, and upgraded the hardware suddenly the network devices were hot-plug-able, thus they did turn up in the “Safely Remove” dialog.

I myself don’t have any trouble with that. The trouble I do have is the people working with those VM’s and their possibly hazardous “uuuh, what’s this ? I don’t need this! <click-click, network-device unplugged>”

VBscript: Query remote OS and SP info (continued)

Mon, 15 Feb 2010 20:34:34 +0000

After some more crunching on my VBscript, I think I finally have a working script that runs through a csv-list I point it to and walk onto each system (by ip-address only sadly) and query the os and the Service Pack that is installed. The CSV may look like this:

1
2
3


Hostname;IP;Model;Description;OS;Service-Pack;BL;Priority
epimetheus;10.0.0.2;VMware guest;File-Server
hades;10.0.0.1;VMware guest;Core-Router

After saving that one, and running a cscript //NoLogo win_sp_level.vbs you should find a completed list like this:

Converting TIVSM RPMs to deb

Mon, 15 Feb 2010 09:59:31 +0000

We received a preinstalled customer server the other day, for which we had declared “as-is” support only, since it is running Lucid Lynx. Now today, I started getting the TSM client to work. Was kinda weird, since at first dsmc was reporting something like this:

# ./dsmc: no such file or directory

After fiddling with it a bit more, here are the control files, as well as the prerm and postinst-scripts for TIVSM-API, TIVSM-API64 and TIVSM-BA:

VMware Data Recovery

Tue, 05 Jan 2010 17:11:35 +0000

I’ve been tinkering with VMware’s Data Recovery for the last two weeks (as in configured it some time before Christmas) and had it running all that time. I have to say the integration into the vCenter Client GUI is amazing, I’d love to see that for VCB also. The Changed Block Tracking is a neat way to minimize the amount backup data as well as your backup window (which is nearly zero anyhow due to vDR using snapshots).

VCP410 exam

Thu, 24 Dec 2009 20:25:55 +0000

I’ve been learning for my VCP-410 exam the last week or so, and what can I say ? It helped … 463 points of a total of 500 points ain’t that bad at all (considering I spend twenty minutes doing it).

Sure, I could have spent more time, and do better than 92,6%, but then again: why should I ?

The achieved points (nor the percentage) don’t appear on the certificate (or at least it didn’t on the old one), so why bother. Anyway, that was my christmas present to myself, it that light; happy christmas ya’ll.

PXEBoot the VMware ESXi installer

Fri, 20 Nov 2009 20:19:46 +0000

Some of you may know, that VMware released vSphere 4.0 Update 1 yesterday. I took this as a reason, to finally wrap my head around booting the VMware ESXi installer from my PXE/TFTP box. Since VMware was kind enough to provide (a somewhat worthless) document, that explains how to extract the necessary files on Windows. But that quite doesn’t work with Linux – and VMware just states that you should be using mount and it’s option offset.

Configuring nagios-plugins-zypper

Thu, 12 Nov 2009 17:45:43 +0000

Since I’m running check_zypper via nrpe (which in turn runs as nobody), I need to set up sudo. In order for the plugin to work, we need to add the following line to /etc/sudoers (by means of visudo):

`1`	`nobody ALL = NOPASSWD: /usr/bin/zypper sl, /usr/bin/zypper --non-interactive --no-gpg-checks --terse list-updates`

(Keep in mind this needs to be a single line …)

Linux: Convert Gigabyte into Blocks

Mon, 02 Nov 2009 16:55:32 +0000

I just googled again for a Gigabyte to Blocks converter when I stumbled upon this:

`1`	`echo "(1510241024*1024)/512" \| bc`

Easy to use and yet handy.

SVC: Migrate VDisks off a MDisk Group onto another

Thu, 29 Oct 2009 18:09:17 +0000

Out of necessity, another SVC shell script was just born. If you ever need to migrate a whole MDisk group onto another, you quickly discover the limited application of the SVC GUI. Now, you could query the VDisks using your original MDisk Group and then copy and paste the VDisk’s name (or the VDisk ID) into a command line and simply reuse that svctask migratevdisk command over and over.

Luckily IBM blessed the SVC with an SSH interface. So again, we can write a (kinda) simple shell script which may look like this:

ESX: Query CDP information from the command line

Thu, 29 Oct 2009 17:20:46 +0000

I’m just tracing some troubles I’m having with a backup server and two (independent) network adapter ports (as in two ports on two different dual-port nics). If I enable the port and set it to auto configuration, it’ll get 100MBit/Half-Duplex, but the Portgroup becomes unavailable.

In order to get the connection back, I need to logon on the console (thank god even the backup server got an iLO2), and manually (as in esxcfg-nics -s 1000 -d full vmnic1) configure the adapter to 1GBit/s and full-duplex.

RPM: Query a specific rpm-file for information

Wed, 28 Oct 2009 18:12:44 +0000

Since I end up googling it each time I need this, it’s about time I write it down … If you want to query a specific RPM for information (Requires/Information/…) you’ll need to use the –package/-p option.

-p <file> — Query a Specific RPM Package File

Up to now, every means of specifying a package to an RPM query focused on packages that had already been installed. While it’s certainly very useful to be able to dredge up information about packages that are already on your system, what about packages that haven’t yet been installed? The -p option can do that for you.

KMP: Define a new subpkg template

Sat, 24 Oct 2009 16:43:36 +0000

There might be reasons, you’d wish you could make the kernel module package do other things. Two already pop into my head: 1) The mpp-Image upgrades for the ibm-rdac kernel module packages and 2) the " adjustments" which need to be done post install for the VMware kernel module package in /etc/vmware-tools/locations.

What you basically do is this:

Add the new subpkg template to your sources list
Call the %suse_kernel_module_package macro with the option -s and then add your source number

For me this looks like this:

SMT: Disable unmirrorable catalogs

Sat, 24 Oct 2009 16:22:50 +0000

Today I got this report from my SMT:

1
2
3
4
5


WARNING: The following repositories cannot be mirrored.
 Maybe you have not enough permissions to download these repositories?

* SLES10-SP2-Pool sles-10-x86_64
* SLES10-SP2-Pool sles-10-i586

However, if you try and disable that repository with smt-catalogs -d, SMT is gonna quit your action with " 0 repositories disabled". Since I didn’t want the error to show up again, here’s a quick way on how to disable it.

Open up a mysql shell (mysql -u root -p preferably) and enter those queries:

Creating a custom RPM repository for SMT

Thu, 22 Oct 2009 17:59:45 +0000

I spent some time yesterday figuring out ways on how to assign custom (as in self-built) RPMs to a installation using SMT. First you obviously need a " external" repository, that can be integrated into the SMT.

So we need to create repository someplace, where the SMT can go and grab it. I ain’t gonna cover the sharing part, since that is your job! I’m just gonna cover the steps on how to create the custom repository and how to integrate it into the SMT.

Active Directory authentification for Samba on SLES11

Wed, 21 Oct 2009 18:46:25 +0000

I recently “redesigned” the PXE-installation server, which comes with a Samba service to easily move files on/off the box. The old one had the restriction, you need to create local user accounts. Since I also did an distribution upgrade, I wanted to try the integration of SLES11 into Active Directory.

And as it turns out, it really is simple. Just follow the steps outlined in the handbook.

Open the Windows Domain Membership module, yast samba-client (or yast, then Network Services -> Windows Domain Membership) and enter your Domain information
Open the Samba Server Module, yast samba-server (or yast, then Network Services -> Samba Server) and also enter your Domain information

Just make sure, you also check the box labeled Also Use SMB Information for Linux Authentication, otherwise it won’t work – don’t ask me why …

Using the integrated kickstart generator

Wed, 14 Oct 2009 16:36:00 +0000

VMware built an kickstart generator into ESX 3.5. You just need to enable it, simply by editing an XML configuration and restarting the webAccess service. Simply edit /usr/lib/vmware/webAccess/tomcat/apache-tomcat-5.5.26/webapps/ui/WEB-INF/struts-config.xml and look for the line saying:

`1`	`<action path="/scriptedInstall" type="org.apache.struts.actions.ForwardAction" parameter="/WEB-INF/jsp/scriptedInstall/disabled.jsp" />`

This line needs to be commented out (<– and –>) and the lines following, having those comment marks around them needs to be removed.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11


<!--
<action path="/scriptedInstall" type="com.vmware.webcenter.scripted.ProcessAction">
 <forward name="scriptedInstall.form1" path="/WEB-INF/jsp/scriptedInstall/form1.jsp" />
 <forward name="scriptedInstall.form2" path="/WEB-INF/jsp/scriptedInstall/form2.jsp" />
 <forward name="scriptedInstall.form3" path="/WEB-INF/jsp/scriptedInstall/form3.jsp" />
 <forward name="scriptedInstall.form4" path="/WEB-INF/jsp/scriptedInstall/form4.jsp" />
 <forward name="scriptedInstall.form5" path="/WEB-INF/jsp/scriptedInstall/form5.jsp" />
 <forward name="scriptedInstall.form6" path="/WEB-INF/jsp/scriptedInstall/form6.jsp" />
 <forward name="scriptedInstall.form7" path="/WEB-INF/jsp/scriptedInstall/form7.jsp" />
</action>
-->

After doing that, you should be able to restart the webAccess service, and after that access your ESX host.

Autoinstalling VMware-Tools

Tue, 13 Oct 2009 16:30:08 +0000

As I wrote before, I have been working on our AutoYaST setup. That entitles determining whether or not we’re currently inside a VMware environment. AutoYaST rules wise, that’s pretty easy (even though the MAC-tag is empty 😳):

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15


 <!-- Addons: Install VMware Tools / KMP -->
 <rule>
 <custom1>
 <script>
if ip a | grep "link/ether 00:50:56" >/dev/null ; then
 echo -n vmware
fi;
 </script>
 <match>*</match>
 <match_type>exact</match_type>
 </custom1>
 <result>
 <profile>addons/@custom1@.xml</profile>
 </result>
 </rule>

The hard part is figuring out ways, to make the VMware Tools installation as pain free as possible. One thing I can’t do, is running vmware-config-tools.pl …

New IBM RDAC version (or not)

Wed, 09 Sep 2009 20:03:30 +0000

A week ago (September 02nd), I received a mail detailing the release of IBM’s new multipathing device driver for the DS4x00 series, which finally works with SLES11 (the available software up till now doesn’t – as in fails with kernels > 2.6.26 iirc).

ESC+ notification detailing the release

There wouldn’t be any trouble, if IBM (or rather the vendor providing the driver – LSI) would actually release the driver … up till today, I have yet to see the new version appear on the download page. I already tried to notify IBM about the trouble, but as usual there is lack of ways to actually get this to the right person.

Tivoli Storage Manager Server 5-5-3

Fri, 28 Aug 2009 14:19:14 +0000

I spent yesterday afternoon upgrading our TS7530, and in my fad I also upgraded TSM to 5.5.3. Now, once I started TSM it quickly started complaining about the paths to the drives.

1
2
3
4
5
6
7
8
9


ANR8873E The path from source TSM1 to destination VTL1_DR03
(/dev/lin_tape/IBMtape03) is taken offline.
ANR8873E The path from source TSM1 to destination VTL1_DR03
(/dev/lin_tape/IBMtape03) is taken offline.
HBA_LoadLibrary: previously unfreed libraries exist, call HBA_FreeLibrary().
ANR8873E The path from source TSM1 to destination VTL1_DR07
(/dev/lin_tape/IBMtape07) is taken offline.
ANR8873E The path from source TSM1 to destination VTL1_DR07
(/dev/lin_tape/IBMtape-07) is taken offline.

I thought maybe this is a mere device problem (we have had them before), so I rebooted the boxes. But still no luck and I went home after about an hour of trying without any luck. In the morning, my co-worker called our trustworthy IBM service partner, and the TSM consultant said he had the exact, same problem yesterday. We would have two options:

IBM RSA II adapter and Java RE (fini)

Wed, 19 Aug 2009 04:32:44 +0000

If you remember back to July, I looked into some troubles I had with the IBM RSA II adapter’s Java interface and the latest JRE updates. I just noticed, that IBM released a new firmware yesterday for the RSA. The ChangeLog states this:

Version 1.13, GFEP35A Problem(s) Fixed:

* Suggested o Fix for Remote Control General Exception in JRE 1.6 update 12 and above. o Corrected a problem that DHCP renew/release may fail after a long time. o Corrected a problem that remote control preference link disapears after creating new key buttons. o Corrected a problem that cause event number shows only from 0 to 255 when views RSA log via telnet session.

rpc-statd starting before portmap

Tue, 04 Aug 2009 14:51:49 +0000

One problem gone, another one turns up. When rpc.statd (nfs-common) tries to start before portmap, it’s gonna result in failure. Now, the logfile (/var/log/daemon.log) is gonna print a rather cryptic error message:

1
2


Aug  4 15:54:25 xen2 rpc.statd[3419]: Version 1.1.2 Starting
Aug  4 15:54:25 xen2 rpc.statd[3419]: unable to register (statd, 1, udp).

After fixing the start order (I really hate SUSE/Debian* for not having init-script dependencies – like Gentoo’s baselayout/Roy’s openrc does have), everything is like it should be and I’m able to put the /srv/xen mount into the fstab …

OFED packages for Debian

Tue, 04 Aug 2009 12:25:08 +0000

As I mentioned yesterday, I’m currently doing some project work. Said project includes InfiniBand technology.

Apparently we bought a “cheap” InfiniBand switch, which comes without a subnet manager. So, in order to communicate between the nodes, you need to install the subnet manager (opensm in my case) on each node.

1
2
3
4


echo 'deb http://pkg-ofed.alioth.debian.org/apt/ofed ./' >
 /etc/apt/sources.list.d/openfabrics.list;
 aptitude update;
 aptitude install opensm

In order to utilize the InfiniBand interface you need to do a few things first though:

TS7530 authentification failure

Sat, 01 Aug 2009 05:20:04 +0000

Today, I had a rather troublesome morning. Once I got to work, Nagios was already complaining about the lin_taped on one of our TSM servers, which apparently failed due to too many SCSI resets. Additionally, I can’t login using the VE console (I can login however using SSH) so I ended up opening up a IBM Electronic Service Call (ESC+).

Using SSH, I can get some information on the VE’s status:

VMware vSphere and templates

Fri, 31 Jul 2009 13:51:01 +0000

I just converted one of my (old) templates, as I wanted to refresh the updates and the virus scanner. After converting, I was asked about the UUID (no clue why), and expected to be done with it. But after looking at the console, I got the following, completely cryptic message:

Unable to connect to MKS

After digging a bit deeper (that is looking at the vmware.log of the virtual machine, since the message of the GUI is real cryptic), I’m a bit wiser:

IBM TS7530 and the Virtualization Engine for Tape Console

Fri, 31 Jul 2009 12:44:16 +0000

I just had yet another support call with IBM, concerning the Tape Console (or VE console, courtesy of Falconstor). My basic problem was/is, that I, as a german person, do have a german Windows Server 2003 installation. Now, if you do have german decimal number format selected in the regional settings, the display is gonna be kinda impaired and you’re gonna see something like this:

VE console with german decimal number format

Custom certificates in VMware vSphere

Fri, 24 Jul 2009 13:34:57 +0000

Finally, after about 6 months (I last talked about that on February 25th, when Virtual Center 2.5U4 was released) our troubles with our " custom" certificates seems to be resolved! As it turns out, it really was our fault and not VMware’s.

When generating the pfx from the signed certificate and the key-file, you need to supply a password, otherwise the vCenter service is unable to utilize the private key of the pfx, since it’s unable to access the PFX with the default password ( testpassword is the default for Virtual Center as well as vSphere).

Windows XP(e) refusing to connect to a terminal server

Thu, 23 Jul 2009 14:48:52 +0000

Today a error message reappeared I thought I wouldn’t see again. We use Wyse Thin Clients and 2X running on two terminal servers, to provide the thin clients with applications. Now, once a while one of the thin clients (not all at once, just a single one) refuse to connect to the terminal server jabbing about this:

`1`	`The remote computer disconnected the session because of an error in the licensing protocol.`

The error message you get from the 2X client ain’t the slightest bit more helpful.

Burning CD/DVDs as unprivileged user

Tue, 21 Jul 2009 15:26:50 +0000

After the last reinstallation of Windows (both at home and at work), I decided that from this day forth, I’d be working as unprivileged user (ie not as local administrator). Now, working as unprivileged user is some kind of relief (since not every program is able to wreak havoc within your computer), but also somewhat of a burden.

Up until now, I had serious troubles burning CDs and/or DVDs as unprivileged user. Basically InfraRecorder refused to work at all. Only way that I had left, was copying the ISO to my local disk and then use runas in conjunction with InfraRecorder, to execute as Administrator.

Updating a Linux VM from Virtual Infrastructure to vSphere

Wed, 08 Jul 2009 13:51:50 +0000

Well, if you’re gonna update a SLES10 (or even a SLES11) VM, you created with Virtual Infrastructure, you’re gonna run into a snag (like I do). Grub (or rather the kernel itself) is gonna barf.

Now, I searched for a while and didn’t find anything specific on the net, so I’m gonna write it down. Up till 3.5U4 the maximal resolution you’d be able to enter within a virtual machine was vga=0x32d (at least for my 19" TFT’s at work). But now, after the upgrade to vSphere that isn’t working anymore.

New vmware-tools-kmp

Wed, 10 Jun 2009 11:31:04 +0000

Disclaimer: I don’t take any responsibility for faults within the software, I just provide the RPM’s! Feel free to ask me about stuff concerning these RPM’s, but I ain’t accountable if your stuff goes kaboom! Oh, and those RPM’s aren’t recommended or supported by VMware!

Since we recently upgraded our VMware Infrastructure to VMware vSphere, I finally had a chance to refresh the RPM’s for the KMP for 2.6.16.60-0.39.3-0.1 and 2.6.27.21-0.1. You can find the source RPM here.

OCF agent for Tivoli Storage Manager: redux

Fri, 05 Jun 2009 09:35:50 +0000

Well, after I finished my first OCF agent back in October 2008, we have it running in production now for about ten months. During that time, we found quite a few points in which we’d like to improve the behaviour with that Linux-HA should handle TSM.

Shutdown TSM nicely if possible (Cancel client sessions, cancel running processes and dismount mounted volumes)
Better error handling

So, after another week of writing and testing with a small instance, I present the new OCF agent for Tivoli Storage Manager. It still has one or two weak points, but they are negligible. I still need to write the documentation for it, but the script should just work …

Weird TS3500 problem: redux

Wed, 03 Jun 2009 12:49:35 +0000

Well, after yesterday’s episode with our tape library today continued to be a taxing day. After restarting a few exports that were hanging yesterday due to our library problems, something similar returned. TSM was unable to locate a few (two to be exact) tapes in the library.

1
2
3
4
5
6
7
8
9


ANR8300E I/O error on library LIB3584 (OP=00006C03, CC=314, KEY=05, ASC=3B,
ASCQ=0E, SENSE=70.00.05.00.00.00.00.0A.00.00.00.00.3B.0E.00.C0.00-.04.,
Description=The source slot or drive was empty in an attempt to move a
volume). Refer to Appendix C in the 'Messages' manual for recommended action.
ANR8312E Volume 000400 could not be located in library LIB3584.
ANR8358E Audit operation is required for library LIB3584.
ANR8381E LTO volume 000400 could not be mounted in drive DR6 (/dev/rmt0).
ANR1402W Mount request denied for volume 000400 - volume unavailable.
ANR1410W Access mode for volume 000400 now set to "unavailable".

Yet the library reported the tapes were still inventoried. 🤷 Here we are again, looking completely baffled. After a short while trying to figure out what to do, we went through the Data Cartridge inventory again. As it turns out, through putting the library in “Pause”-Mode and restarting TSM multiple times, TSM apparently completely forgot that it had these tapes put into drives.

Novell KMP: vmware-tools-kmp and ibm-lin_tape-kmp

Sun, 10 May 2009 11:27:22 +0000

Disclaimer: I don’t take any responsibility for faults within the software, I just provide the RPM’s! Feel free to ask me about stuff concerning these RPM’s, but I ain’t accountable if your stuff goes kaboom … Oh, and those RPM’s aren’t recommended or supported by Novell or IBM!

After working with the novell-kmp solution, I think it’s actually rather easy to create a " Kernel Module Package". In the end, I created two additional KMP’s, one for the tools component of the VMware-Tools shipped with VMware ESX, and another for the lin_tape SCSI driver, used by our IBM TS3400 as well as the IBM TS7530.

TSM: Restoring the database/recovery log to a point-in-time

Fri, 24 Apr 2009 12:38:34 +0000

Well, my co-worker just called on my cell (it’s Friday, 16:00), and asked me which start-up script he needed to change in order to restore the database. My first response was, “ummm, that’s gonna be hard, we’re using heartbeat”.

Okay, so after a bit of asking I got out of him what he wanted to achieve by changing the start-up script. Apparently he did something to crash Tivoli Storage Manager (or rather repeatedly crash it) and wanted to restore the database. He talked to one of the systems partner we do have (and I’m happy we have them most of the time), who in return told him how to do it, but forgot a minute after he hung up the phone.

SLES10: zypper-log

Fri, 03 Apr 2009 04:34:49 +0000

Well, I just stumbled upon something .. My Nagios at work wasn’t working anymore, and I went looking.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11


nagios3 ~ [0] > tail -f /var/log/nagios/nagios.log
[1238658394] Error: Unable to save status file: No space left on device
[1238658403] Error: Unable to save status file: No space left on device
[1238658413] Error: Unable to save status file: No space left on device
[1238658423] SERVICE ALERT: tsm1;POWER WARN;OK;SOFT;4;-u OK - 0
[1238658423] Error: Unable to save status file: No space left on device
[1238658433] SERVICE ALERT: tsm2;LOAD;WARNING;SOFT;1;WARNING - load average: 6.25, 5.72, 5.36
[1238658433] Error: Unable to save status file: No space left on device
[1238658443] Error: Unable to save status file: No space left on device
[1238658453] Error: Unable to save status file: No space left on device
[1238658463] Error: Unable

After that, zip - nada. Next thing, check whether or not the device is really full … Okay, df ..

Nagios: SNMP OID's for IBM's RSA II adapter

Wed, 01 Apr 2009 10:10:15 +0000

Well, after some poking around I finally found some OID’s for the RSA’s (only through these two links: check_rsa_fan and check_rsa_temp).

For Nagios, I dismissed the fans, since the fan speed is only passed on in percent values. So I only added this:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28


define hostgroup{
 hostgroup_name rsa-snmp
 alias Remote Supervisor Adapter (allowing SNMP connections)
}

define service{
 use generic-perfdata

 check_command check_rsa_snmpv1_public!.1.3.6.1.4.1.2.3.51.1.2.1.2.1.1!45!60!°C!Temperature CPU0!
 hostgroup_name rsa-snmp
 service_description TEMP CPU0
}

define service{
 use generic-perfdata

 check_command check_rsa_snmpv1_public!.1.3.6.1.4.1.2.3.51.1.2.1.2.2.1!45!60!°C!Temperature CPU1!
 hostgroup_name rsa-snmp
 service_description TEMP CPU1
}

define service{
 use generic-perfdata

 check_command check_rsa_snmpv1_public!.1.3.6.1.4.1.2.3.51.1.2.1.5.1.0!29!35!°C!Temperature Ambient!
 hostgroup_name rsa-snmp
 service_description TEMP AMBIENT
}

Oh, and if anyone else is curious like me, here’s the list with the OID’s, courtesy of Gerhard Gschlad and Leonardo Calamai.

RPM spec: Installing a custom init-script

Thu, 26 Mar 2009 15:14:51 +0000

Well, I’m sitting again here grinding my head on how to fix up a certain package. Now, I had to look it up again, so this time I’m writing it down!

1
2
3


Source1: ${name}.initd
...
install -o root -g root -m 755 %{S:1} $RPM_BUILD_ROOT/etc/init.d/ndo2db

Windows: Running msconfig as non privileged user

Wed, 25 Mar 2009 10:38:38 +0000

Well, the title is kinda misleading since you need administrator privileges to run msconfig in it’s full scope. But this is just a hint to myself on how to execute msconfig without logging out and then logging in as administrator.

`1`	`runas /user:Administrator C:WINDOWSpchealthhelpctrbinariesmsconfig.exe`

Nagios: Watching Clustered environments (the other way)

Thu, 19 Mar 2009 08:36:22 +0000

Well, recently I stepped up to watch our cluster environments … Michael has a good howto on how to watch Windows Cluster environments in the NSclient++ wiki.

Now, this has it’s own perks … Which I stumbled upon when trying to write a Linux-HA OCF resource agent for the Nagios NRPE server. Combining that Linux-HA with SLES10 is a good thing generally, but using startproc in that resource agent is not such a good idea.

Apparently Novell (or SuSE GmbH) thought it might be wise to include some additional logic into the wrapper. startproc, checkproc and killproc do check for the name of the executable. So if you try to start an additional process with the same name, you need to dig a bit deeper.

For this to work, you need two additional things (quotations directly from man 8 startproc):

-p pid_file (Former option -f changed due to the LSB specification.) Use an alternate pid file instead of the default (/var/run/ .pid). The pid read from this file is being matched against the pid of running processes that have an executable with specified path of the program. In order to avoid confusion with stale pid files, a not up-to-date pid will be ignored.

Now, then apparently this isn’t enough. startproc is still refusing to start a second process.

-i ignore_file The pid found in this file is used as session id of the same binary program which should be ignored by startproc.

Linux-HA: Creating a random authkey

Wed, 18 Mar 2009 13:00:47 +0000

I just looked over the slides of a presentation one of my trainees bought back from Chemnitz, and there was this nifty one-line command, with which you can generate a random sha1sum for your authkeys file.

Now, since I’m a bit lazy here’s the full command line to fill /etc/ha.d/authkeys for you:

1
2


node2 ~ [0] > echo "auth 1
1 sha1 $( dd if=/dev/urandom count=4 2> /dev/null | openssl dgst -sha1 )"

TSM client: Backing up files with umlauts on SLES

Mon, 02 Mar 2009 13:21:10 +0000

In the past, I always had problems with SLES and our Tivoli Storage Manager client’s when backing up files with german umlauts. Well, today I looked a bit harder, and quite quickly found a solution.

1
2


sles9 root [0] > env | grep ^LC
LC_CTYPE=de_DE.UTF-8

As you can see from the above, SLES9/10 ain’t setting LANG or LC_ALL (which I searched for first), but is setting LC_CTYPE.

So, simply changing the LC_CTYPE in the init-script and/or prepending the dsmc command line with a new LC_CTYPE fixes my umlauts problems!

Nagios: check_snmp again

Fri, 27 Feb 2009 10:59:25 +0000

Well, today I had to grind my head again, regarding the way check_snmp handles WARNING and CRITICAL events. From my point of view, check_snmp is really just retarded sometimes.

As you know, all the other plugins accept WARNING and CRITICAL-thresholds based on the calculation, if the return integer is above this threshold it reached WARNING/CRITICAL state. But check_snmp doesn’t play that way.

It expects only ranges, which are NOT gonna result in warning or critical events. Which is kinda stupid, since you gotta rethink twice about the thresholds 😛

Nagios: NSclient++ in a clustered Environment

Thu, 26 Feb 2009 10:50:36 +0000

Well, most of you already know that I’m a Nagios fanatic. I like to watch as many aspects as I possibly can. So, yesterday I started figuring out ways to watch our different cluster groups (housing a bunch – try above 20.000 – of file shares).

Now, my first tries failed horribly. I brought down a complete cluster group, resulting in a major annoyance. Now, today I went at it a bit smarter 😛 I cloned myself two VM’s off my Windows Server 2003 Enterprise R2 template, created a new cluster.

MySQL: Beware of sync_binlog on EXT3

Mon, 23 Feb 2009 21:35:38 +0000

Well, I just glazed again over my my.cnf for our web-cluster because I just moved a database from one cluster to another and getting quite different performance from it. So, as I expected, there is a slight difference between both configuration files:

1
2
3
4
5


@@ -55,8 +58,6 @@
 innodb_log_group_home_dir = /var/lib/mysql/db
 innodb_log_file_size = 512M
 innodb_thread_concurrency = 8
-sync_binlog = 1

And apparently, according to the MySQL Performance Blog that’s really, really bad (as well, we’re currently running without write caching, as the battery module of the storage is dead).

Tivoli Storage Manager Client and Microsoft Cluster Services (continued)

Mon, 16 Feb 2009 15:19:57 +0000

As you might recall from my first article about this topic, I had some troubles with the Microsoft Cluster Services and the registration replication. Now, today as we tried switching the TSM-Server for some resources, we ran into this again.

We were using the service install tool (dsmcutil install scheduler) to set the new password as well as the GUI. Now, as we brought the resource online with the local service manager, everything was honky dory. But as soon as we brought it online using the Cluster Manager, it failed horribly. Why ?

MySQL: Replication and hostname wild cards

Sun, 15 Feb 2009 19:06:13 +0000

Yeah, yeah .. I know, it’s weekend. But I usually can think much better when no one is rattling my cage. So I had another look at my replication problems.

Don’t you never ever change InnoDB settings when migrating between hardware, because InnoDB is rather sensitive regarding those parameters.
When you’re setting up the replication (don’t ask me why) and copying over the database to the second replication partner, be aware if you’re using wild cards you’re gonna get seriously bitten in the back.

Now, let’s look at the constellation.

Linux: Getting information about an EXT3 filesystem

Fri, 13 Feb 2009 11:52:01 +0000

You know, I’m not getting any younger. It’s getting harder remembering every damn command … so here is how you get information out of your EXT3 filesystem:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10


sles10sp2 ~ [0] > tune2fs -l /dev/sda2 | grep "^Filesystem"
Filesystem volume name: <none>
Filesystem UUID: 8eec8235-4d9e-4b58-acf9-3c68c977d5ea
Filesystem magic number: 0xEF53
Filesystem revision #: 1 (dynamic)
Filesystem features: has_journal resize_inode filetype
 needs_recovery sparse_super large_file
Filesystem state: clean
Filesystem OS type: Linux
Filesystem created: Tue May 27 10:48:56 2008

Restarting the NSclient++ service without the management applet

Wed, 11 Feb 2009 14:21:36 +0000

For people, who are as click and point-lazy as me, here is how you restart the service without using the service management applet.

1
2


net stop "NSClientpp (Nagios) 0.3.5.2 2008-09-24 w32"
net start "NSClientpp (Nagios) 0.3.5.2 2008-09-24 w32"

MySQL: Setting up an InnoDB raw device

Wed, 11 Feb 2009 13:34:45 +0000

Well, since I had to brood about this (again I might add), I’m gonna write it down this time …

Setting up the InnoDB raw device isn’t that hard, just make sure the device has proper permissions (either add mysql to the disk group or create a udev rule).

`1`	`KERNEL="sdb2", OWNER="mysql", GROUP="mysql"`

Now after that (and a reboot/udevcontrol reload_rules later), you should be able to initialize the InnoDB device. Yes, the InnoDB device needs initializing.

Monitoring the IBM BladeCenter chassis with Nagios

Tue, 10 Feb 2009 11:40:00 +0000

Today I ended up working out the details on what we want to monitor regarding our BladeCenter. The most interesting details (for us that is) are these:

Fan speeds for Chassis Cooling/Power Module Cooling Bay(s)
Temperature
Power Domain utilization

It wasn’t * that* hard to implement. Only trouble(s) I ran into, were ( 1) IBM did a real shitty job with the MIB’s. If you look closely into the mmblade.mib, you’re gonna notice, that not a single OID is specified for the events. ( 2) As the MIB’s weren’t documented anywhere, I had to look them up via snmpwalk (which I had never used before). So as a reminder (to myself), here’s how it is done:

`1`	`snmpwalk -v1 -c public -O n 10.0.0.35 .1.3.6.1.4.1.2.3.51.2.2`

This will get you a list, with a lot of output (5154 lines to be exact). Lucky me, the web interface of the management module/ssh interface is rather verbose, so all you need to do is compare those values with what you are looking for.

So for myself (and anyone interested) read ahead for the list of checks we are currently running on the management module.

Installing SLES10 via network with no DHCP available

Fri, 30 Jan 2009 11:24:08 +0000

In our current fight against the BladeCenter switches, we’re currently facing the problem that the blades ain’t able to send/receive DHCP-traffic.

So in order to move forward, we had to use static IP addresses. And since SLES10 ain’t straight forward on that, I had to look it up. Now, here’s for me (and everyone else tired of searching) how to do it:

1
2
3
4
5
6
7
8
9


install=http://ftp.barfoo.org/install/SLES10-x64/CD1
 netdevice=eth0
 hostip=10.0.1.240
 netmask=255.255.255.0
 gateway=10.0.1.1
 nameserver=10.0.1.2
 domain=home.barfoo.org
 insmod=bnx2
 splash=verbose

Setting up the BladeCenter H

Fri, 30 Jan 2009 10:50:40 +0000

Well, we finally had our maintenance window today, in which we planned the hardware exchange for our current Dell Blade Chassis (don’t ask!). The exchange went fine, but as we started exploring the components (like the IBM BladeCenter SAN switches – which are in fact Cisco MDS 9100) we hit a few road blocks.

First, the default user name/password combo for the Cisco MDS 9100 for the BladeCenter is USERID/PASSW0RD (just as the rest of the password combinations).

Updating path information for TSM

Wed, 28 Jan 2009 15:11:31 +0000

As I did some switching today (between the new lin_tape version by IBM and our own lin_tape version), I ended up writing those lines a dozen times. Here is (just for me, if you don’t care .. skip ahead) on how to generate a list of commands:

1
2
3
4
5
6


for i in $( seq -w 1 32 ); do
 token="${i/0/}"
 path_prefix="/dev/lt/IBMtape12245775"
 echo "DELETE PATH TSM1 VTL1_DR$i SRCTYPE=SERVER DESTTYPE=DRIVE LIBRARY=VTL1"
 echo "DEFINE PATH TSM1 VTL1_DR$i SRCTYPE=SERVER DESTTYPE=DRIVE LIBRARY=VTL1 DEVICE=$path_prefix$((token+11))A1"
done

which should get you a list like this:

Sidenote: Amount of Slots per Virtual Tape Library

Tue, 27 Jan 2009 20:32:57 +0000

Well, I just stumbled about this again (and I don’t know right now whether or not this is documented inside a RedBook or not) today, so I thought maybe I’m gonna write it down.

Slot-Amount Property of a Virtual Tape Library

Please keep in mind, when creating the virtual library to think hard about the amount of slots you might need. It ain’t that bad, you just can’t decrease the amount anymore. So if you think about creating 50 different virtual tape libraries with 500 slots each on your TS7530, think again. The current software level only supports 25.000 slots on a global level.

Working with IBM's Virtualization Engine Console

Tue, 27 Jan 2009 17:27:31 +0000

Recently, we got the recommendation from our system partner to use static allocated tape cartridges instead of dynamic allocated ones. Apparently using dynamic allocating cartridges comes with a performance penalty if more than a few nodes are backing up a large amount of data at once.

And yet again, I noticed that the IBM Virtualization Engine Console (aka Falconstor Software) is really error prone.

In order to change the allocation type, we had to shred the old cartridges first (500 x ~100M up till now), chance the allocation type at the virtual tape library level, and then recreate the 500 cartridges with a fixed size (500x 102400MB). Now, as I was kinda optimistic, I decided to create all 500 cartridges at once.

Opsview installation reviewed

Sat, 24 Jan 2009 11:33:50 +0000

Well, I recently (well, yesterday) built the opsview RPM’s for SLES10, and started fiddeling about with it today. Alex " recommended" I should rather look at Opsview instead of Centreon, but boy was there a surprise waiting for me …

Opsview has the advantage that it at least lets you use the package manager. But, it also needs a lot of handy work (just like Centreon, which I really dislike since it’s real error prone).

Automatic updates on SUSE Linux Enterprise 10

Fri, 23 Jan 2009 11:30:01 +0000

I had the problem, that the automatic update function of YaST doesn’t work like I want it to. I just wanted it to install only those updates, that ain’t interactive, don’t need a service restart and don’t need a reboot.

YaST does only feature an online update that skips “interactive” updates (I’ve never even encountered an interactive update up till now). So I went ahead and wrote a (hackish) script, that achieves what I need.

Trusted vendors in SUSE Linux Enteprise 10

Mon, 19 Jan 2009 20:31:19 +0000

The other day I had a closer look at the zypper logs (well, I was digging for a time-history of installed packages). First … damn does zypper produce a lot of logs on a " productive" (or rather on a maintained - as in up-to-date) system.

But glazing over the logs, I found out something new about zypper. It actually has an internal list, which only purpose is to identify a trusted vendor …

Cleaning up /tmp

Sun, 18 Jan 2009 13:26:59 +0000

Well, I just looked into /tmp on one of our boxes and noticed that SSHd left behind some (try 400) directories .. Now, I could use a simple rm -rf /tmp/ssh-*, but I didn’t want to kill my current agent forward file.

After looking at the man-page of find, I stumbled about -mtime. And that seems to work out quite well.

`1`	`find /tmp -name "ssh-*" -mtime +5 \| xargs rm -rf`

Automating zypper updates

Sat, 17 Jan 2009 18:16:27 +0000

Well, I just looked into using ` zypper up ` to update some of our boxen (I do have a script, which holds the boxen it needs to process in a variable and simply goes through them one by one) – yes, I could activate auto-update, I just don’t want that at this point 😉

So at first I tried just using zypper to automatically update that given list, but even if you pass –no-confirm, zypper would still ask for your confirmation (which seems kinda stupid). After a short while thinking about it, a lesson from solar@gentoo.org came to mind. When working in a chroot, he simply used this:

VI Client: Changing the language from the system default

Wed, 14 Jan 2009 08:12:45 +0000

Well, as I am in fact running a german Windows XP, the VI Client started displaying all menus and operations in German when I updated to 2.5u2. Normally, I wouldn’t have much of a problem with that, but recently it started to annoy me, since the translation is a bit off from the real meaning of much of the operations.

So today, in the morning I started looking for ways to revert the VI Client back to displaying everything in English. And guess what. There’s no way to switch the language from the VI Client itself. There’s just a workaround.

Distribution running on IBM TS7530 Virtualization Engine

Fri, 09 Jan 2009 14:18:35 +0000

Well, I was just a bit curious earlier what distribution might be running on our IBM TS7530 Virtualization engines .. well, I just had a look-see ..

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27


vetapeservice@VTL-B:~> cat /etc/SuSE-release
SUSE Linux Enterprise Server 10 (x86_64)
VERSION = 10
PATCHLEVEL=1

vetapeservice@VTL-B:~> uname -a
Linux VTL 2.6.16.46-229-smp #1 Sun Apr 13 05:21:49 UTC 2008 x86_64 GNU/Linux

vetapeservice@VTL-B:~> free -m
 total used free shared buffers cached
Mem: 4022 1378 2643 0 342 552
-/+ buffers/cache: 484 3537
Swap: 3815 0 3815

vetapeservice@VTL-B:~> cat /proc/cpuinfo
processor : 1
vendor_id : AuthenticAMD
cpu family : 15
model : 65
model name : Dual-Core AMD Opteron(tm) Processor 8218
stepping : 3
cpu MHz : 2600.186
cache size : 1024 KB
physical id : 0
siblings : 2
core id : 1
cpu cores : 2

Main difference to a " normal" SUSE Linux Enterprise Server 10 installation (there’s about zip normal with that kind of installation, thus the quotes) thus far are:

the build for the VE uses busybox as init
IBM stripped man/info
they are running Xorg/Fluxbox on it

Just don’t ask me why there’s a DE ( d esktop e nvironment) running, it ain’t even hooked up to a monitor. Only reason would be for the RSA’ remote monitor stuff … lala

VBscript undamp; Active Directory and printers (continued)

Fri, 05 Dec 2008 21:54:09 +0000

As I posted earlier, I tried working around some limitations in Microsoft’s Active Directory by teaching the script some intelligence.

But, since we recently started using Thin Clients, all the stuff I did with the fancy vbs was just a waste-of-time. Turns out, Windows XP Embedded doesn’t work quite the same as a " normal" Windows XP (that’s where I tested the script on), and it simply dies when running the WMI Query. Bollocks.

IBM TS7530 and DNS

Sat, 22 Nov 2008 17:31:50 +0000

Well, we had our TS7530 delivered in late September, the day after the IBM service guys came by to prep the VTL for our needs (IBM sells the thing as black box). Now, since that day; they fought with the Call Home functionality. The trouble was simply, that the Call Home Service running on the Virtualization Engines just didn’t start.

After about 6 weeks of trial and error (and the IBM service guys popping in every second week), they finally found the cause of the Call Home Service not being able to start. Domain Name Resolution. Neither the IP addresses of the VE’s nor the VE console were registered in our DNS/or local host files.

Nagios and check_ram yet again

Fri, 14 Nov 2008 18:08:30 +0000

As some people know, I previously " created" (mostly modified the check_swap plug-in to print RAM usage) check_ram in C. Now one of my problems for the past few months was putting the C plug-in as well as " supported" environment under the same hat. Today I had another look at the amount of available plug-ins in NagiosExchange. There are quite a few plug-ins available, but as I do have some experience with Python, I used the one written in Python.

It was rather easy hacking in support for performance data into it, as the below shows. Someone else already posted a non-unified diff for performance data support, but that ain’t quite right according to the Nagios plug-in development guidelines.

IBM TS7530 engine failover and HBA mode

Tue, 28 Oct 2008 15:21:06 +0000

Well, when they delivered the VTL about four weeks ago, nobody figured this thing would be such a mess. Apparently IBM hasn’t set up that much VTL’s with engine failover.

Point being, the VE’s have eight HBA ports (four inside, four outside the black box). Now, as they configured the VTL, the ports were all in initiator mode. And we needed the fourth port in target mode as well, as it’s better to have 4 independent paths to the VTL. The only problem was, the VE console didn’t think so.

IBM TS7530 zoning

Mon, 27 Oct 2008 15:41:59 +0000

At first, as we prepped the zoning for the VTL, we did it WWN-based. Now the trouble with the HBA’s of the VTL is simply that it has different WWPN’s on the same WWN. And WWN-based zoning simply doesn’t allow access to that.

So off we went and switched to Switchport-based zoning, and see. It just works 🤷

Defragmenting all fragmented MyISAM tables

Thu, 16 Oct 2008 20:18:22 +0000

I just had another look at what I wrote the week before last (you know, being home-sick/on vacation has it’s advantages) and additionally read up on " OPTIMIZE TABLE" again. The comments in the manual mention " SHOW TABLE STATUS", which gives you a complete list, but it doesn’t allow you to filter certain kinds of things out (like I only wanted to see MyISAM tables in the list, I only wanted database and table).

Adapter bonding on Linux

Thu, 16 Oct 2008 16:47:23 +0000

Well, today I had a rather weird error. I was testing the adapter bonding on one of the boxen designated as Tivoli Storage Manager Server, when I noticed that the bonding wasn’t working as expected when simulating an error (that is unplugging one of the TP cables for the bond).

Now, the bond had “mode=6 miimon=100” as options. After running “linux bond debug” through Google (which turned up nothing useful, besides one document on the Oracle Wiki about IOS/Linux adapter teaming), I figured “Hey, just lets test switching the arguments.” And guess what ?

Windows Server 2003 Terminal services

Wed, 15 Oct 2008 12:11:18 +0000

Well, once you thought you don’t have any more problems, another one just pops up. I’m currently bashing my head against the wall, why the hell the forwarded (or is it redirected ?) drives are not shown in the in the “My Computer” explorer view. I pretty sure have an idea why (basically, HKEY_CURRENT_USERS\Software\Classes isn’t writeable, but that’s where Windows, or rather the Terminal Services – or whatever is creating the associations), just don’t know a clever way around/by it.

Setting up Linux-HA

Wed, 01 Oct 2008 08:17:09 +0000

Well, initially I thought writing the OCF resource agent for Tivoli Storage Manager was the hard part. But as it turns out, it really ain’t. The hard part, is getting the resources into the heartbeat agent (or whatever you wanna call it). The worst part about it, is that the hb_gui is completely worthless if you want to do a configuration without quorum.

First of all, we need to setup the main Linux-HA configuration file ( /etc/ha.d/ha.cf). Configuring that, is rather simple. For me, as I do have two network devices, over which both nodes see each other (one is an adapter bond of comprising of two simple, plain, old 1G copper ports; the other is the 1G fibre cluster port), the configuration looks like this:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22


udpport 694
autojoin none
crm true
use_logd on
debug false
coredumps false

auto_failback on

ucast bond0 10.0.0.10
ucast bond0 10.0.0.20
ucast eth2 10.0.0.29
ucast eth2 10.0.0.30

node tsm1
node tsm2

respawn root /usr/lib64/heartbeat/pingd -m 100 -d 5s
ping 10.0.0.1

respawn root /sbin/evmsd
apiauth evms uid=hacluster,root

After configuring the service itself is done, one just needs to start the heartbeat daemon on both nodes. Afterwards, we should be able to configure the cluster resources.

I find it particularly easier to just update the corresponding sections with cibadmin (the man-page really has some good examples). So here are my configuration files for two resource groups ( crm_mon doesn’t difference between resources and grouped resources, it’ll just show you that you configured two resources).

Subversion via HTTP(s) and mod_rewrite

Sun, 28 Sep 2008 19:17:01 +0000

Well, I just finished my wild-goose chase with Apache and subversion regarding a rather weird error. I recently reinstalled our subversion box, and ever since then I was unable to commit anything new to any of the repositories. Subversion told me this:

1
2
3


svn-client admin-scripts [1] > svn ci -m "Directories for Tivoli Storage Manager Scripts."
svn: Commit failed (details follow):
svn: MKACTIVITY of '/svn/admin-scripts/!svn/act/someid': 302 Found

Apache didn’t say much about it either, besides this particular line:

What's up, dude

Tue, 16 Sep 2008 08:49:49 +0000

Some people might ask, " Gosh, what’s up with you ?" .. Well I’ve been incredibly busy with work the last month (sheeesh, another month already passed by). We finally finished the public tender; gonna get some of the components next week (that is the library extension, the LTO4 drives and cartridges, two brand new Cisco MDS9134 and some Fibre channel hard disks for the existing DS4700). After work, I made my usual visits in the gym and well, after I was home some relaxing, kicking up my feet and doing nothing (as in zip).

zypper-update-report (was: patch2mail for SLES10)

Tue, 09 Sep 2008 23:11:01 +0000

Well, after some more refining I think I finally have a script I ain’t never gonna touch again (unless something breaks, which can happen quick as we all know).

The script now uses a sysconfig file for the common settings (like sender, receipents, categories to scan for), so it may be deployed en mass.

/etc/sysconfig/zypper-update-report

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20


## Type: string
## Default: root
## Config: ""
#
# Sender address for the update report
FROM="Yourupdatemonkey "

## Type: string
## Default: root
## Config: ""
#
# Receiver address for the update report
#RECEIPENTS="tehsysadmin@barfoo.org"

## Type: string
## Default: "securty recommended optional"
## Config: ""
#
# List of groups, to include in the report
CLASSES="security recommended optional"

/usr/local/sbin/zypper-update-report

Custom macros in host definitions

Sat, 16 Aug 2008 19:10:37 +0000

Well, I was playing with the hostgroup inheritance earlier. One problem with that is, if you define a duplicate service Nagios is really unpredictable or rather inconsistent. Now, as Thomas Guyot-Sionnest told me, I should try custom macros for the check definition. So what I did was the following:

templates/host-windows.cfg

1
2
3
4
5


define host {
 name         generic-windows
 register 0
 _RDPPORT 3389
}

hostgroups/windows.cfg

Debugging rug

Fri, 15 Aug 2008 19:19:06 +0000

Well, it’s 7pm. I’m sitting at home and thinking about why in gods name rug isn’t adding my update repository. I can add the service using yast inst_source, but when yast then syncs with ZenWorks, it tells me something like:

Failed to get repomd/repodata.xml; Reason: 530 - Access denied

So my fellow co-worker turned on the debug-logging and we quickly found out why: rug isn’t using the command line credentials I was passing.

Suspected NRPE weirdness

Sun, 10 Aug 2008 07:55:28 +0000

Well, I just noticed a really weird thing, when you have command line arguments enabled.

Here’s a snippet from my nrpe.cfg:

1
2


dont_blame_nrpe=1
command[check_disk]=/usr/lib/nagios/plugins/check_disk -E -w $ARG1$ -c $ARG2$ -p $ARG3$

Now, if you’d check the free space for the root, it ain’t gonna show any inode percentage (that one isn’t what I’m talking about). But if you have to use bind mounts like I do (Tivoli needs a separate " domain" -- that is a separate mount point for each domain), you might wanna check the free space on the real device, rather than the free space on the bind mount (which is gonna show you the free space of the parent file system - in my case the root fs).

Nagios 3 and hostgroup inheritance

Fri, 08 Aug 2008 12:24:01 +0000

As I wrote some time ago, I was trying to utilize Nagios 3.x’s neat feature of " nested" hostgroups. Well, as it turned out I thought it worked differently; basically like this:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24


define hostgroup {
 hostgroup_name a-parent-hostgroup
 alias Our toplevel parent hostgroup
}

define service {
 use generic-service
 check_command check_dummy!0!
 service_description SSH
 hostgroup_name a-parent-hostgroup
}

define hostgroup {
 hostgroup_name a-child-hostgroup
 hostgroup_members a-parent-hostgroup
 alias Our child hostgroup
}

define service {
 use generic-service
 check_command check_dummy!0!
 service_description LOAD
 hostgroup_name a-child-hostgroup
}

As you can cleary see on line 14, I thought you define the relation between two hostgroups in the child hostgroup. The problem with it was basically (as I said in the earlier posts), that all the services defined for the child hostgroups are handed on upwards to the parent hostgroup(s).

Nagios 3-x and check_pcmeasure-pl

Thu, 07 Aug 2008 18:23:18 +0000

Recently we purchased a MessPC station for our server room, and my co-worker and myself had the wish it to be integrated within Nagios. Well, so far so good. The first I did was put both keywords into Google.

That pretty fast brought up the manufacturer’s page (sorry it’s German only) about the device supporting Nagios by means of either SNMP or a specific plugin called pcmeasure. So I went ahead and tried both ways.

More VirtualCenter troubles (fini)

Thu, 07 Aug 2008 18:03:46 +0000

Well, today the support request came back. Seems one of the originally linked VMTN dicussions really is the only way:

Export the customization specification
Edit the XML file
Import it again

The related part inside the customization specification should then look like this:

1
2
3
4


<type>vim.vm.customization.Password</type>
<plainText>true</plainText>
<value>Password01</value>
</password>

So if you ever think about switching the default VirtualCenter certificate (for whatever reason), make sure you use the above workaround. Otherwise VirtualCenter is gonna fail miserably during the customization phase of the cloning process.

SUSE Linux Enterprise Server 10 on VMware ESX (finished)

Wed, 30 Jul 2008 18:08:59 +0000

As I pointed out in my past posts, I was having some weird errors with SLES10 regarding mounting CD images inside the guest (as well, as it turned out, on the physical hardware).

Now, finally, after about a week or so Novell finally released an updated kernel package today. So the error per se is fixed, I can use my CD drives again, as well as update my virtual machines by means of virtual center, without the trickery of copying the linux.iso from /vmimages/tools to the guest and mount it by means of mount -o loop.

Yet another VMware error

Wed, 30 Jul 2008 18:00:49 +0000

Today I was moving a pretty standard SLES10 virtual machine to another host, when the migration dialog showed me this:

fault.MemorySizeNotRecommended

And if you now think, the virtual machine is something special take a look at those settings:

Virtual machine configuration

I don’t know what to think about that error message. Googling for it doesn’t reveal that much about it. If anyone out there got an idea, I’m open for suggestions.

Fixing vmkernel symlinks

Wed, 30 Jul 2008 07:19:32 +0000

Since I do happen to be in the situation pretty often where the kernel inside a VM is newer than what VMware currently has in their tools (as in the SUSE kernel is newer than the binary modules built by VMware), here’s a quick reminder for myself on how to to fix the .ko symlinks.

1
2
3
4
5


for file in /lib/modules/$( uname -r )/misc/*.ko ;
do
 rm $file &&
 ln -s /lib/modules/$( uname -r )/misc/${file/.ko/.o} $file
done

Microsoft Cluster Services powered by IBM

Sat, 26 Jul 2008 18:47:46 +0000

If you think back, I talked about my problems with MSCS while utilizing the IBM RDAC Multipath driver for Windows.

Everyone I talked to about this, including our IBM business partner and it’s systems engineers; as well as some IBM systems engineer (who in fact was an freelance guy hired by IBM), told me it had to do with how we did the zoning (stuffing every controller into a single zone), and that would be the reason why the x3650 was seeing that many drives.

Connecting to a remote console with MSTSC 6-0-6001

Wed, 23 Jul 2008 13:02:00 +0000

Well, as one can read in about every damn post you can find for that topic, the /console switch is now silently ignored, as well as the rdp file option " connect to console:i:1".

Now, what you don’t find anywhere (only in some scenario explanation), that it is allowed to specify the mode (ie /console previously and now /admin) within the full address parameter.

Scenario: In the RDC client UI, you specify Computer_name /console in the Computer box (where Computer_name represents the name of the remote computer to which you want to connect), and then click Connect.

Nagios Hostgroup Inheritance (continued)

Sat, 19 Jul 2008 09:41:21 +0000

Well, it turns out that my thought was ultimativly flawed. When defining the hostgroup_members in the lower tiers, nagios is association the checks from the lower tier with the upper tiers. Thus propagandating all checks upwards, and me ending up with ~250 checks instead of ~150.

Gonna have to try to define the dependency backwards, maybe that’ll help. But that’s a topic for Monday. Guess I’ll finish viewing Ghost in the Shell - Stand Alone Complex first.

Extending VMotion compatiblity (continued)

Mon, 14 Jul 2008 17:41:40 +0000

Remember my last post about cpu masking ? Well, turns out that you can do it to a “template”.

The only point you don’t need to do, is to mark the VM as a " template". You still can clone it and move it around and all that other stuff, but the good part is, that the cloned VM keeps the cpu mask set to the " template" 🤷

Nagios3 with Active Directory authorization on SLES10

Mon, 14 Jul 2008 17:32:04 +0000

Well, it seems to be getting a “trend” for me, to integrate stuff into our Active Directory. Now that I know why, and how easy that is, I expect to add more stuff. The good thing about the integration is, that you only need to maintain a single source for authorization.

The bad thing about that is, that stuff becomes dependent on the Active Directory (we do have four domain controllers, so that should be fine).

Managing unixODBC connections on SLES10

Thu, 03 Jul 2008 06:42:27 +0000

Recently I got the task, to implement unixODBC/freetds on one (well, it’s really three) of our web servers, as someone wanted to use Microsoft SQL Server 2005 with PHP (without using the MSSQL functions, which PHP provides soo nicely; don’t ask me why).

With that I also got a set of " instructions" on how to install freetds from source (remember, I was a Gentoo dev, so I know my way around, when it comes to building from source), as well as a small set of instructions on how to create the connection.

subversion on WebDAV with Active Directory authorization on SLES10

Sun, 29 Jun 2008 17:49:29 +0000

Okay, so I ended up toying with subversion via WebDAV on SLES today (I know, I know .. it’s bloody Sunday). It wasn’t much of a hassle though, after reading this. Sure, I made a few errors at first (simply confused the logic behind " Location" and " Directory"), but after that plain subversion commits via WebDAV (thus utilizing Apache) worked fine.

For POC or as a hint to myself, here’s where and what I needed to add/change:

The clue to build ppc64 RPM's

Thu, 26 Jun 2008 15:52:22 +0000

Remember, I talked about building RPM’s on SLES10SP2 on ppc64 ? Well, turns out I was rather stupid .. and it was rather simple (don’t ask me why I didn’t think of that). I tried asking solar, I used Google (apparently with the wrong search parameters), nothing though. Not a clue.

Today it bugged me again, so I used Google again. This time with " ppc64 suse rpmbuild", and guess what I saw within the preview of the second hit ..

Updated `check_ram' plugin

Tue, 17 Jun 2008 09:25:21 +0000

As I was building the updated RPMs for SLES10, though needed to refresh my old patch. Also, I was getting strange messages from autotools and fixed these quirks, Now, I prepped a patch for it, and finished building new RPMs for i586 and x86_64.

GPO (behind the scenes)

Thu, 05 Jun 2008 06:46:20 +0000

Well, to begin with we had this really weird problem that the thin clients as well as the terminal server would only load user based group policy if you are a member of the group of local administrators. While that’s ok for the thin clients (users can’t actually change something unless they log in as “Administrator” - don’t ask me why), it’s a real no-no on the terminal server.

Windows XP Embedded, Windows Server 2003 and GPO settings (the solution)

Wed, 04 Jun 2008 19:35:57 +0000

OK, so about an hour (yeah, yeah; I know .. I shouldn’t be working at that time, but it really gave me sleepless nights) ago, I finally figured out why the hell both my Windows XP Embedded thin clients as well as my Windows Server 2003 systems where showing this real weird behaviour when applying group policies, or more precise the user based configuration of a group policy.

Rescuing a rebooting machine that's hanging

Sat, 24 May 2008 17:14:56 +0000

One of my co-worker approached me today with a weird problem. Yesterday he had a disk in a 900GiB array failing which he replaced. After that, he run a rebuild/verification, fsck’ed the file system and tried to mount the volume again.

Apparently the mount produced a kernel oops (guess what, the 900GiB is running reiserfs), thus leaving the kernel tainted (or what do they call it ?). So he tried to reboot the box but it didn’t reboot. It started rebooting but then hung (as in not continuing the reboot). He tried to ssh back to the box, and it worked just fine.

IBM (Tivoli) Integrated Solutions Console

Fri, 23 May 2008 17:28:03 +0000

Here I am, preparing our environment for the first (of hopefully many) tester for our upcoming VTL project. So I ended up installing the ISC and Administration Center for Tivoli Storage Manager on a 64bit guest (that is SLES10 for AMD64), just because I forgot to include support for later versions with our current running one. Guess what, na- na na na na. Exactly, didn’t work, the same errors I got while trying it before in a virtual environment. “Portlet is not available.”

Getting a FC HBA to rescan it's attached devices

Mon, 19 May 2008 11:50:48 +0000

If you’re using a 2.6 based distribution, the FC HBA (or more correctly the corresponding driver) should create entries in /sys/class/scsi_host. Now you only need to get the host-number (basically the # of the host bus adapter) and you can get started ..

Simply doing this, is going to tell the FC HBA “rescan” and discover new devices ..

`1`	`echo "1" > /sys/class/fc_host/host/issue_lip`

That should do the trick, and you should be able to get udev to recognize the new devices attached via FibreChannel without the need to reboot the whole box (which might be a bit tricky).

open-vm-tools for Debian Etch

Sun, 20 Apr 2008 12:16:29 +0000

Well, after a loooong time of trying to get the modules and all the other stuff (read: init-script for the guest daemon and modules) working, I think I’m about there.

I finally fixed a long-standing issue, with the postinst/prerm scripts, and the tools should be about ready. Gonna try and send it Daniel Baumann’s way (that is the Debian Maintainer), for proper inclusion into Lenny.

I (successfully) tried splitting the Xorg parts from the " normal" open-vm-tools, as I usually don’t want Xorg installed on any of my virtual machines. Thus leaving me with open-vm-tools, open-vm-modules and open-vm-toolbox (and open-vm-source) as a list of packages one could install.

Software support and key account managers

Mon, 14 Apr 2008 12:24:59 +0000

As Mike wrote about his experiences with hardware vendors, I’m gonna devote this here post to my favorite software company in the world. We recently bought two copies of a software called " 2X Application Server Enterprise Edition". As one would think from reading the spe cs of the software, it’s near a Citrix solution (which it is, at least for a small part); but in return it’s faaaar away concerning the price. Just so you get an idea, about what I’m meaning with " faaar":

1
2
3
4
5
6


Windows Server 2003:
Standard Edition: 2 * 91,00
CAL: 50 * 6,00
Terminal Server CAL: 50 * 17,00
 ___________
 1.332,00

The above are fixed costs, you need them anyway as both Citrix as well as the 2X solution is only working * on top* of Windows Server 2003 Terminal Services.

Now, here’s the real comparison between 2X Application Server & Loadbalancer and Citrix XenApp Platinum Edition:

1
2


2X 2 * 1510,00 = 3.020,00
Citrix 50 * 393,00 = 19.650,00

While 2X is licensed per terminal server, XenApp is licensed per user. As you can see from the above prices, the 2X solution is roughly 1/6 of the Citrix XenApp solution.

TYPO3 hogging

Tue, 08 Apr 2008 21:28:44 +0000

Well, we do appear to be having some strange load problems with our main TYPO3 box hosting several home pages of the local universities, as you can see below.

LOAD on t3node1 between 05:00-19:00 on 2008/04/07

We repeatedly tried to figure out which of them was the one responsible, but neither I nor the other Unix sysadmin knew a better way to figure out the load each TYPO3 installation was causing (since there ain’t no phptop or something similar). But since today the new semester started, we figured it might be good to finally figure which one it was. And a few minutes (as in one or two) wouldn’t be much of a problem compared to the advantage we’re getting out of it.

Creating multi-distribution RPM/XML repositories

Wed, 02 Apr 2008 14:36:05 +0000

Well, as we do have quite a few custom built RPM’s, I was searching for a new solution to manage the repo(s). Currently I do have a single repository per distribution.

One thing one needs to know about createrepo (from createrepo), it doesn’t support this type of thing in the first place. So I had to come up with another way of doing it. First, I created a proper layout (much like the Debian Official Repository layout):

Integrating Windows XPe into Active Directory

Fri, 07 Mar 2008 22:12:44 +0000

As the guys over at FreeWyseMonkeys demonstrated with JoinDomain.zip, it ain’t hard to integrate a Windows XP Embedded system into Active Directory.

You basically need this:

A system powered by Windows XP Embedded
netdom.exe (from any Windows XP - SP2 in your MUI language)
some know-how, on how to use netdom to integrate it into your AD

Everything else is already present on the Windows XP Embedded systems I’ve seen. Then let’s get it on !

Backup solutions

Fri, 07 Mar 2008 21:38:52 +0000

Well some people apparently completely don’t understand the use of a backup client like dsmc, additionally they don’t seem to have the slightest clue on how to draw up a “clever” backup solution.

Lemme describe the situation for you. We do have two Solaris systems at work, housing our mailing system(s). Now apparently, people are unable to install the Tivoli Storage Manager Client on Solaris (or get it working properly - which people are blaming on the software not working).

OCFS2 follow-up

Fri, 07 Mar 2008 07:29:26 +0000

OK, it turned out that said colleague wasn’t responsible at all. Turns out, the real trigger was me creating a new volume on our SAN, on the same array that houses the OCFS2 volume.

Apparently, during creation of an additional SAN volume, all other SAN volumes in this array are either read-only or delayed during that time, as you can see from the following log:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29


kernel: (13,3):o2hb_write_timeout:242 ERROR: Heartbeat write timeout to device sdd1 after 12000 milliseconds
kernel: Heartbeat thread (13) printing last 24 blocking operations (cur = 4):
kernel: Heartbeat thread stuck at waiting for read completion, stuffing current time into that blocker (index 4)
kernel: Index 5: took 0 ms to do submit_bio for read
kernel: Index 6: took 0 ms to do waiting for read completion
kernel: Index 7: took 0 ms to do bio alloc write
kernel: Index 8: took 0 ms to do bio add page write
kernel: Index 9: took 0 ms to do submit_bio for write
kernel: Index 10: took 0 ms to do checking slots
kernel: Index 11: took 0 ms to do waiting for write completion
kernel: Index 12: took 2002 ms to do msleep
kernel: Index 13: took 0 ms to do allocating bios for read
kernel: Index 14: took 0 ms to do bio alloc read
kernel: Index 15: took 0 ms to do bio add page read
kernel: Index 16: took 0 ms to do submit_bio for read
kernel: Index 17: took 0 ms to do waiting for read completion
kernel: Index 18: took 0 ms to do bio alloc write
kernel: Index 19: took 0 ms to do bio add page write
kernel: Index 20: took 0 ms to do submit_bio for write
kernel: Index 21: took 0 ms to do checking slots
kernel: Index 22: took 0 ms to do waiting for write completion
kernel: Index 23: took 2004 ms to do msleep
kernel: Index 0: took 0 ms to do allocating bios for read
kernel: Index 1: took 0 ms to do bio alloc read
kernel: Index 2: took 0 ms to do bio add page read
kernel: Index 3: took 0 ms to do submit_bio for read
kernel: Index 4: took 9995 ms to do waiting for read completion
kernel: (13,3):o2hb_stop_all_regions:1682 ERROR: stopping heartbeat on all active regions.
kernel: Kernel panic - not syncing: *** ocfs2 is very sorry to be fencing this system by panicing ***

OCFS2 fun

Thu, 06 Mar 2008 10:31:05 +0000

Turns out, that said colleague has been playing with NFS on one off the web nodes, thus apparently rendering the remaining nodes offline (or semi-offline).

Now after all web nodes hung themselves, we had to hard reset them, now everything is tingly again .. yay for a great first day …

Getting the 2X service up and running

Thu, 21 Feb 2008 19:31:18 +0000

As I mentioned before, we decided against the Citrix Presentation Server solution in favour of the 2X LoadBalancer and ApplicationServer combination. You’re gonna say, but Citrix does the same and it’s only one " application frame". 2X is exactly the same.

Zend Optimizer again

Tue, 19 Feb 2008 19:15:54 +0000

Well, I happen to be back at my favorite application. Today I stumbled upon a " nice" thing. If you turn on the Zend Optimizer (doesn’t matter whether it is 2.6.2 or 3.3.0), one of the TYPO3 back ends ain’t showing any content in the preview pane. Once you turn the Zend Optimizer stuff off, it works without a problem.

O RLY ?

And as Zend stated on their " Support Forum", they don’t really support the Zend Optimizer stuff in the first place. Which is nice, what for do you need the Zend Guard shit in the first place ??

Been a while

Sun, 17 Feb 2008 08:53:16 +0000

Well, it’s been quite a while since most of the people last heard a word from me. The last few months I’ve been extremely busy with work-related tasks (and as a side-effect of that, didn’t want to spend much time in front of the computer after 9 hours of work). I also started spending more and more time in the gym, like nearly two hours every Tuesday and Thursday.

I finally fixed our replication issues, we do now have a working! MySQL Multi-Master ( 1. Node, 2. Node -- bear in mind, this boxes are only serving MySQL and nothing else, so don’t use these configurations on mixed setups) Replication Setup as database back end for our TYPO3-vHosts.
all the web nodes are now serving the content from a clustered, shared SAN volume (is that a good thing ? 😛 - don’t know yet …)
our VI environment is getting more and more acceptance (even if you hear some complaints now and then, like “awww, damn that crap my 4GiB RAM, 2x3.0GHz Windows 2008 is running soooo choppy” - simple answer, don’t use Windows Server 2008 and/or Windows Vista!)
I finished prepping our VM templates (at least the Windows ones)
we’re still putting together the plans on whether or not invest into a VDI solution.

The next few weeks are gonna be as frantic as the weeks before, I still have to migrate a lot of TYPO3 installations to our new cluster (which sadly needs time, as we need to wait for DNS changes to propagate). Honestly, I might be ending up extending the SAN volume for the MySQL data storage, as even with only three somewhat busy sites, the binary log of the last 5 days is about 2GiB in size. And we still have ~ 20 other busy sites on a separate box.

Windows Server 2008

Sat, 16 Feb 2008 17:41:11 +0000

Well, as it is Saturday and I’m having lots of time (whereas I’d usually spend it working), I thought I’d give Windows Server 2008 a try. What interested me most, is the Windows Server 2008 " Server Core Installations", as it’s supposed to lower the security risk (as there is no Internet Explorer, no Explorer nothing running by default, only a simply cmd.exe).

As one of my co-workers requested me to upload the Standard/Enterprise/Datacenter DVD (which he got through our Microsoft Select 6.0(?) agreement) to our ISO’ VMFS, I had the DVD already at hand. As for that, I really love the feature set of VMware.

PacketPro 1-7-0

Tue, 15 Jan 2008 08:37:41 +0000

After blogging the last time about the PacketPro 450 LoadBalancer appliance, the guys over at teamix seem to have taken that to heart and implemented a rather nifty thing for their new release.

It’s called " Port forwarding", which is basically what you’d figure from the name. It bounces ports around the load balancer, but saves you from creating a separate virtual server (and adding the physical servers to that one), but also saves you from modifying the syslog-ng configuration on the balanced servers.

Deploying VM templates

Tue, 08 Jan 2008 09:38:03 +0000

Ok, so after my first day yesterday after a rather long vacation I today wanted to look at the problem that the Administrator password isn’t changed when using VirtulCenter’s clone customization functionality (which relies at least for Windows on sysprep).

After a short googling, I stumbled upon this.

Simple problem short … Don’t specify an Administrator password for the template. Then you should be able to change the Administrator password when cloning the template. It’s " should", as the VM’s are still updating.

Looking back (yet again)

Mon, 31 Dec 2007 16:10:53 +0000

Well, it’s yet again New Year’s Eve. Yet again a whole year passed by blazing fast, I didn’t manage to get everything done like I wanted.

That includes the following things:

getting a better job (and probably better paid too!)
getting a better life (well, it’s as it sounds like - my current life is rather unhealthy, and thanks to a friend I got the grip onto myself and started changing a few things - like doing a small workout every day, a bit more movement all over the day and so forth)

Which also means I do have some resolutions for the next year …

Device CAL's ain't no Device CAL's

Tue, 04 Dec 2007 08:21:13 +0000

I stumbled upon a real weird problem. Apparently the terminal server licenses called " per Device" ain’t a real per device. From reading on it Microsoft states it like this:

Device-based versus User-based Terminal Server CALs

Two types of Terminal Server Client Access Licenses are available: TS Device CAL or TS User CAL.

A TS Device CAL permits one device (used by any user) to conduct Windows Sessions on any of your servers.

A TS User CAL permits one user (using any device) to conduct Windows Sessions on any of your servers.

You may choose to use a combination of TS Device CALs and TS User CALs simultaneously with the server software.

Windows terminal services undamp; network printers

Sun, 04 Nov 2007 14:08:05 +0000

Yes, yes. I do list a lot of crappy products (go on, laugh; I don’t really care). Yesterday I had quite a struggle with Microsoft Windows Server 2003 and Terminal services (or more precisely with their way on how to deal with network printers).

As most of you know, there a two (possibly three) different ways on how to do network printers.

would be, to simply share a local connected printer by simply creating a share for the printer
buy a smart printer with integrated print server
a combination of 1. and 2.

We luckily enough do have printers with integrated print servers, so that wouldn’t be a problem. But you get a problem if you’re trying to monitor the printer queue if you simply create a new TCP/IP connection from another target. You simply can’t tell who’s printing what.

Customizing Thin Clients

Fri, 12 Oct 2007 22:14:53 +0000

As some of you know, the company I’m currently working for, recently acquired some thin clients to replace our old computers for the students to work on. Those PC’s are like P3 800 MHz with 512MB RAM and sadly don’t run Office 2007 anymore, so we replaced them with thin clients and are streaming those applications from a Windows Terminal Server cluster (created by and with 2X Application LoadBalancer).

So far so good, getting them to display the applications ain’t hard, the real hard part starts when you want additional things from this Windows XPe (Embedded), like lets say getting them to display a German language.

Nagios undamp; plugins

Sat, 06 Oct 2007 15:12:43 +0000

Since we started utilizing Nagios’s power two months ago, I finally came up with a C-based ram-plugin for nagios. The biggest problem I had with the python and perl based plugins, that some distributions (yes, SLES and Debian) don’t install either Python or Perl.

Since I wanted a manageable setup (as in unified code base across all distributions), I wanted it to work without installing too much. So I took the swap plugin and basically removed what wasn’t necessary and voila!

Thin clients

Sat, 06 Oct 2007 14:13:44 +0000

As some of you people know, we (as in the University) recently purchased some Thin Clients in order to replace some oldish’ computers and solve the software management at the same time.

The Thin Clients ain’t bad, they are Wyse V90L’s and they (as in Wyse) use their own management software to manage and deploy those thin clients and software. The bad thing about that, is it’s using it’s own “Scripting Language” (if you can call it that way - it’s more pseudo scripting since you can’t do much with it besides some basic actions).

TYPO3 and MySQL replication

Sat, 08 Sep 2007 20:33:13 +0000

Apparently the TYPO3 version we are using, doesn’t play too nice with the MySQL MasterMaster replication.

Sometimes, something like this is going to happen:

1
2


070826 0:44:32 [ERROR] Slave: Error 'Duplicate entry '75-222419149' for key 1' on query. Default database: 't3nb'. Query: 'INSERT INTO cache_pagesection
070826 0:44:32 [ERROR] Error running query, slave SQL thread aborted. Fix the problem, and restart the slave SQL thread with "SLAVE START". We stopped at log 'dbc-mysql1.000192' position 611861372

Well, as you can see from the last line in the log, the Slave-SQL thread found a duplicate entry and thought it is smart to just turn off the thread instead of disregarding the just made entry. So from now on, both databases drift since there ain’t no replication anymore until someone kick starts the replication again (someone being me).

PacketPro 450 and SSH checks

Sun, 05 Aug 2007 14:51:56 +0000

As apparently the guys at Teamix read my recent blog post about their cluster solution, someone of their technical support called me on friday at work 😳

And pointed out

That I’m free to express my thoughts about their product (which I recently did)
That there is a better way to workaround this issue

He also said, its something which they had asked multiple times. It’s as simple as editing the Virtual Server and changing the service inspection from “Connection” to “None” .. duh

Fujitsu Siemens, onboard NIC's, Quality assurance and vendors

Thu, 12 Jul 2007 19:15:56 +0000

So we bought some Fujitsu Siemens P5916 Intel vPro back in January/February for the Boss and his secretary.

These boxes are quite nice, come with a Core 2 Duo (which is waaay to overrated for simple business applications like Word, Excel, Access and Outlook), but he insisted on having Windows Vista Ultimate ready PC’s.

We got them, as expected completely blank. Wasn’t so much of a problem though, since we have a Select 5.0 6.0 contract with M$. Only problem was, they refused to install Vista (as in freezing after preping the HDD). So I called our local vendor, who told me " Go, grab the latest BIOS from the support page and perform a BIOS update!" - Which I wasn’t so happy about to hear and to do … That didn’t work, the box would freeze on boot now …

SLES, ZendOptimizer and IBM PowerPC(4)+

Wed, 11 Jul 2007 08:40:57 +0000

What would you figure from the above ? Hopefully the rather obvious, that it’s a really shitty combination.

So we figured it would be a nice thing to test our new setup before going into pre-production testing or production, but we don’t have an extra spare box. So we took one of the power4 boxes we have mounted in the rack basically consuming energy all day (that’s about 38kWh a day) and installed SLES10 onto it. Which wasn’t all that bad (at first the box repeatedly started back to AIX, from CD and after convincing the SMS - that’s basically the bios on the power*-boxes also known as System Management Services with a hammer to boot from the first hard disk).

Dell PowerEdge 1855, DRAC/MC, firmware updates, telnet and csr's

Fri, 06 Jul 2007 09:39:38 +0000

Today I played a bit with our PE Chassis, or more specifically the DRAC/MC (remote management console). One of the things I’ve been experiencing was that the DRAC/MC was rather slow when browsing on the web interface (as in waiting a minute for the jnlp for the KVM to download). So I went ahead, fired up net-misc/atftp on my notebook, put the firmware update provided by Dell in the TFTPROOT and executed this in my telnet session on the DRAC/MC:

miimon, arp_interval and the code

Wed, 04 Jul 2007 19:50:14 +0000

After today’s adventure with the kernel bonding, I just took a look at the code …

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28


 if (miimon) {
 printk(KERN_INFO DRV_NAME
 ": MII link monitoring set to %d msn",
 miimon);
 } else if (arp_interval) {
 int i;

 printk(KERN_INFO DRV_NAME
 ": ARP monitoring set to %d ms, validate %s, with %d target(s):",
 arp_interval,
 arp_validate_tbl[arp_validate_value].modename,
 arp_ip_count);

 for (i = 0; i < arp_ip_count; i++)
 printk (" %s", arp_ip_target[i]);

 printk("n");

 } else {
 /* miimon and arp_interval not set, we need one so things
 * work as expected, see bonding.txt for details
 */
 printk(KERN_WARNING DRV_NAME
 ": Warning: either miimon or arp_interval and "
 "arp_ip_target module parameters must be specified, "
 "otherwise bonding will not detect link failures! see "
 "bonding.txt for details.n");
 }

If I read it right, you only get the KERN_WARNING for “either miimon or arp_interval” only if miimon or arp_interval isn’t set … but at least my config says it is .. 🤷 .. bed time for me :rolling_eyes:

SLES10 on pSeries

Wed, 04 Jul 2007 15:58:39 +0000

Okay, yet another day passed by blazing fast. I had a good day at work, spent nearly the whole day trying to get my bloody systems hooked up to our SAN (which was interrupted by a non-working SAN-switch, disappearing WWN’s, lunch and my trainees), messing around with our internal network, hacking our Blade Chassis switches to get me what I want and some random paperwork.

But first things first .. We installed SLES10 on a pSeries box the other day (I think on Monday), and now I’m trying to get the WWN of it’s Emulex HBA, out of either sysfs or procfs. But whatcha’ thinking ?

Continuing on SLES10

Sat, 16 Jun 2007 16:14:15 +0000

OK, it turns out that I was rather stupid when configuring the my.cnf. As it turned out, the effect I was seeing was due to the presence of two log-bin lines, which looked like the following:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12


[mysqld]
port = 3306
datadir = /mysql/dbase
log = /mysql/logs/dbc-mysql1.log
log-error = /mysql/logs/dbc-mysql1.err
socket = /var/lib/mysql/mysql.sock
bind = 172.16.234.31

# custom paths for binary logs
log-bin = /mysql/binlogs/dbc-mysql1
log-bin-index = /mysql/binlogs/dbc-mysql1.idx
relay-log = /mysql/binlogs/dbc-mysql1.relay

And some lines down there was this:

Back at SLES10

Tue, 12 Jun 2007 19:42:22 +0000

Here I am, sitting at my desk on a Thuesday evening thinking about what happened the last few days.

I finally got to play around with our PacketPro 450 Cluster(nifty LoadBalancing appliance)
We reworked the network the way we want it (and not that tool of a wannabe sysadmin)
We mostly figured out how to do the LoadBalancing right, we just need to find some bugs in the LoadBalancer software (like the thing is failing over to its slave from time to time, but keeping the IP address for himself) or let the guys at teamix do their work and hopefully get a working release within the next week or so
I figured out how to setup interface bonding with SLES10 (it was quite straight forward, thanks to the excellent in-kernel documentation), and we’re using an active-backup mode for now
I still need to figure out how to do the MySQL Master<->Master replication right .. I’m currently building fresh RPM’s on one of those Dell blades (yes, they ROCK!) which will hopefully be finished till I’m at the office tomorrow.
Pt. 5 also includes figuring out how to pass MySQL a custom location for the binary-log, at least that’s what the handbook says in Chapter “5.11.3. The Binary Log” …

When started with the –log-bin[=base_name] option, mysqld writes a log file containing all SQL commands that update data. If no base_name value is given, the default name is the name of the host machine followed by -bin. If the basename is given, but not as an absolute pathname, the server writes the file in the data directory. It is recommended that you specify a basename; see Section B.1.8.1, “Open Issues in MySQL”, for the reason.

Personal progress

Mon, 28 May 2007 15:03:44 +0000

Well, I haven’t blogged in ages (like about a month).

To answer some of the questions I got either via mail or on IRC …

yes, I’m still alive; you ain’t gonna get away that easy 😛
yes, I finally managed to buy my shiny new car

new car
I’m still working on apache, virtualization, mobile, kernel, hardened …

Though I may haven’t been that active recently (I don’t look at CIA or do I count my own commits), I think I managed to close some apache bugs and some of my own. I’m still looking for someone interested in working on the ipw3945 related packages, as I’m kinda sick of that buggish thing.

Bla-bla-bla, yada-yada-yada

Fri, 04 May 2007 06:48:36 +0000

Interesting title … 🤷 I’m sitting back home, in my bed, while I’m supposed to be at work. I’m now a certified sicko due to a common cold.

Watching the toaster series once again (that’s Battlestar Galactica) and trying to think about some things. Heh, well thinking isn’t the right term; it’s more like meditate over stuff happening in the last few months.

Guess this stupid cold is dictating my weekend 😠 and hopefully I’m gonna be better on Monday. Have fun.

AIX 5-3 Linux Toolkit

Fri, 20 Apr 2007 13:07:04 +0000

OK, so I skipped rebuilding a newer RPM version (for now) and I’m currently rebuilding anything that fit’s into app-dev according to IBM …

The list reads like this:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19


$ ls -al ~/app-dev
autoconf-2.59-1.aix5.1.noarch.rpm
automake-1.8.5-1.aix5.1.noarch.rpm
binutils-2.14-3.aix5.1.ppc.rpm
bison-1.875-3.aix5.1.ppc.rpm
coreutils-5.2.1-2.aix5.1.ppc.rpm
cpio-2.5-1.aix5.1.ppc.rpm
diffutils-2.8.1-1.aix4.3.ppc.rpm
flex-2.5.4a-6.aix4.3.ppc.rpm
gawk-3.1.3-1.aix5.1.ppc.rpm
gcc-4.0.0-1.aix5.1.ppc.rpm
gcc-cplusplus-4.0.0-1.aix5.1.ppc.rpm
gdb-6.0-1.aix5.1.ppc.rpm
libgcc-4.0.0-1.aix5.1.ppc.rpm
libstdcplusplus-4.0.0-1.aix5.1.ppc.rpm
libstdcplusplus-devel-4.0.0-1.aix5.1.ppc.rpm
libtool-1.5.8-1.aix5.1.ppc.rpm
m4-1.4.1-1.aix5.1.ppc.rpm
make-3.80-1.aix5.1.ppc.rpm

OK, I’m not exactly rebuilding these old versions, I’m actually using their old specs to compile newer versions of these. I’m currently at coreutils-6.7, which really takes ages. But will see about the rest.

grub via serial console

Tue, 10 Apr 2007 15:09:17 +0000

As I have quite some trouble every time I need grub via serial console, here’s just my personal reminder on how to do it right:

1
2
3
4
5
6
7


# cat /boot/grub/grub.conf
serial --unit=0 --speed=38400
terminal --timeout=15 console serial

title hardened-sources-2.6.20-r1
root (hd0,0)
kernel (hd0,0)/boot/vmlinuz-2.6.20-hardened-r1 root=/dev/hda1 console=ttyS0,38400 console=tty0

Additionally the agetty entry for ttyS0 in /etc/inittab needs to be uncommented and changed accordingly to the serial speed

Cisco sucks

Wed, 14 Mar 2007 16:10:38 +0000

OK, today we had somewhat of an emergency. The core-router for our entire network at work had some kind of hardware defect and repeatedly rebooted every three minutes caused the whole network to go cabooom. Usually (you would think), stuff in the same subnet (or VLAN) would still see each other (again, you would think) .. but apparently the VLAN/subnet database is stored on the core router and took all subnets with it.

SLES-9 (once again)

Thu, 08 Mar 2007 22:55:36 +0000

OK, so today was the highlight of the week … We updated apache2 on Tuesday (yeah, that’s still 2.0.49, so if you have some exploits - try them 😛 ) and now out of the sudden we have major performance issues. We looked nearly the whole forenoon for a reason, why the frackin’ apache was using 236% of the CPU’s.

In the afternoon, when my co-worker decided to go home (that was ~1500), I decided to revert back to the old patch level. But that isn’t as easy as you think (at least on SLES). The only thing I wanted to do, was something like this:

Waiting

Wed, 28 Feb 2007 14:58:05 +0000

We are still waiting for the money promised by the state and the country for our HBFG (again, it’s “Hochschulbauförderungsgesetz”), that hopefully is reducing or eliminating our storage/SAN problem we have currently. Right now we have to Cisco MDS9216 (that’s a 16-port 2GBps SAN-switch, two for redundancy), which means we only have 16 SAN-ports. That isn’t much, but still is to less, as we have like 30 machines or so, that really need access to the SAN, so we either end up unplugging some of them from the SAN or merge them onto some big machines (like our x366).

Shibboleth (WTF is that)

Wed, 28 Feb 2007 08:51:05 +0000

OK, I’m sitting now again in train (hrm, I get the feeling I’ve done that already in the last few days - oh wait, I was doing that just on Monday) this time to Berlin.

My boss ordered me to attend a workshop covering the implementation of Shibboleth (for those of you, who can’t associate anything with that term - it’s an implementation for single sign-on, also covering distributed authorization and authentication) somewhere in Berlin Spandau (Evangelisches Johannesstift Berlin).

AIX-5-3 undamp; rpm-4-4-7

Mon, 05 Feb 2007 14:24:50 +0000

OK, so I tried to install the AIX Toolkit today, to build some newer rpm’s (yaaaaah, I hate RPMS myself, still it’s way better than distributing plain tar.gz archives) but looks like either AIX or rpm-4.4.7 doesn’t like me.

Now I’ve to figure out how to get libm (that’s /lib/libm.so) installed on AIX. Will see about that later and/or tomorrow.

Qualcomm Eudora vs- Microsoft Outlook

Thu, 25 Jan 2007 20:44:52 +0000

OK, so I’ve spent the last day working on my bosses $MAILPROGRAM. In detail, I’ve been trying to get his mails from Eudora (which really is complete crap) to Outlook (yeah, yeah I KNOW but he really needs a decent calendar with his E-Mail program, which neither KMail, nor Thunderbird nor Mozilla Suite - aka SeaMonkey can provide).

Problem with all that is Eudora’s crappy way of saving mails (and their attachments). Eudora is saving the mails in regular MBX format, but is putting the attachments into a separate folder. I’ve looked all over the web, and only found one application capable of importing his mails into Outlook including the attachments from the Attach folder.

Depression coming up

Tue, 05 Sep 2006 11:10:18 +0000

As you may now (or not), I haven’t been pretty active in the last couple of days (OK, I admit it has been the last two weeks).

Work is currently being a bitch (I have a 6,36 h day normally, but lately that evolved into a 8+ h day which I’m not even payed for), personal life is also a bit frustrating (yeah you know those people-people relations called friendship and stuff).

IBM

Mon, 14 Aug 2006 10:16:28 +0000

We just received the long awaited shipment of sixteen 300GB FC-HDD’s (2Gbps with 10000rpm) for our SAN (a pretty old DS4500/FaStT 900).

But there’s still the software option missing we ordered within the same breath. So I called our trustworthy IBM distributor (hah!) and asked the guy responsible for sales, what the ETA on this software option is (if someone is interested its VolumeCopy/FlashCopy).

He told me, that we’ll receive a letter with the license key about 4 weeks after commission !!!!!!

SLES-9 SP2 (continued)

Wed, 09 Aug 2006 12:27:50 +0000

OK, so after yesterdays battle with SLES and rpm, I decided to simply upgrade rpm (again rpmbuild -bb rpm.spec as in from source).

I had to tune (hah, remove stuff that isn’t working on this ancient gcc-3.3.3 ie. -fstack-protector) a bit to get it working, wasted 20 minutes worth of CPU time and ~30 of my time. I finally gave up. I couldn’t persuade neither rpm, nor git to compile on the damn SLES/Dell.

Live sucks (again)

Sun, 21 May 2006 10:46:25 +0000

Now is again such a time in live, where you have the motivation or wish to just fade away.

I’m just listening to Fort Minor - Where’d you go and thinking about the stuff Mike Shinoda is singing …

I want you to know it’s a little fucked up, That I’m stuck here waitin’, no longer debatin’, Tired of sittin’ and hatin’ and makin’ these excuses, For while you’re not around, and feeling so useless, It seems one thing has been true all along, You don’t really know what you got ’til it’s gone.

Bitching

Fri, 10 Mar 2006 19:57:11 +0000

Once again, I’m compelled to play (other call it administering 😛) with our TYPO3 cluster (which is sadly running SLES).

One thing I just learned about SLES (for the ones curious, its Novell’s SuSE Linux Enterprise Server and yes, it suffers the same pain as SuSE/openSuSE). They split one single config file (at least the apache2 one) into 9 (or more) different files.

Another thing is, for what the hell does a simple LAMP need a full blown Xorg w/ KDE installed ?

Work sometimes sucks

Fri, 10 Mar 2006 16:22:46 +0000

Today they finally let me fiddle around on an AIX 5.3 system. Well AIX ain’t bad, but misses (by default installation) some features and comfort.

The first thing I noticed, they only! install telnet and don’t even give an option to install sshd .. That sucks, especially if you’re supposed to log in via the internet (yay! an telnet open to the internet 😯)

So I googled a bit around and found out, that we should have some Bonus CD’s (from IBM of course) featuring openssl/openssh, looked into the cubicle behind the rack and look what I’ve found … Go, go … 😛

vserver related stuff

Tue, 21 Feb 2006 08:07:42 +0000

Well, so I finally got around to fix some pending bugs ..

vzquota and vzctl is finally bumped ( #122532), new baselayout-vserver is in the tree (fixing some issues with rebooting a vps) and I’m finally working on getting dietlibc updated.

So let’s see what the day reveals 😄