Ibm on BAFM

Mass-updating Tivoli Storage Manager drive status

Wed, 11 Oct 2017 19:48:40 +0000

I was fighting with our VTL again, and TSM was thinking all the drives were offline. In order to update the drive status, you’d need to go into the ISC and select each drive and set them to ONLINE. Since I’m a bit click-lazy, I wrote a simple nested for-loop, which gives me the output to update all the drives at once:

1
2
3
4
5


for i in 1 2; do
 for k in $( seq -w 1 32 ); do
 echo "UPDATE DRIVE VTL$i VTL${i}_DR${k} ONLINE=YES"
 done
done

Result is a list like this:

Summer - finally

Sat, 16 Aug 2014 10:25:27 +0000

Well it’s mid of July and the weather seems to be my friend. 25°C ain’t that bad. I really liked the weather last week (although everyone at work was bitching about it being tooo warm 😛) and would like to keep it (for the rest of the year of course!).

Hrm, for everyone who loved the music within Kill Bill - Volume 1: Tomoyasu Hotei really rocks (playing Battle without Honor or Humanity).

IBM RDAC and Windows Cluster Service

Sat, 16 Aug 2014 09:56:43 +0000

Okay, so we received a brand new x3650 the other day entitled to replace one (or better two) of our NAS frontend servers. We installed Windows on it the other day (had to create a custom Windows Server 2003 CD first, since the default one doesn’t recognize the integrated ServeRAID), and we prepped the box during the week with the usual things.

On Monday I started installing the " IBM StorageManager RDAC" MultiPath driver (since the box got two single port PCIe FC-HBA’s) and figured I’d be nice if we had this. I asked a IBM Systems Engineer of one of our partners, which told me generally there wouldn’t be a problem with Microsoft Cluster Services (MSCS) and the IBM MPIO driver. Only requirement would be that I’d install the new storport.sys driver (version 5.2.3790.4021) first (as in Microsoft KB932755).

IBM RDAC: Installing the driver for a (not yet) running version

Fri, 08 Aug 2014 09:04:36 +0000

Well, kernel updates on our Linux servers running IBM’s RDAC driver (developed by LSI) is a real pest .. especially if you have to reboot the box two times in order to install the drivers/initrd correctly.

So I sat down and looked at the Makefile. Turns out, it just needs four tweaks in order to be working with a different kernel version (which you have to pass using environment variables to make).

Novell KMP: KMP'ing IBM's RDAC driver

Fri, 08 Aug 2014 09:03:51 +0000

Well, after yesterday’s lesson about getting the IBM RDAC to install for a not-yet-running kernel, I decided to take it a step further. Novell does have some documentation about KMP’s, which is actually rather good, especially the guide written by Andreas Grünbacher.

After a short tinkering, I got it actually working. I was kinda surprised, at how easily it actually is. One problem I still have to deal with, is modifying the %post, to generate the mpp-initrd image. For now, the KMP only contains the default %post, which updates the modules.* stuff.

Novell KMP: Useable version of ibm-rdac-ds4000

Fri, 08 Aug 2014 09:03:03 +0000

After some more tinkering, a lot more looking at the macros in /usr/lib/rpm/rpm-suse-kernel-module-subpackage and /usr/lib/rpm/suse_macros, I think I finally have a usable RPM’ified version of IBM’s Multipathing driver ready for use.

There is still one major annoyance left: each time you install a new ibm-rdac-ds4000-kmp RPM, you also need to reinstall the corresponding ibm-rdac-ds4000-initrd package, as the macros in /usr/lib/rpm don’t allow for custom %post or %postun.

As mentioned before, I’m gonna send them to LSI/IBM for review, and maybe, MAYBE they are actually gonna make use of that.

IBM RSA II adapter and Java RE

Fri, 08 Aug 2014 08:43:14 +0000

Today, after a short break (you can call it break, I think), I sat down and looked at the IBM RSA II adapter’s remote management GUI and it’s trouble with JRE versions. Ever since the last Java updates, I was unable to access the RSA console because Java would throw an error like this:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21


Initializing RemoteDisk v2.2
MCS v.3.6 initialized
Established connection to rsa.home.barfoo.org:1045
Connected via socket: Socket[addr=rsa.home.barfoo.org/10.0.0.150,port=2000,localport=4292]
Closing socket
java.lang.NullPointerException
 at mcsClient.Row.isValid(Unknown Source)
 at java.awt.Component.invalidateIfValid(Unknown Source)
 at java.awt.Component.setLocale(Unknown Source)
 at javax.swing.JComponent.(Unknown Source)
 at javax.swing.JPanel.(Unknown Source)
 at javax.swing.JPanel.(Unknown Source)
 at javax.swing.JPanel.(Unknown Source)
 at mcsClient.Row.(Unknown Source)
 at mcsClient.Options.(Unknown Source)
 at mcsClient.McsToolBar.(Unknown Source)
 at mcsClient.McsClient.begin(Unknown Source)
 at mcsClient.McsClient.init(Unknown Source)
 at sun.plugin2.applet.Plugin2Manager$AppletExecutionRunnable.run(Unknown Source)
 at java.lang.Thread.run(Unknown Source)
Ausnahme: java.lang.NullPointerException

In the end, I downloaded every version since JRE 1.5.0.11 (that is 20 different versions ❗), as wittnessed by Michael Ellerbeck that the last working version for him was JRE 1.5.0.11, and gave each one a try (since I want to report the issue to IBM, so that they gonna release a fix sometime soon).

IBM SVC: Copy VDisk Host-Mapping from one host to another

Fri, 08 Aug 2014 08:28:32 +0000

As I wrote a few days ago, I started a new job. One of my first (voluntary) tasks was writing a shell script which would copy a VDisk Host-Mapping from a given host to another. This is useful, if you do have a lot of ESX servers for example and a few roaming ones.

Now, if say, you need to do some ESX-Updates and you would like to add the roaming one to a given farm, you would be in a dark an deary place. You would be required to either click through the GUI a dozen times (in my case, it might have needed ~200 clicks) or type svcinfo lshostvdiskmap and svctask mkhostvdiskmap -force (these are incomplete command references) a few times.

SVC: Find WWPN

Mon, 15 Nov 2010 17:50:56 +0000

Today we had (once again) hardware troubles. We ended up replacing a lot of things, but in the end it was a) the HBA and b) apparently some memory DIMMs. Now, that isn’t SVC related. However, we built in another HBA (from our Standby hardware), which apparently already had been assigned to a host.

Now, since you can’t search for a WWPN (at least not that I know of), I ended up writing a little script (yup, AGAIN) in order to do that for me!

SVC: Migrate VDisks off a MDisk Group onto another

Thu, 29 Oct 2009 18:09:17 +0000

Out of necessity, another SVC shell script was just born. If you ever need to migrate a whole MDisk group onto another, you quickly discover the limited application of the SVC GUI. Now, you could query the VDisks using your original MDisk Group and then copy and paste the VDisk’s name (or the VDisk ID) into a command line and simply reuse that svctask migratevdisk command over and over.

Luckily IBM blessed the SVC with an SSH interface. So again, we can write a (kinda) simple shell script which may look like this:

New IBM RDAC version (or not)

Wed, 09 Sep 2009 20:03:30 +0000

A week ago (September 02nd), I received a mail detailing the release of IBM’s new multipathing device driver for the DS4x00 series, which finally works with SLES11 (the available software up till now doesn’t – as in fails with kernels > 2.6.26 iirc).

ESC+ notification detailing the release

There wouldn’t be any trouble, if IBM (or rather the vendor providing the driver – LSI) would actually release the driver … up till today, I have yet to see the new version appear on the download page. I already tried to notify IBM about the trouble, but as usual there is lack of ways to actually get this to the right person.

Tivoli Storage Manager Server 5-5-3

Fri, 28 Aug 2009 14:19:14 +0000

I spent yesterday afternoon upgrading our TS7530, and in my fad I also upgraded TSM to 5.5.3. Now, once I started TSM it quickly started complaining about the paths to the drives.

1
2
3
4
5
6
7
8
9


ANR8873E The path from source TSM1 to destination VTL1_DR03
(/dev/lin_tape/IBMtape03) is taken offline.
ANR8873E The path from source TSM1 to destination VTL1_DR03
(/dev/lin_tape/IBMtape03) is taken offline.
HBA_LoadLibrary: previously unfreed libraries exist, call HBA_FreeLibrary().
ANR8873E The path from source TSM1 to destination VTL1_DR07
(/dev/lin_tape/IBMtape07) is taken offline.
ANR8873E The path from source TSM1 to destination VTL1_DR07
(/dev/lin_tape/IBMtape-07) is taken offline.

I thought maybe this is a mere device problem (we have had them before), so I rebooted the boxes. But still no luck and I went home after about an hour of trying without any luck. In the morning, my co-worker called our trustworthy IBM service partner, and the TSM consultant said he had the exact, same problem yesterday. We would have two options:

IBM RSA II adapter and Java RE (fini)

Wed, 19 Aug 2009 04:32:44 +0000

If you remember back to July, I looked into some troubles I had with the IBM RSA II adapter’s Java interface and the latest JRE updates. I just noticed, that IBM released a new firmware yesterday for the RSA. The ChangeLog states this:

Version 1.13, GFEP35A Problem(s) Fixed:

* Suggested o Fix for Remote Control General Exception in JRE 1.6 update 12 and above. o Corrected a problem that DHCP renew/release may fail after a long time. o Corrected a problem that remote control preference link disapears after creating new key buttons. o Corrected a problem that cause event number shows only from 0 to 255 when views RSA log via telnet session.

IBM TS7530 and the Virtualization Engine for Tape Console

Fri, 31 Jul 2009 12:44:16 +0000

I just had yet another support call with IBM, concerning the Tape Console (or VE console, courtesy of Falconstor). My basic problem was/is, that I, as a german person, do have a german Windows Server 2003 installation. Now, if you do have german decimal number format selected in the regional settings, the display is gonna be kinda impaired and you’re gonna see something like this:

VE console with german decimal number format

OCF agent for Tivoli Storage Manager: redux

Fri, 05 Jun 2009 09:35:50 +0000

Well, after I finished my first OCF agent back in October 2008, we have it running in production now for about ten months. During that time, we found quite a few points in which we’d like to improve the behaviour with that Linux-HA should handle TSM.

Shutdown TSM nicely if possible (Cancel client sessions, cancel running processes and dismount mounted volumes)
Better error handling

So, after another week of writing and testing with a small instance, I present the new OCF agent for Tivoli Storage Manager. It still has one or two weak points, but they are negligible. I still need to write the documentation for it, but the script should just work …

Weird TS3500 problem: redux

Wed, 03 Jun 2009 12:49:35 +0000

Well, after yesterday’s episode with our tape library today continued to be a taxing day. After restarting a few exports that were hanging yesterday due to our library problems, something similar returned. TSM was unable to locate a few (two to be exact) tapes in the library.

1
2
3
4
5
6
7
8
9


ANR8300E I/O error on library LIB3584 (OP=00006C03, CC=314, KEY=05, ASC=3B,
ASCQ=0E, SENSE=70.00.05.00.00.00.00.0A.00.00.00.00.3B.0E.00.C0.00-.04.,
Description=The source slot or drive was empty in an attempt to move a
volume). Refer to Appendix C in the 'Messages' manual for recommended action.
ANR8312E Volume 000400 could not be located in library LIB3584.
ANR8358E Audit operation is required for library LIB3584.
ANR8381E LTO volume 000400 could not be mounted in drive DR6 (/dev/rmt0).
ANR1402W Mount request denied for volume 000400 - volume unavailable.
ANR1410W Access mode for volume 000400 now set to "unavailable".

Yet the library reported the tapes were still inventoried. 🤷 Here we are again, looking completely baffled. After a short while trying to figure out what to do, we went through the Data Cartridge inventory again. As it turns out, through putting the library in “Pause”-Mode and restarting TSM multiple times, TSM apparently completely forgot that it had these tapes put into drives.

Novell KMP: vmware-tools-kmp and ibm-lin_tape-kmp

Sun, 10 May 2009 11:27:22 +0000

Disclaimer: I don’t take any responsibility for faults within the software, I just provide the RPM’s! Feel free to ask me about stuff concerning these RPM’s, but I ain’t accountable if your stuff goes kaboom … Oh, and those RPM’s aren’t recommended or supported by Novell or IBM!

After working with the novell-kmp solution, I think it’s actually rather easy to create a " Kernel Module Package". In the end, I created two additional KMP’s, one for the tools component of the VMware-Tools shipped with VMware ESX, and another for the lin_tape SCSI driver, used by our IBM TS3400 as well as the IBM TS7530.

Monitoring the IBM BladeCenter chassis with Nagios

Tue, 10 Feb 2009 11:40:00 +0000

Today I ended up working out the details on what we want to monitor regarding our BladeCenter. The most interesting details (for us that is) are these:

Fan speeds for Chassis Cooling/Power Module Cooling Bay(s)
Temperature
Power Domain utilization

It wasn’t * that* hard to implement. Only trouble(s) I ran into, were ( 1) IBM did a real shitty job with the MIB’s. If you look closely into the mmblade.mib, you’re gonna notice, that not a single OID is specified for the events. ( 2) As the MIB’s weren’t documented anywhere, I had to look them up via snmpwalk (which I had never used before). So as a reminder (to myself), here’s how it is done:

`1`	`snmpwalk -v1 -c public -O n 10.0.0.35 .1.3.6.1.4.1.2.3.51.2.2`

This will get you a list, with a lot of output (5154 lines to be exact). Lucky me, the web interface of the management module/ssh interface is rather verbose, so all you need to do is compare those values with what you are looking for.

So for myself (and anyone interested) read ahead for the list of checks we are currently running on the management module.

Setting up the BladeCenter H

Fri, 30 Jan 2009 10:50:40 +0000

Well, we finally had our maintenance window today, in which we planned the hardware exchange for our current Dell Blade Chassis (don’t ask!). The exchange went fine, but as we started exploring the components (like the IBM BladeCenter SAN switches – which are in fact Cisco MDS 9100) we hit a few road blocks.

First, the default user name/password combo for the Cisco MDS 9100 for the BladeCenter is USERID/PASSW0RD (just as the rest of the password combinations).

Updating path information for TSM

Wed, 28 Jan 2009 15:11:31 +0000

As I did some switching today (between the new lin_tape version by IBM and our own lin_tape version), I ended up writing those lines a dozen times. Here is (just for me, if you don’t care .. skip ahead) on how to generate a list of commands:

1
2
3
4
5
6


for i in $( seq -w 1 32 ); do
 token="${i/0/}"
 path_prefix="/dev/lt/IBMtape12245775"
 echo "DELETE PATH TSM1 VTL1_DR$i SRCTYPE=SERVER DESTTYPE=DRIVE LIBRARY=VTL1"
 echo "DEFINE PATH TSM1 VTL1_DR$i SRCTYPE=SERVER DESTTYPE=DRIVE LIBRARY=VTL1 DEVICE=$path_prefix$((token+11))A1"
done

which should get you a list like this:

Sidenote: Amount of Slots per Virtual Tape Library

Tue, 27 Jan 2009 20:32:57 +0000

Well, I just stumbled about this again (and I don’t know right now whether or not this is documented inside a RedBook or not) today, so I thought maybe I’m gonna write it down.

Slot-Amount Property of a Virtual Tape Library

Please keep in mind, when creating the virtual library to think hard about the amount of slots you might need. It ain’t that bad, you just can’t decrease the amount anymore. So if you think about creating 50 different virtual tape libraries with 500 slots each on your TS7530, think again. The current software level only supports 25.000 slots on a global level.

Working with IBM's Virtualization Engine Console

Tue, 27 Jan 2009 17:27:31 +0000

Recently, we got the recommendation from our system partner to use static allocated tape cartridges instead of dynamic allocated ones. Apparently using dynamic allocating cartridges comes with a performance penalty if more than a few nodes are backing up a large amount of data at once.

And yet again, I noticed that the IBM Virtualization Engine Console (aka Falconstor Software) is really error prone.

In order to change the allocation type, we had to shred the old cartridges first (500 x ~100M up till now), chance the allocation type at the virtual tape library level, and then recreate the 500 cartridges with a fixed size (500x 102400MB). Now, as I was kinda optimistic, I decided to create all 500 cartridges at once.

Distribution running on IBM TS7530 Virtualization Engine

Fri, 09 Jan 2009 14:18:35 +0000

Well, I was just a bit curious earlier what distribution might be running on our IBM TS7530 Virtualization engines .. well, I just had a look-see ..

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27


vetapeservice@VTL-B:~> cat /etc/SuSE-release
SUSE Linux Enterprise Server 10 (x86_64)
VERSION = 10
PATCHLEVEL=1

vetapeservice@VTL-B:~> uname -a
Linux VTL 2.6.16.46-229-smp #1 Sun Apr 13 05:21:49 UTC 2008 x86_64 GNU/Linux

vetapeservice@VTL-B:~> free -m
 total used free shared buffers cached
Mem: 4022 1378 2643 0 342 552
-/+ buffers/cache: 484 3537
Swap: 3815 0 3815

vetapeservice@VTL-B:~> cat /proc/cpuinfo
processor : 1
vendor_id : AuthenticAMD
cpu family : 15
model : 65
model name : Dual-Core AMD Opteron(tm) Processor 8218
stepping : 3
cpu MHz : 2600.186
cache size : 1024 KB
physical id : 0
siblings : 2
core id : 1
cpu cores : 2

Main difference to a " normal" SUSE Linux Enterprise Server 10 installation (there’s about zip normal with that kind of installation, thus the quotes) thus far are:

the build for the VE uses busybox as init
IBM stripped man/info
they are running Xorg/Fluxbox on it

Just don’t ask me why there’s a DE ( d esktop e nvironment) running, it ain’t even hooked up to a monitor. Only reason would be for the RSA’ remote monitor stuff … lala

SLES, ZendOptimizer and IBM PowerPC(4)+

Wed, 11 Jul 2007 08:40:57 +0000

What would you figure from the above ? Hopefully the rather obvious, that it’s a really shitty combination.

So we figured it would be a nice thing to test our new setup before going into pre-production testing or production, but we don’t have an extra spare box. So we took one of the power4 boxes we have mounted in the rack basically consuming energy all day (that’s about 38kWh a day) and installed SLES10 onto it. Which wasn’t all that bad (at first the box repeatedly started back to AIX, from CD and after convincing the SMS - that’s basically the bios on the power*-boxes also known as System Management Services with a hammer to boot from the first hard disk).

SLES10 on pSeries

Wed, 04 Jul 2007 15:58:39 +0000

Okay, yet another day passed by blazing fast. I had a good day at work, spent nearly the whole day trying to get my bloody systems hooked up to our SAN (which was interrupted by a non-working SAN-switch, disappearing WWN’s, lunch and my trainees), messing around with our internal network, hacking our Blade Chassis switches to get me what I want and some random paperwork.

But first things first .. We installed SLES10 on a pSeries box the other day (I think on Monday), and now I’m trying to get the WWN of it’s Emulex HBA, out of either sysfs or procfs. But whatcha’ thinking ?

AIX 5-3 Linux Toolkit

Fri, 20 Apr 2007 13:07:04 +0000

OK, so I skipped rebuilding a newer RPM version (for now) and I’m currently rebuilding anything that fit’s into app-dev according to IBM …

The list reads like this:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19


$ ls -al ~/app-dev
autoconf-2.59-1.aix5.1.noarch.rpm
automake-1.8.5-1.aix5.1.noarch.rpm
binutils-2.14-3.aix5.1.ppc.rpm
bison-1.875-3.aix5.1.ppc.rpm
coreutils-5.2.1-2.aix5.1.ppc.rpm
cpio-2.5-1.aix5.1.ppc.rpm
diffutils-2.8.1-1.aix4.3.ppc.rpm
flex-2.5.4a-6.aix4.3.ppc.rpm
gawk-3.1.3-1.aix5.1.ppc.rpm
gcc-4.0.0-1.aix5.1.ppc.rpm
gcc-cplusplus-4.0.0-1.aix5.1.ppc.rpm
gdb-6.0-1.aix5.1.ppc.rpm
libgcc-4.0.0-1.aix5.1.ppc.rpm
libstdcplusplus-4.0.0-1.aix5.1.ppc.rpm
libstdcplusplus-devel-4.0.0-1.aix5.1.ppc.rpm
libtool-1.5.8-1.aix5.1.ppc.rpm
m4-1.4.1-1.aix5.1.ppc.rpm
make-3.80-1.aix5.1.ppc.rpm

OK, I’m not exactly rebuilding these old versions, I’m actually using their old specs to compile newer versions of these. I’m currently at coreutils-6.7, which really takes ages. But will see about the rest.

Waiting

Wed, 28 Feb 2007 14:58:05 +0000

We are still waiting for the money promised by the state and the country for our HBFG (again, it’s “Hochschulbauförderungsgesetz”), that hopefully is reducing or eliminating our storage/SAN problem we have currently. Right now we have to Cisco MDS9216 (that’s a 16-port 2GBps SAN-switch, two for redundancy), which means we only have 16 SAN-ports. That isn’t much, but still is to less, as we have like 30 machines or so, that really need access to the SAN, so we either end up unplugging some of them from the SAN or merge them onto some big machines (like our x366).

AIX-5-3 undamp; rpm-4-4-7

Mon, 05 Feb 2007 14:24:50 +0000

OK, so I tried to install the AIX Toolkit today, to build some newer rpm’s (yaaaaah, I hate RPMS myself, still it’s way better than distributing plain tar.gz archives) but looks like either AIX or rpm-4.4.7 doesn’t like me.

Now I’ve to figure out how to get libm (that’s /lib/libm.so) installed on AIX. Will see about that later and/or tomorrow.

IBM

Mon, 14 Aug 2006 10:16:28 +0000

We just received the long awaited shipment of sixteen 300GB FC-HDD’s (2Gbps with 10000rpm) for our SAN (a pretty old DS4500/FaStT 900).

But there’s still the software option missing we ordered within the same breath. So I called our trustworthy IBM distributor (hah!) and asked the guy responsible for sales, what the ETA on this software option is (if someone is interested its VolumeCopy/FlashCopy).

He told me, that we’ll receive a letter with the license key about 4 weeks after commission !!!!!!

Mood sucks

Thu, 03 Aug 2006 06:50:04 +0000

christel, you remember the mood-swings we were talking about ?

I think I’m undergoing just another 😞 I’m currently pretty much pissed. Basically everything is pestering me currently (except #gentoo-dev and Gentoo work).

Work just ripped another piece of me (hah, thanks VMware & BigBlue). I started the day with a ice cold shower (if I’m talking about ice cold it was ice cold), they’re currently replacing our old gas heating and unfortunately that means no warm water at all! arg

Live sucks (again)

Sun, 21 May 2006 10:46:25 +0000

Now is again such a time in live, where you have the motivation or wish to just fade away.

I’m just listening to Fort Minor - Where’d you go and thinking about the stuff Mike Shinoda is singing …

I want you to know it’s a little fucked up, That I’m stuck here waitin’, no longer debatin’, Tired of sittin’ and hatin’ and makin’ these excuses, For while you’re not around, and feeling so useless, It seems one thing has been true all along, You don’t really know what you got ’til it’s gone.

Work sometimes sucks

Fri, 10 Mar 2006 16:22:46 +0000

Today they finally let me fiddle around on an AIX 5.3 system. Well AIX ain’t bad, but misses (by default installation) some features and comfort.

The first thing I noticed, they only! install telnet and don’t even give an option to install sshd .. That sucks, especially if you’re supposed to log in via the internet (yay! an telnet open to the internet 😯)

So I googled a bit around and found out, that we should have some Bonus CD’s (from IBM of course) featuring openssl/openssh, looked into the cubicle behind the rack and look what I’ve found … Go, go … 😛