Sles10 on BAFM

Adapter teaming on SLES10

Sat, 16 Aug 2014 10:18:13 +0000

Since one of the requirements for my current project is having NIC redundancy, I didn’t get around looking at the available " adapter teaming" (or adapter bonding) solutions available for Linux/SLES.

First I tried to dig into the Broadcom solution (since the Blade I first implemented the stuff uses a Broadcom NetXtreme II card) , but found out pretty soon that the basp configuration tool, which is only available on the Broadcom driver CD’s shipped with the Blade itself, pretty much doesn’t work.

Bloody cluster solutions (continued)

Sat, 16 Aug 2014 10:10:22 +0000

So, as the previous try on getting the teamix people to fix the bloody LoadBalancer (as in sending at least an identification string for the SSH check) didn’t work so well (they told me, I should configure MASQ uerading/ ROUTE ing on the PacketPro(which is kinda icky), I went on today and looked at what SLES10 installs as default logger.

Surprisingly they install a rather new syslog-ng (well, syslog-ng-1.6.8 is what they ship) so it was rather easy to workaround the situation.

OCFS2 fun yet again

Sat, 16 Aug 2014 10:05:13 +0000

I’m coming back today from a six day vacation in the warm south (that is Stuttgart), back at work and find three sheets of paper on my desk. Two tell me something I haven’t done yet, the other one tells me something I haven’t seen yet.

One of my colleagues had to restart one of our web nodes and now the thing can’t mount the logging volume (and thus, logrotate / awstats failed to do it’s job). OCFS2 ain’t spitting any error messages, when trying to mount the volume you see it joining the domain the volume belongs to on the other nodes, so from a first glance at things .. nothing is wrong ?

Building RPMs on SLES10SP2-ppc64

Sat, 16 Aug 2014 09:50:36 +0000

Well, it turns out that building stuff on ppc64 is a real pain in the ass, at least on anything SUSE related. I do have to tweak every damn spec to include this:

1
2
3


%ifarch ppc64
export LDFLAGS="$LDFLAGS -m64"
%endif

Otherwise, ld is gonna fail when linking, as it’s gonna try linking the generated 64bit code ( -m64 is passed on via RPM_OPT_FLAGS to CFLAGS) as 32bit code, which ain’t gonna work at all …

Nagios Hostgroup Inheritance

Sat, 16 Aug 2014 09:48:08 +0000

As I wrote earlier, I recently virtualized our nagios. Along with that came a complete " redesign" of how checks are applied. Up till now, I defined checks for each and every single server, thus ending up with ~25 files, each holding roughly 6 checks which are in the same file just sorted by hostname.

As you can imagine, it gets quite confusing with that amount of checks (~150). So the last two days I spent on reorganizing (with Visio), on which object/hostgroup placing a check would make sense. Now, this is my first result of two days planning, reorganizing, reordering and moving hosts into different hostgroups.

SUSE Linux Enterprise Server 10 on VMware ESX

Sat, 16 Aug 2014 09:46:16 +0000

We’re currently having a really weird problem with our VM’s. Sometime last week, SUSE released a kernel update. Now, once you install it and you reboot the selected VM with a DVD/CD image present, you’re gonna see this:

msg.vmxaiomgr.retrycontabort.unkown

The only workaround so far has been to unmount any cleanse any CD-Drives attached to the VM. And yes, this is reproduceable, even reinstalling from scratch doesn’t change the fact, that after installing the patch the VM quits working.

SUSE Linux Enterprise Server 10 on VMware ESX (continued)

Sat, 16 Aug 2014 09:45:36 +0000

Well, after some searching today (we applied the VMware Update 2 today, thus the VMware Tools update too), I finally found out what is causing that problem.

Though the problem seems to be not limited to virtual systems alone, I just browsed through this Novell Forum thread which pretty much describes my problem. I found the same error in the VM’s I tried to mount a CD image.

patch2mail for SLES10

Fri, 08 Aug 2014 09:51:16 +0000

Well, there is this “nifty” tool called patch2mail, which basically converts the XML for the updates to a more readable format. But you’re screwed if you want to do the same on SLES10. Since it ain’t shipping with the zypper xml wrapper thing, you need to do it a bit different.

So I ended up writing a small (and yet, ugly) shell script to generate me a mail of my liking ..

Linux-HA and Tivoli Storage Manager

Fri, 08 Aug 2014 09:44:38 +0000

Well, since we received part of our shipment on Wednesday, I finally looked at how we’re gonna deploy our active/active Tivoli Storage Manager configuration. Right now, we do have a single pSeries box hosting ~100 client nodes which we’re looking to split by two (since we do have two x366 for that purpose now).

Now, as there ain’t no solution for this scenario yet (neither from International Business Machines nor someone out of the open source community), I sat down and started writing an OCF Resource agent for dsmserv (that is the Tivoli Storage Manager server).

Building opsview for SUSE Linux Enterprise 10

Fri, 08 Aug 2014 09:10:06 +0000

Disclaimer: I don’t take any responsibility for faults within the software, I just provide the RPM’s! Feel free to ask me about stuff concerning these RPM’s, but I ain’t accountable if your stuff goes kaboom …

Well, I just looked at opsview again (haha, thanks Alex 😛). Only trouble is, the people over at opsview don’t distribute RPM’s for that … After registering for their site, to download the SRPM’s (or to download anything), I got the RPM’s and started looking at them.

IBM RDAC: Installing the driver for a (not yet) running version

Fri, 08 Aug 2014 09:04:36 +0000

Well, kernel updates on our Linux servers running IBM’s RDAC driver (developed by LSI) is a real pest .. especially if you have to reboot the box two times in order to install the drivers/initrd correctly.

So I sat down and looked at the Makefile. Turns out, it just needs four tweaks in order to be working with a different kernel version (which you have to pass using environment variables to make).

Novell KMP: KMP'ing IBM's RDAC driver

Fri, 08 Aug 2014 09:03:51 +0000

Well, after yesterday’s lesson about getting the IBM RDAC to install for a not-yet-running kernel, I decided to take it a step further. Novell does have some documentation about KMP’s, which is actually rather good, especially the guide written by Andreas Grünbacher.

After a short tinkering, I got it actually working. I was kinda surprised, at how easily it actually is. One problem I still have to deal with, is modifying the %post, to generate the mpp-initrd image. For now, the KMP only contains the default %post, which updates the modules.* stuff.

Novell KMP: Useable version of ibm-rdac-ds4000

Fri, 08 Aug 2014 09:03:03 +0000

After some more tinkering, a lot more looking at the macros in /usr/lib/rpm/rpm-suse-kernel-module-subpackage and /usr/lib/rpm/suse_macros, I think I finally have a usable RPM’ified version of IBM’s Multipathing driver ready for use.

There is still one major annoyance left: each time you install a new ibm-rdac-ds4000-kmp RPM, you also need to reinstall the corresponding ibm-rdac-ds4000-initrd package, as the macros in /usr/lib/rpm don’t allow for custom %post or %postun.

As mentioned before, I’m gonna send them to LSI/IBM for review, and maybe, MAYBE they are actually gonna make use of that.

Linux-HA and Tivoli Storage Manager (Finito!)

Fri, 08 Aug 2014 08:59:07 +0000

As I previously said, I was writing my own OCF resource agent for IBM’s Tivoli Storage Manager Server. And I just finished it yesterday evening (it took me about two hours to write this post).

Only took me about four work days (that is roughly four hours each, which weren’t recorded in that subversion repository) plus most of this week at home (which is 10 hours a day) and about one hundred subversion revisions. The good part about it is, that it actually just works :-D (I was amazed on how good actually). Now you’re gonna say, “but Christian, why didn’t you use the included Init-Script and just fix it up, so it is actually compilant to the LSB Standard ?”

The answer is rather simple: Yeah I could have done that, but you also know that wouldn’t have been fun. Life is all about learning, and learn something I did (even if I hit the head against the wall from time to time ;-) during those few days) … There’s still one or two things I might want to add/change in the future (that is maybe next week), like

adding support for monitor depth by querying the dsmserv instance via dsmadmc (if you read through the resource agent, I already use it for the shutdown/pre-shutdown stuff)
I still have to properly test it (like Alan Robertson mentioned in his one hour thirty talk on Linux-HA 2.0 and on his slides, Page 100-102) in a pre-production environment
I’m probably configure the IBM RSA to act as a stonith device ( s hoot t he o ther n ode i n t he h ead) - just for the case one of them ever gets stuck in a case, where the box is still up, but doesn’t react to any requests anymore

Loooong time

Fri, 08 Aug 2014 08:42:36 +0000

It’s been very quiet around here, I’ve been rather busy with my real life. During that busy time, a lot of things happened. I switched jobs starting on October 1st, I’m now working in Karlsruhe (as compared to the 870km northern Greifswald). It may sound far, but it’s actually quite pleasant. You know, I was born down here (well not exactly here – 70 kilometers afar) and I still had the feeling that this is my home.

AutoYAST and custom swap partitioning

Thu, 03 Jun 2010 18:34:24 +0000

Well, we’ve been discussing our swap partitioning the last few days at work, and I finally got around to implementing it. Again, it proved to be kinda hard, basically because AutoYAST decides to do things differently.

SLES10 not installing boot loader in MBR

Thu, 03 Jun 2010 18:15:23 +0000

Well, as I mentioned in my earlier post, I had some trouble during the week. I was having issues with SLES10 installations not finishing during the bootloader installation phase. After trying out different flavors (as in 10SP2 x64/x86, …), and not having any luck with this, I went searching on Google as a last effort try. Guess what, yet again Google helped me out!

It was pretty simple. Putting /dev/cciss/c0d0 into /boot/grub/device.map as (hd0) made the grub-installer finish. Now, figuring out how to transfer those information during the installation proved difficult. I was just about to give up, while reading through the AutoYAST documentation, when it struck me. There is even an extra chapter for this stuff, so simply putting the following into my profile solved my issues:

Configuring nagios-plugins-zypper

Thu, 12 Nov 2009 17:45:43 +0000

Since I’m running check_zypper via nrpe (which in turn runs as nobody), I need to set up sudo. In order for the plugin to work, we need to add the following line to /etc/sudoers (by means of visudo):

`1`	`nobody ALL = NOPASSWD: /usr/bin/zypper sl, /usr/bin/zypper --non-interactive --no-gpg-checks --terse list-updates`

(Keep in mind this needs to be a single line …)

Creating a custom RPM repository for SMT

Thu, 22 Oct 2009 17:59:45 +0000

I spent some time yesterday figuring out ways on how to assign custom (as in self-built) RPMs to a installation using SMT. First you obviously need a " external" repository, that can be integrated into the SMT.

So we need to create repository someplace, where the SMT can go and grab it. I ain’t gonna cover the sharing part, since that is your job! I’m just gonna cover the steps on how to create the custom repository and how to integrate it into the SMT.

New vmware-tools-kmp

Wed, 10 Jun 2009 11:31:04 +0000

Since we recently upgraded our VMware Infrastructure to VMware vSphere, I finally had a chance to refresh the RPM’s for the KMP for 2.6.16.60-0.39.3-0.1 and 2.6.27.21-0.1. You can find the source RPM here.

Novell KMP: vmware-tools-kmp and ibm-lin_tape-kmp

Sun, 10 May 2009 11:27:22 +0000

After working with the novell-kmp solution, I think it’s actually rather easy to create a " Kernel Module Package". In the end, I created two additional KMP’s, one for the tools component of the VMware-Tools shipped with VMware ESX, and another for the lin_tape SCSI driver, used by our IBM TS3400 as well as the IBM TS7530.

SLES10: zypper-log

Fri, 03 Apr 2009 04:34:49 +0000

Well, I just stumbled upon something .. My Nagios at work wasn’t working anymore, and I went looking.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11


nagios3 ~ [0] > tail -f /var/log/nagios/nagios.log
[1238658394] Error: Unable to save status file: No space left on device
[1238658403] Error: Unable to save status file: No space left on device
[1238658413] Error: Unable to save status file: No space left on device
[1238658423] SERVICE ALERT: tsm1;POWER WARN;OK;SOFT;4;-u OK - 0
[1238658423] Error: Unable to save status file: No space left on device
[1238658433] SERVICE ALERT: tsm2;LOAD;WARNING;SOFT;1;WARNING - load average: 6.25, 5.72, 5.36
[1238658433] Error: Unable to save status file: No space left on device
[1238658443] Error: Unable to save status file: No space left on device
[1238658453] Error: Unable to save status file: No space left on device
[1238658463] Error: Unable

After that, zip - nada. Next thing, check whether or not the device is really full … Okay, df ..

TSM client: Backing up files with umlauts on SLES

Mon, 02 Mar 2009 13:21:10 +0000

In the past, I always had problems with SLES and our Tivoli Storage Manager client’s when backing up files with german umlauts. Well, today I looked a bit harder, and quite quickly found a solution.

1
2


sles9 root [0] > env | grep ^LC
LC_CTYPE=de_DE.UTF-8

As you can see from the above, SLES9/10 ain’t setting LANG or LC_ALL (which I searched for first), but is setting LC_CTYPE.

So, simply changing the LC_CTYPE in the init-script and/or prepending the dsmc command line with a new LC_CTYPE fixes my umlauts problems!

Installing SLES10 via network with no DHCP available

Fri, 30 Jan 2009 11:24:08 +0000

In our current fight against the BladeCenter switches, we’re currently facing the problem that the blades ain’t able to send/receive DHCP-traffic.

So in order to move forward, we had to use static IP addresses. And since SLES10 ain’t straight forward on that, I had to look it up. Now, here’s for me (and everyone else tired of searching) how to do it:

1
2
3
4
5
6
7
8
9


install=http://ftp.barfoo.org/install/SLES10-x64/CD1
 netdevice=eth0
 hostip=10.0.1.240
 netmask=255.255.255.0
 gateway=10.0.1.1
 nameserver=10.0.1.2
 domain=home.barfoo.org
 insmod=bnx2
 splash=verbose

Opsview installation reviewed

Sat, 24 Jan 2009 11:33:50 +0000

Well, I recently (well, yesterday) built the opsview RPM’s for SLES10, and started fiddeling about with it today. Alex " recommended" I should rather look at Opsview instead of Centreon, but boy was there a surprise waiting for me …

Opsview has the advantage that it at least lets you use the package manager. But, it also needs a lot of handy work (just like Centreon, which I really dislike since it’s real error prone).

Automatic updates on SUSE Linux Enterprise 10

Fri, 23 Jan 2009 11:30:01 +0000

I had the problem, that the automatic update function of YaST doesn’t work like I want it to. I just wanted it to install only those updates, that ain’t interactive, don’t need a service restart and don’t need a reboot.

YaST does only feature an online update that skips “interactive” updates (I’ve never even encountered an interactive update up till now). So I went ahead and wrote a (hackish) script, that achieves what I need.

Trusted vendors in SUSE Linux Enteprise 10

Mon, 19 Jan 2009 20:31:19 +0000

The other day I had a closer look at the zypper logs (well, I was digging for a time-history of installed packages). First … damn does zypper produce a lot of logs on a " productive" (or rather on a maintained - as in up-to-date) system.

But glazing over the logs, I found out something new about zypper. It actually has an internal list, which only purpose is to identify a trusted vendor …

Automating zypper updates

Sat, 17 Jan 2009 18:16:27 +0000

Well, I just looked into using ` zypper up ` to update some of our boxen (I do have a script, which holds the boxen it needs to process in a variable and simply goes through them one by one) – yes, I could activate auto-update, I just don’t want that at this point 😉

So at first I tried just using zypper to automatically update that given list, but even if you pass –no-confirm, zypper would still ask for your confirmation (which seems kinda stupid). After a short while thinking about it, a lesson from solar@gentoo.org came to mind. When working in a chroot, he simply used this:

Distribution running on IBM TS7530 Virtualization Engine

Fri, 09 Jan 2009 14:18:35 +0000

Well, I was just a bit curious earlier what distribution might be running on our IBM TS7530 Virtualization engines .. well, I just had a look-see ..

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27


vetapeservice@VTL-B:~> cat /etc/SuSE-release
SUSE Linux Enterprise Server 10 (x86_64)
VERSION = 10
PATCHLEVEL=1

vetapeservice@VTL-B:~> uname -a
Linux VTL 2.6.16.46-229-smp #1 Sun Apr 13 05:21:49 UTC 2008 x86_64 GNU/Linux

vetapeservice@VTL-B:~> free -m
 total used free shared buffers cached
Mem: 4022 1378 2643 0 342 552
-/+ buffers/cache: 484 3537
Swap: 3815 0 3815

vetapeservice@VTL-B:~> cat /proc/cpuinfo
processor : 1
vendor_id : AuthenticAMD
cpu family : 15
model : 65
model name : Dual-Core AMD Opteron(tm) Processor 8218
stepping : 3
cpu MHz : 2600.186
cache size : 1024 KB
physical id : 0
siblings : 2
core id : 1
cpu cores : 2

Main difference to a " normal" SUSE Linux Enterprise Server 10 installation (there’s about zip normal with that kind of installation, thus the quotes) thus far are:

the build for the VE uses busybox as init
IBM stripped man/info
they are running Xorg/Fluxbox on it

Just don’t ask me why there’s a DE ( d esktop e nvironment) running, it ain’t even hooked up to a monitor. Only reason would be for the RSA’ remote monitor stuff … lala

Adapter bonding on Linux

Thu, 16 Oct 2008 16:47:23 +0000

Well, today I had a rather weird error. I was testing the adapter bonding on one of the boxen designated as Tivoli Storage Manager Server, when I noticed that the bonding wasn’t working as expected when simulating an error (that is unplugging one of the TP cables for the bond).

Now, the bond had “mode=6 miimon=100” as options. After running “linux bond debug” through Google (which turned up nothing useful, besides one document on the Oracle Wiki about IOS/Linux adapter teaming), I figured “Hey, just lets test switching the arguments.” And guess what ?

zypper-update-report (was: patch2mail for SLES10)

Tue, 09 Sep 2008 23:11:01 +0000

Well, after some more refining I think I finally have a script I ain’t never gonna touch again (unless something breaks, which can happen quick as we all know).

The script now uses a sysconfig file for the common settings (like sender, receipents, categories to scan for), so it may be deployed en mass.

/etc/sysconfig/zypper-update-report

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20


## Type: string
## Default: root
## Config: ""
#
# Sender address for the update report
FROM="Yourupdatemonkey "

## Type: string
## Default: root
## Config: ""
#
# Receiver address for the update report
#RECEIPENTS="tehsysadmin@barfoo.org"

## Type: string
## Default: "securty recommended optional"
## Config: ""
#
# List of groups, to include in the report
CLASSES="security recommended optional"

/usr/local/sbin/zypper-update-report

Debugging rug

Fri, 15 Aug 2008 19:19:06 +0000

Well, it’s 7pm. I’m sitting at home and thinking about why in gods name rug isn’t adding my update repository. I can add the service using yast inst_source, but when yast then syncs with ZenWorks, it tells me something like:

Failed to get repomd/repodata.xml; Reason: 530 - Access denied

So my fellow co-worker turned on the debug-logging and we quickly found out why: rug isn’t using the command line credentials I was passing.

Yet another VMware error

Wed, 30 Jul 2008 18:00:49 +0000

Today I was moving a pretty standard SLES10 virtual machine to another host, when the migration dialog showed me this:

fault.MemorySizeNotRecommended

And if you now think, the virtual machine is something special take a look at those settings:

Virtual machine configuration

I don’t know what to think about that error message. Googling for it doesn’t reveal that much about it. If anyone out there got an idea, I’m open for suggestions.

Fixing vmkernel symlinks

Wed, 30 Jul 2008 07:19:32 +0000

Since I do happen to be in the situation pretty often where the kernel inside a VM is newer than what VMware currently has in their tools (as in the SUSE kernel is newer than the binary modules built by VMware), here’s a quick reminder for myself on how to to fix the .ko symlinks.

1
2
3
4
5


for file in /lib/modules/$( uname -r )/misc/*.ko ;
do
 rm $file &&
 ln -s /lib/modules/$( uname -r )/misc/${file/.ko/.o} $file
done

Nagios Hostgroup Inheritance (continued)

Sat, 19 Jul 2008 09:41:21 +0000

Well, it turns out that my thought was ultimativly flawed. When defining the hostgroup_members in the lower tiers, nagios is association the checks from the lower tier with the upper tiers. Thus propagandating all checks upwards, and me ending up with ~250 checks instead of ~150.

Gonna have to try to define the dependency backwards, maybe that’ll help. But that’s a topic for Monday. Guess I’ll finish viewing Ghost in the Shell - Stand Alone Complex first.

Nagios3 with Active Directory authorization on SLES10

Mon, 14 Jul 2008 17:32:04 +0000

Well, it seems to be getting a “trend” for me, to integrate stuff into our Active Directory. Now that I know why, and how easy that is, I expect to add more stuff. The good thing about the integration is, that you only need to maintain a single source for authorization.

The bad thing about that is, that stuff becomes dependent on the Active Directory (we do have four domain controllers, so that should be fine).

Managing unixODBC connections on SLES10

Thu, 03 Jul 2008 06:42:27 +0000

Recently I got the task, to implement unixODBC/freetds on one (well, it’s really three) of our web servers, as someone wanted to use Microsoft SQL Server 2005 with PHP (without using the MSSQL functions, which PHP provides soo nicely; don’t ask me why).

With that I also got a set of " instructions" on how to install freetds from source (remember, I was a Gentoo dev, so I know my way around, when it comes to building from source), as well as a small set of instructions on how to create the connection.

subversion on WebDAV with Active Directory authorization on SLES10

Sun, 29 Jun 2008 17:49:29 +0000

Okay, so I ended up toying with subversion via WebDAV on SLES today (I know, I know .. it’s bloody Sunday). It wasn’t much of a hassle though, after reading this. Sure, I made a few errors at first (simply confused the logic behind " Location" and " Directory"), but after that plain subversion commits via WebDAV (thus utilizing Apache) worked fine.

For POC or as a hint to myself, here’s where and what I needed to add/change:

The clue to build ppc64 RPM's

Thu, 26 Jun 2008 15:52:22 +0000

Remember, I talked about building RPM’s on SLES10SP2 on ppc64 ? Well, turns out I was rather stupid .. and it was rather simple (don’t ask me why I didn’t think of that). I tried asking solar, I used Google (apparently with the wrong search parameters), nothing though. Not a clue.

Today it bugged me again, so I used Google again. This time with " ppc64 suse rpmbuild", and guess what I saw within the preview of the second hit ..

IBM (Tivoli) Integrated Solutions Console

Fri, 23 May 2008 17:28:03 +0000

Here I am, preparing our environment for the first (of hopefully many) tester for our upcoming VTL project. So I ended up installing the ISC and Administration Center for Tivoli Storage Manager on a 64bit guest (that is SLES10 for AMD64), just because I forgot to include support for later versions with our current running one. Guess what, na- na na na na. Exactly, didn’t work, the same errors I got while trying it before in a virtual environment. “Portlet is not available.”

Creating multi-distribution RPM/XML repositories

Wed, 02 Apr 2008 14:36:05 +0000

Well, as we do have quite a few custom built RPM’s, I was searching for a new solution to manage the repo(s). Currently I do have a single repository per distribution.

One thing one needs to know about createrepo (from createrepo), it doesn’t support this type of thing in the first place. So I had to come up with another way of doing it. First, I created a proper layout (much like the Debian Official Repository layout):

OCFS2 follow-up

Fri, 07 Mar 2008 07:29:26 +0000

OK, it turned out that said colleague wasn’t responsible at all. Turns out, the real trigger was me creating a new volume on our SAN, on the same array that houses the OCFS2 volume.

Apparently, during creation of an additional SAN volume, all other SAN volumes in this array are either read-only or delayed during that time, as you can see from the following log:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29


kernel: (13,3):o2hb_write_timeout:242 ERROR: Heartbeat write timeout to device sdd1 after 12000 milliseconds
kernel: Heartbeat thread (13) printing last 24 blocking operations (cur = 4):
kernel: Heartbeat thread stuck at waiting for read completion, stuffing current time into that blocker (index 4)
kernel: Index 5: took 0 ms to do submit_bio for read
kernel: Index 6: took 0 ms to do waiting for read completion
kernel: Index 7: took 0 ms to do bio alloc write
kernel: Index 8: took 0 ms to do bio add page write
kernel: Index 9: took 0 ms to do submit_bio for write
kernel: Index 10: took 0 ms to do checking slots
kernel: Index 11: took 0 ms to do waiting for write completion
kernel: Index 12: took 2002 ms to do msleep
kernel: Index 13: took 0 ms to do allocating bios for read
kernel: Index 14: took 0 ms to do bio alloc read
kernel: Index 15: took 0 ms to do bio add page read
kernel: Index 16: took 0 ms to do submit_bio for read
kernel: Index 17: took 0 ms to do waiting for read completion
kernel: Index 18: took 0 ms to do bio alloc write
kernel: Index 19: took 0 ms to do bio add page write
kernel: Index 20: took 0 ms to do submit_bio for write
kernel: Index 21: took 0 ms to do checking slots
kernel: Index 22: took 0 ms to do waiting for write completion
kernel: Index 23: took 2004 ms to do msleep
kernel: Index 0: took 0 ms to do allocating bios for read
kernel: Index 1: took 0 ms to do bio alloc read
kernel: Index 2: took 0 ms to do bio add page read
kernel: Index 3: took 0 ms to do submit_bio for read
kernel: Index 4: took 9995 ms to do waiting for read completion
kernel: (13,3):o2hb_stop_all_regions:1682 ERROR: stopping heartbeat on all active regions.
kernel: Kernel panic - not syncing: *** ocfs2 is very sorry to be fencing this system by panicing ***

OCFS2 fun

Thu, 06 Mar 2008 10:31:05 +0000

Turns out, that said colleague has been playing with NFS on one off the web nodes, thus apparently rendering the remaining nodes offline (or semi-offline).

Now after all web nodes hung themselves, we had to hard reset them, now everything is tingly again .. yay for a great first day …

Been a while

Sun, 17 Feb 2008 08:53:16 +0000

Well, it’s been quite a while since most of the people last heard a word from me. The last few months I’ve been extremely busy with work-related tasks (and as a side-effect of that, didn’t want to spend much time in front of the computer after 9 hours of work). I also started spending more and more time in the gym, like nearly two hours every Tuesday and Thursday.

I finally fixed our replication issues, we do now have a working! MySQL Multi-Master ( 1. Node, 2. Node -- bear in mind, this boxes are only serving MySQL and nothing else, so don’t use these configurations on mixed setups) Replication Setup as database back end for our TYPO3-vHosts.
all the web nodes are now serving the content from a clustered, shared SAN volume (is that a good thing ? 😛 - don’t know yet …)
our VI environment is getting more and more acceptance (even if you hear some complaints now and then, like “awww, damn that crap my 4GiB RAM, 2x3.0GHz Windows 2008 is running soooo choppy” - simple answer, don’t use Windows Server 2008 and/or Windows Vista!)
I finished prepping our VM templates (at least the Windows ones)
we’re still putting together the plans on whether or not invest into a VDI solution.

The next few weeks are gonna be as frantic as the weeks before, I still have to migrate a lot of TYPO3 installations to our new cluster (which sadly needs time, as we need to wait for DNS changes to propagate). Honestly, I might be ending up extending the SAN volume for the MySQL data storage, as even with only three somewhat busy sites, the binary log of the last 5 days is about 2GiB in size. And we still have ~ 20 other busy sites on a separate box.

Being oblivious

Sat, 08 Sep 2007 20:25:31 +0000

Well, the title nearly says everything .. I managed to loose my second pair of car keys, today I somehow found out that I was driving without a drivers license, so I have to go to the registration office and apply for a new one, hopefully should be done in about 4-6 weeks. Oh hell, and I have to spend about 40 € on it ..

Well, live kinda sucks if you’re oblivious. Anyway, work is giving me a ass-load of fun right now, so I’m kinda happy, though it’s Saturday evening, I’m sitting back home, just lost all my custom build Debian packages (yes, I happen to use that at work, right after SLES) and listening to Hed PE.

SLES, ZendOptimizer and IBM PowerPC(4)+

Wed, 11 Jul 2007 08:40:57 +0000

What would you figure from the above ? Hopefully the rather obvious, that it’s a really shitty combination.

So we figured it would be a nice thing to test our new setup before going into pre-production testing or production, but we don’t have an extra spare box. So we took one of the power4 boxes we have mounted in the rack basically consuming energy all day (that’s about 38kWh a day) and installed SLES10 onto it. Which wasn’t all that bad (at first the box repeatedly started back to AIX, from CD and after convincing the SMS - that’s basically the bios on the power*-boxes also known as System Management Services with a hammer to boot from the first hard disk).

miimon, arp_interval and the code

Wed, 04 Jul 2007 19:50:14 +0000

After today’s adventure with the kernel bonding, I just took a look at the code …

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28


 if (miimon) {
 printk(KERN_INFO DRV_NAME
 ": MII link monitoring set to %d msn",
 miimon);
 } else if (arp_interval) {
 int i;

 printk(KERN_INFO DRV_NAME
 ": ARP monitoring set to %d ms, validate %s, with %d target(s):",
 arp_interval,
 arp_validate_tbl[arp_validate_value].modename,
 arp_ip_count);

 for (i = 0; i < arp_ip_count; i++)
 printk (" %s", arp_ip_target[i]);

 printk("n");

 } else {
 /* miimon and arp_interval not set, we need one so things
 * work as expected, see bonding.txt for details
 */
 printk(KERN_WARNING DRV_NAME
 ": Warning: either miimon or arp_interval and "
 "arp_ip_target module parameters must be specified, "
 "otherwise bonding will not detect link failures! see "
 "bonding.txt for details.n");
 }

If I read it right, you only get the KERN_WARNING for “either miimon or arp_interval” only if miimon or arp_interval isn’t set … but at least my config says it is .. 🤷 .. bed time for me :rolling_eyes:

SLES10 on pSeries

Wed, 04 Jul 2007 15:58:39 +0000

Okay, yet another day passed by blazing fast. I had a good day at work, spent nearly the whole day trying to get my bloody systems hooked up to our SAN (which was interrupted by a non-working SAN-switch, disappearing WWN’s, lunch and my trainees), messing around with our internal network, hacking our Blade Chassis switches to get me what I want and some random paperwork.

But first things first .. We installed SLES10 on a pSeries box the other day (I think on Monday), and now I’m trying to get the WWN of it’s Emulex HBA, out of either sysfs or procfs. But whatcha’ thinking ?

Continuing on SLES10

Sat, 16 Jun 2007 16:14:15 +0000

OK, it turns out that I was rather stupid when configuring the my.cnf. As it turned out, the effect I was seeing was due to the presence of two log-bin lines, which looked like the following:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12


[mysqld]
port = 3306
datadir = /mysql/dbase
log = /mysql/logs/dbc-mysql1.log
log-error = /mysql/logs/dbc-mysql1.err
socket = /var/lib/mysql/mysql.sock
bind = 172.16.234.31

# custom paths for binary logs
log-bin = /mysql/binlogs/dbc-mysql1
log-bin-index = /mysql/binlogs/dbc-mysql1.idx
relay-log = /mysql/binlogs/dbc-mysql1.relay

And some lines down there was this:

Back at SLES10

Tue, 12 Jun 2007 19:42:22 +0000

Here I am, sitting at my desk on a Thuesday evening thinking about what happened the last few days.

I finally got to play around with our PacketPro 450 Cluster(nifty LoadBalancing appliance)
We reworked the network the way we want it (and not that tool of a wannabe sysadmin)
We mostly figured out how to do the LoadBalancing right, we just need to find some bugs in the LoadBalancer software (like the thing is failing over to its slave from time to time, but keeping the IP address for himself) or let the guys at teamix do their work and hopefully get a working release within the next week or so
I figured out how to setup interface bonding with SLES10 (it was quite straight forward, thanks to the excellent in-kernel documentation), and we’re using an active-backup mode for now
I still need to figure out how to do the MySQL Master<->Master replication right .. I’m currently building fresh RPM’s on one of those Dell blades (yes, they ROCK!) which will hopefully be finished till I’m at the office tomorrow.
Pt. 5 also includes figuring out how to pass MySQL a custom location for the binary-log, at least that’s what the handbook says in Chapter “5.11.3. The Binary Log” …

When started with the –log-bin[=base_name] option, mysqld writes a log file containing all SQL commands that update data. If no base_name value is given, the default name is the name of the host machine followed by -bin. If the basename is given, but not as an absolute pathname, the server writes the file in the data directory. It is recommended that you specify a basename; see Section B.1.8.1, “Open Issues in MySQL”, for the reason.