Mscs

Okay, so we received a brand new x3650 the other day entitled to replace one (or better two) of our NAS frontend servers. We installed Windows on it the other day (had to create a custom Windows Server 2003 CD first, since the default one doesn’t recognize the integrated ServeRAID), and we prepped the box during the week with the usual things. On Monday I started installing the " IBM StorageManager RDAC" MultiPath driver (since the box got two single port PCIe FC-HBA’s) and figured I’d be nice if we had this. I asked a IBM Systems Engineer of one of our partners, which told me generally there wouldn’t be a problem with Microsoft Cluster Services (MSCS) and the IBM MPIO driver. Only requirement would be that I’d install the new storport.sys driver (version 5.2.3790.4021) first (as in Microsoft KB932755). ...

Well, guess my " solution" didn’t work sooo good. Lemme tell you what’s happening. I successfully added the node to the cluster group, but I can’t get any resources online. The node tries bringing it online, then shows a failure and immidiately moves them over to the next node. There the resource is being successfully moved online .. So again, I’m out of ideas .. Already tried reinstalling the box, after that I could get the third node successfully into the cluster, without the " Advanced (minimum)" trick … 🤷 still ain’t bringing any resources online.

Well, I just had another look at our client scheduler services on our Microsoft Cluster. A while back we noticed that those scheduler services were going nuts after some time. Well, as it turns out, I can tell why. Microsoft Cluster Services have a feature called registration replication, which replicates a given key, if changed when the resource is online, to all connected cluster nodes. Now, we added the obvious registry key to the settings of our cluster resources for the scheduler services ( SOFTWAREIBMADSMCurrentVersionBackupClientNodes) and the scheduler service would use the same registry key to store it’s passwords. But it seems we were far off with that assumption. ...

Well, once again the Microsoft Cluster on VMware bit my ass … As you might know, MSCS on VMware is a particular kind of pain, with each upgrade you end up with the same problem over and over again (SCSI reservations on the RDM-LUNs being one, and the passive node not booting being the other). So I opened up another support case with VMware, and the responded like this: Please see this kb entry: http://kb.vmware.com/kb/1016106 ...

Well, recently I stepped up to watch our cluster environments … Michael has a good howto on how to watch Windows Cluster environments in the NSclient++ wiki. Now, this has it’s own perks … Which I stumbled upon when trying to write a Linux-HA OCF resource agent for the Nagios NRPE server. Combining that Linux-HA with SLES10 is a good thing generally, but using startproc in that resource agent is not such a good idea. Apparently Novell (or SuSE GmbH) thought it might be wise to include some additional logic into the wrapper. startproc, checkproc and killproc do check for the name of the executable. So if you try to start an additional process with the same name, you need to dig a bit deeper. For this to work, you need two additional things (quotations directly from man 8 startproc): -p pid_file (Former option -f changed due to the LSB specification.) Use an alternate pid file instead of the default (/var/run/ .pid). The pid read from this file is being matched against the pid of running processes that have an executable with specified path of the program. In order to avoid confusion with stale pid files, a not up-to-date pid will be ignored. Now, then apparently this isn’t enough. startproc is still refusing to start a second process. -i ignore_file The pid found in this file is used as session id of the same binary program which should be ignored by startproc.

If you think back, I talked about my problems with MSCS while utilizing the IBM RDAC Multipath driver for Windows. Everyone I talked to about this, including our IBM business partner and it’s systems engineers; as well as some IBM systems engineer (who in fact was an freelance guy hired by IBM), told me it had to do with how we did the zoning (stuffing every controller into a single zone), and that would be the reason why the x3650 was seeing that many drives. ...