Well, as we do have quite a few custom built RPM’s, I was searching for a new solution to manage the repo(s). Currently I do have a single repository per distribution. One thing one needs to know about createrepo (from createrepo), it doesn’t support this type of thing in the first place. So I had to come up with another way of doing it. First, I created a proper layout (much like the Debian Official Repository layout): ...
OK, it turned out that said colleague wasn’t responsible at all. Turns out, the real trigger was me creating a new volume on our SAN, on the same array that houses the OCFS2 volume. Apparently, during creation of an additional SAN volume, all other SAN volumes in this array are either read-only or delayed during that time, as you can see from the following log: 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 kernel: (13,3):o2hb_write_timeout:242 ERROR: Heartbeat write timeout to device sdd1 after 12000 milliseconds kernel: Heartbeat thread (13) printing last 24 blocking operations (cur = 4): kernel: Heartbeat thread stuck at waiting for read completion, stuffing current time into that blocker (index 4) kernel: Index 5: took 0 ms to do submit_bio for read kernel: Index 6: took 0 ms to do waiting for read completion kernel: Index 7: took 0 ms to do bio alloc write kernel: Index 8: took 0 ms to do bio add page write kernel: Index 9: took 0 ms to do submit_bio for write kernel: Index 10: took 0 ms to do checking slots kernel: Index 11: took 0 ms to do waiting for write completion kernel: Index 12: took 2002 ms to do msleep kernel: Index 13: took 0 ms to do allocating bios for read kernel: Index 14: took 0 ms to do bio alloc read kernel: Index 15: took 0 ms to do bio add page read kernel: Index 16: took 0 ms to do submit_bio for read kernel: Index 17: took 0 ms to do waiting for read completion kernel: Index 18: took 0 ms to do bio alloc write kernel: Index 19: took 0 ms to do bio add page write kernel: Index 20: took 0 ms to do submit_bio for write kernel: Index 21: took 0 ms to do checking slots kernel: Index 22: took 0 ms to do waiting for write completion kernel: Index 23: took 2004 ms to do msleep kernel: Index 0: took 0 ms to do allocating bios for read kernel: Index 1: took 0 ms to do bio alloc read kernel: Index 2: took 0 ms to do bio add page read kernel: Index 3: took 0 ms to do submit_bio for read kernel: Index 4: took 9995 ms to do waiting for read completion kernel: (13,3):o2hb_stop_all_regions:1682 ERROR: stopping heartbeat on all active regions. kernel: Kernel panic - not syncing: *** ocfs2 is very sorry to be fencing this system by panicing ***
Turns out, that said colleague has been playing with NFS on one off the web nodes, thus apparently rendering the remaining nodes offline (or semi-offline). Now after all web nodes hung themselves, we had to hard reset them, now everything is tingly again .. yay for a great first day …
Well, it’s been quite a while since most of the people last heard a word from me. The last few months I’ve been extremely busy with work-related tasks (and as a side-effect of that, didn’t want to spend much time in front of the computer after 9 hours of work). I also started spending more and more time in the gym, like nearly two hours every Tuesday and Thursday. I finally fixed our replication issues, we do now have a working! MySQL Multi-Master ( 1. Node, 2. Node -- bear in mind, this boxes are only serving MySQL and nothing else, so don’t use these configurations on mixed setups) Replication Setup as database back end for our TYPO3-vHosts. all the web nodes are now serving the content from a clustered, shared SAN volume (is that a good thing ? 😛 - don’t know yet …) our VI environment is getting more and more acceptance (even if you hear some complaints now and then, like “awww, damn that crap my 4GiB RAM, 2x3.0GHz Windows 2008 is running soooo choppy” - simple answer, don’t use Windows Server 2008 and/or Windows Vista!) I finished prepping our VM templates (at least the Windows ones) we’re still putting together the plans on whether or not invest into a VDI solution. The next few weeks are gonna be as frantic as the weeks before, I still have to migrate a lot of TYPO3 installations to our new cluster (which sadly needs time, as we need to wait for DNS changes to propagate). Honestly, I might be ending up extending the SAN volume for the MySQL data storage, as even with only three somewhat busy sites, the binary log of the last 5 days is about 2GiB in size. And we still have ~ 20 other busy sites on a separate box. ...
Well, the title nearly says everything .. I managed to loose my second pair of car keys, today I somehow found out that I was driving without a drivers license, so I have to go to the registration office and apply for a new one, hopefully should be done in about 4-6 weeks. Oh hell, and I have to spend about 40 € on it .. Well, live kinda sucks if you’re oblivious. Anyway, work is giving me a ass-load of fun right now, so I’m kinda happy, though it’s Saturday evening, I’m sitting back home, just lost all my custom build Debian packages (yes, I happen to use that at work, right after SLES) and listening to Hed PE.
What would you figure from the above ? Hopefully the rather obvious, that it’s a really shitty combination. So we figured it would be a nice thing to test our new setup before going into pre-production testing or production, but we don’t have an extra spare box. So we took one of the power4 boxes we have mounted in the rack basically consuming energy all day (that’s about 38kWh a day) and installed SLES10 onto it. Which wasn’t all that bad (at first the box repeatedly started back to AIX, from CD and after convincing the SMS - that’s basically the bios on the power*-boxes also known as System Management Services with a hammer to boot from the first hard disk). ...
After today’s adventure with the kernel bonding, I just took a look at the code … 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 if (miimon) { printk(KERN_INFO DRV_NAME ": MII link monitoring set to %d msn", miimon); } else if (arp_interval) { int i; printk(KERN_INFO DRV_NAME ": ARP monitoring set to %d ms, validate %s, with %d target(s):", arp_interval, arp_validate_tbl[arp_validate_value].modename, arp_ip_count); for (i = 0; i < arp_ip_count; i++) printk (" %s", arp_ip_target[i]); printk("n"); } else { /* miimon and arp_interval not set, we need one so things * work as expected, see bonding.txt for details */ printk(KERN_WARNING DRV_NAME ": Warning: either miimon or arp_interval and " "arp_ip_target module parameters must be specified, " "otherwise bonding will not detect link failures! see " "bonding.txt for details.n"); } If I read it right, you only get the KERN_WARNING for “either miimon or arp_interval” only if miimon or arp_interval isn’t set … but at least my config says it is .. 🤷 .. bed time for me :rolling_eyes:
Okay, yet another day passed by blazing fast. I had a good day at work, spent nearly the whole day trying to get my bloody systems hooked up to our SAN (which was interrupted by a non-working SAN-switch, disappearing WWN’s, lunch and my trainees), messing around with our internal network, hacking our Blade Chassis switches to get me what I want and some random paperwork. But first things first .. We installed SLES10 on a pSeries box the other day (I think on Monday), and now I’m trying to get the WWN of it’s Emulex HBA, out of either sysfs or procfs. But whatcha’ thinking ? ...
OK, it turns out that I was rather stupid when configuring the my.cnf. As it turned out, the effect I was seeing was due to the presence of two log-bin lines, which looked like the following: 1 2 3 4 5 6 7 8 9 10 11 12 [mysqld] port = 3306 datadir = /mysql/dbase log = /mysql/logs/dbc-mysql1.log log-error = /mysql/logs/dbc-mysql1.err socket = /var/lib/mysql/mysql.sock bind = 172.16.234.31 # custom paths for binary logs log-bin = /mysql/binlogs/dbc-mysql1 log-bin-index = /mysql/binlogs/dbc-mysql1.idx relay-log = /mysql/binlogs/dbc-mysql1.relay And some lines down there was this: ...
Here I am, sitting at my desk on a Thuesday evening thinking about what happened the last few days. I finally got to play around with our PacketPro 450 Cluster(nifty LoadBalancing appliance) We reworked the network the way we want it (and not that tool of a wannabe sysadmin) We mostly figured out how to do the LoadBalancing right, we just need to find some bugs in the LoadBalancer software (like the thing is failing over to its slave from time to time, but keeping the IP address for himself) or let the guys at teamix do their work and hopefully get a working release within the next week or so I figured out how to setup interface bonding with SLES10 (it was quite straight forward, thanks to the excellent in-kernel documentation), and we’re using an active-backup mode for now I still need to figure out how to do the MySQL Master<->Master replication right .. I’m currently building fresh RPM’s on one of those Dell blades (yes, they ROCK!) which will hopefully be finished till I’m at the office tomorrow. Pt. 5 also includes figuring out how to pass MySQL a custom location for the binary-log, at least that’s what the handbook says in Chapter “5.11.3. The Binary Log” … When started with the –log-bin[=base_name] option, mysqld writes a log file containing all SQL commands that update data. If no base_name value is given, the default name is the name of the host machine followed by -bin. If the basename is given, but not as an absolute pathname, the server writes the file in the data directory. It is recommended that you specify a basename; see Section B.1.8.1, “Open Issues in MySQL”, for the reason. ...