I decided to try to get more than an IP address clustered on my new OpenAIS setup. I was successful. I have two MySQL db nodes (db1, db2) running at 10.0.2.10[1,2]. Make sure you turn off all mysqld instances before proceeding…pacemaker will start/manage them for you.

# chmod a-x /etc/rc.d/rc.mysqld	(run on both nodes!)
# crm configure property no-quorum-policy="ignore"
# crm configure primitive ip_mysql ocf:heartbeat:IPaddr2 params ip="10.0.2.21"
# crm configure primitive mysqld \
	ocf:heartbeat:mysql \
	params binary="/usr/local/bin/mysqld_safe" \
	config="/etc/my.cnf" \
	socket="/srv/tmp/sockets/mysql.sock" \
	datadir="/srv/var/mysql" \
	user="mysql" \
	test_user="ha" \
	test_password="lololol" \
	test_table="beeets.omglolwtf" \
	op monitor interval="20s" timeout="20s" \
	meta migration-threshold="10" target-role="Started"
# crm configure group mysql mysqld ip_mysql

That should do it. Here’s a step-by-step:

1. First, we turn off automatic execution of mysql on node startup. Since pacemaker will be managing mysql, we don’t want two instances conflicting.
2. Turn off the quorum checking. Since we’re only running two nodes, it doesn’t make sense to stop all services if on of them goes down. In fact, that would sort of defeat the purpose.
3. Add an IP address that will represent our MySQL cluster.
4. Add the actual definition for MySQL. Obviously, you’ll be substituting your own values in for mine.
5. Configure a group. Now MySQL will always run on the same machine as the ip_mysql (we want this!).

With this setup, you can contact MySQL at 10.0.2.21. A failure of db1 will result in db2 assimilating the 10.0.2.21 IP and the MySQL service. Pretty wicked. Next up, How this can relate to replication and/or DRBD.

If you’re experiencing LVM snapshot problems, namely:

  LV system/rootsnap in use: not deactivating
  Couldn't deactivate new snapshot

It may be related to udev. In some kernel version (from what I’ve been reading here and there), there’s a race condition between udev (the thing that makes /dev tick) and LVM. LVM creates a snapshot volume, udev grabs it, LVM loses control. You can fix this by editing your udev rules under /etc/udev/rules.d (keep in mind this fix was made on Slackware 12):

1. Open /etc/udev/rules.d/50-udev.rules
2. Find the line with `LABEL=”persistent_input_end”` and after this line, add

3. KERNEL=="dm-[0-9]*",	OPTIONS+="ignore_device"

4. Restart

That should fix it. Please keep in mind this is for Slackware 12, and even still may not work. If you really want to solve all your problems, please download and use Slack 13 :).

So maybe you’re like me and wanted to play with LVM to speed up MySQL backups. Maybe you didn’t realize that to take LVM snapshots, you can’t use the entire volume when you format it. Fret not, here’s a simple way to reduce the size of an LV, giving you some breathing room for your backups:

	# umount /dev/db/data
	# e2fsck -f /dev/db/data
	# resize2fs /dev/db/data 200M
	# lvreduce -L 200M /dev/db/data

You cannot reduce the volume or filesystem size to less than the amount of space the data takes up (without losing data). But if you figure out how, you’ll be pretty rich. And never do this to anything you cherish without taking a backup.

There it is. Now check out mylvmbackup if you haven’t already.

Having taken my programming roots in QBASIC (shut up), C, C++, and a very healthy self-administered dose of x86 assembly, I can say that for the most part I have a good sense of what programming is. All of what I’ve learned up until now has helped me develop my sense for good code, and helped me to write programs and applications that I can sit back and be proud of. I’ve been working with PHP for over 4 years now, and I have to say it’s the most ugly language I’ve ever used.

Let me explain. PHP itself is wonderfully loosely-typed, C-like syntactically, and all around easy to write code for. The syntax is familiar because of my background. The integration with web is apparent down to its core, and it’s a hell of a lot easier than assembly to write. When perusing through a project filled to the brim with C source code, I’m usually left thinking about how it works, why the developer did what they did, and why that makes sense for that particular application. I’m usually able to figure out these questions and there’s one main reason: the code isn’t shit. With PHP, I’m usually left wondering what the developer was thinking, the 100s of ways I could have done it more efficiently, and why this person is actually making money doing this.

With roughly 90% of open-source PHP projects, everything works great. I love it, clients love it, everyone kisses eachother’s ass. But then one day you get that inevitable change request…I want it to do THIS. A quick look at the source code reveals that, omg, it’s been written by a team of highly trained ape-like creatures! It surprises me that WordPress plugins that get 100s of downloads a day throw errors (unless you disable error output, which I never do on my dev machines). Whole architectures are written with random indentation, or indentation with spaces (sorry Rubyers, but space-indentation is an evil scourge on humanity). No effort is put into separating pieces of code that could so easily be modularized if only they were given a second thought.

Do I hate PHP? No, I love PHP. I think it’s a well-written, high-level web development language. It’s fast, portable, and scalable. It allows me to focus on the problems I face, not the syntax of what I’m trying to do. Paired with an excellent editor like Eclipse (w/ PHPeclipse) I’m unstoppable. But why can’t any other PHP developers share my love of well-written code? It’s the #1 critique of PHP, and rightly so. I’m pretty sure that all programming languages, save Python, allow you to write awful, unreadable code…but PHP’s culture seems to be built around shitty code, amateurish hacks, and lack of elegance. PHP isn’t the problem, it’s the people writing it who suck!

So I do love the language, but hate most of the implementations. I have to say though, nothing is worse than Coldfusion.

Notice the “clone” in quotes. Why? You can’t actually clone a VM technically. We can work around that, though. Keep in mind, this guide is for VirtualBox <= 3.0.8 (later versions may have a clone button or something).

One thing you CAN do is export to OVF and re-import, but I’ve found that OVF loses many settings (like video ram, network settings, whether you use SATA or not, etc). I prefer not to even bother with this method.

The next thing you can do is just clone your VM’s hard disk(s):

1. Go into the ~/.VirtualBox/HardDisks/ folder. Copy and paste (windows) or cp (linux/unix) from db_master1.vdi to db_master2.vdi. If you try to import this into the Virtual Media Manager, it will piss and moan about the UUID being duplicate or some shit.
2. VBoxManage internalcommands setvdiuuid db_master2.vdi – this is the magic command that allows you to import that new HD.
3. Create a new VM, and set db_master2.vdi as the primary drive.
4. Configure your new VM to have the same settings. (this is a pain, but there really aren’t that many settings).

There are a few things you’ll have to dick with once you have your VM cloned. If you’re into networking/cluster/HA crap like me, you’ll probably have a static IP. This obviously needs to be changed. It’s different for every distro, but it’s in /etc/rc.d/rc.inet1.conf for Slackware, and /etc/networking/interfaces for Debian (any other distro can go to hell).

Your old network interfaces, eth[0...n] will now be eth[(n+1)...(n*2)]: eg, if you had eth0 and eth1 before, they will now be eth2 and eth3. To reset this, (in Slackware):

1. Open/etc/udev/rules.d/75-network-devices.rules in your favorite editor
2. Remove all the entries.
3. Restart. (note – someone please correct me if you don’t need a restart… perhaps /sbin/udevtrigger will fix this?)
4. You will now have eth0 and eth1 again. Hoo-fucking-ray.

The process is the same for Debian, but the 75-network-devices.rules will most likely have a different name.

Good luck.

Recently I read about compute clusters and how they’re used. I just had to try it. I successfully installed (from source) OpenAIS and Pacemaker (guide here) on Slackware 12. The experience was, overall, extremely smooth. I had a few hiccups I can only attribute to my not being able to follow directions, but with a few Makefile tweaks and some very small code tweaks, I got everything to compile and run.

Keep in mind, this is all using VirtualBox VMs, so once the cluster stuff is installed on one machine, I can more or less copy and paste and have a 3-computer linux cluster running from the comfort of Windows 7 (it’s just for games, I swear!!). Aside from having to mess a bit with the networking in VirtualBox, everything was almost completely automatic.

The next step is to figure out DRBD and how it fits in with all this HA stuff. I’ve been trying to find a guide on using Pacemaker with MySQL, but no guide is written JUST for MySQL…it’s all MySQL with DRBD. I’d rather not complicate things too much until I can figure out how the hell this is all working.

Anyway, I’ll report back with my findings sometime soon.

Please note – at the time of installing (about 6 or 7 days ago), there was a bug in the Debian packaged distribution of OpenAIS/Pacemaker that makes the “expected votes” value of the quorum formula > 3 billion. Unless you have 3 billion machines laying around, you will NOT be able to start any resources unless you configure to ignore the quorum. I’ve confirmed this bug on the IRC channel, and to my knowledge, it still exists.

My advice is to compile from source (but that’s always my advice anyway :) )

<rant>

We’ve all heard of howtobepunk.com, the world’s best guide on the punk subculture. Weighing in at over 15-billion page views per day, it’s one of those sites people just expect to see up…and every person on earth checks it on average 2.3 times per day. The domain recently expired, and oddly enough, I wasn’t warned.

I’m not too bent out of shape about that, it’s happened before (and many times it has been my fault). What really chaps my caboose is the transfer process. See, I hate GoDaddy with a passion. Always have. Since NFS.net started doing domain registration, I’ve been pretty gung-ho about using them. I decided to transfer HTBP.com to Nearly Free Speech.

GoDaddy, to the full extent the ICANN let’s them, makes this the most difficult and tedious process you can possibly imagine. Every guide you find on their site about transfer domains assumes that you want to transfer the domain TO them. Why would anyone, after all, want to transfer a domain away from the best registrar/host/whatever else the fuck they do in the world? You’d have to be crazy.

Call me a fruitcake. Anyway, I finally got through the whole process, activated the transfer, blah blah. GoDaddy sends me an email saying it’s all successful. Great, I can sit back now.

NO!!!!! I can’t. There’s some sort of domain transfer “pending” system that holds on to the domain for a week (or until you approve it AGAIN). GoDaddy decided to hide this at the bottom of the email they sent telling me everything is fine.

Anyway, dealing with those guys is a nightmare. Only one domain left on there, and once it comes up for renewal I’ll never host another domain on GoDaddy again. Their system itself works fine, but even a highly skilled web developer has trouble using their shitty interface. I can’t imagine how some average douche who wants to register mydogsparkyiscool.com would make heads or tails of it.

</rant>

Compared are Linux 2.4, 2.6, FreeBSD, NetBSD, and OpenBSD. Really well-performed benchmarks, with graphs.

http://bulk.fefe.de/scalability

Linux 2.6 was hands down the winner, which makes me feel good about Slackware (2.6 linux but actually stable) as a server. I’m sure Windows would have won if only it was benchmarked. One thing to keep in mind – from what I gathered, the machine tested was a single-processor, single-core machine…this means that SMP scalability was not tested, a HUGE consideration for more modern servers (what server now doesn’t have multiple cores?) and may skew the modern-day results, especially between the two leads, FreeBSD and Linux 2.6.

So, being a web developer, I like to sometimes see when problems arise in my code. Thanks to the latest and greatest from firefox, this is now that much harder. Great. Another “feature” of theirs…almost as useful as the memory leak feature that still haunts me to this day. Here’s what I’m talking about:

Ever see this? The file upload.php exists at that location. Checking the source shows that, indeed, the script loaded and threw and error. Why then, pray tell, does firefox insist on persisting in its endeavour to destroy my life and career and steal all my friends?

I’ve found no documentation of this “feature” so have been thus far unable to get rid of it. Any ideas?