Another SysAdmin morning

Wake up, solve problems, keep going.

If you know my life, you know that I operate (with a clued team around me) a critical piece of the internet, and I also, casually, host some sites and servers for friends and family. My dad’s website from when he was writing books, some art pages for friends, my own hasnt-been-updated-since-1999-old-enough-to-drink-personal-site.

On a given morning, I can wake up to find problems with the big global service I run for my day job, or I can find that something’s awry with my train set.

So, this morning, as I woke up and blearlily looked at FaceBook Messenger (our IM platform of choice these days since AIM died), I find a message from a long-time client/friend of mine. “Hey, Gushi, I’m trying to do my monthly update, and the site’s running slow.”

And to know about this, you need to know about Jenny. Jenny is an artist that I’ve hosted for over a decade. She updates her site once a month, and when she does, this thundering herd of internet comes and tries to load the pages. We’ve seen this before. But it keeps getting worse every time.

Now, if this were a static site, it wouldn’t be a problem. A quick look at the logs, and a glace at top tells me the real problem. Jenny’s site is written in PHP, and it was a custom build. As a custom build, there’s no obvious security holes to exploit like there might be for a copy of Wordpress, but there’s also no useful tuning advice out there. And then we come to the greater problem: PHP in general.

PHP, as you know, is a language I hate for a number of reasons I won’t get into here, but the short version of it is: PHP with a webserver can either be performant, or secure. I chose secure. Those methods are called mod_php and suPHP, respectively.

On my system, with suPHP, if you have a user who has a PHP page (say, a guestbook), and someone tries to load that, the system will go look up the user (say, bob) and will fire up a setuid program called suphp, which will then do some permissions checking and fire up a copy of the PHP interpeter, as bob just to serve that page. This means that bob's scripts can’t overwrite alice's files. (As has happened with a number of Wordpress and PHPBB exploits in the past.) It means if bob has some poorly behaved script that gets compromised, and tries to write files elsewhere (like dropping executables in /var/tmp, I know, because those files are all owned by bob. This setup is sort of clever because it lets PHP scripts run stock, and as a bonus, lets me run multiple PHP interpreters, for scripts that are old and can’t easily be upgraded — like my Gallery install.

If I have a site that I *know* is going to be doing a lot of PHP, for example, my webmail site, which runs SquirrelMail, I can give it its own Apache process — bind Apache to a different IP on the box, and let it run mod_php, were the PHP is built into the webserver, and doesn’t need to fork.

So, this morning, a tail -50000 httpd-access-log | grep jenny | wc -l yielded a number in the 48000s. None of the traffic looked DDOS’y, it all looked to be coming from the correct regions, it really was just that popular. And, it would seem, getting moreso.

So, being tired of this, I did what any nerd would do:

• I grabbed a spare IP from my pool, and ifconfig‘d it onto the box.
• I checked the site’s DNS TTL (6 minutes, cool), and updated the A record to the new IP address. With DNSSEC signing, natch. (Even though she doesn’t have DS records, I like to live in a world where everyone ultimately will).
• I went in and copied my systems httpd-webmail.conf over to another IP, and changed the user and group definitions in it. (I didn’t need to update the PID file, because FreeBSD’s startup scripts handle that by themselves.)
• I copied her <vhost> stanza from the main vhost.conf over to this new file — no need to delete it from the old one.
• I added one line in FreeBSD’s standard /etc/rc.conf configuration file, defining a new configuration file: apache24_hersite_configfile="/usr/local/etc/apache24/httpd-hersite.conf"
• And then just ran service apache24 start hersite.
• Unlike Debian, there was no mucking around with sites_enabled or silly wrappers like a2ensite.

That was it. Load started dropping immediately. Things got fast, and easy. It all just worked.

Total time: about 20 minutes. Not even enough time to get through a decent mug of coffee. Heck, it took me longer to write this blog.