I’m a system administrator. An operations Engineer. I manage physical servers, as well as ethernet switches, routers, and the infrastructure that keeps those devices alive.
And there’s a problem there. Sometimes, those devices go end-of-support, and as a SysAdmin, you still need to manage them — even if just to shut them down or wipe their drives.
Now, the OS on the main piece of hardware in our stable is an x86_64 copy of Linux or FreeBSD. It can be updated to modern standards trivially. Even a ten-years-old machine can take a modern copy of BSD, albeit not quite as performantly.
But when that system also has a tiny embedded management board, typically a UPS or a Rack-mount server, or even something more vague like an air-conditioner, or a print server, or a generator transfer switch, you’re out of luck. You can’t upgrade the CPU or RAM in that tiny management board, and more than likely, it’s limited on how much code can fit in its tiny Boot Flash, too. In most cases, this means upgrading it to support more modern SSL/TLS algorithms is a non-starter. (Modern SSL routines are orders of magnitude more CPU intensive than what existed even a decade ago).
What this means, in practical terms, is that talking to that little controller, to do anything useful with the big server that it manages becomes way more difficult.
In a previous post, I detailed what it might take to get a modern copy of Java to talk to an older Sun Microsystems ILOM. It involved turning every knob in Java to enable old hashing algorithms like MD5 and old ciphers like RC4, and adding your server to the “trusted” list and override lists in every possible case. That advice may no longer apply, unfortunately. The code that handles those old ciphers may not even be there anymore.
Thus, one of the requisite tools in my arsenal has long been a Lenovo Ideapad running Windows XP. It was my personal-carry laptop in 2009, and since I replaced the hard drive with an SSD, it’s been pretty immune to failures. It lived in a drawer at my office. It was my go-to tool for driving things like ID badge printers for a Sci-Fi con I worked with. When I started playing more with virtualization, I found a way to image the hard drive (and release the activation) and keep it with me so I no longer needed to keep a physical machine around.
I’ve also kept a mac mini around with an older Core 2 Duo processor, in order to play Command and Conquer: Tiberian Sun. For the longest time, it just never ran right under more modern versions of Windows, and it was always the best Stress Reliever game for me. Upgrading shouldn’t mean losing access to what I love!
Once I made the jump to Apple Silicon, rather than virtualizing my trusty XP machine with VMWare Fusion (which can no longer virtualize the X86 architecture), I instead emulated it with UTM. It’s interesting how in the UTM docs, they tell you the SHA256 checksum of a known-good Windows ISO, but don’t tell you where to get it. “If you know, you know.”
Once I got Windows XP up and running, I attempted to patch it, and found that someone had managed to beautifully, faithfully, re-create the Windows Update experience as “Legacy Update” so that XP could still be patched up to what was the most current patch level back in the day. Legacy Update also fixes your SSL trust store and enables a few more modern ciphers, so that if you have an older machine (like an iDRAC 6) that’s signed by a CA with a SHA256-hashed cert, it has a chance of working.
Side Note: Windows XP (both Embedded and Not) is still Alive and Well out in the world, but hopefully not on the internet. It’s Running on everything from ATMs to Kiosk Machines, to CT Scanners, to Door Controllers, to C&C Machines and Industrial Lathes. Some of these run via 32-bit PCI interfaces, or parallel port interfaces, or even ISA cards. Some of these were developed by now-defunct companies that aren’t going to produce new software just because Microsoft puts out a new OS. It’s ignorant of Microsoft to claim that “upgrading everywhere” is an option. There’s a thriving market of people buying old motherboards to continue to run these types of machines, and it would be really great if Microsoft would make it so people don’t need to commit wink-wink-nudge-nudge levels of piracy in order to use the $50,000+ hardware installs that happened to be built around a now-EOL OS. I would buy a license for this OS if I could. Instead, I console myself with the fact that the license was salvaged from another piece of hardware I legitimately owned, but I’m still forced to violate their EULA to do my job.
Will Legacy Update make it safe to use your Windows XP VM for everyday use, on the modern internet? Absolutely not.
Is this new bastard emulated version of XP performant enough to run games needing more CPU than FreeCell? No.
But will it launch IE6 and Firefox 52.9.0 ESR (the last version known to work on XP) and Java 6, and let you manage that box that you can’t talk to any other way?
Yup.
The only other utility I really need there is a copy of Putty. Since file sharing with the host computer is sketchy, this leaves me with the easy ability to sftp/scp things up and down to a unix server on the outside. And because networking is NAT’d within my laptop, it means if my mac is on our VPN, so is the XP machine, so I don’t have to track down a current VPN client for XP.
Let me be clear. I shouldn’t have to keep this retrocomputing hobby alive in order to keep doing my job. This shouldn’t be necessary. There should be some switch, deep in Safari or Chrome’s internals where I can start it in a mode that forces me to type out “Yes I know what I am doing, I am a sysadmin, and I need the Old Magic and the old less-strict Javascript Interpreter”. I would type that every time, rather than feel dirty booting up a legally dubious copy of a 20-years-dead OS.
For your reading joy, I present three tales of having-to-talk-to-old-stuff:
Tale Number One: OpenSSH and APC PDU’s.
Several years ago, I was sitting in the Portland area over the Christmas holidays, and needed to SSH in to one of my APC PDUs. These machines sit on their own protected network segment, and are not reachable from the outside world.
These machines had 768 bit RSA host keys. (And in fact, they could handle a 1024 bit key, but couldn’t generate that key onboard). OpenSSH outright refused to talk to these devices without being recompiled.
While the OpenSSH developers have cautiously set the default set of supported ciphers to best-in-class versions, and have said “if you need to connect to something a little older, here’s a set of legacy options you can set”, that set doesn’t include everything.
I reached out on the OpenSSH Users list saying “Gee, I know 768 bit keys are bad, but it would be super cool if we had an option, perhaps one that must be specified on the command line, that let this still work”. Remember, this is not the removal of old swaths of code or broken algorithms, this was an arbitrary #define about key size. It was a compile-time option, and I was asking if it could be run-time.
What I got instead was a Flame War. (Note that I only followed up once. Also note how the last comment is about List Conduct).
For the record, The OpenSSH devs’ answer was “You should upgrade all your smart $1200 power strips.”
Cool, thanks guys.
At the time, we had over 20 of them, which work perfectly fine as power strips, and they still give us power utilization via SNMPv1, but we’ve since had to build an alternative SSH binary just to talk to them.
Tale Number Two: SuperMicro Virtual Media and SMB
While my regular readers know I extoll the love for the Dell iDRAC, I recently was dealing a device made by Netgate (the people who make the PFSense firewall). And as a small integrator, they went with SuperMicro hardware. Supermicro servers are reasonable, affordable, machines in everything but their management boards. This machine supported an HTML5 console, so at least I didn’t need to jump through dumb java hoops to get a remote login.
However, when things went far south enough that my only options were to reinstall the OS, I found myself with a dilemma: Yes, the machine could talk to an ISO image, but not one where you browsed on your local machine and uploaded the ISO to an onboard sdcard (such as with Dell’s iDSDM’s)— nor even an emulation layer provided by the HTML5 virtual console. No, this thing would only talk to an ISO mounted on a Windows share.
I even tried chasing down a copy of Supermicro’s IPMIview, which used to be on their FTP site, but is now in a weird ftp-like directory on Supermicro.com. It refused to play ball with my Windows XP VM (which claimed the zip file was corrupted), and Windows 10 worked, but at the end of the day, told me I needed to buy a $180 license for the server to do Virtual Media, which wasn’t available for this motherboard anyway.
It gets even worse: This is not modern windows share. Not remotely. This board needs SMB version 1.0, and NTLMV1 Authentication. (So, effectively, Windows 95-era networking. Stuff for which there are known vulnerabilities.)
Now, there are ways to make Samba do this kind of dangerous, broken, don’t-try-this-at-home-kids networking, because the Samba developers (unlike the OpenSSH and OpenSSL devs) are willing to leave dangerous off-by-default options in place for the 1% of those who really need them. (Thank you for this). And while I fired up a copy of Samba on one of our jumphosts, firewalled it to only my ILOM, and deleted it right after, it still felt icky.
Credit where it’s due: A blog post by Tom Rogers helped me determine most of what was necessary. Tom stops short of telling you how to actually do this, so that may be its own post here.
Even after getting Samba set up, the iLOM did some stupid things with regard to “Optional” usernames. (If you leave it blank in the iLOM, it doesn’t try a guest account, it defaults to trying an Administrator account.)
In the end, I got the OS reinstalled, but man, it felt like trying to get the Voyager probe to talk to Houston again.
Tale Number Three. The HP ILO3:
Sitting in one of our datacenter spaces, we have a machine that is an HP DL360 G7. We bought this machine to match the exact spec a customer had for their own installs, so that we could replicate not only hardware-specific problems (for example, related to ECC memory timing), but also so that we could benchmark what performance that customer would meet in production.
It’s still a perfectly valid machine to spin up fuzzing tests on, but when it came time to connect to its iLOM and monitor it (for example, for a failed PSU, something that’s mostly invisible to the OS), it was a nonstarter.
Every modern browser threw warnings about it. Once again, I broke out my Windows XP VM. From there, I was able to download the latest thankfully-not-behind-a-paywall iLOM firmware for it (dated 2020, up from 2012).
While the new firmware happily updated the HP Logo to be their new standard (I guess that was important), it also gave me a picture of a bridge to go jump off of while trying to get this working…
What it didn’t do was enable TLS1.2, or enable any modern Ciphers. It did not fix the fact that the best available cipher suite for this machine was still awful (as an older copy of openssl s_client tells me).
New, TLSv1/SSLv3, Cipher is RC4-SHA
Server public key is 1024 bit
Secure Renegotiation IS NOT supported
Compression: NONE
Expansion: NONE
No ALPN negotiatedModern copies of OpenSSL wouldn’t even speak to it. I had to install the openssl-insecure FreeBSD package just to run openssl s_client to get the above results.
The RC4 cipher has been completely removed from OpenSSL entirely, which on the one hand is reasonable, but on the other, makes it much, much harder for me to do something like use an Apache server’s proxy module to talk to it, so I can at least talk to it with A Perl Script — something that logs in, downloads an XML file, and parses it. There is no remote execution here; no opportunity for a man-in-the-middle attack; no stealing of sensitive data. I either get an XML file which tells me about my hardware’s health, or I don’t.
(That Perl script uses the system OpenSSL, which of course, cannot talk to the ILO either.)
If I were feeling insanely stubborn about this, I might custom build my perl packages against the openssl-unsafe port and put them on their own VM, limited entirely for this purpose, and make it so there’s a 1:1 proxy for the old host, but it would be nice if there were a modern version of the OpenSSL code, that lets me set some crazy environment define that says “I’m a sysadmin, I have old shit, and I need to monitor it”, and use a stock mod_proxy or Squid.
I could go on. I’ve got a number of fun war stories, and honestly, most of the people in my line of work have many similar stories that we share with each other. They are the stuff of legend, and the thing that makes the job interesting and worthwhile. Solving stuff like this with the most basic of you-wont-believe-what-I-had-to-do builds dopamine akin to climbing everest.
I’ve been working for a non-profit for many years, and that often involves dealing with secondary-market and donated hardware, in a building that — I swore — could detect that I was too far away and threw temper tantrums.
In the Netflix Documentary, “The Last Blockbuster”, we are shown that one tiny Blockbuster Video store in Bend (Oregon) hoards the PCs bought up from other long-shuttered Blockbuster stores, carefully imaging the hard drives, because they’re the only ones that work with their POS software.
The US Government, famously, still used 8-Inch Floppy disks for its missile defense system well into this century. Do you really think that every computer on a Nuclear Aircraft Carrier is going to upgrade to Windows 11 because 10 is EOL?
A line of Cisco Servers famously used not just Java, but also Flash for their management UI, which you can now not even officially download.
Even if you’re not a nonprofit (or the last Blockbuster on Earth), it doesn’t mean you’re going to rip out every door controller in your building, just because the airgapped tower computer sitting in the closet to provision key fobs runs Windows 95 (or even DOS). You’re going to leave it alone, image the hard drive, and pray it never breaks. And you’re going to search eBay for the same exact machine with the same motherboard, quietly buy it and put it somewhere safe, and you’re going to figure out how to crack the activation on that copy of Windows when you need it. And you’re going to look like a hero for doing that, when (not if) the time comes.
Part of being a sysadmin means having to talk to old hardware over a TCP/IP network is still a requirement. These systems are not going away, despite both Microsoft and Theo’s best assertions that they should. Someone hired us to keep them online, and that means finding a way that if there’s copper between the two machines, to make them play ball, one way or another.
There is a fallback, of course: In many cases, we have DataCenter staff who can do Smart Hands for us, who can burn a CD and put it in a drive, or who can even (for a price) reinstall an OS. And usually, we won’t get a second thought from management for making the call and spending the money on Smart Hands.
But what would be the fun in that?
