Mass-Updating your Dell Servers with DRM (Dell Repository Manager)

At the Day Job, we manage a fleet of Dell Poweredge servers out in the wild. They are often donated to us, and often in odd, remote locations where smart hands are either difficult to coordinate because of access, or time zones — or pandemics.

And occasionally, they have interesting issues where they need a firmware update of some sort. Sometimes it’s because the firmware has a security issue. Sometimes, it’s to enable the HTML5 console so you don’t need Java to launch a remote window. Sometimes, it’s so you can get them to take a reasonable SSL certificate. Sometimes, it’s so that you can solve a problem like a Dell-branded network card that decides to put out LLDP packets on its own that are invisible to TCPdump. (Yes, this happened).

You would think that it would be possible to go into the iDRAC settings, hit a button that says “Please check for updates from Dell, wherever they are” and life would be good. There’s even a tab in the iDRAC that makes you think something like this exists.

I truly, truly wish it worked that way. Failing that, it would be great if Dell gave you some sort of service that gave you a private URL that you could use when you paid for support, where your machine used some HTTPS client cert to log in to an official update server. That’s not the reality we live in.

You might then assume that you’re reduced to trawling the Dell support site, downloading packages to your local machine, and then uploading them one by one. Dell does give you an option that’s better than downloading a ton, however, and that is the Dell EMC repository manager. (That’s a link to the Dell Page). This little Java-based tool is a GUI for Windows or Linux that will let you do three things:

• Connect to a single machine and build a catalog from it.
• Let you specify a class (or classes) of machine and let you build a catalog from that. For example, “R330”.
• Lets you create an empty repository. (I still haven’t figured out the use of this).

Once you have a repository, you can download items for it, or update it, or clone it, or whatever, and you can export that repository to a catalog file, in a windows share, bootable CDROM, or to a file path.

The key here, though, is that there’s also a command line version of this tool, as well as a command line way to drive the iDRAC, and figuring those out made our life awesome. In this article, I’ll share some of our Secret Sauce.

Prerequisites

I’m not going to take you through how to install Ubuntu from scratch. We’ve got an Ubuntu VM, running 20.04 LTS with some nominal amount of ram (you’re running a Java app, under X, use more than you think you need). For disk, we gave it a 200G /data partition that we symlinked most of the Dell stuff in to. We installed X on it, which we access via our VMWare console (typically using VMWare Fusion or Workstation). We downloaded and ran the Dell repository manager manually from the above link. (We do have a Dell repository configured in /etc/apt but it didn’t provide this tool).

We created a basic apache virtual host, and we’ve made it so the drmuser user can write to this directory:

<VirtualHost *:80>
  ServerAdmin webmaster@localhost
  DocumentRoot /var/www/html/drm
</VirtualHost>

We also created an HTTPS version, just in case, but since the files are signed by Dell (and since some iDRAC’s cant even check over HTTPS, and even then, the iDRAC doesn’t know how to check most SSL certs), it’s largely pointless.

Some iDRACs can also check FTP or Windows (SMB/CIFS) file shares, but we haven’t found enabling that over the public internet to be necessary. HTTP is a pretty reasonable common denominator.

Running the tool — building a mega-catalog for all your stuff

Repository builds happen in two steps: Create, and Export.

After using a puppet command (specific to our install base, so I can’t really share it) to gather the list of server types, we generated a master repository like this:

drm --create --repository=everything --inputplatformlist="R220,R330,R340,R420,R430,R440,R620,R630,R640,R740,R840,R6525" --dupformat=win64

(That’s one line, but Medium wraps it weird)

We use dupformat=win64 because the iDRAC knows how to digest those (Windows 64-bit) files natively. If we were building a Windows or Linux CD with these tools we might do things differently.

Note that while you can limit categories of things to install down to certain components like RAID cards, or BIOS drivers, if you just do the above, it will fetch everything for all those platforms. So yes, you’ll be pulling down firmware files for Adaptec tape devices you probably will never own. Disk is cheap. There’s also ways to limit things to “critical” and “optional” but we never want to do this.

Here are the various “types” and “categories” drm knows about:

drm:/data/dell/drm$ drm --list=component-types
Component types
---------------
BIOS
APPLICATION
FIRMWARE
DRIVER
drm:/data/dell/drm$ drm --list=category
Categories
----------
BIOS
Chassis System Management
Chipset
Diagnostics
Drivers for OS Deployment
Embedded Solutions
ESM
Express Flash PCIe SSD
Fibre Channel
Firmware
iDRAC with Lifecycle Controller
Memory
Network
SAS Drive
SAS Non-RAID
SAS RAID
Serial ATA
Systems Management
Tape Drives
Video

Note: there’s no way I’ve found to actually list all the supported platforms (i.e. Model Numbers). The GUI shows them but this is a shortcoming in the CLI.

From there, drm goes in and builds catalogs for every one of those machines, for every type of firmware (above), in the background. This is relatively quick, much like an apt-get update is quick. This is not the part where it pulls down all the firmware files you’re going to need.

You can get a status of the job with:

drm --li=job

You can update your “everything” repository thusly:

drm:~$ drm --update --repository="everything"
Refreshing repo: everything
Job Refresh_07/02/2022 20:50 completed successfully for all selected repositories.
drm:~$

And from there, you just need to export it to somewhere on disk:

drm:~$ drm --deployment-type=share --location=/var/www/html/drm --repository=everything
Job Export_07/02/2022_20:51 is scheduled to run at: Sat Jul 02 20:52:01 UTC 2022
Export Repository job started.

This will take a while. From there, you’ll see it running for, perhaps a while, in the output of drm -li=job:

Medium is not the best place for screen pastes, this got mangled in <tt> mode.

But eventually you’ll see it finish:

Success!

Now, when drm exports a catalog, it sticks a version number on it, so the catalog file would have been exported to /var/www/html/drm/everything_1.01_Catalog.xml and then the various installable files in subfolders under that.

Applying the update catalog you built via the iDRAC GUI

Now, once that export is done, you’re ready. You could go into your iDRAC’s web UI section, under “iDrac Settings -> Update and Rollback” and just fill this in:

iDrac screenshot, showing an update

In case the screenshot is too ugly, I’ve only filled in the HTTP address (with an IP address) and the catalog filename, which is everything_1.00_Catalog.xml.

That filename is a bit unwieldy though. What if, instead, you just symlinked that everything_1.00_Catalog.xml to the default filename, Catalog.xml? Then, you’d only have to fill in the IP address, or hostname (hostname requires your iDRAC has DNS configured), and hit “Check for Update”. One box, and go.

You’ll be presented with a list of firmware, and for things that require a reboot, the iDRAC will shut down your server cleanly, reboot into a special installer, then bring your system back up. Naturally, you want to have scheduled downtime for this! Checking for updates is fine, but when you click “Install”, it’s going to try to do that for you! You’ve been warned!

Checking for and Doing updates from the iDRAC command line

If you don’t want to mess with the GUI, or feel bold enough to script this, it’s possible! You can ssh to an iDRAC, and from there, you can use the built-in racadm command to check for updates:

racadm update -e 10.1.1.10 -t HTTP

If you want it to just go ahead and install whatever’s in the catalog even if it’s a downgrade you can add -a TRUE

racadm update -e 10.1.1.10 -a TRUE -t HTTP

From there, the command will give you a command that you can use to view the status of the actual update check (it will be some variant of racadm jobqueue view), and you can also use

racadm update viewreport

To see which packages are being updated.

Here’s a full output from a system that’s already up to date (only the IP is changed):

racadm>>racadm update -e 10.1.1.10 -t HTTP
racadm update -e 10.1.1.10 -t HTTP
RAC1118 : Update from repository operation has been initiated. Check the progress of the operation using "racadm jobqueue view -i JID_568229942185" command.
racadm>>racadm jobqueue view -i JID_568229942185
racadm jobqueue view -i JID_568229942185
---------------------------- JOB -------------------------
[Job ID=JID_568229942185]
Job Name=Repository Update
Status=Downloaded
Start Time=[Not Applicable]
Expiration Time=[Not Applicable]
Message=[RED002: Package successfully downloaded.]
Percent Complete=[NA]
----------------------------------------------------------
racadm>>racadm update viewreport
racadm update viewreport
ERROR: RAC1194: Successfully compared the firmware versions on the server to the versions defined in the specified catalog.
The server firmware is up to date and matches the catalog definitions.
racadm>>

For completeness, here’s what a viewreport looked like when there was an update available:

racadm>>racadm update viewreport
racadm update viewreport
ComponentType     = Firmware
ElementName       = Broadcom Gigabit Ethernet BCM5720 - 44:A8:DE:AD:BE:EF
FQDD              = NIC.Embedded.1-1-1
Current Version   = 21.81.3
Available Version = 22.00.6
-------------------------------------------------------------------

Building a catalog for an individual machine

There’s one final caveat here: that catalog file is pretty big (11 megs). It doesn’t sound huge, but if you have a machine out in the world at the end of a throttled network connection, this can take a long time to download, and parse out, so it *might* also be worth your while to download and build a single catalog file for that one machine, instead of the “everything” package.

Here’s the command that will let drm log into your iDRAC, inventory your system, and build exactly what it needs as a catalog, no more, no less:

drm --create --repo=somename -it=idrac -ih=hostnameofbox --dupformat=win64 --auth=root:calvin

followed by:

drm --deployment-type=share --location=/var/www/html/drm --repository=somename

Where:

• somename is a name for the repo
• hostnameofbox is the ip or hostname of the iDRAC to log in to
• --auth=root:calvin is a current working user. This can also be a user with read-only access if you prefer to create one of those.
• -it=idracis literal, it’s short for “integration type”
• --deployment-type=share is literal
• --location=/var/www/html/drm is also literal

This will get created in your HTML docroot as somename_1.00_Catalog.xml or, if that already exists, somename_1.01_Catalog.xml and so on. You can manually feed the update screen that file if the “big one” is too massive.

Wrapping it up

In this article we’ve:

• Shown you the base install for the VM we use as well as the location we downloaded the Dell tool from. We installed in GUI mode, but haven’t looked back since. (Trying to update the tool seems to want to be done in GUI mode, to be clear).
• Shown you how to build a mega-catalog for all your installed machines. You could also build a single catalog for each of a Dell “generation” (i.e. R340, 440, 740 are all 14th generation servers), or just for an individual model.
• Shown you how to build a catalog for a single machine by letting the iDRAC probe the hardware.
• Shown you how to update that catalog, and re-extract it.
• Shown you how to use both the iDRAC web UI as well as the SSH racadm command to grab and apply those updates.

Exercises for the reader might be:

• Periodically using cron to download updates for given systems, and rewrite the Catalog.xml symlink (we do it manually).
• Find a way to script logins to your Dell machines (say, with an SSH key) to run an update check, and then provide a report, by parsing the output. This is something else we do manually, but in a fleet of, say, thousands of machines we would script this.

If you have any questions, feel free to leave a note here, or contact me at medium[at]gushi[dot]org! Happy updating!