Have you ever looked at the backplane of a server? You know, the kind that’s meant to go into a rack in a datacenter? They’ll usually have more than one ethernet port, but oddly, a lot of the time one of those ethernet ports is gonna be off to the side, over with the USB ports instead of with the other ones.
That’s your server’s IPMI interface, and it can do a lot.
What is IPMI?
IPMI stands for Intelligent Platform Management Interface. It’s a standard for datacenter and server administrators to enable out of band management of their servers that was first standardized in 1998. Nowadays, basically every vendor supports it or has something like it. Dell calls theirs DRAC or iDRAC, HP has ILO, and so on. Basically, it’s a built-in back door to your server, one that’s explicitly designed into the board and that people want to be there. It consists of a main controller called the Baseboard Management Controller (BMC) that handles all the heavy lifting, assisted by other sub-controllers distributed amongst the various modules and parts of the motherboard that are referred to as "satellite controllers". All of these controllers communicate via the Intelligent Platform Management Bus (IPMB), which is basically souped-up I2C. From the perspective of the datacenter administrator, who has to manage hundreds or thousands of computers, IPMI provides some pretty sweet advantages. You can power-cycle all your servers safely, check out consoles at will, collect detailed sensor information and more, all from the comfort of your computer chair and without having to get up and pull boxes out of cabs. The IPMI standard even includes a wire protocol called "Remote Management Control Protocol", which is used for programmatic management on the LAN. As a further convenience, the vendor may provide a web or BIOS/UEFI-based management console so that the IPMI can be configured and managed without needing custom tools.
In short, IPMI is a powerful tool in the operations arsenal that can help make remote management of large numbers of servers extremely simple and removes the need to run physical crash carts around on your datacenter floors. It’s available on basically every server-grade motherboard you can buy these days.
What can IPMI do for Datto?
Here at Datto, we strive to ensure every part of our product is safe, secure, and easily manageable. To that end, we saw a lot of advantages to having IPMI web interface available remotely for our customers. For one, if one of our on-premise appliances had some sort of issue that caused it to lose internet access from the OS, having access to IPMI would be a godsend for troubleshooting any potential issues. We could also trigger a reboot remotely through the web interface if needed, or access it remotely to do things like mount and boot from a rescue ISO. Having these options available aids our support team, and providing those capabilities to our customers means that they can self-service in more situations, which is a win-win all around.
While we were messing around with our boards and thinking up ways to expose IPMI remotely, we had a lot of fun poking around our IPMI firmware images and inspecting running IPMI on our boards. This blog post is a distillation of what we discovered about the particulars of interacting with and dissecting IPMI software. It’s split into two sections: how to see what a running motherboard’s IPMI is doing, and how to dig into IPMI firmware images to see how the sausage is being made.
Inspecting a Physical IPMI
As the IPMI is entirely an embedded system, the only reliable method of access you have is via its serial console, which is typically exposed via a standard 4-pin UART interface somewhere on the main motherboard. As an example, these are the IPMI UART pins for this board.
The very first thing you should do when presented with a new board is to find the serial console pins. This is largely a matter of heuristics, experience, and guessing. Some of the more helpful heuristics we’ve found are:
- The pins may be labeled “DEBUG” or “IPMI” or similar
- They will be in a straight line next to each other, with Tx and Rx next to each other, and ground at one of the ends
- Having a multimeter in continuity mode lying around is handy, as it’ll help identify that pattern of pin layout: 2 in a row that beep, one that doesn’t
- The pins will generally be near the main IPMI processor, so try to look around there.
Some boards may have more than one interface that matches these heuristics, so it’s up to you to figure out which one is for the IPMI console and which is for the fan controller or something. Once you have a candidate, the next step is to figure out which pin is which. Here’s the UART interface from above with TX (green), RX (white), and ground (black) connected in that order.
The next question is what do you plug these pins into? In the olden days we would be plugging these directly into a serial port on the computer, but it is 2019 and nobody has those on their laptops anymore. So we use the next best thing: a USB-to-UART adapter. We’ve found these to be reliable, and a search for “USB to UART” adapter on Amazon will give you plenty of other options. From there, you can use GNU Screen, Minicom, or any other serial console emulator to interact with your IPMI.
On most boards, the IPMI will start booting as soon as power is supplied, whether or not the rest of the motherboard is booting or not. So the easiest way to reboot the IPMI is just yanking the power cable and plugging it back in every so often while you probe the pins. If you’re lucky, there will be an LED on the board that lights up to show the IPMI is running.
Eventually you will see something on your serial console emulator that resembles text. You may have been lucky enough to have guessed the correct baud rate, in which case you will see intelligible text fly by. If you are not, try playing with the baud rate until you find one that works. Common baud rates are 96800 and 115200. This is entirely a matter of guesswork, so be patient, you’ll get it eventually.
Once you have access to the serial console, you’ll probably be presented with a login prompt. Hacking passwords is beyond the scope of this post, and every vendor will have a different login, so we’ll leave this as an exercise for the curious reader. Even without console access, however, the text from the boot can tell you a lot about what services are running and if your IPMI is working correctly.
Tearing open the firmware image
Another interesting angle to take is to tear open the firmware image and inspect the files contained therein. You should be able to find firmware images available to download on your vendor’s website. Since the firmware runs Linux, these firmware images tend to be tiny disk images that use standard Linux tools and filesystems. This means you can trivially pull them apart and take a look inside. Each vendor has their own way of doing things, but there are a few classical hallmarks:
- The root partition is going to be a compressed cramfs or squashfs that is mounted read only
- There may or may not be a separate read/write config partition
- It will almost certainly be using busybox to provide common utilities like sh(1)
- The bootloader is very likely u-boot
- The kernel image will most likely be uncompressed
If any of these points do not hold, then this is a cause for some closer examination (did you perhaps download the UEFI image instead?). In general, vendors are not going to give you detailed instructions for how to work with their firmware images, so we’re on our own again. Having said that, there are tools at our disposal to extract the packed firmware images into something we can explore.
Start by using binwalk to take a look at the structure of your image. With luck, you’ll see some partitions extracted and mounted, something like this
$ binwalk -e ./S4P2143_P1.00.00.ima DECIMAL HEXADECIMAL DESCRIPTION -------------------------------------------------------------------------------- 159084 0x26D6C CRC32 polynomial table, little endian 207100 0x328FC CRC32 polynomial table, little endian 393216 0x60000 JFFS2 filesystem, little endian 2490368 0x260000 uImage header, header size: 64 bytes, header CRC: 0xAFD449, created: 2019-04-03 06:26:13, image size: 18472960 bytes, Data Address: 0x0, Entry Point: 0x0, data CRC: 0x9CB8E0F4, OS: Linux, CPU: ARM, image type: RAMDisk Image, compression type: none, image name: "" 2490432 0x260040 CramFS filesystem, little endian, size: 18472960 version 2 sorted_dirs CRC 0xDD99F5F4, edition 0, 11537 blocks, 4017 files 20971584 0x1400040 uImage header, header size: 64 bytes, header CRC: 0xB7530D6C, created: 2019-04-02 06:38:01, image size: 2731032 bytes, Data Address: 0x80008000, Entry Point: 0x80008000, data CRC: 0xA05B2067, OS: Linux, CPU: ARM, image type: OS Kernel Image, compression type: none, image name: "Linux-3.14.17-ami" 20988703 0x140431F gzip compressed data, maximum compression, from Unix, NULL date (1970-01-01 00:00:00) 23789568 0x16B0000 CramFS filesystem, little endian, size: 5398528 version 2 sorted_dirs CRC 0x3E0A9697, edition 0, 1542 blocks, 120 files
Binwalk should have created a folder in the working directory with the extracted partitions and contents for you to peruse.
From here, you can basically start poking around at your leisure. Our firmware images had a lot of fun and interesting things to explore. Every firmware is made differently, however! So what holds true for us may not be for you.
From the binwalk output, we can see that this image uses a minimal cramfs partition as root. Taking a look inside that extracted partition, we see busybox providing the core utilities, and good old sysvinit as PID 1 (/etc/rc5.d, ah the memories!). We can also see that it’s compiled for 32-bit ARM with file(1).
$ ls -al bin/ls lrwxrwxrwx 1 sramanujam sramanujam 7 Aug 22 04:11 bin/ls -> busybox $ file bin/busybox bin/busybox: ELF 32-bit LSB executable, ARM, EABI5 version 1 (SYSV), dynamically linked, interpreter /lib/ld-, for GNU/Linux 2.6.16, stripped
Looking inside /etc, we can see a lot of interesting config files. It seems like this root filesystem was developed on or tested with an older Redhat-based Linux install, because all the startup scripts in /etc/init.d, even the ones for custom programs, have chkconfig(1) comments at the top, even though chkconfig is not present in the rootfs itself:
$ head -8 etc/init.d/cdserver #!/bin/sh # /etc/init.d/cdserver: start CD Server # # chkconfig: 2345 10 90 #Runlevel : 3 = S35 # Restart the service on warm reboot #Runlevel : 9 = K35 #Runlevel : 9 = S35
Curiously, all these init scripts use start-stop-daemon, which comes from Debian. Only on a random embedded IPMI board will you find this mixture of RHEL and Debian conventions in a single place. Weird!
On top of that are several more interesting programs: lighttpd for serving the web interface, Redis (for some reason), and some other binary blobs that are named things like “cdserver” and “fdserver”, clearly meant to handle IPMI functionality like fan control and providing the backend for console redirection.
Configuration files are stored in a JFFS2 partition that is mounted read-write. JFFS2 is a filesystem designed to work with raw flash devices, which makes sense considering the embedded environment this image is meant to run in. It seems that most of the configuration you can do from the web UI just ends up writing configuration files and snippets out to the configuration partition. There’s even a script that handles the factory reset functionality: the configuration on the jffs2 partition is simply overwritten with a read-only copy of the config stored on the root partition.
It’s not straightforward to mount JFFS2 filesystems on Linux, since the driver itself expects to be presented with a raw flash device and can’t use loop devices. However, there is a way!
$ sudo modprobe mtdblock $ sudo modprobe mtdram total_size=32768 erase_size=128 # total_size is the size of your filesystem image in megabytes $ sudo mknod /dev/mtdblock0 b 31 0 # the base name of the block node you create here must begin with “mtd” $ sudo dd if=/path/to/extracted/jffs2.img of=/dev/mtdblock0 $ mount /dev/mtdblock0 /mnt
Now you should be able to see inside:
$ ls -alh /mnt total 62K drwxr-xr-x 9 root root 0 Dec 31 1969 . drwxr-xr-x 8 datto datto 4.0K Aug 29 16:27 .. -rw-r--r-- 1 root root 397 Apr 2 02:40 activedir.conf -rw-r--r-- 1 root root 10 Apr 2 02:32 adjtime -rwxr-xr-x 1 root root 105 Apr 2 02:42 adviserd.conf -rw-r--r-- 1 root root 694 Apr 2 02:40 autorecord.conf drwxr-xr-x 3 root root 0 Apr 3 02:26 BMC1 …
There’s quite a bit of interesting material to dig through here as well. Notable highlights include an (expired?) certificate:
Certificate: Data: Version: 1 (0x0) Serial Number: 17342999179200669294 (0xf0aeb5ddc825a66e) Signature Algorithm: sha256WithRSAEncryption Issuer: C=US, ST=Georgia, L=Norcross, O=AMI, OU=MegaRAC, CN=ami-redfish.com/emailAddressemail@example.com Validity Not Before: Oct 19 14:55:16 2015 GMT Not After : Oct 18 14:55:16 2016 GMT Subject: C=US, ST=Georgia, L=Norcross, O=AMI, OU=MegaRAC, CN=ami-redfish.com/emailAddressfirstname.lastname@example.org
Other fun things to poke at here include the web server configuration and configuration for the various vendor utilities included on the image. From this, you should be able to get a good idea of what’s going on when the image is booted.
As a final curiosity, examination of the firmware images themselves with hexdump(1) shows that around each of the partitions there’s some sort of header that seems to carry information (metadata perhaps?). At the very end of the image there’s some metadata about the image itself. We’re still unsure what that’s about. If anybody knows, please reach out!
In the end, we worked with our vendors to integrate our in-house remote-access client into our servers’ IPMI images to provide remote functionality for our users, rather than doing something more radical on the board itself. We learned a lot about IPMI and how it worked along the way. Once we started digging, we quickly realized IPMI implementations are based on a familiar Linux stack under the hood, and that allowed us to have confidence in our efforts to extend and augment traditional IPMI capabilities to deliver value to our customers.