[+] Introduction

Everyone is probably tired of me yapping on about pwning printers, so I decided to write another blog post related to that subject.

After my brief stint with lexmark at pwn2own in late 2022 I decided to park the Lexmark printer for good and move on to other adventures. Until.. I got a message from Rick telling me lexmark pushed an update on the day of the pwn2own 2023 signup deadline where they changed the firmware update encryption and added firmware rollback prevention.

Rick’s (or rather, PHP HOOLIGANS’) pwn2own entry was in shambles due to changes in the affected binaries/libraries and they needed a last minute miracle to revive it. My lexmark printer was still on the older firmware. He kindly provided me with a persistent backdoor that could be installed prior to applying the update. Luckily the backdoor actually properly persisted after the update and didn’t result in a brick. Great! I could pull the binaries and libraries off of the device and PHP HOOLIGANS managed to successfully compromise the Lexmark once again.

I had a brief look at what Lexmark actually added that prevented us from decrypting the root filesystem like we were previously able to using my tool, but quickly decided I didn’t really care much at that point and moved on again.

In March 2024 I gave a talk at NULLcon berlin called “Printer Hacking Adventures” in which I briefly mentioned this new firmware encryption and how it had not been broken so far, but that it would be broken soon™. I was bluffing very much at this point, though I knew it would probably be a walk in the park given the right mindset and some dedication.

Fast forward again to July 2024, I finally somehow mustered up the motivation to have a stab at this. A month or so later I was asked if I was interested in giving a talk at SAS24 and I decided this might be a fun subject. The SAS24 talk is only 20 minutes long.. so here we are with this blogpost (and a follow-up blogpost) that are a little more indepth.

[+] Lexmark firmware encryption, the old way

When you want to reverse engineer the software of some closed source proprietary device you generally have two options: dump the firmware from the flash storage of the actual device, or unpack the firmware from a firmware update file. The latter obviously being the more favorable option since it’s less (potentially) destructive and easier to reproduce for newer firmwares. Typically you run into a chicken-egg situation though with the latter option since vendors like to obfuscate and/or encrypt their firmware update files to prevent people from being able to stare at the code easily. Luckily for me, due to prior research on lexmark printers by others (1,2) I’ve never had to get rid of a single screw in my lexmark printer. My printer shipped with an older firmware and I was able to get a root shell on it by reproducing a vulnerability documented by Crowdstrike.

Using this root shell I was able to pull off the entire root filesystem and started analyzing the firmware update mechanism in the hopes of being able to unpack firmware files “offline”.

Lexmark printer firmware update files are shipped as big blobs with an .FLS extension. Contained therein you will find a brief stanza that is actually some ASCII text containing a PJL (Printer Job Language) stream that tells the printer we’re about to update its firmware, followed by a big blob of binary data containing the actual update data.

$ head -n3 fw/CXLBL.230.037.fls
-12345X@PJL
@PJL COMMENT NETFLASH ID="CXLBL,CXLBN,CSLBN" RIP="230.037" ENG="BL.230.E038" IOT="1.1.26+git0+db74e4f338" DATE="20231011"
@PJL LPROGRAMRIP SOCKET=1 KERNELCOUNT=9131440 TYPECOUNT=152106320 KERNELENCR=3 FKSIGNSZ=9129888 FLASHOPTS="AF=1;INF=1;MV=2;TV=1;LJF=1;MD=1;CV=230.037;NUH=1;SMA=1;SV=2.0;IS=1" RIPNAME="granite2-color-lite"

The big blob of binary data has a high entropy and appears to be obfuscated/encrypted in some way. All of this was already documented by Claroty in 2020.. which I obviously found out when I had already reverse engineered most of it myself. Claroty did not dare to post any key material though, so some retracing of effort was justified no matter what.

The next section will be a brief summary of the format of the encrypted container, rather than going too deep into the actual reversing of /usr/bin/hydra (the big daemon on lexmark printers responsible for a lot of things, amongst which is firmware updates).

The encrypted data consists out of a kernel and a data region. Both of these regions start with a 0x128 bytes RSA encrypted header (PKCS padded).

These headers are followed by:

• a RSA encrypted signature
• a RSA encrypted AES key
• the AES encrypted data

The RSA public keys needed for the RSA decrypt operation are hardcoded inside of /usr/bin/hydra. The AES-CBC IV is derived by mangling the AES key with a rolling XOR operation that is seeded with the constant 0x49. If you are interested in the actual implementation, you can review the code of the utility I wrote for decrypting these blobs.

We’re actually only interested in the data blob. Let’s use an older firmware as an example, we’ll use CXLBL.076.308. After decrypting the data.bin blob we can use unblob to list any chunks:

$ poetry run unblob -s -v ./input/CXLBL.076.308_data.bin \
    | grep 'Found valid chunk' | awk -F 'chunk=' '{ print $2 }'

chunk=0x8bb14-0x1703cb handler=gzip pid=22302
chunk=0x175bdc-0xd0cbdc handler=cramfs pid=22302
chunk=0xd0cdf4-0x7d87df4 handler=squashfs_v4_le pid=22302
chunk=0x7e70e14-0x80dae14 handler=squashfs_v4_le pid=22302

We can spot two SquashFS filesystems that have been identified, extracting them using unblob gives us the following file trees:

$ ls -la output/CXLBL.076.308_data.bin_extract/132582932-135114260.squashfs_v4_le_extract
total 24
drwxrwxr-x 6 root root 4096 Sep 25 09:46 .
drwxr-xr-x 6 root root 4096 Sep 25 09:46 ..
drwxrwxr-x 2 root root 4096 Mar  1  2022 images
drwxrwxr-x 2 root root 4096 Sep 25 09:46 opt
drwxrwxr-x 4 root root 4096 Mar  1  2022 thineng
drwxrwxr-x 5 root root 4096 Mar  1  2022 thineng-bluering

$ ls -la output/CXLBL.076.308_data.bin_extract/13684212-131628532.squashfs_v4_le_extract
total 92
drwxrwxr-x 22 root root 4096 Sep 25 09:46 .
drwxr-xr-x  6 root root 4096 Sep 25 09:46 ..
drwxrwxr-x  2 root root 4096 Sep 25 09:46 bin
drwxrwxr-x  2 root root 4096 Mar  1  2022 boot
drwxrwxr-x  2 root root 4096 Mar 11  2021 dev
drwxrwxr-x  2 root root 4096 Nov 16  2021 .devtool
drwxrwxr-x 53 root root 4096 Sep 25 09:46 etc

[ .. ]

drwxrwxr-x  6 root root 4096 Sep 25 09:46 home
drwxrwxr-x 13 root root 4096 Sep 25 09:46 var
lrwxrwxrwx  1 root root   13 Sep 25 09:46 web -> usr/share/web
-rw-r--r--  1 root root  980 Mar  1  2022 Build.Info

One contains some boring data assets, and the other one in fact contains the root file system. Great! That’s all that was needed to start your vulnerability research journey on old Lexmark printers.

[+] Lexmark firmware encryption, the new way

The first firmware to introduce this new encryption was CXLBL.230.037. Let’s start by unpeeling the outer layers with the existing tooling:

$ python3 fw_decrypt.py CXLBL.230.037.fls
> SECTION HEADER:
  - decrypted signature size : 0x00000014
  - decrypted aes key size   : 0x00000010
  - aes key byte size        : 0x00000010
  - aes mode                 : 0x00000001
  - decrypted data size      : 0x008b51b0

> signature : 8628db0a74e88d66c2d3d0cc5ae7f42abe5fd99f
> AES key   : fd9844b269f9de4620b46bc19f2f2786
unpacking: .... [...] ... done! (0x8b5707)

> SECTION HEADER:
  - decrypted signature size : 0x00000014
  - decrypted aes key size   : 0x00000010
  - aes key byte size        : 0x00000010
  - aes mode                 : 0x00000001
  - decrypted data size      : 0x0910f150

> signature : 1129b982bf0b3f7e3a7a4caceac7d2cfad896b3c
> AES key   : a559bb2efd4f5dd2dfa104ee9ad3d05c
unpacking: .... [...] ... done! (0x99c4c57)

$ ls -la kernel.bin data.bin
-rw-r--r--  1 user  staff  152105296 Sep 25 09:20 data.bin
-rw-r--r--  1 user  staff    9130416 Sep 25 09:20 kernel.bin

$ file kernel.bin data.bin
kernel.bin: POSIX shell script executable (binary data)
data.bin:   data

So far, so good. we’re still able to get some data out of this. Let’s peep at unblob again:

$ poetry run unblob -s -v ./input/CXLBL.230.307_data.bin \
    | grep 'Found valid chunk' | awk -F 'chunk=' '{ print $2 }'

chunk=0x9fb44-0x18d88a handler=gzip pid=22423
chunk=0x192e9c-0xcd9e9c handler=cramfs pid=22423

Oh no, where did our precious SquashFS filesystem(s) go?

Examining the ramdisk (CramFS) filesystem for the new firmware we find stuff that belongs to the U-boot bootloader environment setup in the ucmdline file. An interesting tidbit here is the bootargs that are passed to the linux kernel.

setenv bootargs "$bootargs dm-mod.create=\"${VERITY_DEVNAME},CRYPT-VERITY-${rootuuid}-${VERITY_DEVNAME},0,ro,0 ${sectors} verity 0 /dev/${VERITY_DEV} /dev/${VERITY_DEV} 4096 4096 ${blocks} ${hash_offset} sha256 ${roothash} - ;crypt_root,,1,ro,0 ${sectors} crypt aes-cbc-plain64be:plain :32:user:wtm:rootfs 0 /dev/dm-0 0\" root=/dev/dm-1"

Using the module parameter dm-mod.create it is possible to configure a device mapper device at boot that holds your root filesystem (for example). Infact, this one here appears to configure two device mapper devices. The format of this module parameter is quite convoluted and hard to follow, let’s consult the documentation for a bit and reformat this kernel parameter argument value list a bit to be more readible:

First device mapper entry:

Second device mapper entry:

The first entry relates to a configuration of verity that is used for integrity checking of a block device. The second one describes a crypt target for a device mapper entry called “crypt_root”, this looks like what we’re interested in!

If we want to break down the parameters passed to this crypt target we’ll have to consult the documentation for dm-crypt for the order of the arguments, which gives us:

The documentation also tells us this about the key parameter:

Key used for encryption. It is encoded either as a hexadecimal number or it can be passed as <key_string> prefixed with single colon character (‘:’) for keys residing in kernel keyring service. You can only use key sizes that are valid for the selected cipher in combination with the selected iv mode. Note that for some iv modes the key string can contain additional keys (for example IV seed) so the key contains more parts concatenated into a single string.

I’ve highlighted in bold a relevant tidbit here, as our key value is prefixed with a colon, we are infact dealing with a key that is managed by the ‘kernel keyring service’. If you want to learn more about this subsystem of the linux kernel I once again refer you to the documentation on kernel.org.

Curious, so where does this key come from? Who is registering it with the kernel? And what is this ‘wtm’ acronym? More and more questions, we’ll have to go deeper!

[+] What is a WTM?

If we cruise a bit through the CramFS filesystem containing the U-boot stuff, looking for references to ‘wtm’ we find the following files, contained inside the initramFS that is stored inside uInitramfs:

$ (cd uInitramfs_unpacked ; find . | grep wtm)
/lib/modules/5.4.254-yocto-standard/extra/drivers/wtm-client
/lib/modules/5.4.254-yocto-standard/extra/drivers/wtm-client/wtm-client.ko
/lib/modules/5.4.254-yocto-standard/extra/drivers/wtm-controller
/lib/modules/5.4.254-yocto-standard/extra/drivers/wtm-controller/wtm-controller.ko
/usr/share/wtm-crypt
/usr/share/wtm-crypt/wkey2.bin
/usr/share/wtm-crypt/wkey4.bin
/usr/share/wtm-crypt/wkey3.bin
/usr/share/wtm-crypt/wkey1.bin

Oh wow, key blobs! It can’t be that easy, right? Yeah it’s not. There’s also some kernel modules related to this ‘wtm’ thing, what do they do?

There’s also references to ‘wtm’ inside the /sbin/init binary contained in this ramdisk:

$ strings uInitramfs_unpacked/sbin/init | grep wtm
/usr/src/debug/libstd-rs/1.66.0-r0/rustc-1.66.0-src/library/core/src/str/pattern.rs src/main.rs/lib/modulespath to strextradrivers.komodule paramsinit_module/dev/ubiblock0_2no dm-mod.create/dev/mmcblkp/mnt/bin/etc/lib/usr/varproc/sys/devrootinitBasecstrcramfsmount boot/mnt/rootfs.keyno rootfs.keyread keyno indexno flagsno tabledevtmpfssquashfsumount bootwtm-controllerload wtm-controllerwtm-clientload wtm-clientkeyringwtm:rootfsadd_keydm-mod.createno nameno uuidindex not intstart not intno sectorssectors not intno typedm error /sbin/procmount procsyssysfsmount sysdevmount dev/proc/cmdline
cannot access a Thread Local Storage value during or after destruction/usr/src/debug/libstd-rs/1.66.0-r0/rustc-1.66.0-src/library/std/src/thread/local.rs()GenlBufferDeserializing data type neli::types/usr/src/debug/ramdisk-init/0.1.0.AUTOINC+8743cb507e-r0/cargo_home/bitbake/neli-0.6.4/src/types.rsDeserializing field type Buffer to be deserialized: Field deserialized: socketsrc/wtm.rswtm-mailboxresolve nlnlattr
recvwkey.binnetlink sendget payloadNo message responsecalled `Option::unwrap()` on a `None` value/sys/firmware/devicetree/base/upc@f9808000/upc,wkeyopen upc,wkeyread upc,wkey/usr/share/wtm-cryptopen key

.. which reveals their init was written in rustlang, not uncommon in lexmark land (they also have their own IPC broker written in rustlang, for example).

So without having done any actual reverse engineering yet (and by applying some good old common sense) we can reasonably conclude their init program is registering this wtm key with the kernel, probably by interfacing with the wtm related kernel modules, somehow.

If we inspect the metadata of one of the kernel modules we see:

$ modinfo wtm-client/wtm-client.ko
filename:       wtm-client.ko
alias:          net-pf-16-proto-16-family-wtm-mailbox
description:    WTM Mailbox Client
license:        Proprietary
alias:          of:N*T*Cmarvell,wtm-mailbox-clientC*
alias:          of:N*T*Cmarvell,wtm-mailbox-client
depends:
name:           wtm_client
vermagic:       5.4.254-yocto-standard SMP preempt mod_unload ARMv7 p2v8

The module has an alias that includes the string marvell, which happens to be the vendor of the SoC used in these lexmark printers. Maybe WTM is some existing marvell thing?

If we have a peep at /proc/cpuinfo on out rooted printer we see:

root@ET788C77F816DD:~# grep '^Hardware' /proc/cpuinfo
Hardware	: Marvell Pegmatite (Device Tree)

Googling for marvell pegmatite gives close to zero results, except for some lexmark printer bootlog that is part of a writeup by NCC group. :)

Googling for marvell wtm and sifting through some noise we start to find some interesting information though. For example this FIPS 140-2 security policy which contains the following introduction:

The PXA-2128 and PXA-610 SoCs are equipped with a dedicated security hardware module known as WTM (Wireless Trusted Module) that offers the trusted computing services required for user authentication, identity management, secure storage as well as secure communication. Within WTM, there is a pool of the hardware cryptographic engines that performs at high throughput of the cryptographic operations over a set of FIPS-Approved algorithms, such as AES, TDES, SHA, HMAC, RSA, and EC-DSA. In addition, the on-chip hardware entropy-bit-generator under WTM is a reliable source of the entropy seeding to the FIPS-Approved DRBG schemes. The dedicated WTM secure firmware is responsible for device trusted boot, access control, authentication, and key management.

Bingo, looks like this is our WTM. The “Wireless Trusted Module” that is (apparently) part of some Marvell SoC’s and can be used for crypto stuff, secure boot stuff & key management stuff.

[+] More WTM OSINT

Some more searching for this WTM leads us to some patent on ‘Trusted modular firmware update using digital certificate’ that was originally filed by Marvell back in 2008. Patents being patents it’s very verbose yet still very vague and contains some functional diagrams and a lot of word soup. The original intent for this module was to work over-the-air, that’s where the “Wireless” in WTM comes from, I think.

Another interesting tidbit of information is this kernel driver that was originally written for the One Laptop Per Child project (OLPC). OLPC was this non-profit initiative that aimed to bring affordable laptop computers to the developing world. Earlier OLPC laptops were based on AMD Geode (x86), but later models (XO-1.75 for example) featured a Marvell Armada PXA2128 SoC.

So back to this kernel driver, OLPC’s certainly did not implement any kind of secure boot, so how is it related to Marvell WTM? This is where it gets hilarious (in my opinion, anyway); let’s look at this code:

/*
 * The OLPC XO-1.75 and XO-4 laptops do not have a hardware PS/2 controller.
 * Instead, the OLPC firmware runs a bit-banging PS/2 implementation on an
 * otherwise-unused slow processor which is included in the Marvell MMP2/MMP3
 * SoC, known as the "Security Processor" (SP) or "Wireless Trusted Module"
 * (WTM). This firmware then reports its results via the WTM registers,
 * which we read from the Application Processor (AP, i.e. main CPU) in this
 * driver.
 *
 * On the hardware side we have a PS/2 mouse and an AT keyboard, the data
 * is multiplexed through this system. We create a serio port for each one,
 * and demultiplex the data accordingly.
 */

So the super secure processor is being used to.. bitbang PS/2, to give these cute little laptops access to their keyboard and mouse. :)

This means though that at least some people on the OLPC software engineering team had access to some documentation/information on the Marvell WTM, to implement their PS/2 bitbanging firmware. And since OLPC is an OSS initiative, we are able to browse their public git repositories hosted on dev.laptop.org.

There we find the cforth repository which is described as:

Compact Forth interpreter written in C. Used for booting from the Security Processor on OLPC’s ARM systems.

Hey, that’s cool! Maybe this repository can learn us something about how the WTM is programmed, since we don’t have an actual manual for the thing. Indeed, the repository contains quite some code related to interfacing with the various peripherals the WTM has access to. (albeit a lot of it is implemented as actual forth code rather than C or assembly).

From the Makefiles in this repository we can tell the cross-compiler they invoke to build this forth interpreter is a bog standard arm-gcc toolchain. So, the WTM is in fact an ARM core! (which is not a super big surprise, but it’s still nice to have some confirmation)

[+] WTM on Lexmark printers

Okay that was a lovely detour, but we need to get back to cracking this new encryption thingie. If you paid attention to the introductionary rant of this post you might remember I had this persistent backdoor installed on my lexmark printer. So we do infact already have arbitrary code execution- and exploration-capabilities (at least on the main application processor, AP from here on) in the form of a shell on the printer which is running a firmware that is infact using this new root filesystem encryption.

Supposedly there is some way for the AP to talk to the WTM. It would be nice if we could start talking to the WTM from the AP ourselves. But first we need to learn about the interface that allows us to do this. We have two options at this point:

• reverse rustlang init binary
• reverse non-rustlang kernel modules (wtm-client.ko/wtm-controller.ko)

Obviously the latter option is better for our sanity and probably has a faster turnaround. To my surprise (and above all my relief) these kernel modules have their symbols intact. both the kernel modules make heavy use of the common mailbox framework that is part of the Linux kernel API’s. wtm-client talks over mbox to wtm-controller. wtm-controller in turn pokes some MMIO to talk to the WTM, and relays back any responses (from the same MMIO space) to wtm-client over mbox.

So how does userland talk to wtm-client? netlink sockets.

[+] wtm-client.ko

Doing some light reversing of wtm-client.ko we learn it accepts various netlink messages to accomplish things. The command table is limited and looks like this:

0x01: cmd_generate_random
0x02: cmd_hmac_operation
0x03: cmd_key_wrap
0x04: cmd_aes_operation
0x05: cmd_aes_operation

Under the hood, these netlink commands translate into actual commands the WTM firmware understands, and that are submitted through the WTM MMIO command interface, with the help of wtm-controller.ko. The lower level WTM commands we find used by wtm-client.ko are:

0x3007: key_wrap
0x3008: key_unwrap_load
0x4000: generate_random
0x5000: aes_init
0x5001: aes_zeroize
0x7004: hmac_init
0x7006: hmac_update
0x7007: hmac_final

Some juicy ones that are likely relevant to our interests in there. Of course, this coming from the wtm-client code, it doesn’t tell the full story. There might be more commands? We also know nothing about the actual key loading process yet, as we haven’t bothered to actually reverse engineer that dreaded init binary written in rustlang.

[+] The Lexmark WTM firmware

During this whole excercise I had been wondering (and ignoring at the same time) one thing: where does the WTM firmware code running on the lexmark come from anyway? It probably wasn’t always there and the WTM has to be bootstrapped at some point during the bootchain..

When stunt-hacking at high speed and with little care you sometimes miss obvious details. Like this gzip’d blob somewhere in the data region of the printer firmware update which contains a small CPIO archive containing another tiny filesystem:

cpio_unpacked$ find . -type f
./etc/version
./etc/timestamp
./lib/modules/5.4.90/modules.alias
./lib/modules/5.4.90/modules.devname
./lib/modules/5.4.90/modules.dep
./lib/modules/5.4.90/modules.order
./lib/modules/5.4.90/modules.dep.bin
./lib/modules/5.4.90/modules.symbols.bin
./lib/modules/5.4.90/extra/wtm-linux.ko
./lib/modules/5.4.90/modules.builtin.bin
./lib/modules/5.4.90/modules.builtin
./lib/modules/5.4.90/modules.symbols
./lib/modules/5.4.90/modules.alias.bin
./lib/modules/5.4.90/modules.softdep
./lib/modules/5.4.90/modules.builtin.modinfo
./init
./bin/sh
./dev/console

cpio_unpacked$ wc -c init
0 init

Interesting, what is that used for? It has another wtm related kernel module as well, called wtm-linux.ko. Also what’s up with the kernel version in those paths? The kernel version string for the other stuff we saw is 5.4.254-yocto-standard, that doesn’t match 5.4.90. then it hit me: the WTM on Lexmark printers is running a full linux kernel! I was expecting something teensy and more self-contained after seeing the cute approach taken by the OLPC engineers.

So we do infact have easy access to the code running on the WTM. Following our gut instinct we guesstimate the majority of the actual logic is implemented inside of wtm-linux.ko, let’s reverse that one for a bit.

Quickly I identified the command handling logic in the wtm-linux driver and we are now able to reconstruct the full table of commands supported by the WTM firmware:

0x0000: wtm_reset
0x0001: wtm_init
0x0002: wtm_self_test
0x0003: wtm_configure_dma
0x0004: wtm_set_batch_count
0x0005: wtm_ack_batch_error
0x0006: wtm_freq_change
0x1000: wtm_get_trust_status_register
0x1001: wtm_kernel_version_read
0x1002: wtm_software_version_advance
0x1003: wtm_software_version_read
0x1004: wtm_lifecycle_advance
0x1005: wtm_lifecycle_read
0x2000: wtm_oem_platform_bind
0x2001: wtm_oem_platform_verify
0x2002: wtm_oem_jtag_key_bind
0x2003: wtm_oem_jtag_key_verify
0x2004: wtm_otp_write_platform_config
0x2005: wtm_otp_read_platform_config
0x2006: wtm_rkek_provision
0x2007: wtm_set_aes_use_rkek
0x2008: wtm_otp_block_write
0x2009: wtm_otp_block_read
0x200a: wtm_oem_usbid_provision
0x200b: wtm_oem_usbid_read
0x200c: wtm_set_jtag_permanent_disable
0x200d: wtm_set_temporary_fa_disable
0x200e: wtm_device_pin_provision
0x3000: wtm_get_context_info
0x3001: wtm_load_engine_context
0x3002: wtm_store_engine_context
0x3003: wtm_load_engine_context_external
0x3004: wtm_store_engine_context_external
0x3005: wtm_purge_context
0x3006: wtm_get_next_cache_slot_id
0x3007: wtm_key_wrap
0x3008: wtm_key_unwrap_load
0x4000: wtm_drbg_gen_ran_bits
0x4001: wtm_x931_drbg
0x5000: wtm_aes_init
0x5001: wtm_aes_zeroize
0x5002: wtm_aes_process
0x5003: wtm_aes_finish
0x6000: wtm_des_init
0x6001: wtm_des_zeroize
0x6002: wtm_des_process
0x6003: wtm_des_finish
0x7000: wtm_hash_init
0x7001: wtm_hash_zeroize
0x7002: wtm_hash_update
0x7003: wtm_hash_final
0x7004: wtm_hmac_init
0x7005: wtm_hmac_zeroize
0x7006: wtm_hmac_update
0x7007: wtm_hmac_final
0x8000: wtm_emsa_pkcs1_v15_verify
0x8001: wtm_emsa_pkcs1_v15_verify_init
0x8002: wtm_emsa_pkcs1_v15_verify_update
0x8003: wtm_emsa_pkcs1_v15_verify_final
0x8004: wtm_emsa_pkcs1_v15_zeroize
0x8005: wtm_emsa_pkcs1_v15_sign
0x8006: wtm_emsa_pkcs1_v15_sign_init
0x8007: wtm_emsa_pkcs1_v15_sign_update
0x8008: wtm_emsa_pkcs1_v15_sign_final
0x9000: wtm_dh_public_key_derive
0x9001: wtm_dh_shared_key_gen

That is quite a bit more than what is indirectly exposed by the wtm-client driver we were looking at earlier. And we can see the actual implementation of these commands on the WTM side now, great!

By looking at some devicetree related strings in the kernel image for the kernel running on the WTM we learn the platform is called mckinley5 internally. So we have pegmatite (the main application processor) talking to mckinley5 (the WTM).

[+] Avoiding rustlang RE take 2

Ok, let’s not deviate too much (for now!) from our original goal, getting the key(s) required to decrypt the root filesystem for pegmatite. We know the init process on pegmatite talks to the wtm-client driver over netlink. Netlink sockets are.. just regular sockets, with their own address family AF_NETLINK. So I decided it might be fun to carefully patch the init binary and make it use a regular AF_INET socket to relay all the netlink traffic over good ol’ TCP. This allows us to run the init binary in something like qemu-arm and easily see/respond to the netlink messages, without having to deal too much with rustlang. The other benefit of emulating the init process instead of directly executing it on a rooted printer that has already gone through the init proces is we can tread more carefully, we don’t want to end up with a brick or having to take out a screwdriver.

Let’s look at the patches I created for the init binary:

We’ll no-op the mount/umount operations, as we don’t actually want any of those to happen. Another no-op patch is introduced in this small routine that calls SYS_init_module to load the wtm related kernel modules, since we don’t actually want to load those.

At some point the init binary will parse the kernel cmdline by open-ing /proc/cmdline, we’ll patch this string to be /prac/cmdline, so we can easily substitute it with a fake kernel cmdline that lives in /prac inside of our chroot.

Patching socket to always return 3 doesn’t magically give us a valid file descriptor 3, of course. We could inject a small hook that sets up the actual TCP socket, but we’re lazy so we will build a ‘preinit’ instead:

#include <stdio.h>
#include <arpa/inet.h>
#include <sys/socket.h>
#include <unistd.h>

#define PORT 3232

int main() {
    struct sockaddr_in servaddr, cli;
    int sockfd = socket(AF_INET, SOCK_STREAM, 0);
    if (sockfd == -1) {
        return -1;
    }

    servaddr.sin_family = AF_INET;
    servaddr.sin_addr.s_addr = inet_addr("127.0.0.1");
    servaddr.sin_port = htons(PORT);

    if (connect(sockfd, (struct sockaddr*)&servaddr, sizeof(servaddr)) != 0) {
        return -1;
    }

    char *a_argv[]={ "init", NULL };
    execve("/init", a_argv, NULL);
}

This small shim will set up the TCP connection for us, and never close the socket file descriptor (3) before execve-ing into the actual init binary. So now when the real init runs the nescessary fd 3 will be present.

We can now build a highlevel TCP daemon in python that pretends to be the kernel-side of the netlink socket and emulate the patched init binary in a chroot’ed qemu-arm to get a better idea of the actual netlink messages and (expected) responses, without having to worry about trying to make sense of rustlang madness in our favorite disassembler/decompiler. :)

[+] Netlink

Netlink messages start with a fixed size (0x10 bytes) header that looks like this:

struct nlmsghdr {
      __u32   nlmsg_len;      /* Length of message including headers */
      __u16   nlmsg_type;     /* Generic Netlink Family (subsystem) ID */
      __u16   nlmsg_flags;    /* Flags - request or dump */
      __u32   nlmsg_seq;      /* Sequence number */
      __u32   nlmsg_pid;      /* Port ID, set to 0 */
};

Typically, the nlmsg_type is either 1, 2, 3 or 4. However the first netlink message we get from the init process has a nlmsg_type of 0x10:

00000000: 2400 0000 1000 0500 0000 0000 0000 0000  $...............
00000010: 0302 0000 1000 0200 7774 6d2d 6d61 696c  ........wtm-mail
00000020: 626f 7800                                box.

If we consult /usr/include/linux/netlink.h we find:

#define NLMSG_MIN_TYPE          0x10    /* < 0x10: reserved control messages */

So apparently you can come up with your own control message types as well, as long as they are >= 0x10. Let’s ignore netlink for the most part, and simply send the exact same netlink packet to the actual wtm-client module on the printer, we get these two response packets:

00000000: 5c00 0000 1000 0000 0000 0000 a403 0000  \...............
00000010: 0102 0000 1000 0200 7774 6d2d 6d61 696c  ........wtm-mail
00000020: 626f 7800 0600 0100 1700 0000 0800 0300  box.............
00000030: 0100 0000 0800 0400 0000 0000 0800 0500  ................
00000040: 0200 0000 1800 0600 1400 0100 0800 0100  ................
00000050: 0100 0000 0800 0200 0a00 0000            ............

00000000: 2400 0000 0200 0001 0000 0000 a504 0000  $...............
00000010: 0000 0000 2400 0000 1000 0500 0000 0000  ....$...........
00000020: 0000 0000                                ....

This appears to be part of some fairly boring init/handshake stanza, so let’s just copy paste the packets into our netlink emulator daemon for now.

Next, we get a fairly large netlink packet from the init process:

00000000: 8c02 0000 1700 0100 0000 0000 0000 0000  ................
00000010: 0101 0000 7502 0200 0506 0101 2000 0000  ....u....... ...
00000020: 0105 8000 0008 0000 0069 4e38 d798 91f0  .........iN8....

                [ .. 8< snip snip 8< ... ]

00000220: 13fe a4d1 d064 252c 88ff f5fe a1ad b444  .....d%,.......D
00000230: 76dc 9576 842c cc5b 3398 7403 7772 f5c3  v..v.,.[3.t.wr..
00000240: c858 72e6 2d2a 2296 a244 39f3 cefe 17be  .Xr.-*"..D9.....
00000250: 17d6 3a48 1393 2e01 d24f 96d1 4276 fd7f  ..:H.....O..Bv..
00000260: 7987 bb8d cef1 0228 d88b aa6b 9d88 5d8b  y......(...k..].
00000270: ad5f 4d89 3f3c ed0b bc05 661d 55ba 7549  ._M.?<....f.U.uI
00000280: 8071 ae5e 1f55 fc9f 7b00 0000            .q.^.U..{...

That’s quite a big message, and where does all the high entropy data come from? It looks to be about ~0x200+ bytes in size (ignoring any netlink header/framing stuff). If we follow the emulated init process with strace(1) we can see these two open calls:

openat(AT_FDCWD, "/mnt/rootfs.key", O_RDONLY|O_CLOEXEC) = 6
openat(AT_FDCWD, "/usr/share/wtm-crypt/wkey4.bin", O_RDONLY|O_CLOEXEC) = 7

And by staring at the hex of the big netlink message and matching it with the contents of those files we can establish that the message format is :

> netlink header bullshit
> content of wkey4.bin
> last 0x20 bytes of rootfs.key
> first 0x10 bytes of rootfs.key
> 3 bytes of padding

if we also replay this message against wtm-client we get the following response:

000000: 40 00 00 00 17 00 00 00 01 00 00 00 00 00 00 00  @...............
000010: 01 01 00 00 06 00 01 00 00 00 00 00 24 00 02 00  ............$...
000020: 79 30 68 20 64 30 30 64 2c 20 67 30 20 64 75 6d  =..iS!...m.$...v
000030: 70 20 6a 30 30 72 20 30 77 6e 20 6b 33 79 7a 21  #..]c.16..;.5...

Could it be that simple? Turns out it is, those 0x20 high entropy bytes are infact the AES-256 key that is used for crypting the root filesystem! And we managed to snag it by .. simply replaying some netlink messages on a rooted printer. This means we can turn our rooted printer into an oracle for unwrapping these root filesystem keys. And that’s exactly what I implemented.

$ python3 rootfs_decrypt.py ./CXLBL.230.037 192.168.0.13
> wrapped rootfs key   : 05661d55ba [..] 93f3ced0bbc
> unwrapped rootfs key : -- 8< snip snip 8< --
100%|███████████████████████| 270736/270736 [00:02<00:00, 132357.71it/s]
> decrypted rootfs written to ./CXLBL.230.037/main/content_rootfs_dec.bin

$ file CXLBL.230.037/main/content_rootfs_dec.bin
CXLBL.230.037/main/content_rootfs_dec.bin: Squashfs filesystem, little endian, version 4.0, xz compressed, 137513245 bytes, 14547 inodes, blocksize: 131072 bytes, created: Wed Oct 11 18:15:44 2023

Pretty weird, they could easily have added some mechanism on the WTM side that prevents the key unwrap mechanism from being invoked repeatedly. After all the rootfs key only needs to be derived once. But judging by the WTM firmware I stared at, it wasn’t written by lexmark in the first place – it appears to be some general purpose WTM firmware developed by Marvell with all kinds of bells and whistles.

Kind of anti-climatic, right? That’s why in the follow-up blogpost we’ll do some more exploration of the WTM. We should figure out a way to issue arbitrary commands to the WTM, and maybe we can even get arbitrary code execution on it? :-)

[+] Closing Words

I hope you enjoyed the read. Vendors playing cat & mouse to frustrate security researchers is nothing new. Bolting on additional obfuscation on devices that have already been pwned is not likely to make a big difference or cause a big slow down. If you want to learn more about the WTM, make sure to read the follow up post “Let’s PWN WTM!” as well!