Add device security verification guide

[tdewey-rpi]

• Introduced a comprehensive guide for verifying the security configuration of provisioned Raspberry Pi devices, covering secure boot enforcement, firmware version checks, device-specific encryption key validation, encryption key usage, and JTAG status verification.
• Included detailed prerequisites, verification procedures, and troubleshooting tips to assist users in ensuring device security compliance.
• Added an automated verification script and API-based verification methods for enhanced usability and efficiency in security checks.

[tdewey-rpi]

Written with the liberal help of Claude - I've took a few review passes at it, but I'm too close to the doc to find more problems now.

[octal-ip]

Thanks for the effort on this one, it's a great start. I've had a few issues with the verification process against a newly provisioned CM4 using rpi-sb-provisioner v2.0.5:

• /var/log/rpi-sb-provisioner/<device_serial>/bootstrap.log doesn't contain any program_pubkey=1 line, however Writing key and EEPROM configuration to the device is present. The log files seem to be missing a lot of content compared to what I can see streaming past in the web UI service logs?
  The same applies when searching provisioner.log for any references to cryptinit, luks and crypt - nothing is found.
  I'm definitely looking at the right log files as I can see them being created/updated during the provisioning process, they just seem to be missing a lot of content.
• Running vcgencmd version on the provisioned device appears to return a hash of the firmware version (e.g. 5560078dcc8591a00f57b9068d13e5544aeef3aa), which isn't comparable to the firmware version name (typically a date, e.g. pieeprom-2025-05-16.bin).
• Running vcgencmd bootloader_config on the provisioned device returns a binary output that isn't human readable.
• Running lsinitramfs /boot/initramfs8 | grep -i key on the provisioned device returns cannot open /boot/initramfs8: No such file errors.
• The documented rpiboot metadata fetch process is incomplete. On the CM4, I had to follow the secure-boot-example process to sign a recovery image, which was then able to return the metadata file. However, the CUSTOMER_KEY_HASH didn't match the hash of my private key PEM generated with the openssl/sha256 command. This is surprising as the CM4 boots the OS and runs as expected.
• journalctl on the provisioned device doesn't have any references to initramfs or luks.

What did work:

• After provisioning with rpi-sb-provisioner v2.0.5 the CM4 boots successfully, and I can verify JTAG is disabled.
• vcgencmd otp_dump | grep "30:" shows a non-zero value that is 8 characters long.
• Unique device keys were created.
• lsblk -f and cryptsetup status cryptroot show the LUKS encrypted partition as expected.

Happy to provide additional information on any of the items above if it'd be helpful.

[stu-spp]

The documented rpiboot metadata fetch process is incomplete. On the CM4, I had to follow the secure-boot-example process to sign a recovery image, which was then able to return the metadata file. However, the CUSTOMER_KEY_HASH didn't match the hash of my private key PEM generated with the openssl/sha256 command. This is surprising as the CM4 boots the OS and runs as expected.

Could you go into a bit more detail about how to do this? I'd like to try to reproduce.

[stu-spp]

Regarding the scaling recommendations documentation that you removed: what is a provisioning station and also what is a provisioning station head? Is that a raspberry pi 5 device running rpi-sb-provisioner?

[octal-ip]

The documented rpiboot metadata fetch process is incomplete. On the CM4, I had to follow the secure-boot-example process to sign a recovery image, which was then able to return the metadata file. However, the CUSTOMER_KEY_HASH didn't match the hash of my private key PEM generated with the openssl/sha256 command. This is surprising as the CM4 boots the OS and runs as expected.

Could you go into a bit more detail about how to do this? I'd like to try to reproduce.

Certainly.
I generated the hash of my private key using the following command:
openssl rsa -in ${CUSTOMER_KEY_FILE_PEM} -pubout -outform DER | sha256sum | awk '{print $1}'
This is the same file referenced in the /etc/rpi-sb-provisioner/config file that was used to provision the device.

I then downloaded and compiled a fresh copy of the secure-boot example:

git clone https://github.com/raspberrypi/usbboot secure-boot
cd secure-boot
git submodule update --init
make

Then I signed the EEPROM image and booted the CM4 from it to generate the metadata file:

cd secure-boot-recovery
../tools/update-pieeprom.sh -k "${CUSTOMER_KEY_FILE_PEM}"
cd ..
./rpiboot -d secure-boot-recovery -j secure-boot-recovery/metadata

The "CUSTOMER_KEY_HASH" value in the metadata json file doesn't match the SHA256 of the private key used to provision the device.

[tdewey-rpi]

The documented rpiboot metadata fetch process is incomplete. On the CM4, I had to follow the secure-boot-example process to sign a recovery image, which was then able to return the metadata file. However, the CUSTOMER_KEY_HASH didn't match the hash of my private key PEM generated with the openssl/sha256 command. This is surprising as the CM4 boots the OS and runs as expected.

Could you go into a bit more detail about how to do this? I'd like to try to reproduce.

Certainly. I generated the hash of my private key using the following command: openssl rsa -in ${CUSTOMER_KEY_FILE_PEM} -pubout -outform DER | sha256sum | awk '{print $1}' This is the same file referenced in the /etc/rpi-sb-provisioner/config file that was used to provision the device.

I then downloaded and compiled a fresh copy of the secure-boot example:

git clone https://github.com/raspberrypi/usbboot secure-boot
cd secure-boot
git submodule update --init
make

Then I signed the EEPROM image and booted the CM4 from it to generate the metadata file:

cd secure-boot-recovery
../tools/update-pieeprom.sh -k "${CUSTOMER_KEY_FILE_PEM}"
cd ..
./rpiboot -d secure-boot-recovery -j secure-boot-recovery/metadata

The "CUSTOMER_KEY_HASH" value in the metadata json file doesn't match the SHA256 of the private key used to provision the device.

I'm technically on PTO for another week, but the curiosity of this got the better of me - until I realised it was likely a framing problem.

I've pushed an update to the guide that should address most of the concerns - most notably, newer versions of rpi-sb-provisioner capture much more of the relevant data in the manufacturing database.

[timg236]

The manual calculation of the key hash in the comment is wrong because it’s using the DER encoding so you can’t attempt to manually verify it this way. The bootrom uses the big number values directly.

[timg236]

See https://github.com/raspberrypi/rpi-eeprom/blob/master/tools/rpi-bootloader-key-convert for how to extract the components of the key used by the bootrom

[timg236]

I suggest using https://github.com/raspberrypi/rpi-eeprom/blob/master/tools/rpi-bootloader-key-convert

[timg236]

I'm not suer USER_BOARDREV has been used

[timg236]

rpi-eeprom-config without arugments is preferred over vcgencmd. We should use the direct device-tree interface wherever possible instead of vcgencmd / vcmailbox

[octal-ip]

Thanks for all the extra effort on this. I've re-provisioned a CM4 using v2.1.1 of rpi-sb-provisioner, however I'm still running into a few issues while verifying the status after provisioning is complete:

• Sections 1.1 and 1.2: devkey_revoked, signed_boot_enabled, provision_ts are all empty in the provisioning database.
• Section 1.4: The expected hash doesn't match the CUSTOMER_KEY_HASH retrieved from metadata when in secure boot mode - both are completely different. I note the advice provided by timg236, but I haven't been able to figure out how to apply or check this.
• Section 2.2: rpi-eeprom-update is not enabled by default on CM4 (this is easily fixed, but may not work in default Debian installs).
• Section 4.1: same result as section 1.1 and 1.2.
• Section 4.2: No initramfs file exists in any of the specified locations, or even the entire filesystem.
• Section 4.3 result:

=== OTP SECURE BOOT VERIFICATION ===
Device: BCM2711 (Pi 4 family)
OTP row 17: 0x0000c8b0
⚠ Bit 28 NOT SET: Public key hash not programmed
⚠ Bit 29 NOT SET: Development key not revoked
⚠ SECURE BOOT MAY NOT BE FULLY ENFORCED
  Both bits 28 and 29 must be set for full enforcement

Below is an output of the comprehensive verification script when run on the provisioned device:

=== On-Device Security Verification ===
Device: SB-Test

Device Model: Raspberry Pi Compute Module 4 Rev 1.1

1. OTP KEY PROGRAMMING VERIFICATION
Device: BCM2711 (Pi 4 family)
⚠ No customer public key found in OTP
  Checked rows: 36 37 38 39 40 41 42 43
  This may indicate secure boot was not enabled

1.5. SECURE BOOT ENFORCEMENT VERIFICATION
Secure boot OTP row 17: 0x0000c8b0
⚠ Bit 28 NOT SET: Public key hash not programmed
⚠ Bit 29 NOT SET: Development key not revoked
⚠ SECURE BOOT MAY NOT BE FULLY ENFORCED

2. FIRMWARE VERSION
✓ Firmware version: Apr 30 2025 13:33:39
  Bootloader info:
    rpi-eeprom-update is not enabled by default on CM4.
    The recommended method for flashing the EEPROM is rpiboot.
    See: https://github.com/raspberrypi/usbboot/blob/master/Readme.md

3. STORAGE ENCRYPTION VERIFICATION
✓ LUKS encrypted storage detected:
    └─mmcblk0p2   crypto_LUKS 2     root  21f70df2-9c0c-4578-b326-a6907305e486
✓ cryptroot mapping is active:
    /dev/mapper/cryptroot is active and is in use.
      type:    LUKS2
      cipher:  xchacha12,aes-adiantum-plain64
      keysize: 256 bits
      key location: keyring
      device:  /dev/mmcblk0p2
      sector size:  512
      offset:  32768 sectors
      size:    14172160 sectors
      mode:    read/write

4. JTAG STATUS VERIFICATION
Device: BCM2711 (Pi 4 family)
JTAG OTP row 16: 0xcc280000
✓ JTAG lock bits 26-27 are SET (JTAG debugging disabled)

5. BOOT PROCESS VERIFICATION
⚠ Root filesystem not mounted from device mapper
  Mount: /dev/mapper/cryptroot / ext4 rw,relatime 0 0
✓ Boot command line shows initramfs-based boot (root=/dev/ram0)

=== On-Device Verification Complete ===

I'm happy to assist in any way I can with this, please let me know if there's any further tests or details required.

[tdewey-rpi]

@octal-ip Just to confirm - did you use the latest version of this PR?

[octal-ip]

@octal-ip Just to confirm - did you use the latest version of this PR?

I believe so, I followed this version: 4a8bb03