MKII secure bootloader issues

[Rafał Wojdyła]

Hi,

I'm preparing to activate secure boot on the MKII Armory (imx6ulz) and
have some issues. Here's what I've done (all steps assume uSD card as
the target media):

1. Prepared signing keys as described here:
https://github.com/f-secure-foundry/usbarmory/wiki/Secure-boot-(Mk-II)
2. Built the signed bootloader as described here:
https://github.com/f-secure-foundry/armory-boot#secure-boot
3. Built the Debian image based on the official repo
(https://github.com/f-secure-foundry/usbarmory-debian-base_image). Here
I needed to modify the Makefile to generate proper
/boot/armory-boot.conf and /boot/armory-boot.conf.sig. I've verified
that the files look like they supposed to. Is there no built-in way to
do this btw?
4. Wrote the signed bootloader into the resulting image:
sudo dd if=armory-boot-signed.imx of=usbarmory.raw bs=512 seek=2
conv=fsync conv=notrunc
5. Wrote the image onto the sd card:
sudo dd if=usbarmory.raw of=/dev/mmcblk0 bs=1M conv=fsync

The armory doesn't seem to boot, the blue LED is on but then nothing
happens. I wish I had the debug accessory...

I tried using the official release of Debian image, unmodified one boots
fine. But if I try to overwrite the bootloader with armory-boot, it's
back to the blue LED. I've tried both signed and unsigned armory-boot,
the result is the same. Any ideas?

Thanks,

--
Rafał Wojdyła
Invisible Things Lab

[Rafał Wojdyła]

I should add, secure boot is not enabled yet -- testing without it for now.

[Andrea Barisani]

Have you tried armory-boot without any signing (therefore without `PUBLIC_KEY` and `HAB_KEYS`) first?

Thanks

[Andrea Barisani]

The only thing I cannot do from your post is evaluate whether armory-boot.conf has, for whatever reason, an incorrect format or some error that causes this.

If you want to debug it yourself without a debugging accessory may I suggest you move the line usbarmory.LED("blue", true) after each step in main.go so that you can identify where it panics? (tedious I know, but without a debug accessory there is nothing else I can suggest).

Thanks

[Rafał Wojdyła]

My armory-boot.conf from
linux-image-5.4-usbarmory-mark-two_5.4.82-0_armhf/boot looks like this:

{
"kernel": [
"/boot/zImage-5.4.82-0-usbarmory",
"4e1da74d188ac0b3bb8b17171c95f580ccc5a4c2e5267bce8612f55f7bdc948e"
],
"dtb": [
"/boot/imx6ulz-usbarmory-default-5.4.82-0.dtb",
"1227288d7b726c1d6eaf74ce6cf8cbf55fb843aa04c8623eeee378ce39aefe03"
],
"cmdline": "console=ttymxc1,115200 root=/dev/mmcblk1p1 rootwait rw"
}

I'll dig into this more tomorrow, thank you for the suggestion with the LED.

Edit: I quickly checked these files with sha256sum and voila, the kernel
one is different for some reason, probably something wrong with my
Makefile modification.

> <https://github.com/f-secure-foundry/usbarmory-debian-base_image>).

[Rafał Wojdyła]

Alright, after mounting the image and updating the kernel hash
everything seems to work as expected.

Merry Christmas! :)

--
Rafał Wojdyła
Invisible Things Lab

[Rafał Wojdyła]

On 2020-12-23 10:17, Rafał Wojdyła wrote:
> Alright, after mounting the image and updating the kernel hash
> everything seems to work as expected.
>
> Merry Christmas! :)
>

Well, turns out it's not all good when I enabled secure boot ;)

From the debian host where I have the secure boot keys:
xxd -ps -c 32 $HAB_KEYS/SRK_1_2_3_4_fuse.bin
74554e436cb0eab1d5f8be638f035df4774bc50516cb181632edc731448da84c

Commands on armory that were used to fuse the keys and enable secure boot:
sudo -E go/bin/crucible -m IMX6UL -r 1 -b 16 -e little blow SRK_HASH
74554e436cb0eab1d5f8be638f035df4774bc50516cb181632edc731448da84c
sudo -E go/bin/crucible -m IMX6UL -r 1 -b 2 -e big blow SRK_LOCK 1
sudo -E go/bin/crucible -m IMX6UL -r 1 -b 2 -e big blow SEC_CONFIG 0b11
sudo -E go/bin/crucible -m IMX6UL -r 1 -b 2 -e big blow DIR_BT_DIS 1
sudo -E go/bin/crucible -m IMX6UL -r 1 -b 2 -e big blow SJC_DISABLE 1
sudo -E go/bin/crucible -m IMX6UL -r 1 -b 2 -e big blow JTAG_SMODE 0b11
sudo -E go/bin/crucible -m IMX6UL -r 1 -b 2 -e big blow JTAG_HEO 1
sudo -E go/bin/crucible -m IMX6UL -r 1 -b 2 -e big blow KTE 1
sudo -E go/bin/crucible -m IMX6UL -r 1 -b 2 -e big blow SDP_READ_DISABLE 1
sudo -E go/bin/crucible -m IMX6UL -r 1 -b 2 -e big blow
UART_SERIAL_DOWNLOAD_DISABLE 1

Now the armory won't boot, both LEDs are dimly lit.

I've tried to build signed armory-ums to check if that'll run, with no
results (HAB_KEYS is set):

make CROSS_COMPILE=arm-none-eabi- imx_signed
sudo dd if=armory-ums-signed.imx of=/dev/mmcblk0 bs=1M conv=fsync

Host with the armory plugged in shows just this in dmesg:

[ 2145.395358] xhci_hcd 0000:39:00.0: xHCI Host Controller
[ 2145.395364] xhci_hcd 0000:39:00.0: new USB bus registered, assigned
bus number 3
[ 2145.396499] xhci_hcd 0000:39:00.0: hcc params 0x200077c1 hci version
0x110 quirks 0x0000000200009810
[ 2145.396741] usb usb3: New USB device found, idVendor=1d6b,
idProduct=0002, bcdDevice= 5.04
[ 2145.396743] usb usb3: New USB device strings: Mfr=3, Product=2,
SerialNumber=1
[ 2145.396744] usb usb3: Product: xHCI Host Controller
[ 2145.396746] usb usb3: Manufacturer: Linux 5.4.0-58-generic xhci-hcd
[ 2145.396747] usb usb3: SerialNumber: 0000:39:00.0
[ 2145.396889] hub 3-0:1.0: USB hub found
[ 2145.396899] hub 3-0:1.0: 2 ports detected
[ 2145.397452] xhci_hcd 0000:39:00.0: xHCI Host Controller
[ 2145.397458] xhci_hcd 0000:39:00.0: new USB bus registered, assigned
bus number 4
[ 2145.397462] xhci_hcd 0000:39:00.0: Host supports USB 3.1 Enhanced
SuperSpeed
[ 2145.397518] usb usb4: New USB device found, idVendor=1d6b,
idProduct=0003, bcdDevice= 5.04
[ 2145.397519] usb usb4: New USB device strings: Mfr=3, Product=2,
SerialNumber=1
[ 2145.397521] usb usb4: Product: xHCI Host Controller
[ 2145.397522] usb usb4: Manufacturer: Linux 5.4.0-58-generic xhci-hcd
[ 2145.397524] usb usb4: SerialNumber: 0000:39:00.0
[ 2145.397660] hub 4-0:1.0: USB hub found
[ 2145.397673] hub 4-0:1.0: 2 ports detected

[lcar...@gmail.com]

imx images should be dd’ed with bs=512 seek=2

On 23 Dec 2020, at 18:41, Rafał Wojdyła <om...@invisiblethingslab.com> wrote:

--
You received this message because you are subscribed to the Google Groups "USB armory" group.
To unsubscribe from this group and stop receiving emails from it, send an email to usbarmory+...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/usbarmory/b841c735-27f5-76da-b076-ab91d8119843%40invisiblethingslab.com.

[Rafał Wojdyła]

I tried imx_usb_loader and the results are interesting.
Unsigned armory-ums.imx fails to load as expected:

config file <.//mx6ull_usb_work.conf>
parse .//mx6ull_usb_work.conf
Trying to open device vid=0x15a2 pid=0x0080
Interface 0 claimed
HAB security state: production mode (0x12343412)
== work item
filename ../armory-ums.imx
load_size 0 bytes
load_addr 0x00000000
dcd 1
clear_dcd 0
plug 1
jump_mode 3
jump_addr 0x00000000
== end work item
loading DCD table @0x910000

<<<952, 952 bytes>>>
succeeded (security 0x12343412, status 0x128a8a12)
clear dcd_ptr=0x8000f42c

loading binary file(../armory-ums.imx) to 8000f400, skip=0, fsize=179c00
type=aa

<<<1547264, 1547264 bytes>>>
succeeded (security 0x12343412, status 0x88888888)
jumping to 0x8000f400
failed (security 0x12343412, status 0x33220a00)

...but then, signed ums image is correctly loaded, although SDP was
disabled via crucible:

Trying to open device vid=0x15a2 pid=0x0080
Interface 0 claimed
HAB security state: production mode (0x12343412)
== work item
filename ../armory-ums-signed.imx
load_size 0 bytes
load_addr 0x00000000
dcd 1
clear_dcd 0
plug 1
jump_mode 3
jump_addr 0x00000000
== end work item
loading DCD table @0x910000

<<<952, 952 bytes>>>
succeeded (security 0x12343412, status 0x128a8a12)
clear dcd_ptr=0x8000f42c

loading binary file(../armory-ums-signed.imx) to 8000f400, skip=0,
fsize=17dc00 type=aa

<<<1563648, 1563648 bytes>>>
succeeded (security 0x12343412, status 0x88888888)
jumping to 0x8000f400

So it seems HAB is correctly enabled, but SDP is not disabled?

[Rafał Wojdyła]

The same armory-ums-signed.imx image that worked via imx_usb_loader (and
correctly exposed emmc/sd as usb storage devices) does not boot when
written into the sd card by

sudo dd if=armory-ums-signed.imx of=/dev/mmcblk0 bs=512 seek=2 conv=fsync

So I am at a loss for now.

[lcar...@gmail.com]

When loading signed images over SDP the —serial option needs to be passed to the signing tool.

This is fine by default in the Makefile, if you want to flash armory-ums you have to remove —serial when signing.

We only document USB loading for armory-ums as that is the predominant use case.

> On 23 Dec 2020, at 20:07, Rafał Wojdyła <om...@invisiblethingslab.com> wrote:
>
> The same armory-ums-signed.imx image that worked via imx_usb_loader (and correctly exposed emmc/sd as usb storage devices) does not boot when written into the sd card by

> --
> You received this message because you are subscribed to the Google Groups "USB armory" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to usbarmory+...@googlegroups.com.

> To view this discussion on the web visit https://groups.google.com/d/msgid/usbarmory/99c3eadd-e697-984b-e4a4-42fba3d2f647%40invisiblethingslab.com.

[Rafał Wojdyła]

Thanks, that got me to armory-ums booting from the sd card. So it seems
signed bootloader is able to correctly run, now off to debug why
armory-boot doesn't work. :)

[lcar...@gmail.com]

I thought you identified that the incorrect hash was the issue?

Ensure that armory-boot is dd’ed with the correct flags and signed without —serial if booted from SD or MMC.

Let me know how I can help.

> On 23 Dec 2020, at 20:33, Rafał Wojdyła <om...@invisiblethingslab.com> wrote:
>
> Thanks, that got me to armory-ums booting from the sd card. So it seems signed bootloader is able to correctly run, now off to debug why armory-boot doesn't work. :)

[Rafał Wojdyła]

After I fixed the hash issue the armory-boot was working (without secure
boot). Now with secure boot enabled it doesn't boot. Here's some image
verification I've done to confirm that the armory-boot.conf is valid:

$ sudo mount -o loop,offset=5242880 -t ext4 usbarmory.raw rootfs
$ cd rootfs/boot
$ sha256sum zImage-5.4.82-0-usbarmory
$ 926e786e55c1049778367fb817a4b6ec00920d071ae18d837a9f5b05717fede2
zImage-5.4.82-0-usbarmory
$ sha256sum imx6ulz-usbarmory-default-5.4.82-0.dtb
1227288d7b726c1d6eaf74ce6cf8cbf55fb843aa04c8623eeee378ce39aefe03
imx6ulz-usbarmory-default-5.4.82-0.dtb
$ cat armory-boot.conf
{
"kernel": [
"/boot/zImage-5.4.82-0-usbarmory",
"926e786e55c1049778367fb817a4b6ec00920d071ae18d837a9f5b05717fede2"

],
"dtb": [
"/boot/imx6ulz-usbarmory-default-5.4.82-0.dtb",
"1227288d7b726c1d6eaf74ce6cf8cbf55fb843aa04c8623eeee378ce39aefe03"
],

"cmdline": "root=/dev/mmcblk1p1 rootwait rw"
}

$ cat armory-boot.conf.sig
untrusted comment: verify with armory-boot.pub
RWQ0pxGNC6bCEz9BjYW694bEv1f4JhOlSPykm9rG7Bs8RNU9gbnTX//PpipgWSPKBd4GlrgUE/3vFzivuv+23sK54VlgYKGvkgY=
$ signify-openbsd -V -p $HAB_KEYS/armory-boot.pub -x
armory-boot.conf.sig -m armory-boot.conf
Signature Verified

I modified the armory-boot's main.go source to blink some LEDs and it
seems to do that, so the bootloader is executing. I'll try to pinpoint
where it stops later.

[lcar...@gmail.com]

Could you try with minisign instead and see if that makes a difference?

We should support both but maybe we have an error there (I use signify myself).

Thanks

> On 23 Dec 2020, at 22:21, Rafał Wojdyła <om...@invisiblethingslab.com> wrote:
>
> After I fixed the hash issue the armory-boot was working (without secure boot). Now with secure boot enabled it doesn't boot. Here's some image verification I've done to confirm that the armory-boot.conf is valid:

[Rafał Wojdyła]

I tried minisign with fresh second stage keys but that didn't change the
outcome. Through LED debugging I've determined that armory-boot goes up
to the exec() call in boot.go and after a few seconds it reboots. That's
enough for now, I'll dig into it more after Christmas.

[Rafał Wojdyła]

Well, I'm not sure what I did but it's now working with armory-boot:

[ 11.389946] mxs-dcp 2280000.crypto: mxs_dcp: initialized
[ 11.402923] mxs_dcp: Trusted State detected

Before that I tried the stock debian image with signed u-boot and that
worked as well. I'll probably prepare a step-by-step writeup later if
anyone runs into some issues like these (and to verify my own setup).

[Andrea Barisani]

I also just updated armory-boot on my secure booted armory, to try out the new kernel, and I confirm it works for me.

I could add a composite debugging mode which exposes Ethernet over USB for an SSH based debugging console, that would be quite easy to do as we have all that code in our GoKey project anyway. I'll get around to that as soon as I have the chance. Of course it would only allow to debug any error after armory-boot is started, but still would help folks that don't have the debugging accessory. What do you think?

Thanks for your patience :)

[Rafał Wojdyła]

Ethernet debugging could be nice as an optional feature. I have some
experience with ARM SOCs like the later Arduinos so I could probably get
a console somehow, but thankfully that wasn't needed.

Thank you for this great project :)

> <https://groups.google.com/d/msgid/usbarmory/99c3eadd-e697-984b-e4a4-42fba3d2f647%40invisiblethingslab.com>.