Can't read AP ROM content using flashrom & suzy-q cable

[Martin R]

Hello,

I try to read the ROM content of CML (Hatch) Chromebook using suzy-q cable & flashrom. I have opened the CCD using /dev/ttyUSB0 on a host machine and now the CCD returns following state:

> ccd

State: Opened

Password: none

Flags: 0x400004

Capabilities: 5555555515000000

  UartGscRxAPTx   Y 1=Always

  UartGscTxAPRx   Y 1=Always

  UartGscRxECTx   Y 1=Always

  UartGscTxECRx   Y 1=Always

  FlashAP         Y 1=Always

  FlashEC         Y 1=Always

  OverrideWP      Y 1=Always

  RebootECAP      Y 1=Always

  GscFullConsole  Y 1=Always

  UnlockNoReboot  Y 1=Always

  UnlockNoShortPP Y 1=Always

  OpenNoTPMWipe   Y 1=Always

  OpenNoLongPP    Y 1=Always

  BatteryBypassPP Y 1=Always

  UpdateNoTPMWipe Y 1=Always

  I2C             Y 1=Always

  FlashRead       Y 1=Always

  OpenNoDevMode   Y 1=Always

  OpenFromUSB     Y 1=Always

read_tpm_nvmem: object at 0x100a not found

[11190.635058 Console unlock allowed]

TPM:

Capabilities are modified.

Use 'ccd help' to print subcommands

So it looks fine. Now I do "sudo ./flashrom -p raiden_debug_spi:target=AP -r ap.rom" and flashrom reads ROM content and saves it to a file ap.rom. All is fine, the size is correct but the content of it is nothing but zeros 0x00...

If on the other hand I read rom content on the DUT itself it reads AP firmware just fine. However, on the DUT I have a different version of the flashrom application and it prints a warning: "Found chipset "Intel Cometlake. Enabling flash write... WARNING: SPI Configuration Lockdown activated."

Does the last sentence explain why I read only zeros over SPI? If so how to unlock this?

Also, I use the exact same setup to read AP ROM of two other chromebooks GLK and another CML Chromebook - it works just fine.

Thanks for help.​

[Vadim Bendebury]

this is odd, could it be that flashrom ran on the host does not recognize the flash chip properly? What flash chips do the other chromebooks have?

What do you see when you run "sudo ./flashrom -p raiden_debug_spi:target=AP --flash-name" ?

--vb    

--
--
Chromium OS Discussion mailing list: chromium-...@chromium.org
View archives, change email options, or unsubscribe:
https://groups.google.com/a/chromium.org/group/chromium-os-discuss

[Martin R]

Hi, 

thanks for the answer. The flash chip is: W25Q256JV_M on both platforms. I was able to read the ap.rom once and its content was not all zero, however the important section SI_DESC was only zeroes which is certainly not correct. 

Do we have some more ideas? :|

Martin

[Vadim Bendebury]

I am not sure what else could be wrong without probing the SPI bus during attempts to read/write the chip..

-vb

[Martin R]

I have noticed sth else which is very suspicious - I can't set nor clear the password ussing CCD. I try: ccd password test or ccd password clear and in both cases cd returns "Access Denied". Sounds like there is a password set already maybe but this is not the case - even ccd says Password: none". So I am confused

PS: is it possible to replace the flash chip easly? I start to think maybe this is a HW issue?

Martin

[Vadim Bendebury]

what is printed on the Cr50 console in response to the 'version' command?

[Martin R]

> version
Chip:    g cr50 B2-C
Board:   0
RO_A:  * 0.0.10/29d77172
RO_B:    0.0.10/c2a3f8f9
RW_A:    0.4.15/cr50_v1.9308_B.557-975a31a
RW_B:  * 0.4.18/cr50_v1.9308_B.644-724f23f
BID A:   46464646:00000000:00000010 Yes
BID B:   46464646:00000000:00000010 Yes
Build:   0.4.18/cr50_v1.9308_B.644-724f23f
         tpm2:v0.0.329-35ec5fa
         cryptoc:v0.0.10-78c366f
         2019-05-22 18:17:23 @chromeos-legacy-release-us-east1-d-x32-126-ise3

Is there any way to confirm SPI chip itself is healthy and can communicate just fine?

Martin

[Vadim Bendebury]

this in fact is a very old Cr50 version, the most recent one is 0.0.11/0.5.7. What is your device Chrome OS version?

-vb

[Martin R]

One of the devices is running ChromeOS: 79.0.3940.0 and the other one is running Ubuntu 18.04

[Vadim Bendebury]

you should update Cr50 to the latest available version.

Also, I doubt anything is wrong with the flash chip, otherwise the computer would not be able to boot.

[Martin R]

Ok, can you point me to the documentation for Cr50 FW update? I can't find anything on that matter.

[Vadim Bendebury]

Are both of your Cr50 images of the same version?

-vb

[Martin R]

Actually no, the other CML (on which the SPI works fine) has slightly newer version:

> version
Chip:    g cr50 B2-C
Board:   0
RO_A:  * 0.0.10/29d77172
RO_B:    0.0.10/c2a3f8f9
RW_A:    0.4.18/cr50_v1.9308_B.644-724f23f
RW_B:  * 0.4.23/cr50_v1.9308_B.730-c4e1bbc
BID A:   46464646:00000000:00000010 Yes
BID B:   46464646:00000000:00000010 Yes
Build:   0.4.23/cr50_v1.9308_B.730-c4e1bbc
         tpm2:v1.9308_26_0.48-bb9caa2
         cryptoc:v1.9308_26_0.2-78c366f
         2019-10-02 18:17:22 @chromeos-ci-legacy-us-central2-d-x

[Vadim Bendebury]

interesting. and what do both of them say in response to the 'bid' console command?

[Martin R]

The same for both -> Board ID: 5a5a4352, flags 00007f7f

[Martin R]

Hi,

I have performed few additional steps:

1. So far I have only been using flashrom from chromeOS: 

- Fetch URL: https://chromium.googlesource.com/chromiumos/third_party/flashrom

- top commit: c5121eb03f1a8b30aa46d8e517539aa5df98b275

Now, I have run: sudo flashrom -p raiden_debug_spi:target=AP -w ap.rom followed by a read back -> sudo flashrom -p raiden_debug_spi:target=AP -r ap_read_back.rom then I compared md5sums of both of them and guess what, they are different!

2. I also compiled the upstream flashrom and gave it a try:

- Fetch URL: https://github.com/flashrom/flashrom.git

- top commit 5ab46567df4bcc470db769d584683e67d784084e

But here the behaviour is quite different -> Erasing and writing flash chip... FAILED at 0x00000000! Expected=0xff, Found=0x00, failed byte count from 0x00000000-0x00000fff: 0x1000

ERASE FAILED!

Above error repeats itself indefinitely.... Even plain flashrom -E fails. I have also checked what flash chip this version of flashrom recognizes and they match the one detected by chrome flashrom. However the very first call returned:

$ sudo ../flashrom_upstream/flashrom -p raiden_debug_spi:target=AP --flash-name
flashrom v1.2-170-g5ab4656 on Linux 5.4.0-58-generic (x86_64)
flashrom is free software, get the source code at https://flashrom.org Using clock_gettime for delay loops (clk_id: 1, resolution: 1ns).
Raiden target: 2
No EEPROM/flash device found.
Note: flashrom can never write if the flash chip isn't found automatically.

The consecutive ones:

sudo ../flashrom_upstream/flashrom -p raiden_debug_spi:target=AP --flash-name
flashrom v1.2-170-g5ab4656 on Linux 5.4.0-58-generic (x86_64)
flashrom is free software, get the source code at https://flashrom.org Using clock_gettime for delay loops (clk_id: 1, resolution: 1ns).
Raiden target: 2
Found Winbond flash chip "W25Q256JV_M" (32768 kB, SPI) on raiden_debug_spi.
vendor="Winbond" name="W25Q256JV_M"

PS: Vadim: can we add somebody from HW division to the discussion?

[dragon788]

Have you ever done a Powerwash with the option to "update the firmware for security" selected on either of the systems? I wonder if this would explain the differing versions.

You can also see how ChromeOS is using Flashrom on your ChromeOS system by running the following command.

sudo grep -iIr 'flashrom -p' /usr/ /opt/

I also saw that there is a cr50 folder under /opt/google/ but I haven't figured out what exactly is in it yet.

---
To unsubscribe from this group and stop receiving emails from it, send an email to chromium-os-dis...@chromium.org.

[Vadim Bendebury]

On Wed, Dec 30, 2020 at 7:30 AM Martin R <martin...@gmail.com> wrote:

Hi,

I have performed few additional steps:

1. So far I have only been using flashrom from chromeOS: 

- Fetch URL: https://chromium.googlesource.com/chromiumos/third_party/flashrom

- top commit: c5121eb03f1a8b30aa46d8e517539aa5df98b275

Now, I have run: sudo flashrom -p raiden_debug_spi:target=AP -w ap.rom followed by a read back -> sudo flashrom -p raiden_debug_spi:target=AP -r ap_read_back.rom then I compared md5sums of both of them and guess what, they are different!

 

This could be explained by ME (the management engine) modifying the flash contents, there could be some logs saved in the flash in the process. You should be comparing parts of the flash which are code and RO data,

-vb

[Martin R]

Hi,

It looks like I root caused the issue. I was using yet another chromebook as a host device (flashrom) and that machine was running on a battery only, so I suspect the voltage on chrombooks from a battery source is not stable enough... Since I have only one voltage adapter for chrombooks currently I used another host machine and the problem is gone.

Thanks for support Vadim! 

By the way, could you provide the guide for cr50 update?

PS: Is the AP rom content stored in flash chip itself?

Martin 

[Tim Wawrzynczak]

Martin,

You can find more information on updating cr50 here: https://chromium.googlesource.com/chromiumos/platform/ec/+/cr50_stab/docs/case_closed_debugging_cr50.md#Troubleshooting

Yes, the platform SPI flash contains the AP boot firmware (coreboot, etc.); the layout for Hatch devices is here: https://review.coreboot.org/plugins/gitiles/coreboot/+/refs/heads/master/src/mainboard/google/hatch/chromeos-hatch-16MiB.fmd

Cheers,

 - Tim

On Wed, Dec 30, 2020 at 4:01 PM Martin R <martin...@gmail.com> wrote:

Hi,

It looks like I root caused the issue. I was using yet another chromebook as a host device (flashrom) and that machine was running on a battery only, so I suspect the voltage on chrombooks from a battery source is not stable enough... Since I have only one voltage adapter for chrombooks currently I used another host machine and the problem is gone.

Thanks for support Vadim! 

By the way, could you provide the guide for cr50 update?

PS: Is the AP rom content stored in flash chip itself?

Martin 

śr., 30 gru 2020 o 18:33 Vadim Bendebury <vbe...@chromium.org> napisał(a):

On Wed, Dec 30, 2020 at 7:30 AM Martin R <martin...@gmail.com> wrote:

Hi,

I have performed few additional steps:

1. So far I have only been using flashrom from chromeOS: 

- Fetch URL: https://chromium.googlesource.com/chromiumos/third_party/flashrom

- top commit: c5121eb03f1a8b30aa46d8e517539aa5df98b275

Now, I have run: sudo flashrom -p raiden_debug_spi:target=AP -w ap.rom followed by a read back -> sudo flashrom -p raiden_debug_spi:target=AP -r ap_read_back.rom then I compared md5sums of both of them and guess what, they are different!

 

This could be explained by ME (the management engine) modifying the flash contents, there could be some logs saved in the flash in the process. You should be comparing parts of the flash which are code and RO data,

-vb

--
--
Chromium OS Discussion mailing list: chromium-...@chromium.org
View archives, change email options, or unsubscribe:
https://groups.google.com/a/chromium.org/group/chromium-os-discuss