How to build and install custom BIOS for Chromebox

[Broc Seib]

Hi all,

I have a Stumpy i5 Chromebox from I/O and added more memory and a larger mSATA drive. The extra memory works fine. But the new mSATA drive does not cold boot into AHCI mode (see Terry Lambert's hypothesis here: http://goo.gl/6uwqY). In reading that thread, it seems that many off the shelf mSATA devices from different vendors all have this problem.

I am ready to hack around on u-boot to try to get the device into the right mode earlier in the cold boot sequence. I looked at the AHCI spec and found the right bit to flip, and then I found in the Chromium u-boot code where that bit gets flipped (it line 109: http://goo.gl/qxxSi). I feel I am capable of tracking this down, but I'm needing a bit of Chromium orienteering.

First, I lack the knowledge of the boot sequence and what loads from ROM, RAM, disk, etc. Nor do I have the right nomenclature -- e.g., shall I still call it a "BIOS"? I would like to know where u-boot sits among this sequence of cold boot participants, and where it loads from.

Second, I could use some pointers on how to build and install my own BIOS (or firmware, or ???)  that contains u-boot. (?)  I was able to successfully build and install my own ChromiumOS (http://goo.gl/BhPmH), and I've found some coreboot and u-boot code in the src tree, but it is not clear to me if the u-boot code is somehow packaged into the big monolithic chromiumos_image.bin file, or how just the u-boot part can be (presumably) installed to the flash memory of the chromebox.

I need a little chromebox/cros education please. :-)

Broc

[Mike Frysinger]

we typically refer to it as "the firmware", but it serves the same
function as a standard BIOS. coreboot takes care of the really low
level cpu/mem bootstrapping before handing off to its payload
(u-boot).

the chromiumos_image.bin does contain a firmware partition (you can
see with `cgpt show <bin>`), but that is the RW firmware, not the RO
firmware. the latter sits on a difference device than the main SSD
and is what initializes the disk, parses the GPT, and jumps to the RW
firmware.

so if your problem is with initializing the SSD, the stuff in the
chromiumos_image.bin won't help. you'll need to disassemble the
stumpy to make the RO storage writable, then boot a system and
manually flash that, then hope you didn't mess it up & reboot :).

i don't think we have documentation on how to do this for
stumpy/lumpy. it's a pretty easy way to brick your systems. maybe if
i can get a screwdriver tomorrow, i'll take some pictures of stumpy's
guts.
-mike

> --
> Chromium OS discuss mailing list: chromium-...@chromium.org
> View archives, change email options, or unsubscribe:
> http://groups.google.com/a/chromium.org/group/chromium-os-discuss?hl=en

[Broc Seib]

Once I built a 68332 embedded system where you had to hold your breath while your volatile memory ran the code to reprogram the boot flash. Maybe here I'll at least plug into a UPS. And still hold my breath.

Here are some photos of my chromebox main board, since I already had my unit apart. (p.s. - evernote links to the full size image, so if you save the image(s) locally, you'll get the full resolution, if needed.) I haven't studied the chips to try to locate a flash device. I didn't see any obvious jumpers either. Perhaps I need to solder a couple pads together or flywire a pin somewhere...

  https://www.evernote.com/pub/brocseib/public-pics

Photo annotations welcome. :-)

Broc

[Mike Frysinger]

see if this helps

-mike

[Broc Seib]

So the read-only firmware has only minimal responsibility of locating the read-write firmware on the disk, loading it to some fixed place in memory, and then jumping to execute it? It sounds like very minimal code.

What happens in the RO firmware when you do a CTRL-U to indicate you wish to boot from USB? I presume the same thing, except it looks around on the USB for the R/W firmware to load and execute?

Maybe I should write some firmware code that lives on a low-profile USB that I leave plugged in all the time. It would do nothing more than put the mSATA disk in the right mode, and then execute precisely what the RO firmware would have done to load the RW firmware from disk and execute it. Yes, I would have to do CTRL-U at hard boot, but I'm already doing CTRL-D.

Is there source code available for the contents of the RO flash?

If not, I'll want to investigate how to read the contents of the flash device, i.e. know where it is mapped in memory.

Broc


On Monday, November 19, 2012 3:16:25 AM UTC-5, Mike Frysinger wrote:

[Broc Seib]

Yes this is useful info Mike. (for all: Winbond W25Q64CV  in the SOIC-8 package.  Spec sheet: http://goo.gl/HUySH )

I think I will investigate the safer road for now (write some firmware to live on a USB device). But if I come back to this more dangerous road, do you know of source code available that can write the flash device? Or do we write our own?

Broc

On Tuesday, November 20, 2012 8:25:15 AM UTC-5, Mike Frysinger wrote:

see if this helps

-mike

[Trever]

Thank you Mike and Broc.  If no one else does, I will put up some documentation on the Chrome OS wiki about this, though it probably belongs with similar documentation for Snow, i.e. here:

http://www.chromium.org/chromium-os/developer-information-for-chrome-os-devices/samsung-sandy-bridge

I think only Googlers can edit those pages though.

Trever

[Mike Frysinger]

@chromium.org get write access to dev.chromium.org.  you don't need to be a googler to get @chromium.org, but the bar isn't exactly low either.

if there's content that'd be better off there, feel free to write up some stuff and forward it along to me.

-mike

[Trever]

Mike,

I will write something up and pass it along.  If you'd rather not be the transcription person here, is there a way to let me edit just a page?  I don't personally care, just trying to make it easier if it is.  I do have edit access to https://sites.google.com/site/chromeoswikisite/, but not the same site/thing.

What I'd like to get completed and posted clearly and "authoritatively" is documentation on how to make RO into RW for each device where that's possible.

My goal is a bit different than Broc's.  What I have in mind should work on stock Chrome device hardware.  Full verified boot of HEX-like builds anyone?  Or, build your own self signed for full verified boot?  Maybe other chain loader/OS payloads too eventually.

That is, I'd like to get posted specific step-by-step documentation on how to flash in a different public RO key but one that will work to boot chromium builds signed with the corresponding private key- a key pair that anyone can generate, of course.  Just instructions on how to change the key in the (normally) RO firmware.  Leave everything else in RO alone.  Then change it back to RO and install your self signed (or whatever) build (kernel and root fs and I guess the RW firmware will need to be resigned too- still haven't done a build myself yet, so not sure yet what hurdles might await), and have a verified boot in non-dev mode. 

Famous last words but seems like figuring all of this out and getting it documented so that any fairly competent tinkerer can do it should be fairly straightforward.

Questions then:

1.  I am looking at my board and find the flash and jumper.  It looks like two pins, so you insert a jumper in there and that toggles to RW.  Correct?  I downloaded the spec sheet, but double checking.  

2.  What kind of jumper goes in there?  I tried a few I had lying around, no love.  They don't fit.  Is there a way to refer to a part number or a standard of some sort?  Beats me.  This is ridiculous.  Bending the pins to touch, does that work?  Seems more destructive than need be though.  Would like a repeatable way of doing this.

3.  (Moving to the software side of things.)  Is there already scripts in the source for modifying the public root key in the RO firmware? (Assuming of course you've made the RO firmware RW.)

P.S.  Hmmmm... maybe my response here should be another thread on this email list?

[Trever]

So in case what I said below isn't clear:  document how someone can "become their own Google" if they so choose (create and install fully verified builds on Chrome OS hardware).  I have read something about an update server too.  So you could be your own chromium distro provider that updates Chrome devices you've tinkered with to use your builds (or tinkered with to use someone else's builds, say if HEX builds ever offered this, and you burned in his public key, eg.).  All of this would work in non-dev mode, giving fully verified boot, after tinkering and setting things back to normal.  Seems possible to me...

I can imagine various use cases though what I have in mind is just hobbyist stuff.

[Mike Frysinger]

On Wed, Nov 21, 2012 at 3:18 AM, Trever <trr...@gmail.com> wrote:

I will write something up and pass it along.  If you'd rather not be the transcription person here, is there a way to let me edit just a page?  I don't personally care, just trying to make it easier if it is.  I do have edit access to https://sites.google.com/site/chromeoswikisite/, but not the same site/thing.

you could write it up there and when finished, lemme know and i can move the content over

That is, I'd like to get posted specific step-by-step documentation on how to flash in a different public RO key but one that will work to boot chromium builds signed with the corresponding private key- a key pair that anyone can generate, of course.  Just instructions on how to change the key in the (normally) RO firmware.  Leave everything else in RO alone.  Then change it back to RO and install your self signed (or whatever) build (kernel and root fs and I guess the RW firmware will need to be resigned too- still haven't done a build myself yet, so not sure yet what hurdles might await), and have a verified boot in non-dev mode. 

the gbb_utility is used to add the key.  check out the vboot_platform repo.  some interesting subdirs:

 - example set of keys for signing (obviously you don't want to use these yourself since pub & priv keys are posted)

http://git.chromium.org/gitweb/?p=chromiumos/platform/vboot_reference.git;a=tree;f=tests/devkeys

 - these are the scripts that we run in order to sign official images with official keys -- the keys are the only part we keep secret (well, we also currently have the signer daemon private too, but that merely runs security/sanity checks before executing these vboot scripts).  look at sign_official_build.sh as the main driver script, and sign_firmware.sh as the file that does signing specifically for firmware images.  that shows you that the gbb_utility can be used to take an existing firmware image and replace the keys found in there.  perfect for extracting the RO firmware, replacing just the keys, and then writing that back to the RO storage.

http://git.chromium.org/gitweb/?p=chromiumos/platform/vboot_reference.git;a=tree;f=scripts/image_signing

- these are the scripts we use to generate keys

http://git.chromium.org/gitweb/?p=chromiumos/platform/vboot_reference.git;a=tree;f=scripts/keygeneration

1.  I am looking at my board and find the flash and jumper.  It looks like two pins, so you insert a jumper in there and that toggles to RW.  Correct?  I downloaded the spec sheet, but double checking.

easy to test: run crossystem on the device.  look at the last field (called like wps or something).  it'll be 1 when the write protect is enabled, and 0 when it's been cleared.

 

2.  What kind of jumper goes in there?  I tried a few I had lying around, no love.  They don't fit.  Is there a way to refer to a part number or a standard of some sort?  Beats me.  This is ridiculous.  Bending the pins to touch, does that work?  Seems more destructive than need be though.  Would like a repeatable way of doing this.

i'd have to open mine and/or talk to someone who knows

 

3.  (Moving to the software side of things.)  Is there already scripts in the source for modifying the public root key in the RO firmware? (Assuming of course you've made the RO firmware RW.)

gbb_utility, but it operates on files, not spi flashes.  you have to do the extract/modify/write steps yourself.

-mike

[Trever]

Thanks Mike, very helpful, *much* appreciated.  My write up will be coming.

Curious:  signer daemon?

[Mike Frysinger]

the way official images are produced:

 - the release/firmware/factory cbuildbot configs run on internal buildbots for the respective branches and compile official (unsigned) images (they have access to all the private source code/binaries that makes it Google Chrome OS rather than Chromium OS)

 - hardware/vm/security tests are run on those images

 - the buildbots upload the test results & images ("artifacts") to an internal Google Storage bucket with restricted access

 - an internal signer daemon running on internal machines finds those artifacts, runs sanity/security checks, and then signs with official Google keys iff everything passes

 - the signed images get uploaded to another internal Google Storage bucket with restricted access

 - this bucket is organized by channel/board/version and are all "candidate" images

from there, release managers (TPMs) work with dev/QA teams to evaluate the test results and overall release/branch status for a particular branch/version before pushing to the rest of the world.  we have configs so that the channels for a board can be on a different version if need be (although for obvious reasons we try to avoid that).  this allows us to skip a release for a board while not impacting other boards (which we've done in the past with e.g. CR-48/mario and the initial aura release), or to push a release for a specific board ahead of others (which you can see with the ARM Chromebook today).

i'm glossing over some details, but you get the gist

-mike

On Mon, Nov 26, 2012 at 12:08 AM, Trever <trr...@gmail.com> wrote:

Curious:  signer daemon?

[Vadim Bendebury]

On Tue, Nov 20, 2012 at 9:12 AM, Broc Seib <broc...@gmail.com> wrote:
> So the read-only firmware has only minimal responsibility of locating the
> read-write firmware on the disk, loading it to some fixed place in memory,
> and then jumping to execute it? It sounds like very minimal code.
>

The read-write firmware is also in the SPI flash, it is actually a
NOOP unless there is a need to fix a problem found in RO section
(which has not happened yet AFAIK).

> What happens in the RO firmware when you do a CTRL-U to indicate you wish to
> boot from USB? I presume the same thing, except it looks around on the USB
> for the R/W firmware to load and execute?
>

Again, it loads the kernel blob (not RW firmware) from either SSD of USB Flash

cheers,
/vb

[Eric Cyb]

So is it at least possible, by pressing Ctrl+U, to boot a custom-made image from USB?
For instance, a live distribution?

[Ottavio Caruso]

On 30 November 2012 16:10, Eric Cyb <cybe...@gmail.com> wrote:
> So is it at least possible, by pressing Ctrl+U, to boot a custom-made image
> from USB?

It is possible and it has been done, we're talking here of custom BIOS.

--
Ottavio

[Eric Cyb]

Sorry for being a bit off topic here, but as Broc said, we coul leave a USB key always plugged.
This key coul reset the AHCI controler or boot anither OS.
But I was not able to find instructions to create a custom bootable from USB.
Do you have pointers, please?

Thanks
Eric

[Broc Seib]

Hey Ottavio, where can I find sources for a custom made BIOS?

[Broc Seib]

Hey all, I did find some sources to help with flashing the Winbond SPI device. It's in the u-boot code, e.g.,

  src/third_party/u-boot/files/drivers/mtd/spi/winbond.c (our device is indeed 0x4017, the W25Q64CV)
  src/third_party/u-boot/files/drivers/mtd/spi/spi_flash.c (calls spi_flash_probe_winbond)

Are there sources available for the contents of the read-only firmware?

Broc

[Mike Frysinger]

partially.  please consult:
http://dev.chromium.org/chromium-os/developer-information-for-chrome-os-devices/custom-firmware

-mike

[Sebastian Oliva]

Sorry to revive this old thread. What is the procedure to disassemble stumpy far enough to reach that jumper?

Thanks!

[Mike Frysinger]

the wiki has a search function which should readily take you to the relevant page:

http://dev.chromium.org/chromium-os/developer-information-for-chrome-os-devices/samsung-sandy-bridge#TOC-Disassembly

-mike

--

[Matthew Swanson]

I realize I'm a little late to the party, but I have this chromebox and am trying to install linux on it in place of chrome os. I connected the 2 pins together as shown in Mike's picture farther down. What are my next steps for this? I want to install Lubuntu or something similar

[Trever]

Hi Mike,

Fast forward 6 years (it looks like!).

I accidentally put the tin foil in the wrong jumper initially (note to self:  don't do this in the kitchen when people are around to distract)- I put it in the identical looking jumper just to the left on the other side of the little cut out in your photo below of stumpy.  Would you happen to know what that jumper does?

Everything seems fine now, but stumpy won't power off.  Under Chrome OS I get a white screen at the end, with power remaining on.  After correcting the jumper situation (removing it from the wrong place and putting it in the right place), I reflashed firmware and NIXOS is doing the same thing- it won't power down the machine, I see a kernel message about waiting for a clock (I could get exact message if it matters).

Wondering if I did that with my tin foil oops, and if so if there's a fix.

Here's to the Google I/O i5 Chromeboxes from years back- the hardware is still going strong!  So easy to take the machines apart and play...

Thanks,

T

[Mike Frysinger]

sorry, but i don't know what that does.  if you reflash the original firmware back in, does it still not shut down properly ?

i vaguely recall some reports related to stumpy about power shutdown not going smoothly, so you might be running into that too.

-mike

--
--
Chromium OS discuss mailing list: chromium-os-discuss@chromium.org

[Luigi Semenzato]

On Mon, Feb 26, 2018 at 11:52 AM, Mike Frysinger <vap...@chromium.org> wrote:
>
> sorry, but i don't know what that does. if you reflash the original firmware back in, does it still not shut down properly ?
>
> i vaguely recall some reports related to stumpy about power shutdown not going smoothly, so you might be running into that too.
> -mike

You recall correctly. We have seen this happen on a few stumpies. It
seems to be a hardware issue. I compared two stumpies, same hardware
ID, same OS image, and same firmware (I compared the firmware
byte-by-byte to make sure). One stumpy worked fine. The other would
call into ACPI reset at shutdown, and just hang there without turning
power off as expected. We don't have firmware sources for the stumpy,
so that's where my efforts ended.

I can't even say this wasn't caused by your tin foil experiments,
because the units we have here go through all kinds of abuse, and,
although unlikely, it cannot be excluded that something similar
happened to them.

>
>
> On Sun, Feb 25, 2018 at 4:50 PM, Trever <trr...@gmail.com> wrote:
>>
>> Hi Mike,
>>
>> Fast forward 6 years (it looks like!).
>>
>> I accidentally put the tin foil in the wrong jumper initially (note to self: don't do this in the kitchen when people are around to distract)- I put it in the identical looking jumper just to the left on the other side of the little cut out in your photo below of stumpy. Would you happen to know what that jumper does?
>>
>> Everything seems fine now, but stumpy won't power off. Under Chrome OS I get a white screen at the end, with power remaining on. After correcting the jumper situation (removing it from the wrong place and putting it in the right place), I reflashed firmware and NIXOS is doing the same thing- it won't power down the machine, I see a kernel message about waiting for a clock (I could get exact message if it matters).
>>
>> Wondering if I did that with my tin foil oops, and if so if there's a fix.
>>
>> Here's to the Google I/O i5 Chromeboxes from years back- the hardware is still going strong! So easy to take the machines apart and play...
>>
>> Thanks,
>>
>> T
>>
>>
>>
>>
>>
>> On Tuesday, November 20, 2012 at 8:24:45 AM UTC-5, Mike Frysinger wrote:
>>>
>>> see if this helps
>>> -mike
>>>
>>>

[Trever Nightingale]

Thanks guys for your responses, I appreciate it.  At least you didn't say "Oh God, you used THAT jumper?  And your house hasn't burned down yet!?"

And weirdly, I can no longer reproduce the behavior.  At least for now.  Everything is working normally.

Makes no sense.  Only thing I did was have the power off for an hour versus 1 minute.  Shrug.  Now I have two good (if older) repurposed home lab machines.  Yay.

Chell is holding me over for now, but waiting for the 2018 announced Chromeboxes, I hope they will be available to consumers and not just enterprise enrolled type units.

-T