Firmware update

[Frank Hunleth]

There have been some side discussions on firmware update with Nerves, but I wanted to let the larger list know what was happening. 

The summary is that the current approach that uses ZIP files is broke. The Erlang standard library doesn't let you incrementally decompress files, so the current code decompresses the images all at once. The ZIP library also uses several times as much memory to do the decompression than what I'd expect, and this leads to out-of-memory errors and ridiculously slow performance on big images or on small memory platforms.

A side issue is that I had gotten accustomed to u-boot in all of my embedded development and made some assumptions that the x86 and Raspberry Pi bootloaders couldn't do. I patched fwtool to support these use cases, but it's a hack and I'd like to fix that too. (BTW, Raspberry PI firmware update is broke since the upstream RPi kernel config hardcodes kernel parameters for which rootfs to use. Easy to fix, but I haven't done it yet.)

Since I'm going to be revisiting firmware update, I'd like to share my thoughts on requirements and design and get comments from others.

Requirements

1. All parts of firmware update are stored in a compressed archive (probably not ZIP going forward)
   
2. The firmware archive contains a manifest file that contains metadata (version info, etc.) and instructions for how to apply the update
3. The format supports the use of digital signatures - probably not implemented in v1.
4. Format contains checks for corruption so that updates can be aborted quickly if there's a problem
5. The format supports the partition A/partition B setup to alternate where firmware root filesystems are stored to minimize the window of time where updates can brick the device and enable the feature where the system can go back to a previous firmware if the new one is bad.
6. Applying a firmware update should be bound by the Flash write time. The C-based firmware update system I use is very close to this, so it should be possible here too.
7. MBR systems only for v1. GPT and EFI for later since I currently don't run on any platforms that use these yet.

Design thoughts

1. I'm thinking about moving from the JSON-based instructions list to a small scripting language. I originally thought that firmware update installation logic should be trivial based on my u-boot experience, but I've since had to add an "if" construct to it. My C-based firmware update system uses shell scripts, but I don't want to go there. Thoughts are to embed Lua or implement a simple custom scripting language. I'm leaning to the custom route, since firmware update logic is rule based, and it seems like there should be a simple way to use Erlang's/Elixir's pattern matching to do it.
2. I'm bummed that Erlang ZIP support doesn't work well, since ZIP files are so easy to inspect. But, if ZIP is out, then I'll probably look into the better compression algorithms on TAR or CPIO to see if I can make the images a little smaller.
3. OpenBSD just published a nice package signing utility, http://www.tedunangst.com/flak/post/signify. I can't use it directly since it has OpenBSD-isms in it, but it might not be too hard to move to Nerves and it looks simpler and less error-prone than my experiences with openssl.

Thoughts?

Frank

--
Frank Hunleth
Troodon Software LLC
Embedded Software Development
http://troodon-software.com/

[Omri Rotem]

As for design thoughts, about the custom script: I highly recommend not to.

As far as I can see the needs are hardly new, and as in most cases, custom solutions are kind of messy. you end up building another product with its own life.

my 2 cents.

[Frank Hunleth]

Hi Omri,

On Mon, May 5, 2014 at 4:49 AM, Omri Rotem <omri....@gmail.com> wrote:
>
> As for design thoughts, about the custom script: I highly recommend not to.
>
> As far as I can see the needs are hardly new, and as in most cases, custom solutions are kind of messy. you end up building another product with its own life.

I'm definitely not disagreeing with your statement, but do you have any suggestions for languages to embed or alternatives?

Here are some more thoughts: 

1. It's important to me that I can have a well-defined execution environment so that I can make guarantees that a firmware update will either always apply or always fail harmlessly no matter what version of firmware is executing the upgrade. For the "C" based firmware updates that I do, I use bash scripts, but there's a certain list of commands that are guaranteed to exist. I don't like the bash route, since it's amazingly easy for a command or commandline option to creep in that isn't supported on an old release and that causes a failure that's hard for tech support people to debug without my help. I suppose that I could address some of this via regression testing, but I don't know how to automate it and I know from experience that manual coverage is not 100%.
2. I had wanted to embed a little Erlang script, since that seemed logical. However, that approach feels like the bash script approach in that scripts have open ended access. I'd liked to do better than bash, so if there's some way of easily checking that the script doesn't access Erlang modules that aren't on the guaranteed list, that would be nice.
3. This isn't the biggest problem (once firmware update signatures are implemented), but since firmware updates run as root, it would be nice if the amount of damage that someone could do with a malicious firmware update script were limited. For example, I don't know how to prevent a call to os:cmd/1 for code running on an Erlang VM. At least, it's felt to me that the Erlang VM has no support for restricted execution environments or anything like that. The route to doing this seems to be handled outside of the VM.

I know how to solve my concerns in C by embedding a Lua interpreter that only has bindings to a guaranteed list of operations. Therefore, 1. I know that if the script works once, that it doesn't access functions that aren't guaranteed, and 2. I can verify by inspection that the limited execution environment sandboxes the damage that a malicious user can do. I know that point 2 can never be perfect, but allowing trivial unlimited access seems wrong too.

Having said all this, it seems really silly to create a language if all I want is a restricted execution environment. Figuring out how to turn this into something completely declarative may be the route as well, but I'm not sure that that reduces the amount of code that I need to write and maintain.

Frank

> --
> You received this message because you are subscribed to the Google Groups "nerves-project" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to nerves-projec...@googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.

[Frank Hunleth]

One status update regarding firmware update: I just switched out the
use of Erlang's zip module with calls to the unzip commandline
utility. On my laptop, it reduced programming time of a 137 MB image
from 58 seconds to a little over 2 seconds. I believe that it can be
reduced a little more, but I probably won't address that until I get a
chance to revisit other aspects of firmware update.

Frank

[Omri Rotem]

Hi Frank,

I will open by saying that I am hardly in the position to advise on these issues, as this is not my day to day work.. wish I could be of more concise help here.

Having said that, I will add that there are examples out there, one that I know of is nginx - the web server:

Scripting in nginx script is very very foreign to people, and creates a lot of confusion. the way around it was lua plugins. I have not written any lua plugins, and opted to learn the nginx script, but this is only because I had a very small task to achieve.

Point is: any scripting you may come up with will probably satisfy the current list of features, but as time passes, I fear that more features will be introduced, thus complicating the script, and taking much time both in learning and developing.

All the best,

Omri.

[Frank Hunleth]

Hi Omri,

Points taken. Since the big performance issue with firmware updates
seems to be resolved, I'm going to sit on the firmware update redesign
until better ideas arise on the scripting front. Between your email
and other opinions, I'm realizing that the core problem is that I
don't know how create a well-defined and restricted execution
environment in Erlang. If I did, then this would be a no-brainer since
the "scripting" language could either be Erlang code or a beam file.
Right now, if I need something quickly, I'll probably go with Lua
since I'm familiar with it, but I find that direction somewhat
disappointing.

Thanks,
Frank