Some comments on TUF after playing with it a bit

[Corey Minyard]

I'm working on building an update system and I've been playing with TUF for a bit.  I like the design and how well thought out it is.  And it's fairly easy to use.  I do have some comments and questions on it, though.

I'm currently playing with building a manifest-oriented update system.  Basically you have sequentially numbered manifest files on the repo and a list of files in each manifest.  The client would fetch the latest manifest file from the repo, compare that with the current one, download any new or updated files, and then call a program with the list of new, updated, and deleted files from the manifest.  This could be used to download updates for package manager-based systems (like rpm or rpm-ostree), delta image based files (like for ostree), or even image-based systems.  This should be something pretty flexible and a step easier to use than TUF itself.

I have looked at uptane, but that seems very focused not not very flexible.

Now my comments.

TUF is not an update framework.  It is an authenticated file transfer mechanism with protection against many known attacks.  You can use it to build an update system, but it could be used for a lot of things besides that.  I'm not sure what to do with this comment, but it's an observation, I guess.

repo.py and client.py are terrible names for the scripts.  I've renamed them to tuf-repo and tuf-client in my yocto recipes, could the same be done in TUF itself?

I know it's been mentioned before, but a C implementation would be quite handy for small systems.  I guess add a vote for that.

There appears to be no way to update the timestamp or snapshot file time with the repo tool.  You can write a little script to do it easily enough, but that seems like something that should be built in to the repo tool.  Am I missing something?

I'd like to generate some instructions on how to do all this with air-gapped systems.  But I'm not quite sure of the procedure to update a repository without causing possible glitches on the clients, nor am I 100% clear on proper key management.  It seems like for best security, you would have three systems:

   root-system   --->    build-system  --->  web-server

In my mind, the root system(s) is where you create the keys (allowing multiple signing keys on different root systems for surviving the compromise of root keys).  This system is only used for initial creation and if keys are compromised or about to expire.  The root key(s) stays there and are not transferred to anything else (except for secure backup).  You would then transfer everything but the root key(s) to the build system, which would take the output of a build, add everything to the repository, put the client part onto the target repository for the final initial target image.  You would then transfer the repository to the web server along with the timestamp key.  The web server periodically updates the timestamp file.

What is the best way to do the transfer to the web server?  If seems like you could get into a situation where a client transfers the metadata while it is being updated, and the client would be confused, either getting a partial metadata file or files mixed between metadata versions.  In fact, looking at the code, that looks possible even when the repo tool copies the metadata over from staging.  Consistent snapshots don't seem to solve this problem, they seem to solve another problem.

What is the procedure for key update?  If a key has been compromised or is about to expire, how do you handle that?

I'm not sure where the snapshot key and file update falls in all this.  It's default expiry is one week, so there probably needs to be some sort of automated system to update it.  But I don't think you want that on the web server, right?  How is that expected to work?

Sorry for all the newbie questions.

-corey

[Trishank Kuppusamy]

i agree with many things you said

but in 2019, we should use Rust, not C: https://github.com/heartsucker/rust-tuf

--
You received this message because you are subscribed to the Google Groups "The Update Framework (TUF)" group.
Visit this group at https://groups.google.com/group/theupdateframework.
To view this discussion on the web visit https://groups.google.com/d/msgid/theupdateframework/29c956c0-e06d-42a7-bedb-01e3b6f26f52%40googlegroups.com.

--

https://keybase.io/trishankdatadog

[Sebastien Awwad]

These are good questions -- and questions about the system are much appreciated.

> TUF is not an update framework.  It is an authenticated file transfer mechanism with protection against many known attacks.  You can use it to build an update system, but it could be used for a lot of things besides that.  I'm not sure what to do with this comment, but it's an observation, I guess.

Maybe we should call it TNU for "TUF is Not an Updater" instead. :)   There's something to this point, but TUF isn't a "file transfer mechanism", really: much of the focus of the TUF design is on producing a trust structure on the repository side that clients can leverage and that is compromise resilient, and the design doesn't care what file transfer mechanism is employed.  (The reference implementation's verification client uses HTTP(S), but we're working on opening that up a bit, and the spec doesn't care about method of acquisition.)  Personally, on the client/verifier side, I think of TUF as a library for performing verification of data with guarantees for freshness and against various attacks.  On the provider/repository side, I think of TUF as a compromise-resilient trust scheme and metadata definition.

Regarding flexibility, have you taken a look at the specifications for TUF and Uptane?

> There appears to be no way to update the timestamp or snapshot file time with the repo tool.  You can write a little script to do it easily enough, but that seems like something that should be built in to the repo tool.  Am I missing something?

> repo.py and client.py are terrible names for the scripts.  I've renamed them to tuf-repo and tuf-client in my yocto recipes, could the same be done in TUF itself?

repo.py and client.py are mostly tutorial tools.  I updated the tutorials to make this clearer (example) not too long ago.  You'd want to use tuf.client.updater and tuf.repository_tool for the detailed tutorial and real applications.  That should provide all the functionality required -- and if not, please follow up!

> I know it's been mentioned before, but a C implementation would be quite handy for small systems.  I guess add a vote for that.

I lose track of the implementations of TUF, but other folks on this list may be more helpful there.  There are implementations in Go and Rust, among others, if those are helpful.  There might be a C implementation out there, but I can't think of one.  I'm not much of a C developer anymore, so I'm not likely to be useful there anytime soon....  The TUF specification should make it so that it isn't too bad to implement one, and hopefully the reference implementation is useful in that regard, too.

> I have looked at uptane, but that seems very focused not not very flexible.

Uptane is a bit particular to automotive and related fields.  Some folks have expressed interest in using it for IoT, which makes some sense.  For its intended applications, I think the design is fairly flexible.

> What is the procedure for key update?  If a key has been compromised or is about to expire, how do you handle that?

The role that delegates trust to that key updates, removing/replacing that key.  For example, version 1 of Root.json might list key A for role Targets, and version 2 of Root.json can list key B instead.  All sorts of arrangements are possible, but that's the gist of it.

Alternatively, there are additional mechanisms in the works for replacing keys you have control over, if the role delegating trust is held by some other group that is slow to produce new metadata.

> I'm not sure where the snapshot key and file update falls in all this.  It's default expiry is one week, so there probably needs to be some sort of automated system to update it.  But I don't think you want that on the web server, right?  How is that expected to work?

You can set the snapshot expiry wherever makes the most sense for your system.  In some systems, I'd expect snapshot to be automated (for example, in a community repository like PyPI), and in others not.  Snapshot being an offline key is helpful for some small additional security advantages, but an online key often makes sense.

I hope I didn't miss much.  :)  As for your more detailed questions and proposals on repository design and how to configure things, I should take a closer look, but have you seen the TUF papers?  In particular, Survivable Key Compromise might be useful.

--

[Corey Minyard]

On Wednesday, May 1, 2019 at 12:07:08 PM UTC-5, Sebastien Awwad wrote:

These are good questions -- and questions about the system are much appreciated.

> TUF is not an update framework.  It is an authenticated file transfer mechanism with protection against many known attacks.  You can use it to build an update system, but it could be used for a lot of things besides that.  I'm not sure what to do with this comment, but it's an observation, I guess.

Maybe we should call it TNU for "TUF is Not an Updater" instead. :)   There's something to this point, but TUF isn't a "file transfer mechanism", really: much of the focus of the TUF design is on producing a trust structure on the repository side that clients can leverage and that is compromise resilient, and the design doesn't care what file transfer mechanism is employed.  (The reference implementation's verification client uses HTTP(S), but we're working on opening that up a bit, and the spec doesn't care about method of acquisition.)  Personally, on the client/verifier side, I think of TUF as a library for performing verification of data with guarantees for freshness and against various attacks.  On the provider/repository side, I think of TUF as a compromise-resilient trust scheme and metadata definition.

Yes you are right, I was sloppy with my definition :).  But I can see this tool being useful for things that have nothing to do with software updates.  Anything where you need compromise resilient trust could benefit from this.

 

Regarding flexibility, have you taken a look at the specifications for TUF and Uptane?

Yes, I have looked a bit at those.

 

> There appears to be no way to update the timestamp or snapshot file time with the repo tool.  You can write a little script to do it easily enough, but that seems like something that should be built in to the repo tool.  Am I missing something?

> repo.py and client.py are terrible names for the scripts.  I've renamed them to tuf-repo and tuf-client in my yocto recipes, could the same be done in TUF itself?

repo.py and client.py are mostly tutorial tools.  I updated the tutorials to make this clearer (example) not too long ago.  You'd want to use tuf.client.updater and tuf.repository_tool for the detailed tutorial and real applications.  That should provide all the functionality required -- and if not, please follow up!

I have gone through the tutorial and written some code, and it does provide what is required.  As far as the naming, good naming is important, even if it is a tutorial.  At least giving it a name that's unambiguous and unlikely to conflict with something else would be nice.  It is automatically installed by pip, after all.  client.py is kind of useless by itself, really, but repo.py has useful capabilities.

As far as the timestamp updates, I my thought was: "This is fundamental, why isn't it here?  I must be missing something."

 

> I have looked at uptane, but that seems very focused not not very flexible.

Uptane is a bit particular to automotive and related fields.  Some folks have expressed interest in using it for IoT, which makes some sense.  For its intended applications, I think the design is fairly flexible.

I'd agree.  I'm not against Uptane (indeed I'm stealing stuff from it :) but it didn't meet all my needs, and it currently relies on an enormous Java binary, which is problematic for me.

 

> What is the procedure for key update?  If a key has been compromised or is about to expire, how do you handle that?

The role that delegates trust to that key updates, removing/replacing that key.  For example, version 1 of Root.json might list key A for role Targets, and version 2 of Root.json can list key B instead.  All sorts of arrangements are possible, but that's the gist of it.

Alternatively, there are additional mechanisms in the works for replacing keys you have control over, if the role delegating trust is held by some other group that is slow to produce new metadata.

The confusing part for me is root key update.  How does a client know to trust a new Root.json file if you have a new root key?  Especially if metadata is updates several times before a client looks at it again.

 

I hope I didn't miss much.  :)  As for your more detailed questions and proposals on repository design and how to configure things, I should take a closer look, but have you seen the TUF papers?  In particular, Survivable Key Compromise might be useful.

I will look at those papers, they may (well, probably) answer some of my questions.  One that it probably won't answer is:

What is the best way to do the transfer to the web server?  If seems like you could get into a situation where a client transfers the metadata while it is being updated, and the client would be confused, either getting a partial metadata file or files mixed between metadata versions.  In fact, looking at the code, that looks possible even when the repo tool copies the metadata over from staging.  Consistent snapshots don't seem to solve this problem, they seem to solve another problem.

Am I missing something?  It looks like the TUF repository code just copies the files over without any way of assuring atomicity.  That seems like it could confuse clients if the timing was bad.  Or maybe, the client would just error out and get it next time?

-corey

[Corey Minyard]

On Wednesday, May 1, 2019 at 1:58:04 PM UTC-5, Corey Minyard wrote:

On Wednesday, May 1, 2019 at 12:07:08 PM UTC-5, Sebastien Awwad wrote:

> What is the procedure for key update?  If a key has been compromised or is about to expire, how do you handle that?

The role that delegates trust to that key updates, removing/replacing that key.  For example, version 1 of Root.json might list key A for role Targets, and version 2 of Root.json can list key B instead.  All sorts of arrangements are possible, but that's the gist of it.

Alternatively, there are additional mechanisms in the works for replacing keys you have control over, if the role delegating trust is held by some other group that is slow to produce new metadata.

The confusing part for me is root key update.  How does a client know to trust a new Root.json file if you have a new root key?  Especially if metadata is updates several times before a client looks at it again.

 

The paper were helpful with this one.  I can see that you need multiple root keys to handle the compromise of a root key.  And if the root key is not compromised, then you can use it to sign the next root.json file, and that the root.json files are versioned themselves.  This makes more sense now.  The versioning of the root.json files was what I missed, really.  It doesn't seem to be documented in the documentation, but I found some info in the code.

-corey