why dm-crypt AND dm-verity?

[Trever]

Why does /mnt/stateful_partition/encrypted use dm-crypt

whereas

/mnt/stateful_partition/home is dm-verity?

I have done a *lot* of research on this.  Still coming up short in terms of a real explanation...

Thanks

Trever

[Trever]

Bah!

Sorry, I meant ecryptfs, not dm-verity

Why dm-crypt & ecryptfs?

[Trever]

I realize it's the weekend, meaning more patience than usual is best!  :-)

But does anyone know the answer?  

I'm trying to finish up some documentation I plan to post that explains in some detail what people see when they do a df from a local Chrome OS shell.  If I can include information about this (why dm-crypt in a few places and ecryptfs elsewhere), that would be great.  (And if all goes well, this will not be a contentious document :-)

I have seen some things:  hiding (encrypting) filenames, something to do with the rather complicated encryption that attends a user's account, bug discussions that suggest it's just legacy and will go away.  But a less tentative knowledge- if someone has it- would be great.

Trever

[Sonny Rao]

+wad, keescook

On Sun, Mar 17, 2013 at 8:47 PM, Trever <trr...@gmail.com> wrote:
> I realize it's the weekend, meaning more patience than usual is best! :-)
>

Yup, weekends aren't the best time to ask this type of question on the
list and expect a quick answer.

> But does anyone know the answer?
>

Yes, someone does :-) And we should probably put this onto the wiki
(and/or update what is already there)

> I'm trying to finish up some documentation I plan to post that explains in
> some detail what people see when they do a df from a local Chrome OS shell.
> If I can include information about this (why dm-crypt in a few places and
> ecryptfs elsewhere), that would be great. (And if all goes well, this will
> not be a contentious document :-)
>

I'll try to answer, but I've added folks who know better than me to cc
I think there's a lot of history behind these decisions that Will
probably knows better than anyone else.

eCryptFS is used to encrypt files that are specific to a particular
logged-in user, with a user-specific key that requires the user to
authenticate.

There's a somewhat old design doc here:
http://dev.chromium.org/chromium-os/chromiumos-design-docs/protecting-cached-user-data

I say it's old because there are probably pieces of it that are out of date.

That page has some explanation of what is mounted, and Appendix B of
that discusses some of the previous approches which were considered.
AFAIK, ecryptfs has been used since the first public launch of
chromeos on the CR-48.

dm-crypt is being used for the encrypted stateful, which was added
around R22. This was to make sure system level logs and other
information in /var/ are encrypted. The dm-crypt filesystem is a
separate ext4 filesytem and is created on a dm-crypt device which is
on top of a sparsely allocated loopback file in the stateful
filesystem. I think this relies on hole-punching in ext4 which wasn't
available until more recent kernels.

Hope that helps some.

> I have seen some things: hiding (encrypting) filenames, something to do
> with the rather complicated encryption that attends a user's account, bug
> discussions that suggest it's just legacy and will go away. But a less
> tentative knowledge- if someone has it- would be great.
>
>
> Trever
>
>
>
> On Sunday, March 17, 2013 12:34:27 AM UTC-7, Trever wrote:
>>
>> Bah!
>>
>> Sorry, I meant ecryptfs, not dm-verity
>>
>> Why dm-crypt & ecryptfs?
>>
>>
>>
>> On Sunday, March 17, 2013 12:14:13 AM UTC-7, Trever wrote:
>>>
>>> Why does /mnt/stateful_partition/encrypted use dm-crypt
>>>
>>> whereas
>>>
>>> /mnt/stateful_partition/home is dm-verity?
>>>
>>> I have done a *lot* of research on this. Still coming up short in terms
>>> of a real explanation...
>>>
>>> Thanks
>>>
>>> Trever
>

> --
> --
> Chromium OS discuss mailing list: chromium-...@chromium.org
> View archives, change email options, or unsubscribe:
> http://groups.google.com/a/chromium.org/group/chromium-os-discuss?hl=en
>
>
>

[Trever]

On Sunday, March 17, 2013 9:31:33 PM UTC-7, Sonny Rao wrote:

dm-crypt is being used for the encrypted stateful, which was added
around R22.  This was to make sure system level logs and other
information in /var/ are encrypted.  The dm-crypt filesystem is a
separate ext4 filesytem and is created  on a dm-crypt device which is
on top of a sparsely allocated loopback file in the stateful
filesystem.  I think this relies on hole-punching in ext4 which wasn't
available until more recent kernels.

Hope that helps some. 

Yes, helps, thanks! 

What I still don't understand however is why dm-crypt instead of ecryptfs everywhere encryption is wanted.  If it is obvious or explained somewhere, I missed that.  I'd like to understand this.

Thanks

Trever

[Sonny Rao]

Again I'd defer to Will and Kees, but I got the impression that we
like dm-crypt better, probably because it's simpler, easier to
understand and so on, and so we use it wherever we can.

[Trever]

On Wednesday, March 20, 2013 12:02:22 AM UTC-7, Sonny Rao wrote:

> What I still don't understand however is why dm-crypt instead of ecryptfs
> everywhere encryption is wanted.  If it is obvious or explained somewhere, I
> missed that.  I'd like to understand this.

Again I'd defer to Will and Kees, but I got the impression that we
like dm-crypt better, probably because it's simpler, easier to
understand and so on, and so we use it wherever we can.

On Kees's blog, he talks about the "discard, hole-punching, and TRIM".

That particular entry helps explain the mounting (along with Encryption: dm-crypt slide here).  So that's good.

But only makes me wonder in what sense dm-crypt is "simpler", it being based on this seemingly complicated layering:

"ext4, on dm-crypt, on loopback of a sparse file, on ext4, on SSD"

So still left wondering why dm-crypt & ecryptfs.

Kees?

There doesn't *have* to be an explanation.  My goal is understanding the Chrome OS filesystems which includes understanding why they are the way they are.  In some cases, maybe no particular reason.  But if there are reasons, that should be part of any explanation of the filesystems...

Thanks in advance,

Trever

 

[Trever]

On Wednesday, March 20, 2013 12:17:11 AM UTC-7, Trever wrote:

So still left wondering why dm-crypt & ecryptfs.

Kees?

There doesn't *have* to be an explanation.  My goal is understanding the Chrome OS filesystems which includes understanding why they are the way they are.  In some cases, maybe no particular reason.  But if there are reasons, that should be part of any explanation of the filesystems...

I should add that I have looked through the source for mount-encrypted.c, and I've also been trying to come to terms with the different encryption schemes as extensively detailed in the several relevant design docs.

If I'm supposed to surmise why some filesystem decisions led to ecryptfs and why some led to dm-crypt, I've missed it.  Could be no reason, but, of course, I don't know.

I have seen stuff about dm-thin and some expressed dissatisfaction with ecryptfs performance, so maybe the future is dm-crypt and dm-thin, and the current ecryptfs is just legacy.  But now I'm trotting out my theories (I have a few), and no sense in me speculating.

Thanks,

Trever

[Will Drewry]

I'll try to provide some useful backstory:

When we released Chromium OS in 2009, we used[1] per-user sparse files
on dm-crypt. dm-thin didn't exist, discard/trim weren't safe to be
used anywhere, etc. Instead we used a trick on sign-out where we
resize2fs the user volume to the smallest size and then truncated the
sparse file - then we forgot the dm-crypt key. On sign-in, we'd
always sparsely resize the filesystem back to the full sparse-size.

This worked (shock!), but the challenge was that it meant that a
particularly long lived session could introduce a lot of fs churn.
The churn would make resize2fs slow which in turn made sign-out slow.
This meant reboots from a signed-in session also got slow. On the
whole, it seemed unsustainable if we wanted the experience to be
snappy and safe, and not just safe.

Around that time, ecryptfs had been seeing field testing from
Canonical, and I'd be using it for some other things. It seemed like
the right trade off for the time -- deal with VFS complexity, bad
locking decisions, limited ability to take advantage of the buffer
caches, etc in order to get speedy and reliable sign-in/sign-out. We
then took advantage of passthrough files for passing through
directories that have safely-removable data: Cache, Downloads, etc.
(f...@chromium.org did the migration and legwork to make ecryptfs and
our current tpm-brokered key unwrapped exist.)

Unfortunately, I've never been particularly happy with ecryptfs
itself, and I don't think it will get leaps and bounds better any time
soon. I'd like to move all user data back to per-user dm-crypt
volumes. However, ext4->dm-crypt->loop->ext4->stateful is quite a few
layers. Kees has forced it into working reliably for
mount-encrypted.c with discard support, but doing it in one place is
easier to manage that doing it for each user. We're not sure if
complexity in the vfs is worse than complexity across the various
system interfaces. We keep going back and forth, but the ideal
solution is to use dm-thin. It's the elegant solution that avoids
having two filesystems stacked and passing around discards. I'm
hoping it we can move to it as it matures and we figure out a good
migration strategy :)


As to the "Protecting Cached User Data" doc, it is pretty much
up-to-date. Not too much has changed. However, I didn't realize that
revisions were not publicly accessible. Here is the published design
document for the prior version:
https://docs.google.com/document/d/13Epf_DOW8SBkfNhMBP_c8UABmz-l1wyIQ-bb72yN-gc/pub

hth!
will


1 - http://git.chromium.org/gitweb/?p=chromiumos/platform/cryptohome.git;a=blob;f=lib/cryptohome;h=2e62cff6c17f76c6cd7c81a94cbd3fcaefdb6845;hb=c9845b92f049801da53d57eaf9bd549f5eb0c201
History is truncated from the move to git. This originally lived in svn.

[Trever]

Thanks for this information.  Still have to digest it but one quick question for now:

Do your remarks about trade offs for ecryptfs explain why you're not happy with ecryptfs (eg. bad locking decisions and the buffering issues)?  What is the source of the unhappiness with ecryptfs?  Security, performance, several issues?

I have no position on ecryptfs- not that it would even matter if I did (dah).  Point is, this is not a loaded question, my sole agenda here is trying to understand the "why" (explanatory) questions for Chrome OS configuration I'm seeing.

As you have moved among and continue to assess the various Linux encryption options, I realize this could probably quickly balloon into a large set of questions and a long discussion.  That isn't where I'd like to go here, just seeking quick rationalizations that give a quick lay of the land, if that's possible.

Thanks again, appreciate the information!  It is for documentation that I will post, so hopefully that is useful in return.

Trever

[Will Drewry]

On Wed, Mar 20, 2013 at 2:30 PM, Trever <trr...@gmail.com> wrote:
> Thanks for this information. Still have to digest it but one quick question
> for now:
>
> Do your remarks about trade offs for ecryptfs explain why you're not happy
> with ecryptfs (eg. bad locking decisions and the buffering issues)? What is
> the source of the unhappiness with ecryptfs? Security, performance, several
> issues?
>
> I have no position on ecryptfs- not that it would even matter if I did
> (dah). Point is, this is not a loaded question, my sole agenda here is
> trying to understand the "why" (explanatory) questions for Chrome OS
> configuration I'm seeing.

First off, ecryptfs is making great progress under its current
stewardship. However, it still has hurdles to overcome:
- support asynchronous ciphers (ablkcipher): without that support,
hardware-offload is dead in the water (unless a sync-wrapper is used)
- VFS layering woes -- we want unencrypted pages available in memory
during use, but since the filesystems are layered, you also end up
with the underlying filesystem trying to be helpful by caching too (to
some extent). There's also when do you flush data, how slow is
synchronous flushing (data->vfs->ecryptfs->vfs->ext4->blockdev), etc.
If the wrong error messages are propagated up, you end up with file
corruption. We've seen this more recently with disk full scenarios --
-ENOSPC wasn't bubbling up properly and wham, corruption.

The other side of it is the trade off in leaking file/disk structure.
Since Chrome OS disk usage is pretty predictable, I don't think
there's much tradeoff between an opaque crypto-block device, sparse
cryptoblock device, and ecryptfs, but there is a little -- you can
easily connect every encrypted block back to one logical file. Not
the worst, but it probably opens up more analytical attacks.

> As you have moved among and continue to assess the various Linux encryption
> options, I realize this could probably quickly balloon into a large set of
> questions and a long discussion. That isn't where I'd like to go here, just
> seeking quick rationalizations that give a quick lay of the land, if that's
> possible.

I think some of these are captured in the revision doc I linked in the
last one too, so I hope that's helpful. In general, I prefer dm-crypt
because doing this work at the block device layer provides a cleaner
infrastructure and doesn't muddy sensitive decisions further up the
stack, like with VFS caching.

> Thanks again, appreciate the information! It is for documentation that I
> will post, so hopefully that is useful in return.

Documentation is always good - thanks!
will

[Trever]

Okay, thanks Will.

Quick question (belongs ideally in another thread, but):

How does the kernel know where the root filesystem can be found on disk when booting?

The kernel command line parameters coming from the verified boot just say:

root=/dev/dm-0

That's a logical path and physically, there are two root filesystems anyways (of course).

It appears to me that:

/dev/dm-0 tells kernel to use device mapper, and additional kernel command line parameter dm="several critical verity values" tells the kernel where to find the root filesystem physically, specifically, in the payload parameter, where the UUID of the kernel partition gets substituted and then it says +1, which means the next partition (which is indeed either ROOT-A or ROOT-B).  I.e. payload=UUID+1.   (I guess this is irrelevant but as I understand it, hashtree is at hashstart, which is physically the same partition as the root filesystem, but at the end, outside of the filesystem.)

Is this correct?  I'm guessing.

*In other words, there is logic somewhere that knows how to associate a partition number with the given UUID and then add one to find the partition with the root filesystem?*  If I'm correct, where is this logic- which source file?  I think it's upstream in the kernel now?  Is it do_mount_dm.c?  I'm sorry if this seems a lazy question, but I have searched a bit (problem is not locally- don't have all the source pulled down).

By way of trying to more fully understand dm-verity and vb in general, I have also found two papers on the internet which claim verified boot has a vulnerability surrounding the root filesystem.  One talks about changing the kernel command line parameters (I think), and one about how rootfs can be rolled back and thus vulnerable to a rollback attack.  IIRC.  I *can* provide the links to those papers (one is out of MIT), but (a) you probably know the ones? and (b) I've never been too impressed with these supposed vulnerabilities (should I be?).   I'm thinking the %U kernel command line UUID trick, all verified, potentially addresses these vulnerabilities (the papers are old).  Though one might mangle the partition table in a malicious way here?  The root verity hash has still has to match when the root filesystem is mounted though, so bang- caught- I don't see a vulnerability here.  (It seems to me.)  Primarily, it was never clear to me the vulnerabilities were "real" anyways, because they seemed to ignore the explicit threat model (hence, not really fair critiques).

Thanks,

Trever

[Mike Frysinger]

i think you want:

https://git.kernel.org/cgit/linux/kernel/git/torvalds/linux.git/tree/Documentation/device-mapper/verity.txt

there are also all the files in here:

http://git.chromium.org/gitweb/?p=chromiumos/platform/vboot_reference.git;a=tree;f=scripts/image_signing

those are used when signing images, so they handle a lot of the cmdline manipulation

the kernel command line is included in one of the signing checks at runtime (our verified boot docs cover it), so that can't be manipulated

our system can be rolled back between versions signed with the same key (the latest one), but if we're aware of a version/key that is vulnerable, we can trivially bump the keys & the # and then all versions signed by an older key will be rejected.

i'm not really worried about manipulating the partition table.  the worse you could do is modify the flags to tell it to boot a different area on disk, but the thing you load still has to be signed+verified, so it's not like doing that would help.

-mike

[Trever]

On Thursday, March 21, 2013 3:21:14 PM UTC-7, Mike Frysinger wrote:

i think you want:

https://git.kernel.org/cgit/linux/kernel/git/torvalds/linux.git/tree/Documentation/device-mapper/verity.txt

there are also all the files in here:

http://git.chromium.org/gitweb/?p=chromiumos/platform/vboot_reference.git;a=tree;f=scripts/image_signing

Well...

I just want to understand how the kernel knows where to go to attempt to mount the root filesystem (eg. /dev/mmcblk0p5 or /dev/mmcblk0p3)?  That's what I'm not getting.

I have read through all the verity documentation I can find, including the verity text you link to.  That document talks about specifying <dev>, which is the root filesystem in the case of Chrome OS (as I understand it).

So either that's just relevant for building a root filesystem, or, it suggests that the root filesystem is *not* passed in (pace what I was speculating before, the +1 theory).  Instead, it would seem to say when you build a kernel image (and/or modules), you hard code in the path to what will be the root filesystem.  I find that hard to believe, but is that how it works?

Another option that doesn't seem plausible: it tastes all the partitions it can find until it comes to one that has the correct root block hash (based on the verity value).

Sorry if I'm being obtuse.

Trever

[Trever]

Okay, I believe my +1 theory is essentially correct, that the kernel takes the dm= values, specifically the payload, and goes from that to find where the root filesystem resides.

I'm still parsing through do_mounts_dm.c, which I *think* I found here:

http://git.chromium.org/gitweb/?p=chromiumos/third_party/kernel.git;a=blob;f=init/do_mounts_dm.c;h=989904d9a2fbc5c1ca90a51bbe40819376e45d87;hb=HEAD

Confirmations appreciated...  but this looks correct (including documentation in that and other related files).  

It also makes sense.  I don't see any other way the kernel could find the root filesystem.  There's no initramfs when booting from internal SSD, tasting partitions is the opposite of avoiding probing (i.e. time consuming), you really wouldn't (it seems) want to hard code this value, etc..

Trever

[Trever]

Okay, I believe my +1 theory is essentially correct, that the kernel takes the dm= values, specifically the payload, and goes from that to find where the root filesystem resides.

I'm still parsing through do_mounts_dm.c, which I *think* I found here:

http://git.chromium.org/gitweb/?p=chromiumos/third_party/kernel.git;a=blob;f=init/do_mounts_dm.c;h=989904d9a2fbc5c1ca90a51bbe40819376e45d87;hb=HEAD

Confirmations appreciated...  but this looks correct (including documentation in that and other related files).  

It also makes sense.  I don't see any other way the kernel could find the root filesystem.  There's no initramfs when booting from internal SSD, tasting partitions is the opposite of avoiding probing (i.e. time consuming), you really wouldn't (it seems) want to hard code this value, etc..

Trever

On Thursday, March 21, 2013 3:47:27 PM UTC-7, Trever wrote:

[Sonny Rao]

http://git.chromium.org/gitweb/?p=chromiumos/third_party/kernel.git;a=commit;h=9cd2d7f96861151e8437ae41f93ce4f9d6581014