How to leverage external storage in Crostini container

[Josh Reed]

Hello! I have a Chromebook with 64 GB of internal storage, and I'm looking to use to for dev purposes. I got the Linux "Crostini" container up and running easily, and I got docker installed, as many of my workflows involve docker and/or podman.

However, since docker images can be quite large, I want to be able to move them elsewhere, since I expect to quickly run out of space on my drive. I have a 256 GB SD Card that I plan to just leave in the slot, and I can share its folders with the container via the normal means (where it shows up in /mnt/chromeos). But I haven't been able to figure out how to effectively use that as a storage location for docker.

Things I've tried:

1. Symlinking /var/lib/docker to /mnt/chromeos/path/to/my-shared-folder

Result: docker pull complains that it cannot chown the layers once it downloads them

2. Bind-mounting /var/lib/docker to /mnt/chromeos/path/to/my-shared-folder

Result: same as above: docker pull complains that it cannot chown the layers once it downloads them

3. Disabling rootful docker daemon and enabling rootless docker as my user, and bind-mounting or symlinking the ~/.local docker storage area to /mnt/chromeos/path/to/my-shared-folder

Result: docker continues to complain about not being able to chown

4. Creating an external drive to share with the VM as described and then making an LXD filesystem pool and volume to provide the drive to the container when booted. From https://chromium.googlesource.com/chromiumos/docs/+/HEAD/containers_and_vms.md#how-can-i-run-vms-with-data-images-on-external-storage 

Result: works, can use this filesystem, but it is insanely slow. I didn't do much IO testing here but normal operations like `docker run busybox echo hi` take > 15 s, vs almost instantaneous when the storage location is on the normal mount. Also seems to complicate the lxd startup and make it unstable.

Anybody have any ideas on how I can usably share my SD Card for docker storage use with this container?

[Keith I Myers]

From Crosh

vmc create-extra-disk --size=120G "/media/removable/USB Drive/extra-disk.img"

vmc start termina --extra-disk "/media/removable/MicroSD/extra-disk.img"

You will then need to mount the disk to a mount point in your container. 

Use at your own risk, it's been a bit hit and miss. 

--
--
Chromium OS Discussion mailing list: chromium-...@chromium.org
View archives, change email options, or unsubscribe:
https://groups.google.com/a/chromium.org/group/chromium-os-discuss
---
To unsubscribe from this group and stop receiving emails from it, send an email to chromium-os-dis...@chromium.org.

[Josh Reed]

Thanks! That was method 4 that I was talking about above. It resulted in really slow performance, but I think that might be the LXD volume type I used. Any tips on how you’ve mounted it in the container from the VM?

[Josh Reed]

This ended up not really working... I can't even create a disk larger than about 3G.

crosh> vmc create-extra-disk --size=128G --removable-media "SD Card/extra-disk.img"
Error: operation `extra_disk_create` failed: failed to allocate an extra disk at /media/removable/SD Card/extra-disk.img: File too large (os error 27)

I got the device I did because I figured I could always use SD card for extra space to do development work, but it doesn't really seem like you can use the SD card for much of anything.

Starting to feel like getting a chromebook was a mistake

[Peter Marheine]

It sounds like your SD card might be formatted with a filesystem (FAT32?) that doesn't support files larger than 4GB, in that case. exFAT might work, or ext4 should be a safe choice.

That error message is good enough that I was able to easily find the code that implements that operation, and can tell that it should work with large files which means the kernel is unhappy about the size which often means the filesystem can't handle large files.

[Josh Reed]

Thanks! Sorry, I was getting frustrated and didn't even bother searching for the error message. (I know, troubleshooting 101).

I'll try reformatting in ext4.

If I can't get this to work, I guess there's always developer mode.

[Josh Reed]

Reformatted using exFAT, seems to allow larger files now. Creating the disk does take quite some time, and then creating the VM from that with the `vmc start ...` command seems to take forever too. Is that normal?

[Josh Reed]

Okay, it took some doing, but I think I got it working! Getting reasonable speed all things considered.

Couple of things I've found:

- It takes _forever_ to make the larger disks

- I had to start the vm without the disk attached first, then stop it and re-start it with the disk attached

- Then I had to start the container first without the device added, and then add it with `lxc config device add` after it was running

It's a bit complicated, but if it's mainly to function as my container storage, I can live with it

Thanks for all your help!

[dragon788]

The extremely long creation time is likely because they aren't sparsely allocating the space with thin provisioning, they are preallocating it all so you don't accidentally run out of space by putting something else in that storage and causing data loss when an expanding volume doesn't have room to utilize all the space out thought it had.

The other benefit of 'thick' provisioning is you get the best access speeds your storage is capable of because it isn't performing writes and volume extensions at the same time.

[Josh Reed]

Okay after all that, it looks like the disk has trouble staying mounted after sleep. I'm also getting really slow speeds using it for docker storage, but that could be complicated by btrfs. I think I'm just going to try developer mode.