PSA: cros_sdk chroot implementation changing

[Benjamin Gordon]

Once http://crrev.com/c/553462 goes in later today, new chroots will be created on a sparse image mounted through a loopback device instead of directly in the chroot directory.  Existing chroots will continue to work as-is.  If you only access your chroot through cros_sdk, the difference should be transparent.  If you directly poke around in your chroot, you'll need to make sure to run cros_sdk once after each reboot to get everything mounted back into the correct place.

The new implementation does need lvm2 and thin-provisioning-tools.  If you don't have those and don't want to install them, you can continue creating non-loopback chroots by passing --nouse-image to cros_sdk.

Thanks,

Benjamin

[Jiwoong Lee]

Hi Benjamin,

This PSA is heavy to me and I had spent some time to understand. In particular, I pay attention to the behavior changes before and after. So follow-up questions..

On Wed, Aug 2, 2017 at 8:35 AM, Benjamin Gordon <bmgo...@chromium.org> wrote:

If you directly poke around in your chroot, you'll need to make sure to run cros_sdk once after each reboot to get everything mounted back into the correct place.

Q. Before this change, even after I reboot the system, I have a direct directory access to ~/chromium/chroot. After this change, won't I have, until I issue the command "cros_sdk" at least once?

The new implementation does need lvm2 and thin-provisioning-tools. 

Q. After this change, will "cros_sdk" command start to fail, at the time of chroot creation, if those dependencies are not installed?

If you don't have those and don't want to install them, you can continue creating non-loopback chroots by passing --nouse-image to cros_sdk.

Q. Do you plan to update the developer guide?

https://www.chromium.org/chromium-os/developer-guide#TOC-Create-a-chroot

Q. I used to use "mount --bind" to access the outside chroot from the inside. Will it be impacted?

Thanks

Jiwoong 

[Benjamin Gordon]

On Wed, Aug 2, 2017 at 10:32 AM, Jiwoong Lee <po...@chromium.org> wrote:

Hi Benjamin,

This PSA is heavy to me and I had spent some time to understand. In particular, I pay attention to the behavior changes before and after. So follow-up questions..

On Wed, Aug 2, 2017 at 8:35 AM, Benjamin Gordon <bmgo...@chromium.org> wrote:

If you directly poke around in your chroot, you'll need to make sure to run cros_sdk once after each reboot to get everything mounted back into the correct place.

Q. Before this change, even after I reboot the system, I have a direct directory access to ~/chromium/chroot. After this change, won't I have, until I issue the command "cros_sdk" at least once?

Yes, that's right.  After you run cros_sdk, it will leave the chroot mounted until you manually unmount it or reboot, so you should only have to do this once after each reboot.  Even something trivial like "cros_sdk -- true" should be enough to get everything set up correctly.

The new implementation does need lvm2 and thin-provisioning-tools. 

Q. After this change, will "cros_sdk" command start to fail, at the time of chroot creation, if those dependencies are not installed?

Yes, it will fail and should tell you exactly which command is missing.

 

If you don't have those and don't want to install them, you can continue creating non-loopback chroots by passing --nouse-image to cros_sdk.

Q. Do you plan to update the developer guide?

https://www.chromium.org/chromium-os/developer-guide#TOC-Create-a-chroot

Yes, I'll take a look at updating the guide.

 

Q. I used to use "mount --bind" to access the outside chroot from the inside. Will it be impacted?

This should work exactly as before.  If it doesn't, email me and I'll help debug.

Benjamin

[Vadim Bendebury]

On Wed, Aug 2, 2017 at 8:35 AM, Benjamin Gordon <bmgo...@chromium.org> wrote:

The new implementation does need lvm2 and thin-provisioning-tools.  If you don't have those and don't want to install them, you can continue creating non-loopback chroots by passing --nouse-image to cros_sdk.

This is what I get trying to cros_sdk after repo sync:

$ cros_sdk

The tool(s) lvchange, lvcreate, lvs, pvscan, vgchange, vgcreate, vgs were not found.

Please install the appropriate package in your host.

Example(ubuntu):

  sudo apt-get install <packagename>

this message is quite cryptic: what is "the appropriate package",  how does one find out? It would be good to show it at least for ubuntu, the recommended OS.

--vb

Thanks,

Benjamin

--
--
Chromium OS Developers mailing list: chromiu...@chromium.org
View archives, change email options, or unsubscribe:
http://groups.google.com/a/chromium.org/group/chromium-os-dev?hl=en

[Benjamin Gordon]

On Thu, Aug 3, 2017 at 12:47 PM, Vadim Bendebury <vbe...@chromium.org> wrote:

On Wed, Aug 2, 2017 at 8:35 AM, Benjamin Gordon <bmgo...@chromium.org> wrote:

The new implementation does need lvm2 and thin-provisioning-tools.  If you don't have those and don't want to install them, you can continue creating non-loopback chroots by passing --nouse-image to cros_sdk.

This is what I get trying to cros_sdk after repo sync:

$ cros_sdk

The tool(s) lvchange, lvcreate, lvs, pvscan, vgchange, vgcreate, vgs were not found.

Please install the appropriate package in your host.

Example(ubuntu):

  sudo apt-get install <packagename>

this message is quite cryptic: what is "the appropriate package",  how does one find out? It would be good to show it at least for ubuntu, the recommended OS.

Good idea; I'll see if I can improve that message.  In the meantime, the packages you want on Ubuntu are called lvm2 and thin-provisioning-tools.

[Vadim Bendebury]

Great, thank you!

--vb

[Mike Frysinger]

imo requiring manual intervention is still wrong.  if the tools/support aren't available, the user should not have to manually pass --nouse-image.  it should just work transparently.

-mike

--

[Benjamin Gordon]

I agree that installing the packages is an inconvenience, but I think having a silent fallback is going to lead to hard-to-debug setups later.  Once snapshots and other follow-up features that depend on this are available, people will appreciate getting a clear error message instead of having to track down why the thing they requested doesn't happen.

That said, the current check was probably too aggressive.  I've sent out https://crrev.com/c/600540 to avoid making the check when the chroot is already populated.  That should let existing chroots keep working without needing to pass --nouse-image, but will still give an up-front error when attempting to create/replace a new chroot without the needed packages installed.  Please take a look and see if that addresses your concern.

Benjamin

[hannah....@intel.com]

> We are seeing build issues related to CL

> https://chromium-review.googlesource.com/c/553462/12/scripts/cros_sdk.

> py

> 

> 

> chromiumos/chromite/scripts/cros_sdk.py", line 718, in main

> 

> parent_ns_file = open('/proc/%d/ns/mnt' % (os.getppid(),))

> 

> IOError: [Errno 2] No such file or directory: '/proc/56943/ns/mnt'

> 

> Build step 'Execute shell' marked build as failure.

> 

> The issue persists despite using cros_sdk --enter --nouse-image.

> 

> Another related issue is that while trying to delete cros_sdk,

> chroot.build shows as device or resource busy. Trying with sudo,

> unmounting the dir or checking for background processes related to

> chroot does not seem to work either.

> 

> Chroot.build is being created with root user instead of the actual

> user running the build, which is causing an issue with automated builds.

> 

> Please let us know what are we missing?

[russ...@intel.com]

after installing lvm2 and thin-provisioning-tools into Ubuntu 14.04 and clearing out the cros_sdk (cros clean --clobber --debug), doing cros_sdk results in a new mount entry:

/dev/mapper/cros_ssd+chromiumos_3+chroot_000-chroot  493G  5.0G  488G   2% /ssd/chromiumos_3/chroot
/dev/mapper/cros_ssd+chromiumos_2+chroot_000-chroot  493G  4.9G  488G   1% /ssd/chromiumos_2/chroot.build/chroot

[russ...@intel.com]

per a request from Benjamin for more info here:

- I have 2 different trees, chromiumos_2 and chromiumos_3. They were downloaded and built a few weeks ago, so they are coming in as "before the change" repos. I tried various sequences of rep sync, setup_board and build_packages in my testing. The -nouse-image flag did work for me. After reading the above messages I installed the 2 packages. I noticed that just doing cros_sdk with a preformed chroot did not show the mount. After removing the chroot via cros clean and running cros_sdk to download a new one, I then saw the mount point. The output below shows diff end directories: chroot and chroot.build. After leaving the chromiumos_2 chroot they both now display as chroot only. In both, after exiting the cros_sdk command the mount points persist.

[Benjamin Gordon]

For the first part, cros_sdk doesn't convert an existing chroot to a loopback unless you pass --replace or delete it and recreate.  However, it does (incorrectly) require the lvm tools to be installed even if they're not needed.  I have https://crrev.com/c/600540 out for review to make it not check for lvm tools in that scenario, although that shouldn't matter for you now that you've already installed the two packages (and sorry about that).

For the second part, cros_sdk moves the mount point from chroot.build/chroot onto the real chroot after the initial command inside the chroot exits.  If you're running something long-running in there and concurrently accessing the chroot before it exits, the effect will be exactly what you've described.  I'm working on a change that will get rid of the chroot.build directory entirely and make the chroot directory visible immediately; that should be ready within a couple of days.  If you need a workaround in the meantime, you could either pass --nouse-image or run an initial very short command to get it mounted, e.g. cros_sdk -- true.

Thanks,

Benjamin

[Yunlian Jiang]

Previously, I can enter the same chroot with multiple sessions, now I
can enter a chroot with only one session,
For other sessions, I got

14:25:36: INFO: chroot lock: blocking while taking write lock

Is this related to the change? Can we fix this?

Thanks

> ---
> You received this message because you are subscribed to the Google Groups
> "Chromium OS dev" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to chromium-os-d...@chromium.org.

[Benjamin Gordon]

Yes, this is related to this change.  I'm working on a fix, which I'm hoping to have ready in the next day or two.  In the meantime, you can probably get back into a working state by exiting from your first cros_sdk command.  When it finishes, the chroot will be mounted into the correct place and then subsequent cros_sdk commands should enter properly without taking the locks.

Thanks,

Benjamin

> email to chromium-os-dev+unsubscribe@chromium.org.

[Benjamin Gordon]

I've re-worked the loopback implementation to address the problems people have reported.  Those of you who have had problems with the loopback images in cros_sdk, please try patching https://crrev.com/c/608585 and give it another try.  If you still have problems, let me know.  I plan to submit this early next week if I don't hear about any problems.

Thanks,

Benjamin

[Pratik Prajapati]

I see this error with and without using --nouse-image.

I have GBs of disk space in my machine. Any WA or fix?

17:49:48: INFO: Determining required toolchain updates...

17:49:49: INFO: Updating packages:

17:49:49: INFO: ['sys-libs/libcxx', 'sys-kernel/linux-headers', 'sys-libs/libcxxabi']

17:49:49: INFO: RunCommand: /mnt/host/source/chromite/bin/parallel_emerge --oneshot --update --getbinpkg --usepkgonly sys-libs/libcxx sys-kernel/linux-headers sys-libs/libcxxabi

Traceback (most recent call last):

  File "/mnt/host/source/chromite/bin/parallel_emerge", line 168, in <module>

    DoMain()

  File "/mnt/host/source/chromite/bin/parallel_emerge", line 164, in DoMain

    commandline.ScriptWrapperMain(FindTarget)

  File "/mnt/host/source/chromite/lib/commandline.py", line 889, in ScriptWrapperMain

    target = find_target_func(target)

  File "/mnt/host/source/chromite/bin/parallel_emerge", line 139, in FindTarget

    module = cros_import.ImportModule(target)

  File "/mnt/host/source/chromite/lib/cros_import.py", line 43, in ImportModule

    module = __import__(target)

  File "/mnt/host/source/chromite/scripts/parallel_emerge.py", line 122, in <module>

    KILLED = multiprocessing.Event()

  File "/usr/lib64/python2.7/multiprocessing/__init__.py", line 211, in Event

    return Event()

  File "/usr/lib64/python2.7/multiprocessing/synchronize.py", line 302, in __init__

    self._cond = Condition(Lock())

  File "/usr/lib64/python2.7/multiprocessing/synchronize.py", line 147, in __init__

    SemLock.__init__(self, SEMAPHORE, 1, 1)

  File "/usr/lib64/python2.7/multiprocessing/synchronize.py", line 75, in __init__

    sl = self._semlock = _multiprocessing.SemLock(kind, value, maxvalue)

OSError: [Errno 28] No space left on device

> email to chromium-os-d...@chromium.org.

[Benjamin Gordon]

That looks like a failure to create a semaphore rather than a disk that's out of space.  Semaphores are normally created in shared memory, so you could check and make sure that /dev/shm is mounted and hasn't been filled up.

Thanks,

Benjamin

[Pratik Prajapati]

Hi Benjamin,

Yes its mounted and not filled up.

from ubuntu shell:

Filesystem                                           Size  Used Avail Use% Mounted on

tmpfs                                                 16G  133M   16G   1% /dev/shm

--

Regards,

Pratik

[ahas...@chromium.org]

I have a similar problem that once in awhile my workstation runs out of space even though it is a 1TB disk and I have not used much. A simple 'du' on my chromiumos folder shows 692GB of used space, but I have only less than 200GB of data + 536GB of chroot.img. Why chroot.img is such a big file?

Thanks,

Amin.

[Sage, Russ]

My BKM is to periodically nuke the chroot and all the cache(s). This will help manage the disk usage. But you take the hit to download the tarball on next cros_sdk.

Run “cros clean –clobber –debug”

--

[ahas...@chromium.org]

But that's very inconvenient. The mounted chroot folder size for me is only about 50GB. I don't know what are the implementation details of chroot, but doesn't chroot.img should gradually increase in size as more files are added? Why is it a large fixed size? Am I missing something?

Thanks,

Amin.

To unsubscribe from this group and stop receiving emails from it, send an email to chromium-os-dev+unsubscribe@chromium.org.

[Pratik Prajapati]

No WA yet.. :( Still same error. Any inputs from Google/Chromium ?

I did "cros clean --clobber --debug"

and then 

$cros_sdk --nouse-image

 * Generating locale-archive: forcing # of jobs to 1

17:57:38: INFO: Determining required toolchain updates...

17:57:39: INFO: Nothing to update!

17:57:39: INFO: Nothing to clean!

INFO    cros_sdk:make_chroot: Running emerge curl sudo gentoolkit ...

 * Generating locale-archive: forcing # of jobs to 1

Traceback (most recent call last):

  File "/mnt/host/source/chromite/bin/parallel_emerge", line 168, in <module>

    DoMain()

  File "/mnt/host/source/chromite/bin/parallel_emerge", line 164, in DoMain

    commandline.ScriptWrapperMain(FindTarget)

  File "/mnt/host/source/chromite/lib/commandline.py", line 889, in ScriptWrapperMain

    target = find_target_func(target)

  File "/mnt/host/source/chromite/bin/parallel_emerge", line 139, in FindTarget

    module = cros_import.ImportModule(target)

  File "/mnt/host/source/chromite/lib/cros_import.py", line 43, in ImportModule

    module = __import__(target)

  File "/mnt/host/source/chromite/scripts/parallel_emerge.py", line 122, in <module>

    KILLED = multiprocessing.Event()

  File "/usr/lib64/python2.7/multiprocessing/__init__.py", line 211, in Event

    return Event()

  File "/usr/lib64/python2.7/multiprocessing/synchronize.py", line 302, in __init__

    self._cond = Condition(Lock())

  File "/usr/lib64/python2.7/multiprocessing/synchronize.py", line 147, in __init__

    SemLock.__init__(self, SEMAPHORE, 1, 1)

  File "/usr/lib64/python2.7/multiprocessing/synchronize.py", line 75, in __init__

    sl = self._semlock = _multiprocessing.SemLock(kind, value, maxvalue)

OSError: [Errno 28] No space left on device

--

Regards,

Pratik

You received this message because you are subscribed to a topic in the Google Groups "Chromium OS dev" group.
To unsubscribe from this topic, visit https://groups.google.com/a/chromium.org/d/topic/chromium-os-dev/yVEWDR7wDfA/unsubscribe.
To unsubscribe from this group and all its topics, send an email to chromium-os-dev+unsubscribe@chromium.org.

[Simon Glass]

Hi,

On 30 August 2017 at 07:38, <ahas...@chromium.org> wrote:
> I have a similar problem that once in awhile my workstation runs out of
> space even though it is a 1TB disk and I have not used much. A simple 'du'
> on my chromiumos folder shows 692GB of used space, but I have only less than
> 200GB of data + 536GB of chroot.img. Why chroot.img is such a big file?
>

It is a sparse file so does not really use that much space. If you
type 'df' inside the chroot you will see how much space it is really
using.

Regards,
Simon

>
> Thanks,
> Amin.
>
>
> On Wednesday, August 2, 2017 at 8:35:50 AM UTC-7, Benjamin Gordon wrote:
>>
>> Once http://crrev.com/c/553462 goes in later today, new chroots will be
>> created on a sparse image mounted through a loopback device instead of
>> directly in the chroot directory. Existing chroots will continue to work
>> as-is. If you only access your chroot through cros_sdk, the difference
>> should be transparent. If you directly poke around in your chroot, you'll
>> need to make sure to run cros_sdk once after each reboot to get everything
>> mounted back into the correct place.
>>
>> The new implementation does need lvm2 and thin-provisioning-tools. If you
>> don't have those and don't want to install them, you can continue creating
>> non-loopback chroots by passing --nouse-image to cros_sdk.
>>
>> Thanks,
>> Benjamin
>

[Pratik Prajapati]

From my earlier email,

Filesystem                                           Size  Used Avail Use% Mounted on

tmpfs                                                 16G  133M   16G   1% /dev/shm

This issue has really blocked me 100% :( Is this a bug on cros_sdk?

--

Regards,

Pratik

[Simon Glass]

Hi Patrik,

On 30 August 2017 at 11:04, Pratik Prajapati <pratik.p...@gmail.com> wrote:
>
> From my earlier email,
>
> Filesystem Size Used Avail Use%
> Mounted on
> tmpfs 16G 133M 16G 1%
> /dev/shm
>
> This issue has really blocked me 100% :( Is this a bug on cros_sdk?

I'm sorry about that. I have not seen this problem myself. I wonder
what is actually running out of space?

My comment was about the size of chroot.img. You asked 'Why chroot.img
is such a big file?'.

Also I'm not sure about the rules on this list but normally it is
better not to top post on mailing lists, since it then makes the
threads very confusing.

Regards,
Simon

>> chromium-os-d...@chromium.org.
>>
>

[Pratik Prajapati]

Hi Simon,

I guess Amin asked for size of chroot.img.

>>Also I'm not sure about the rules on this list but normally it is

better not to top post on mailing lists, since it then makes the
threads very confusing.

Please let me know if you want me to start a new thread or file a bug!

--

Regards,

Pratik

>> chromium-os-dev+unsubscribe@chromium.org.
>>
>

[Pratik Prajapati]

Hi Benjamin,

This is just fresh chroot.

Steps:

repo init -u https://chromium.googlesource.com/chromiumos/manifest.git

my depot tools are also latest.

I just do "cros_sdk --nouse-image" to create chroot environment and i see the issue.

i get same error without --nouse-image param also.

--

Regards,

Pratik

On Tue, Aug 29, 2017 at 8:14 PM, Benjamin Gordon <bmgo...@google.com> wrote:

On Tue, Aug 29, 2017 at 9:04 PM, Pratik Prajapati <pratik.p...@gmail.com> wrote:

From my earlier email,

Filesystem                                           Size  Used Avail Use% Mounted on

tmpfs                                                 16G  133M   16G   1% /dev/shm

This issue has really blocked me 100% :( Is this a bug on cros_sdk?

I don't think this is a known issue.  Could you provide a list of steps that get into this state, e.g. does it affect a fresh chroot, do you have to run a particular command inside the chroot to see the problem, etc?

Thanks,

Benjamin

[Benjamin Gordon]

On Tue, Aug 29, 2017 at 9:45 PM, Pratik Prajapati <pratik.p...@gmail.com> wrote:

Hi Benjamin,

This is just fresh chroot.

Steps:

repo init -u https://chromium.googlesource.com/chromiumos/manifest.git

my depot tools are also latest.

I just do "cros_sdk --nouse-image" to create chroot environment and i see the issue.

i get same error without --nouse-image param also.

That definitely doesn't sound like the normal behavior.  Would you mind opening a bug on crbug.com with whatever additional details you think might be relevant?  I'll take a look and try to help debug as soon as I get to the office tomorrow morning.

 

Thanks,

Benjamin

[Pratik Prajapati]

https://bugs.chromium.org/p/chromium/issues/detail?id=760660 is filed.

--

Regards,

Pratik

[Zhongze Hu]

I'm not sure whether this is related to the chroot implementation change. When I tried to build packages within chroot, I got error messages:

ERROR   : Fatal: Missing upgrade hook for 151

ERROR   : Chroot version is too new. Consider running cros_sdk --replace

And 'cros_sdk --replace' failed with the same error.

Anyone knows how to fix this?

Thanks,

Zhongze

You received this message because you are subscribed to the Google Groups "Chromium OS dev" group.

To unsubscribe from this group and stop receiving emails from it, send an email to chromium-os-dev+unsubscribe@chromium.org.

[Benjamin Gordon]

That error means you're missing the "151" version upgrade hook from $CROS_ROOT/src/scripts/chroot_version_hooks.d.  The file in my checkout was added on Aug 1.  Maybe you need a repo sync?

Thanks,

Benjamin