losetup: cannot find an unused loop device , kernel config of loopback device

[Colin Williams]

I'm running bullseye / debian 11.0 testing. I have been running a
script that is supposed to write a filesystem image (for chromiumOS).
In short I'm getting the following

losetup: cannot find an unused loop device
partx: stat of failed: No such file or directory
partx: stat of failed: No such file or directory

Or for a greater description http://ix.io/3v3i

Looking up the error someone mentioned that the kernel needed to be
configured to support loopback devices. I was curious where the kernel
config is located. I looked at https://wiki.debian.org/KernelFAQ and
it seems to be extremely out of date. Referencing a 2.6 kernel and
mentioning kernel configs at /boot/ which are not existent on my
system.

Can someone tell me where I should look for the kernel for the
loopback setting?


Better yet, does anyone know how to solve my problem with the script
or losetup? Could this be an issue building from a chroot environment
with a different kernel version? Or building the image from a btrfs
filesystem?

[to...@tuxteam.de]

On Thu, Aug 05, 2021 at 04:36:59AM -0700, Colin Williams wrote:
> I'm running bullseye / debian 11.0 testing. I have been running a
> script that is supposed to write a filesystem image (for chromiumOS).
> In short I'm getting the following
>
> losetup: cannot find an unused loop device
> partx: stat of failed: No such file or directory
> partx: stat of failed: No such file or directory

Could you please do a quick

sudo losetup -a

and perhaps

ls /dev/loop*

and tell us the result?

> Looking up the error someone mentioned that the kernel needed to be

> configured to support loopback devices [...]

It would be an exotic kernel lacking that. Perhaps they are all used?

> Better yet, does anyone know how to solve my problem with the script
> or losetup? Could this be an issue building from a chroot environment
> with a different kernel version? Or building the image from a btrfs
> filesystem?

First let's try to understand what is going on. Perhaps some uncivilised
script has left around busy but unused loop devices.

Cheers
- t

[Thomas Schmitt]

Hi,

> Can someone tell me where I should look for the kernel for the
> loopback setting?

Quite exactly a year ago i learned the hard way that it's
CONFIG_BLK_DEV_LOOP which on amd64 should be set to "m" to get /dev/loop*.
See its description at
https://sources.debian.org/src/linux/5.10.46-1/drivers/block/Kconfig/#L174

But i'm somewhat puzzled by this statement at the end
"Most users will answer N here."

A code search yields no BLK_DEV_LOOP=n
https://codesearch.debian.net/search?q=package%3Alinux+BLK_DEV_LOOP%3Dn&literal=0
but lots of "y" and "m".

(Wasn't there a way to inquire the running kernel's configuration ?)

-----------------------------------------------------------------------

I see that to...@tuxteam.de did not Cc you. In case you are not subscribed,

i repeat what he wrote:

> Could you please do a quick
> sudo losetup -a
> and perhaps
> ls /dev/loop*
> and tell us the result?

> [...] Perhaps they are all used?

Have a nice day :)

Thomas

[The Wanderer]

On 2021-08-05 at 09:04, Thomas Schmitt wrote:

> Hi,
>
>> Can someone tell me where I should look for the kernel for the
>> loopback setting?
>
> Quite exactly a year ago i learned the hard way that it's
> CONFIG_BLK_DEV_LOOP which on amd64 should be set to "m" to get /dev/loop*.
> See its description at
> https://sources.debian.org/src/linux/5.10.46-1/drivers/block/Kconfig/#L174
>
> But i'm somewhat puzzled by this statement at the end
> "Most users will answer N here."

That does seem odd; at least in the modern Linux world, loopback-device
support is going to be expected if not required on the large majority of
systems. I can't rule out that this description may have been accurate
at one point, but unless an N here will mean that loopback-device
support will be provided by some other code path, it does indeed not
seem likely that it is what most users will/should say.

It might be worth filing a documentation-level kernel bug report about
this, or at least posting to the LKML to ask what the reasoning here is.

> A code search yields no BLK_DEV_LOOP=n
> https://codesearch.debian.net/search?q=package%3Alinux+BLK_DEV_LOOP%3Dn&literal=0
> but lots of "y" and "m".
>
> (Wasn't there a way to inquire the running kernel's configuration ?)

AFAIK, that's /proc/config.gz; it's present only if a specific Kconfig
setting is enabled, and Debian stopped enabling that setting quite some
years ago, apparently on the grounds that keeping the kernel config in
memory at all times is unnecessarily wasteful (especially with the
limited RAM that you see on e.g. embedded-type systems) vs. keeping it
on-disk.

At no later than that same time, Debian started putting the kernel
config on-disk instead, under the filename /boot/config-`uname -r`. If -
as the OP stated - the kernel config is not present in that location,
then either there's information we're not being provided, or something
is wrong.

--
The Wanderer

The reasonable man adapts himself to the world; the unreasonable one
persists in trying to adapt the world to himself. Therefore all
progress depends on the unreasonable man. -- George Bernard Shaw

[Thomas Schmitt]

Hi,

The Wanderer wrote:
> It might be worth filing a documentation-level kernel bug report about
> this, or at least posting to the LKML to ask what the reasoning here is.

... together with a handful of sr and isofs bug fixes, if this was an
ideal world. {:)

I wrote:
> > (Wasn't there a way to inquire the running kernel's configuration ?)

The Wanderer wrote:
> Debian started putting the kernel
> config on-disk instead, under the filename /boot/config-`uname -r`.

Ah yes. That exists here and says
CONFIG_BLK_DEV_LOOP=m

(Last year i igot missing CONFIG_BLK_DEV_LOOP by not running
make oldconfig
before compiling the kernel source.)

> If -
> as the OP stated - the kernel config is not present in that location,
> then either there's information we're not being provided, or something
> is wrong.

In any case i should read original posts more carefully.

[Colin Williams]

Hello everyone,

In hindsight after looking at this much too late there were many
mistakes in my initial mail. The issue may or may not be debian
related and involves at least analyzing the script. There is a claim
in the documentation that

Then I'll make another attempt to further expose my ignorance.

1) kernels configs are located at /boot/. Not sure why I couldn't find
them. Perhaps a typo.

2) CONFIG_BLK_DEV_LOOP is set to M as expected. Using a file as a
loopback seems like it would be a common practice

3) I'm running a script /build_image --board=${BOARD}
--noenable_rootfs_verification dev whose output is shown in
http://ix.io/3v3i

4) I assume the script is supposed to create a file used as a loopback
device to write the image contents

5) The issue then might likely be script related and not debian related

6) I also posted to the chromium-dev google group, waiting for approval

7) I modified the build_image script with set -x, then recorded the
output: http://ix.io/3v6K

8) I believe the issues arise in
https://github.com/drocsid/cros-scripts/blob/main/common.sh which
seems to create the loopback file, etc.

9) from the log output shared above we see:
```++ sync -f /mnt/host/source/src/build/images/kukui/R94-14125.0.2021_08_05_1510-a1/chromiumos_base_image.bin
+++ sudo losetup --show -f
/mnt/host/source/src/build/images/kukui/R94-14125.0.2021_08_05_1510-a1/chromiumos_base_image.bin

losetup: cannot find an unused loop device

++ lb_dev=
```

Then it appears that `sync -f
/mnt/host/source/src/build/images/kukui/R94-14125.0.2021_08_05_1510-a1/chromiumos_base_image.bin`
is the file we are trying to attach as a loopback device using
losetup.

10) The path /mnt/host/source/src/build/images/kukui/R94-14125.0.2021_08_05_1510-a1/chromiumos_base_image.bin
does not exist

11) However from the directory I'm working under a relative
./src/build/images/kukui/R94-14125.0.2021_08_05_1510-a1/chromiumos_base_image.bin
exists. Then this appears to be the reason for the script errors

12) Looking around I found GCLIENT_ROOT set here
https://github.com/drocsid/cros-scripts/blob/c361371f37d1e298a3810fcc7b11a502043c56ef/cros_workon_make#L12
. I have tried modifying it to the top level directory that contains
./src/build/images/kukui/R94-14125.0.2021_08_05_1510-a1/chromiumos_base_image.bin
but something from one of the other scripts seems to set it back.

13) There are also related GCLIENT_ROOT functions sprinkled around in
common.sh E.G. https://github.com/drocsid/cros-scripts/blob/main/common.sh#L265

Then I'm trying to determine why GCLIENT_ROOT is not set properly on
bullseye, and also trying to figure out a way to override it if I
can't fix the scripts directly.

[Thomas Schmitt]

Hi,

Colin Williams wrote:
> http://ix.io/3v3i

At least this shows an impressive partition table.
(Among them 5 partitions of size 512 bytes.)


> http://ix.io/3v6K

(Best to be downloaded and viewed in a text editor.)

> I believe the issues arise in
> https://github.com/drocsid/cros-scripts/blob/main/common.sh

> [...]

> +++ sudo losetup --show -f
> /mnt/host/source/src/build/images/kukui/R94-14125.0.2021_08_05_1510-a1/chrom
> iumos_base_image.bin
> losetup: cannot find an unused loop device

According to the man page this aims for acquiring an existing but unused
loop device. But in my local experiments it also creates a new loop device
if all existing ones are occupied.
So this should work if new loop devices can be created at all.

> 10) The path
> /mnt/host/source/src/build/images/kukui/R94-14125.0.2021_08_05_1510-a1/chromiumos_base_image.bin
> does not exist

But

sudo losetup --show -f non_existing_file_name

yields

losetup: non_existing_file_name: failed to set up loop device: No such file or directory

and not "cannot find an unused loop device".


So currently i think that at your point 10 your investigation left the road
to the loop device problem.

You should in any case look how many loop devices are occupied before
your script run:

losetup -l -a

If this does not show a lot of devices, then you will have execute this
command at various places in the script in order to see how the list of
used loop devices evolves during the script run.

[to...@tuxteam.de]

On Fri, Aug 06, 2021 at 09:48:29PM -0700, Colin Williams wrote:
> Hello everyone,
>
> In hindsight after looking at this much too late there were many
> mistakes in my initial mail. The issue may or may not be debian
> related and involves at least analyzing the script. There is a claim
> in the documentation that
>
> Then I'll make another attempt to further expose my ignorance.

[...]

I've rather the impression that you are trying to kill too
many birds with one stone :)

And, to be honest, I don't feel like debugging some random script
off github :-)

Why don't you first try to find out whether (a) loopback is working
on your running machine and (b) not all of your available loopback
devices are exhausted?

I think I posted some proposals already here [1].

Cheers

[1] https://lists.debian.org/debian-user/2021/08/msg00248.html

-- t

[Thomas Schmitt]

Hi,

Colin Williams:
> 3) When trying to create the loopback device the script tries to use a
> path
> /mnt/host/src/rc/build/images/kukui/R94-14125.0.2021_08_05_1510-a1/chromiumos_base_image.bin
> which does not exist

The error message of losetup does not match this theory.

Did you make sure that the file really does not exist when the losetup
command fails ?
E.g. by performing
ls -ld "${image}"
immediately before
lb_dev=$(sudo losetup --show -f "$@" "${image}")


> Let me know if I need to re-read your email.

Is there a hard reason why you don't show the output of

losetup -l -a

before, after, and maybe during the script run ?

(I'm not alone with this proposal. See
https://lists.debian.org/debian-user/2021/08/msg00353.html
which was not Cd'ed to you.
)

[Colin Williams]

Hi Thomas,

It's not entirely clear that what I was trying to express was
understood. Then in short:

1) A file ./src/build/images/kukui/R94-14125.0.2021_08_05_1510-a1/chromiumos_base_image.bin
is created

2) It seems that commands use a variable called GCLIENT_ROOT and it's
value is set to /mnt/host/src

3) When trying to create the loopback device the script tries to use a
path /mnt/host/src/rc/build/images/kukui/R94-14125.0.2021_08_05_1510-a1/chromiumos_base_image.bin
which does not exist

4) I'm stuck trying to figure out why GCLIENT_ROOT can't be set to the
actual mounted filesystem path so that the image can be found and the
loopback device created, etc.

Then I assume I could create a block device use losetup with the
correct path. So I'm debugging the script. There may be some details
related to chroot and GCLIENT_ROOT that I don't understand. Let me

know if I need to re-read your email.

Best Regards

[Colin Williams]

Hi Thomas,

>The error message of losetup does not match this theory.

Re-reading http://ix.io/3v6K and it does appear that possibly
/mnt/src/host/ does exist based on some of the debugging output.
Thanks for making me look back. I made this "theory" on trying to `ls
/mnt/host/src/rc/build/images/kukui/R94-14125.0.2021_08_05_1510-a1/chromiumos_base_image.bin`
after the script had been executed. I received an error the path
didn't exist. But perhaps it was cleaned up after the script finished
executing.

>Is there a hard reason why you don't show the output of losetup -l -a

I will now modify the script and also provide before and after below:

before:

(cr) ((b1688d0...)) colin@M00974055-VM ~/chromiumos/src/scripts $ losetup -l -a
-bash: losetup: command not found
(cr) ((b1688d0...)) colin@M00974055-VM ~/chromiumos/src/scripts $ sudo
losetup -l -a
(cr) ((b1688d0...)) colin@M00974055-VM ~/chromiumos/src/scripts $

no output appears

during:

I've added losetup here:
https://github.com/drocsid/cros-scripts/blob/feature/deb11-II/common.sh#L641

ourput after running ./build_image
https://chromium.googlesource.com/chromiumos/docs/+/main/developer_guide.md#Build-a-disk-image-for-your-board
with above changes: http://ix.io/3vfj

after:

colin@M00974055-VM:~/chromiumos/src/scripts$ cat ~/losetup-error.txt |
curl -F 'f:1=<-' ix.io
http://ix.io/3vfj
colin@M00974055-VM:~/chromiumos/src/scripts$ losetup -l -a
-bash: losetup: command not found
colin@M00974055-VM:~/chromiumos/src/scripts$ sudo losetup -l -a
[sudo] password for colin:
colin@M00974055-VM:~/chromiumos/src/scripts$

no output appears.

[Thomas Schmitt]

Hi,

Colin Williams wrote:
> http://ix.io/3vfj

Where i read

+++ sudo losetup --show -f /mnt/host/source/src/build/ima

ges/kukui/R94-14125.0.2021_08_07_0451-a1/chromiumos_base_

image.bin
losetup: cannot find an unused loop device

++ lb_dev=
++ sudo losetup -l -a
+++ sudo partx -v -d ''

Now that kills the theory about all loop devices being occupied.

I had a look into
https://sources.debian.org/src/util-linux/2.36.1-7/sys-utils/losetup.c
where i find two occurences of

"cannot find an unused loop device"

Each happens after a call to loopcxt_find_unused() at
https://sources.debian.org/src/util-linux/2.36.1-7/lib/loopdev.c/#L1544
which seems to try two different methods to get the number of an unused
loop device.
No other external reason is to see for getting the losetup error message.

Any theory about bad program parameters seems unlikely.

----------------------------------------------------------------------

Do you have a file
/dev/loop-control
?

What is listed by

ls -ld /dev/loop*

?
(/dev/loop files do not vanish after losetup -d. So if there are none, then
it is likely that there never have been any.)

----------------------------------------------------------------------

What happens if you try to create a loop device manually ?

not_yet_existing_file="some_file_path"
dd if=/dev/zero bs=512 count=2 of="$not_yet_existing_file"
sudo losetup loop0 "$not_yet_existing_file"

There should be no messages from losetup.
sudo wc -c </dev/loop0
should yield "1024".

To clean up do:

losetup -d /dev/loop0
rm "$not_yet_existing_file"

[David Wright]

On Sat 07 Aug 2021 at 05:03:10 (-0700), Colin Williams wrote:
> >The error message of losetup does not match this theory.
>
> Re-reading http://ix.io/3v6K and it does appear that possibly
> /mnt/src/host/ does exist based on some of the debugging output.
> Thanks for making me look back. I made this "theory" on trying to `ls
> /mnt/host/src/rc/build/images/kukui/R94-14125.0.2021_08_05_1510-a1/chromiumos_base_image.bin`

↑↑

Is this a typo. This is the second time of posting:

> after the script had been executed. I received an error the path
> didn't exist. But perhaps it was cleaned up after the script finished
> executing.
>

> On Sat, Aug 7, 2021 at 1:19 AM Thomas Schmitt <scdb...@gmx.net> wrote:
> > Colin Williams:
> > > 3) When trying to create the loopback device the script tries to use a
> > > path
> > > /mnt/host/src/rc/build/images/kukui/R94-14125.0.2021_08_05_1510-a1/chromiumos_base_image.bin
> > > which does not exist

Cheers,
David.

[Colin Williams]

> ----------------------------------------------------------------------
>
> Do you have a file
> /dev/loop-control
> ?
>
> What is listed by
>
> ls -ld /dev/loop*
>

colin@M00974055-VM:~$ sudo ls /dev/loop-control
[sudo] password for colin:
ls: cannot access '/dev/loop-control': No such file or directory
colin@M00974055-VM:~$ sudo ls -ld /dev/loop*
ls: cannot access '/dev/loop*': No such file or directory

> ----------------------------------------------------------------------
>
> What happens if you try to create a loop device manually ?
>

not_yet_existing_file="some_file_path"
dd if=/dev/zero bs=512 count=2 of="$not_yet_existing_file"
sudo losetup loop0 "$not_yet_existing_file"

not_yet_existing_file="/tmp/does_not_exist_yet"

dd if=/dev/zero bs=512 count=2 of="$not_yet_existing_file"
sudo losetup loop0 "$not_yet_existing_file"

2+0 records in
2+0 records out
1024 bytes (1.0 kB, 1.0 KiB) copied, 0.000187826 s, 5.5 MB/s
losetup: /dev/loop0: failed to set up loop device: No such file or directory

> There should be no messages from losetup.

But we see one above ^^

> sudo wc -c </dev/loop0
> should yield "1024".

sudo wc -c </dev/loop0
bash: /dev/loop0: No such file or directory

> To clean up do:
>
losetup -d /dev/loop0
rm "$not_yet_existing_file"

colin@M00974055-VM:~$ sudo losetup -d /dev/loop0
losetup: /dev/loop0: failed to use device: No such device
colin@M00974055-VM:~$ rm "$not_yet_existing_file"
colin@M00974055-VM:~$

______________________________________________________________________

>
>
> Have a nice day :)
>
> Thomas

Thank you Thomas. Thanks again for looking at my issue. Not sure where
to go from here.

________________________________________________________________________

colin@M00974055-VM:~$ uname -r
5.10.0-3-amd64
colin@M00974055-VM:~$ cat /etc/debian_version
11.0colin@M00974055-VM:~$ lsmod | grep loop
colin@M00974055-VM:~$ sudo modprobe loop
colin@M00974055-VM:~$ lsmod | grep loop
loop 36864 0
colin@M00974055-VM:~$ dd if=/dev/zero bs=512 count=2 of="$not_yet_existing_file"
2+0 records in
2+0 records out
1024 bytes (1.0 kB, 1.0 KiB) copied, 0.000134838 s, 7.6 MB/s
colin@M00974055-VM:~$ sudo losetup loop0 "$not_yet_existing_file"

No error here ^^ Guess we needed to load the module?

colin@M00974055-VM:~$ sudo wc -c </dev/loop0
bash: /dev/loop0: Permission denied
colin@M00974055-VM:~$ sudo -i
root@M00974055-VM:~# wc -c </dev/loop0
1024
root@M00974055-VM:


wc -c </dev/loop0
1024
root@M00974055-VM:~# losetup -d /dev/loop0
rm "$not_yet_existing_file"
rm: cannot remove '': No such file or directory

I think we are on to something after loading the module

_____________________________________________________________

I re-ran the script after the module. It looks like it's making more
progress but still getting an eventual error related to loopback
devices:

http://ix.io/3vi8


^^ This is probably the most useful to look at moving forward ^^

Now I will re-run the inital commands below

sudo ls -ld /dev/loop*
brw-rw---- 1 root disk 7, 0 Aug 7 13:07 /dev/loop0
brw-rw---- 1 root disk 259, 12 Aug 7 13:07 /dev/loop0p1
brw-rw---- 1 root disk 259, 21 Aug 7 13:07 /dev/loop0p10
brw-rw---- 1 root disk 259, 22 Aug 7 13:07 /dev/loop0p11
brw-rw---- 1 root disk 259, 23 Aug 7 13:07 /dev/loop0p12
brw-rw---- 1 root disk 259, 13 Aug 7 13:07 /dev/loop0p2
brw-rw---- 1 root disk 259, 14 Aug 7 13:07 /dev/loop0p3
brw-rw---- 1 root disk 259, 15 Aug 7 13:07 /dev/loop0p4
brw-rw---- 1 root disk 259, 16 Aug 7 13:07 /dev/loop0p5
brw-rw---- 1 root disk 259, 17 Aug 7 13:07 /dev/loop0p6
brw-rw---- 1 root disk 259, 18 Aug 7 13:07 /dev/loop0p7
brw-rw---- 1 root disk 259, 19 Aug 7 13:07 /dev/loop0p8
brw-rw---- 1 root disk 259, 20 Aug 7 13:07 /dev/loop0p9
brw-rw---- 1 root disk 7, 1 Aug 7 13:01 /dev/loop1
brw-rw---- 1 root disk 259, 0 Aug 7 13:01 /dev/loop1p1
brw-rw---- 1 root disk 259, 9 Aug 7 13:01 /dev/loop1p10
brw-rw---- 1 root disk 259, 10 Aug 7 13:01 /dev/loop1p11
brw-rw---- 1 root disk 259, 11 Aug 7 13:01 /dev/loop1p12
brw-rw---- 1 root disk 259, 1 Aug 7 13:01 /dev/loop1p2
brw-rw---- 1 root disk 259, 2 Aug 7 13:01 /dev/loop1p3
brw-rw---- 1 root disk 259, 3 Aug 7 13:01 /dev/loop1p4
brw-rw---- 1 root disk 259, 4 Aug 7 13:01 /dev/loop1p5
brw-rw---- 1 root disk 259, 5 Aug 7 13:01 /dev/loop1p6
brw-rw---- 1 root disk 259, 6 Aug 7 13:01 /dev/loop1p7
brw-rw---- 1 root disk 259, 7 Aug 7 13:01 /dev/loop1p8
brw-rw---- 1 root disk 259, 8 Aug 7 13:01 /dev/loop1p9
brw-rw---- 1 root disk 7, 2 Aug 7 13:09 /dev/loop2
brw-rw---- 1 root disk 259, 24 Aug 7 13:09 /dev/loop2p1
brw-rw---- 1 root disk 259, 33 Aug 7 13:09 /dev/loop2p10
brw-rw---- 1 root disk 259, 34 Aug 7 13:09 /dev/loop2p11
brw-rw---- 1 root disk 259, 35 Aug 7 13:09 /dev/loop2p12
brw-rw---- 1 root disk 259, 25 Aug 7 13:09 /dev/loop2p2
brw-rw---- 1 root disk 259, 26 Aug 7 13:09 /dev/loop2p3
brw-rw---- 1 root disk 259, 27 Aug 7 13:09 /dev/loop2p4
brw-rw---- 1 root disk 259, 28 Aug 7 13:09 /dev/loop2p5
brw-rw---- 1 root disk 259, 29 Aug 7 13:09 /dev/loop2p6
brw-rw---- 1 root disk 259, 30 Aug 7 13:09 /dev/loop2p7
brw-rw---- 1 root disk 259, 31 Aug 7 13:09 /dev/loop2p8
brw-rw---- 1 root disk 259, 32 Aug 7 13:09 /dev/loop2p9
brw-rw---- 1 root disk 7, 3 Aug 7 12:59 /dev/loop3
brw-rw---- 1 root disk 7, 4 Aug 7 12:59 /dev/loop4
brw-rw---- 1 root disk 7, 5 Aug 7 12:59 /dev/loop5
brw-rw---- 1 root disk 7, 6 Aug 7 12:59 /dev/loop6
brw-rw---- 1 root disk 7, 7 Aug 7 12:59 /dev/loop7
crw-rw---- 1 root disk 10, 237 Aug 7 12:59 /dev/loop-control
(cr) (feature/deb11-II) colin@M00974055-VM ~/chromiumos/src/scripts $
sudo ls /dev/loop-control
/dev/loop-control


_________________________________________________________________

[Colin Williams]

I seem to have resolved the issue above after rebooting. Thanks for
helping to debug Thomas and everyone.