partition reporting full, but not

[Keith Bainbridge]

Good afternoon All

I have just rebooted this laptop to ensure it is 'fresh'

/ is reporting full.

Trying to locate where I ran

sudo du -hPx --max-depth=1 /
0 /mnt
181M /boot
15M /etc
0 /media
236M /opt
336K /root
0 /srv
4.0K /tmp
8.1G /usr
726M /var
9.2G /
keith@dell0 $

Sat 17Feb2024@13:33:29
:~


But:
>> sudo df -h /
Filesystem Size Used Avail Use% Mounted on
/dev/sda3 36G 35G 100M 100% /
keith@dell0 $

Sat 17Feb2024@13:33:39
:~

Where do I start locating the conflicting information please?

--
All the best

Keith Bainbridge

keith.bain...@gmail.com
+61 (0)447 667 468

UTC + 10:00

[Greg Wooledge]

On Sat, Feb 17, 2024 at 01:38:56PM +1100, Keith Bainbridge wrote:
> >> sudo df -h /
> Filesystem Size Used Avail Use% Mounted on
> /dev/sda3 36G 35G 100M 100% /

First off: you don't need sudo for this, ever.

Second: what kind of file system is this?

> sudo du -hPx --max-depth=1 /
> 0 /mnt
> 181M /boot
> 15M /etc
> 0 /media
> 236M /opt
> 336K /root
> 0 /srv
> 4.0K /tmp
> 8.1G /usr
> 726M /var
> 9.2G /

So I guess the question is "Where's the rest of that 35 G used data?"

Conspicuously missing from this output is /home. Is that a separate
file system? If so, you *could* have data inside the /home directory
of the root file system, which is hidden by the /home file system that's
mounted over it. You'd need to unmount /home to check. It's not a
super probable situation, but it's worth checking.

The same applies to any other directory that's got a file system mounted
on it.

Or... it could be some bizarre btrfs crap. If this is a btrfs.

[David Wright]

Typically some space is reserved for root, so that you have a little
headroom to extricate yourself from this situation. It looks like it's
the 5% shown below (from an installer screen):

┌────────────────────────┤ [!!] Partition disks ├─────────────────────────┐
│ │
│ You are editing partition #3 of /dev/nvme0n1. This partition is │
│ formatted with the Ext4 journaling file system. All data in it WILL │
│ BE DESTROYED! │
│ │
│ Partition settings: │
│ │
│ Name: Umbo-B │
│ Use as: Ext4 journaling file system │
│ │
│ Format the partition: yes, format it │
│ Mount point: / │
│ Mount options: defaults │
│ Label: umbo03 │
│ Reserved blocks: 5% │
│ Typical usage: standard │
│ Bootable flag: off │
│ │
│ Resize the partition (currently 31.5 GB) │
│ Erase data on this partition │
│ Delete the partition │
│ Done setting up the partition │
│ │
│ <Go Back> │
│ │
└─────────────────────────────────────────────────────────────────────────┘

Cheers,
David.

[Gremlin]

Do you have any nfs mounts?

[Keith Bainbridge]

No nfs mounts

Thankyou

[Keith Bainbridge]

On 17/2/24 13:52, Greg Wooledge wrote:
> On Sat, Feb 17, 2024 at 01:38:56PM +1100, Keith Bainbridge wrote:
>> >> sudo df -h /
>> Filesystem Size Used Avail Use% Mounted on
>> /dev/sda3 36G 35G 100M 100% /
>
> First off: you don't need sudo for this, ever.
>
> Second: what kind of file system is this?
>
>> sudo du -hPx --max-depth=1 /
>> 0 /mnt
>> 181M /boot
>> 15M /etc
>> 0 /media
>> 236M /opt
>> 336K /root
>> 0 /srv
>> 4.0K /tmp
>> 8.1G /usr
>> 726M /var
>> 9.2G /
>
> So I guess the question is "Where's the rest of that 35 G used data?"
>

My wife is often telling me I'm not clear (obtuse). Thank you for
clarifying the question

> Conspicuously missing from this output is /home. Is that a separate
> file system? If so, you *could* have data inside the /home directory
> of the root file system, which is hidden by the /home file system that's
> mounted over it. You'd need to unmount /home to check. It's not a
> super probable situation, but it's worth checking.
>

I had wondered why /home was missing and concluded that as it's listed if I

df -h /home/

Filesystem Size Used Avail Use% Mounted on

/dev/sda3 ===same partiton as / 36G 35G 100M 100% /home
keith@dell0 $

Then
>> sudo du -hPx --max-depth=1 /home/
[sudo] password for root:
2.2G /home/keith
2.2G /home/
keith@dell0 $

So, part of the missing 35G, but nothing significant

I checked my daily driver - lenv0, and I have the same situation -

sudo du -hPx --max-depth=1 /

[sudo] password for root:
265M /boot
18M /etc
0 /media
947M /opt
1.5M /root
0 /srv
du: cannot access '/tmp/.mount_Espansr8ZHig': Permission denied
12M /tmp
9.6G /usr
1.8G /var
13G /

>> sudo du -hPx --max-depth=1 /home/
3.0M /home/ke1th
3.0M /home/
keith@lenv0 $

df -h

Filesystem Size Used Avail Use% Mounted on

udev 7.2G 0 7.2G 0% /dev
tmpfs 1.5G 1.9M 1.5G 1% /run
/dev/nvme0n1p2 63G 27G 35G 44% /
tmpfs 7.3G 84M 7.2G 2% /dev/shm
tmpfs 5.0M 16K 5.0M 1% /run/lock
/dev/nvme0n1p2 63G 27G 35G 44% /home

In this case /home/keith is a sym link to another partition.

But still 14GB difference between the 2 reporting processes.

> The same applies to any other directory that's got a file system mounted
> on it.

I have had data written to /mnt/someDisk when the partition was NOT
mounted. I get that suggestion. Those instances showed up in du though.

>
> Or... it could be some bizarre btrfs crap. If this is a btrfs.

Yes the / partitions are btrfs
>
The annoying habit of listing /home at df seems to be part of btrfs
standard practice which I dislike. So far the ability to create
timeshift and BackInTime snapshots in the proverbial blink of an eye are
VERY good points for btrfs. Seems I need to allow for more more
'slack' space as well? du is reporting my timeshift directory as 0,
and the internal subdir that lists save file as 108K. This tells me that
btrfs is doing something magic.
Guess I'll be contemplating reinstalling onto ext4 in my sleep tonight.

To answer another question - no NFS

Thanks All

[to...@tuxteam.de]

Absolutely. What I miss, too, is lost+found (not every file system has
that,i though).

> The same applies to any other directory that's got a file system mounted
> on it.

Yes, looking "under" the mounts is a very good idea.

> Or... it could be some bizarre btrfs crap. If this is a btrfs.

We're all curious :-)

Cheers
--
t

[Felix Miata]

Keith Bainbridge composed on 2024-02-17 15:44 (UTC+1100):

> Yes the / partitions are btrfs

df was not designed for the task you gave it. You need to use

btrfs filesystem <task>

commands:
https://btrfs.readthedocs.io/en/latest/btrfs-filesystem.html
--
Evolution as taught in public schools is, like religion,
based on faith, not based on science.

Team OS/2 ** Reg. Linux User #211409 ** a11y rocks!

Felix Miata

[to...@tuxteam.de]

On Sat, Feb 17, 2024 at 03:44:49PM +1100, Keith Bainbridge wrote:

[...]

> df -h
> Filesystem Size Used Avail Use% Mounted on
> udev 7.2G 0 7.2G 0% /dev
> tmpfs 1.5G 1.9M 1.5G 1% /run
> /dev/nvme0n1p2 63G 27G 35G 44% /
> tmpfs 7.3G 84M 7.2G 2% /dev/shm
> tmpfs 5.0M 16K 5.0M 1% /run/lock
> /dev/nvme0n1p2 63G 27G 35G 44% /home

[...]

> Yes the / partitions are btrfs
> >
> The annoying habit of listing /home at df seems to be part of btrfs standard

> practice which I dislike [...]

Oh, wait! This is btrfs. It seems to be able to put more than one file
system in a partition (so no, it's not "standard practice": you have
two different file systems there, which df duly reports, but they seem
to co-habitate one partition)

This being btrfs... quite possibly the missing space is used up in
one or more snapshots?

Cheers
--
t

[Keith Bainbridge]

On 17/2/24 17:08, Felix Miata wrote:
> Keith Bainbridge composed on 2024-02-17 15:44 (UTC+1100):
>
>> Yes the / partitions are btrfs
>
> df was not designed for the task you gave it. You need to use
>
> btrfs filesystem <task>
>
> commands:
> https://btrfs.readthedocs.io/en/latest/btrfs-filesystem.html

Seems I have some serious reading to do

Thanks for thelink

[songbird]

Keith Bainbridge wrote:
...
> No nfs mounts

any swap partition or swap space?

but other than that sharing /home with / is likely your
issue and you mention snapshots and backintime and i do
recall that needing plenty of space.

as for btrfs, i have no clue, i've never touched it.


songbird

[debia...@howorth.org.uk]

Keith Bainbridge <keit...@gmail.com> wrote:

> Yes the / partitions are btrfs

So the apparently missing space is perhaps taken up by btrfs snapshots.

[Stefan Monnier]

> So the apparently missing space is perhaps taken up by btrfs snapshots.

Another possibility is a (few) large file(s) that is/are still open for
some process(es) but have been `rm` (`unlink`) so they don't have a name
any more.


Stefan

[Greg Wooledge]

In the original post, they said they rebooted, in order to rule this out.

[Cindy Sue Causey]

I only know to say this because it just happened a few days ago. Rsync
left some semi-permanent remnants when I was having problems with the
wireless capable hard drive docking station repeatedly cutting out. I
was offloading videos and images from a camera during many of those
times.

There were 3 to 5 instances of discarded remnant files in several
directories while the problem was ongoing (MANY times over a couple
weeks). The residual files were only visible when in CTRL+H view in
Thunar. Sizes ranged from a few MB to almost a GB. All had to be
manually removed.

PS That hardware problem, which I think I mentioned in another thread,
is semi-solved... and appears to be a possible case of [WIFI] jamming,
of all things. If it starts up again, there will possibly be a new
thread asking about potential tracking packages. Current instant fix?
Tweet very loudly about the issue and its potential source....

Cindy :)

NB See Also: Minnesota burglaries via suspected WIFI jamming (e.g. on reddit).
--
Talking Rock, Pickens County, Georgia, USA
* runs with birdseed *

[Max Nikulin]

On 17/02/2024 09:52, Greg Wooledge wrote:
> If so, you *could* have data inside the /home directory
> of the root file system, which is hidden by the /home file system that's
> mounted over it. You'd need to unmount /home to check.

A less intrusive way to inspect shadowed directories is bind mounts.

mkdir /tmp/root
mount --bind / /tmp/root

[Keith Bainbridge]

On 18/2/24 09:19, Cindy Sue Causey wrote:
>
> I only know to say this because it just happened a few days ago. Rsync
> left some semi-permanent remnants when I was having problems with the
> wireless capable hard drive docking station repeatedly cutting out. I
> was offloading videos and images from a camera during many of those
> times.

Thanks Cindy

Interesting

Though as far as I can recall the only files I have rsync'd onto / are
4or5 config files, and all in /home/keith which now on another
partition. And with a gain in free space which virtually matches its
reported space used, yesterday.

[Keith Bainbridge]

Seems to be the prime suspect. If that's the case, btrfs is NOT
hard-linking the snapshots as timeshift claims it does. The only way to
check is install on ext4 and compare. I have saves enough free space to
do this.

My effort to date is to move my home to /mnt/data and sim-link it into
/home. df is now showing 2.3GB free on /. df showed /home as 2.2GB
yesterday. At least there is a little space to play with; and give me
time to consider. A fresh install may be worth checking in snapshots are
as big as this all makes them look.

a few brief answer to other comments will follow

[David Christensen]

>> Keith Bainbridge composed on 2024-02-17 15:44 (UTC+1100):
>>
>>> Yes the / partitions are btrfs

Several years ago, I installed Debian (9?) using btrfs for root (and
boot?). I failed to understand that btrfs required regular maintenance
and/or I was too lazy to figure it out and do it. After a few months,
the systems started running slowly and I seem to recall conflicting
reports of storage usage. I STFW, RTFM, etc., and tried doing the
maintenance by hand. I quickly came to the conclusion that I needed to
run `btrfs balance start ...` many, many times. So, I wrote Perl script
and let it hammer on the file systems for hours at a time (!). I was
able to rescue most of the disks to decent performance, but one was
especially bad and I was only able to rescue it to marginal performance.
I continued running btrfs for a while and running the Perl script
periodically. Ultimatedly, I backed up, put in fresh OS disks, and
installed using ext4. This was one of my reasons to use FreeBSD and ZFS
for my SOHO servers.


David

[Keith Bainbridge]

Thank you Max

This has proved a real boon

[Keith Bainbridge]

On 18/2/24 14:49, Keith Bainbridge wrote:
>
> On 18/2/24 07:34, debia...@howorth.org.uk wrote:
>> Keith Bainbridge <keit...@gmail.com> wrote:
>>> Yes the / partitions are btrfs
>>
>> So the apparently missing space is perhaps taken up by btrfs snapshots.
>>
>
> Seems to be the prime suspect.   If that's the case, btrfs is NOT hard-
> linking the snapshots as timeshift claims it does. The only way to check
> is install on ext4 and compare. I have saves enough free space to do this.
>
> My effort to date is to move my home to /mnt/data and sim-link it into /
> home. df is now showing 2.3GB free on /.  df showed /home as 2.2GB
> yesterday.  At least there is a little space to play with; and give me
> time to consider. A fresh install may be worth checking in snapshots are
> as big as this all makes them look.
>
> a few brief answer to other comments will follow

So later yesterday afternoon I created a new snapshot with no obvious
change is free space.

I then update/upgrade. The initial attempt told me
63 upgraded, 3 newly installed, 0 to remove and 0 not upgraded.
Need to get 337 MB of archives.
After this operation, 473 MB of additional disk space will be used.
Do you want to continue? [Y/n]

But the 3 kernel related packages failed to install a couple of times.
When I finally figured I should check space, there was none. I rolled
back to prior to the upgrade, but still no free space.

I said sometime in this thread that timeshift (and BiT) use hard links
to create progressive copies of the system. The more I think about how
hard links reportedly work, I reckon it can't be simply hard links.

So I'm starting a new thread on that topic.

[Max Nikulin]

On 19/02/2024 06:26, Keith Bainbridge wrote:
>
> So later yesterday afternoon I created a new snapshot with no obvious
> change is free space.

Effect of snapshots is delayed. When you remove a file that does not
belong to any snapshot, some disk space is reclaimed. However to restore
a file (even a removed later) from a snapshot, it must be stored
anywhere. That is why snapshots consume disk space.

Try to remove unnecessary snapshots. I have no idea if btrfs requires
additional maintenance.

[Keith Bainbridge]

On 19/2/24 13:41, Felix Miata wrote:
> would be some places to start. Didn't you do your
> https://btrfs.readthedocs.io/en/latest/btrfs-filesystem.html
> reading yet? ?_?


My eyes have glazed over too often, already. I know I have to get back,
but that NEED to do it is making it harder.

[Felix Miata]

Keith Bainbridge composed on 2024-02-18 14:49 (UTC+1100):

> debia...@howorth.org.uk wrote:

>> So the apparently missing space is perhaps taken up by btrfs snapshots.

> Seems to be the prime suspect.

While snapshotting is obviously a consumer, until you use the right tool for the
job, you won't know anything meaningful about overall space usage on btrfs.

btrfs filesystem df
btrfs filesystem du
btrfs filesysten show

would be some places to start. Didn't you do your
https://btrfs.readthedocs.io/en/latest/btrfs-filesystem.html
reading yet? ?_?

[Keith Bainbridge]

So I'm back to see some more helpful hints. Thanks folk

I am convinced that the missing space is used by btrfs snapshot process.
But WHY is the used space reporting on my daily driver LESS than that on
the spare machine 29G vs 35G? The original install was the same .iso
Ah well

I could add some of the spare space the the / partition, but how much?
Play safe and use the lot, making it 60G compared to 63G on my daily
driver. (And create some free space off the data partition before it's
too late.)

Just as well I have time on my hands

Again, thanks to all for your suggestions

[to...@tuxteam.de]

On Mon, Feb 19, 2024 at 02:20:20PM +1100, Keith Bainbridge wrote:

[...]

> I am convinced that the missing space is used by btrfs snapshot process. But
> WHY is the used space reporting on my daily driver LESS than that on the
> spare machine 29G vs 35G? The original install was the same .iso Ah well

Perhaps because you upgrade your daily driver more often? May be
because it has more packages installed?

Another post of yours upthread suggests that your upgrade process roughly
is:

1. take snapshot
2. upgrade

Do you remove your snapshots if/after all went well? (note that I
have no idea how snapshots are removed under btrfs and whether they
are somehow visible to "du" -- I guess the second is a "no").

If not, the space is taken by all the files which have been overwritten
in the upgrade process. More "versions", less space.

Cheers
--
t

[David Christensen]

On 2/18/24 19:20, Keith Bainbridge wrote:
> I am convinced that the missing space is used by btrfs snapshot process.

Perhaps. But, are you re-balancing your btrfs file systems regularly?

https://manpages.debian.org/bookworm/btrfs-progs/btrfs-balance.8.en.html


Doing it by hand was not practical for me. I wrote a Perl script to
automate the process. On SSD's, the results were decent. On USB flash
drives, not so much.


Searching for a power tool today, I see:

2024-02-18 23:27:43 dpchrist@laalaa ~/stretch-amd64
$ apt-cache search btrfs | grep mainten
btrfsmaintenance - automate btrfs maintenance tasks on mountpoints or
directories

https://packages.debian.org/bookworm/btrfsmaintenance


I suggest installing and trying the btrfsmaintenance package.


David

[debia...@howorth.org.uk]

David Christensen <dpch...@holgerdanske.com> wrote:
> On 2/18/24 19:20, Keith Bainbridge wrote:
> > I am convinced that the missing space is used by btrfs snapshot
> > process.
>
>
> Perhaps. But, are you re-balancing your btrfs file systems regularly?
>
> https://manpages.debian.org/bookworm/btrfs-progs/btrfs-balance.8.en.html

But does balancing a btrfs system actually recover any space? I think
not. It's eternally moving the deckchairs on the Titanic. (though with
a more useful purpose)

I'd recommend installing snapper if you haven't already.
https://wiki.archlinux.org/title/Snapper is helpful for using it.

[Keith Bainbridge]

I just removed 3 snapshots from my daily driver with no change in used
space reported by df

Mindful of the fact that somebody considers it may take time to reflect
changes within the snapshots, I'll check again tomorrow morning

[Keith Bainbridge]

I am sure I saw a response to comment of mine, where I was misunderstood
in the numbers I quoted for used space on my daily driver - 29G; and the
space used by the problem machine - 35G. There was a suggestion that I
had not updated it as often as daily driver. I had kept problem box
as up to date as daily until a few days ago when it refused to update
due to lack of space. This is when I discovered I had a problem. It is
switched off at present, pending my deciding whether to expand /
partition or re-install on the free space on ext4. I will delete a few
snapshots before I proceed, just to see what happens - I'll do that
shortly, in fact, now I can see that it may have a bigger affect than I
figured.

Now a minor amendment to my last note, where deleting snapshots has haad
no bearing on used space. Before I started, df reported 28G used,
compared to 29G used yesterday. Remember my home is sym-linked from
another partition. du is reporting /home is 3M which is the original
/home/keith and re-named to keep it handy IN CASE I need it some day -
like when I did some major surgery on that data partition the other
week. I'm trying to say that nothing I've done overnight has changed
used space. There were no packages to upgrade today.

df is now reporting 27G used on / confirming btrfs seems to take time
to reflect changes in snapshots.

Back later.

[Felix Miata]

Keith Bainbridge composed on 2024-02-20 17:45 (UTC+1100):

> I just removed 3 snapshots from my daily driver with no change in used
> space reported by df

df doesn't know how to calculate freespace on btrfs. You need to be typing

btrfs filesystem df

if you have not aliased df to btrfs filesystem df.

[Keith Bainbridge]

Back. On booting up problem machine I was greeted with warnings in disk
space low on /. I generally don't log into desktop on this machine.
Deleted 4 snapshots. df immediately reported used space 33G (down from
35G) and free space 2.9G, up from ~200M at login. I don't think I've
EVER seen used space and free space equal size before.

I rebooted just to see if anything changed. 10 mins later df is still
reporting

Size Used Avail Use% Mounted on

36G 33G 2.9G 92% /


apt update/upgrade gave me

63 upgraded, 3 newly installed, 0 to remove and 0 not upgraded.
Need to get 337 MB of archives.

which I reckon is what I got yesterday after I moved my home to my data
partition and

Quoted from 19Feb at 10:26 (UTC 18Feb at 23:26):

I then update/upgrade. The initial attempt told me
63 upgraded, 3 newly installed, 0 to remove and 0 not upgraded.
Need to get 337 MB of archives.
After this operation, 473 MB of additional disk space will be used.
Do you want to continue? [Y/n]

But the 3 kernel related packages failed to install a couple of times.
When I finally figured I should check space, there was none. I rolled
back to prior to the upgrade, but still no free space.

And earlier:My effort to date is to move my home to /mnt/data and
sym-link it into /home. df is now showing 2.3GB free on /. df showed

/home as 2.2GB yesterday. At least there is a little space to play
with; and give me time to consider. A fresh install may be worth
checking in snapshots are as big as this all makes them look.

Back to present, apt is setting up the kernel-image & headers. And
completed to upgrade while I typed that last sentence.
df is reporting 36G 35G 1004M - still used + free = size
Rebooted and df reports 36G 35G 888M 98% /



For now, I'll expand / partition and reduce timeshift retention rules


When I've completed the reading assignment I have from Sunday, I'll move
on. I might start with changing timeshift to use rsync instead of
btrfs - just to see how it compares. And put snapshots on another
partition. They should really be on a separate drive, but that machine
doesn't have a 2nd drive bay that I could see when I swapped the system
drive last year when I was given the now 8 year old machine.


I'm looking forward to your responses

[to...@tuxteam.de]

On Tue, Feb 20, 2024 at 02:42:18AM -0500, Felix Miata wrote:
> Keith Bainbridge composed on 2024-02-20 17:45 (UTC+1100):
>
> > I just removed 3 snapshots from my daily driver with no change in used
> > space reported by df
>
> df doesn't know how to calculate freespace on btrfs. You need to be typing
>
> btrfs filesystem df
>
> if you have not aliased df to btrfs filesystem df.

Still, Keith seems to have a real shortage of file system free space,
otherwise Debian upgrades wouldn't fail.

I don't know much about btrfs, but what would be really helpful (if
you do, and it seems so) would be for you to fill us in on how to
asses the space used up by old snapshots (what seems to be the main
suspect currently).

Cheers
--
t

[Keith Bainbridge]

On 20/2/24 18:42, Felix Miata wrote:
> btrfs filesystem df

OK, so please interpret:

>> btrfs filesystem df -h /
Data, single: total=32.80GiB, used=31.94GiB
System, DUP: total=8.00MiB, used=16.00KiB
Metadata, DUP: total=1.50GiB, used=1.10GiB
GlobalReserve, single: total=71.69MiB, used=0.00B
keith@dell0 $

Tue 20Feb2024@20:57:45
:~

>> btrfs filesystem df -h /mnt/data/
Data, single: total=530.02GiB, used=329.55GiB
System, DUP: total=8.00MiB, used=112.00KiB
Metadata, DUP: total=5.00GiB, used=1.10GiB
GlobalReserve, single: total=512.00MiB, used=0.00B
keith@dell0 $

when cfdisk reports:
Device Start End
Sectors Size Type
>> /dev/sda1 2048 1667678207
1667676160 795.2G Linux filesystem
Free space 1667678208 1785522175
117843968 56.2G
/dev/sda2 1785522176 1786245119
722944 353M EFI System
/dev/sda3 1786245120 1933045759
146800640 70G EFI System
/dev/sda4 1933045760 1953523711
20477952 9.8G Linux swap

To clarify
/dev/sda1 is /mnt/data and I don't understand the 'EFI System' note
/dev/sda3 is /

Perhaps the explanation why the reports from btrfs filesystem df make no
sense will come when I get into my assignment?

[Keith Bainbridge]

Tomas, the upgrade failure was earlier than these notes. It has now worked

Sorry, but I don't know how to assess the snapshot space usage.

[to...@tuxteam.de]

On Tue, Feb 20, 2024 at 09:21:15PM +1100, Keith Bainbridge wrote:
>
> On 20/2/24 19:38, to...@tuxteam.de wrote:

[...]

> Tomas, the upgrade failure was earlier than these notes. It has now worked

I see.

> Sorry, but I don't know how to assess the snapshot space usage.

Nor do I -- my question was rather directed at Felix, who seems to
be the only one in this thread with some btrfs experience.

Perhaps it'd be wise to include "btrfs" in the Subject to attract
the attention of btrfs buffs?

Cheers
--
t

[Thomas Schmitt]

Hi,

> when cfdisk reports:
> Device Start End Sectors Size Type

> /dev/sda2 1785522176 1786245119 722944 353M EFI System
> /dev/sda3 1786245120 1933045759 146800640 70G EFI System

> I don't understand the 'EFI System' note /dev/sda3 is /

The partition type does not necessarily match the type of filesystem
which is currently in that partition. It's just a field in the
partition table which must have some value.


Have a nice day :)

Thomas

[debia...@howorth.org.uk]

Felix Miata <mrm...@earthlink.net> wrote:
> Keith Bainbridge composed on 2024-02-20 17:45 (UTC+1100):
>
> > I just removed 3 snapshots from my daily driver with no change in
> > used space reported by df
>
> df doesn't know how to calculate freespace on btrfs. You need to be
> typing
>
> btrfs filesystem df

df [options] <path>
Show a terse summary information about allocation of block
group types of a given mount point. The original purpose of
this command was a debugging helper. The output needs to be
further interpreted and is not suitable for quick overview.


FWIW my root filesystem is btrfs and I use the normal df command all the
time without a problem. I've never used btrfs filesystem df

[Felix Miata]

to...@tuxteam.de composed on 2024-02-20 09:38 (UTC+0100):

My hands-on experience with btrfs is limited to one laptop my brother gave me with
openSUSE installed that I only boot 2-3 times per year. My mentioning of the btrfs
command in this thread is based upon years of frequenting openSUSE and Fedora
mailing lists and forums. openSUSE made btrfs its default filesystem somewhere
around 7-8 years ago. I think Fedora made btrfs default more recently. Google &
DDG should be able to provide much better help than I on how to interpret btrfs
command output. So should sup...@lists.opensuse.org and
https://forums.opensuse.org/c/english/install-boot-login/18 and the comparable
Fedora forums. My own Gnu/Linux installations, other than Knoppix, have always
been on extX.

I suggest Keith's 36G / partition size must be a marginal for btrfs use. IIRC, 40G
may be the officially suggested minimum size for an openSUSE btrfs / filesystem
where /home/ is on a separate (xfs) filesystem. Keith has /home/ within /.

Surely somewhere on debian.org such things must be addressed if Bookworm's default
has also been changed to btrfs.

[Greg Wooledge]

On Tue, Feb 20, 2024 at 02:47:26PM -0500, Felix Miata wrote:
> Surely somewhere on debian.org such things must be addressed if Bookworm's default
> has also been changed to btrfs.

That has not happened. The default file system is still ext4.

[Felix Miata]

Greg Wooledge composed on 2024-02-20 14:56 (UTC-0500):

I didn't think so, which begs the question why OP Keith is using it. :p

[Felix Miata]

to...@tuxteam.de composed on 2024-02-20 09:38 (UTC+0100):

A current thread from elsewhere that should be helpful:
<https://forums.opensuse.org/t/btrfs-disk-full-how-to-fix-it-is-that-really-the-solution/172576>

btrfs filesystem usage /
snapper list
btrfs qgroup show /

Was requested by a btrfs expert.

[Keith Bainbridge]

On 21/2/24 11:38, Felix Miata wrote:
> A current thread from elsewhere that should be helpful:
> <https://forums.opensuse.org/t/btrfs-disk-full-how-to-fix-it-is-that-
> really-the-solution/172576>
>
> btrfs filesystem usage /
> snapper list
> btrfs qgroup show /

Thanks for the prompt, Felix

>> sudo btrfs filesystem usage /
[sudo] password for root:
Overall:
Device size: 70.00GiB ===== remember I expanded this partition
yesterday
Device allocated: 35.82GiB
Device unallocated: 34.18GiB
Device missing: 0.00B
Device slack: 0.00B
Used: 34.30GiB
Free (estimated): 34.90GiB (min: 17.81GiB)
Free (statfs, df): 34.90GiB
Data ratio: 1.00
Metadata ratio: 2.00
Global reserve: 71.69MiB (used: 0.00B)
Multiple profiles: no

Data,single: Size:32.80GiB, Used:32.09GiB (97.82%)
/dev/sda3 32.80GiB

Metadata,DUP: Size:1.50GiB, Used:1.11GiB (73.77%)
/dev/sda3 3.00GiB

System,DUP: Size:8.00MiB, Used:16.00KiB (0.20%)
/dev/sda3 16.00MiB

Unallocated:
/dev/sda3 34.18GiB
keith@dell0 $

Wed 21Feb2024@11:44:26
:~

At least I can make sense from some of these output numbers

>> sudo btrfs qgroup show /
ERROR: can't list qgroups: quotas not enabled
keith@dell0 $

Wed 21Feb2024@11:47:13
:~

installed snapper and
>> sudo snapper list
The config 'root' does not exist. Likely snapper is not configured.
See 'man snapper' for further instructions.
keith@dell0 $

More to learn about, I see

[Felix Miata]

Keith Bainbridge composed on 2024-02-21 11:57 (UTC+1100):

> Felix Miata wrote:

>> A current thread from elsewhere that should be helpful:
>> <https://forums.opensuse.org/t/btrfs-disk-full-how-to-fix-it-is-that-
>> really-the-solution/172576>

>> btrfs filesystem usage /
>> snapper list
>> btrfs qgroup show /

> Thanks for the prompt, Felix

> >> sudo btrfs filesystem usage /
> [sudo] password for root:
> Overall:
> Device size: 70.00GiB ===== remember I expanded this partition
> yesterday

How? Did you resize the filesystem too?
<https://btrfs.readthedocs.io/en/latest/btrfs-filesystem.html#man-filesystem-resize>

[Keith Bainbridge]

On 21/2/24 10:47, Felix Miata wrote:
> I didn't think so, which begs the question why OP Keith is using it. :p
> --

I read somewhere about 2 years ago, that it automagically de-duped data
when it detected I was copying the same file to different directories.
It's not deliberate, but I have often found copies of my files amongst
files I have backed up for my wife (where I have put copies of my
important stuff occasionally. And more common if I rsync a usb to a
new destination - the usb I use at a volunteer job with portable apps on
a foreign PC. I found what I think was that article the other week;
and it now talks about a manual process to de-dupe the data. I don't
believe I could have skipped the most important paragraph in the
article. I prefer rdfind -makehardlinks. Writing to btrfs feels
quicker than ext4

But why use btrfs on a system partition. I like to try new stuff. Like I
get a nagging thought of trying arch linux regularly. I resist
mostly...... I should use sid as my daily driver, but I'm not good at
asking for help; and I get the feeling that some people here would want
to tell me something like 'I made my bed, lay in it.'


So I make do with using developer versions of firefox & thunderbird
and beta libreOffice (24.02 I think) They are on a separate partition
from my system.

[to...@tuxteam.de]

On Wed, Feb 21, 2024 at 12:21:05PM +1100, Keith Bainbridge wrote:
>
> On 21/2/24 10:47, Felix Miata wrote:
> > I didn't think so, which begs the question why OP Keith is using it. :p
> > --
>
> I read somewhere about 2 years ago, that it automagically de-duped data

> when it detected I was copying the same file to different directories [...]

I think the Wikipedia [1] is a good ref, at least at the level we are
discussing.

Deduplication is mentioned there requiring userspace tools, so it seems
you'll have to run a process (as a daemon, from cron, whatever) to achieve
that.

It also mentions "reflinks", which is a kind of COW file copy (not to be
confused with a hardlink, which all civilised file systems have).

Cheers
[1] https://en.wikipedia.org/wiki/Btrfs#List_of_features
--
t

[David Wright]

On Mon 19 Feb 2024 at 10:26:05 (+1100), Keith Bainbridge wrote:
> On 18/2/24 14:49, Keith Bainbridge wrote:
> > On 18/2/24 07:34, debia...@howorth.org.uk wrote:

> > > Keith Bainbridge wrote:
> > > > Yes the / partitions are btrfs
> > >
> > > So the apparently missing space is perhaps taken up by btrfs snapshots.
> > >
> >
> > Seems to be the prime suspect.   If that's the case, btrfs is NOT
> > hard- linking the snapshots as timeshift claims it does. The only
> > way to check is install on ext4 and compare. I have saves enough
> > free space to do this.
> >
> > My effort to date is to move my home to /mnt/data and sim-link it
> > into / home. df is now showing 2.3GB free on /.  df showed /home
> > as 2.2GB yesterday.  At least there is a little space to play
> > with; and give me time to consider. A fresh install may be worth
> > checking in snapshots are as big as this all makes them look.
> >
> > a few brief answer to other comments will follow
>
>
> So later yesterday afternoon I created a new snapshot with no obvious
> change is free space.

That would seem reasonable, as there's not much more to do than note
which files the snapshot contains.

> I then update/upgrade. The initial attempt told me
> 63 upgraded, 3 newly installed, 0 to remove and 0 not upgraded.
> Need to get 337 MB of archives.
> After this operation, 473 MB of additional disk space will be used.
> Do you want to continue? [Y/n]
>
> But the 3 kernel related packages failed to install a couple of times.
> When I finally figured I should check space, there was none. I
> rolled back to prior to the upgrade, but still no free space.
>
> I said sometime in this thread that timeshift (and BiT) use hard links
> to create progressive copies of the system. The more I think about how
> hard links reportedly work, I reckon it can't be simply hard links.

Reading a tiny bit of the BTRFS docs, I would suggest that you're not
taking Block Groups into account. If BTRFS allocates data chunks in
blocks of 1GB, then it's conceivable that an upgrade involving 473MB
of data, plus the consequential increase in the size of the snapshot,
could all reside in a very recently created Block Group.

Put another way, df would see no change in Used and Available (as no
new Block Group allocation), whereas I would expect btrfs filesystem df
to report a change here (because it knows what's within its Block
Groups):

> Data, single: total=32.80GiB, used=31.94GiB
> Tue 20Feb2024@20:57:45

Cheers,
David.