du showing more disk usage than df
bzaman
Surprisingly , the du command in my desktop is showing more disk usage
than df . I came across systems where df command shows more disk usage
than du because df includes metadata information about partition and
also considers disk space consumed by files which are removed but not
yet released by some process .
=====
$ sudo du -sh /
13G /
$ sudo df -h /
Filesystem Size Used Avail Use% Mounted on
/dev/md0 9.5G 8.1G 903M 91% /
---
$ sudo df -h /usr/
Filesystem Size Used Avail Use% Mounted on
/dev/sda5 8.6G 4.6G 3.6G 56% /usr
$ sudo du -sh /usr/
5.0G /usr/
======
Can somebody explain why this discrepancy in df and du results .
Thanks in Advance
Badiuzzaman
David Brown
If you've got lots of sparse files, they will show up in the du disk
usage, but not on the df for disk free.
You'd also get discrepancies if you have other file systems mounted
within these trees - they count for du but not for df.
Doug Freyburger
> bzaman wrote:
>
>> Surprisingly , the du command in my desktop is showing more disk usage
>> than df . I came across systems where df command shows more disk usage
>> than du because df includes metadata information about partition and
>> also considers disk space consumed by files which are removed but not
>> yet released by some process .
This is extremely common.
>> Can somebody explain why this discrepancy in df and du results .
>
> If you've got lots of sparse files, they will show up in the du disk
> usage, but not on the df for disk free.
>
> You'd also get discrepancies if you have other file systems mounted
> within these trees - they count for du but not for df.
Listing at handful of reasons and explaining why they happen is a
standard interview question for anyone claiming to be intermediate or
more in skill and knowledge - Understanding inodes is on the SAGE scale
for intermediate.
Additional reasons include use of hard links.
bzaman
wrote:
Thanks for the reply.
Is there any way to identify these sparse files . Should not be an
issue with the mounts as I have rebooted the host and still du is
showing more disk space than df .
Thanks
Zaman
Keith Keller
> On 29/04/2010 13:17, bzaman wrote:
>>
>> =====
>> $ sudo du -sh /
>> 13G /
>> $ sudo df -h /
>> Filesystem Size Used Avail Use% Mounted on
>> /dev/md0 9.5G 8.1G 903M 91% /
>> ---
>> $ sudo df -h /usr/
>> Filesystem Size Used Avail Use% Mounted on
>> /dev/sda5 8.6G 4.6G 3.6G 56% /usr
>> $ sudo du -sh /usr/
>> 5.0G /usr/
>> ======
>>
>>
>> Can somebody explain why this discrepancy in df and du results .
>
> within these trees - they count for du but not for df.
In particular, your first du command includes the 5GB on /usr.
Subtracting that away gets you closer to df's reported 8GB used.
There are plenty of ways to detect a sparse file (see google), but I
don't know of a way using find directly to locate them. Of course you
could combine a detect program with find, but that could be slow on a
large filesystem. You might start in directories that often have sparse
files--/var, for example, often has them (though those particular sparse
files you wouldn't want to mess with!).
--keith
--
kkeller...@wombat.san-francisco.ca.us
(try just my userid to email me)
AOLSFAQ=http://www.therockgarden.ca/aolsfaq.txt
see X- headers for PGP signature information
David Brown
I thought of that one too - but hard links are only counted once by du
unless you use the -l switch.
unruh
> Hi ,
>
> Surprisingly , the du command in my desktop is showing more disk usage
> than df . I came across systems where df command shows more disk usage
> than du because df includes metadata information about partition and
> also considers disk space consumed by files which are removed but not
> yet released by some process .
>
>=====
> $ sudo du -sh /
> 13G /
> $ sudo df -h /
> Filesystem Size Used Avail Use% Mounted on
> /dev/md0 9.5G 8.1G 903M 91% /
> ---
> $ sudo df -h /usr/
> Filesystem Size Used Avail Use% Mounted on
> /dev/sda5 8.6G 4.6G 3.6G 56% /usr
> $ sudo du -sh /usr/
> 5.0G /usr/
Sparse files? du uses the size in the directories to add up the total
used. A sparse file may show a size of 2GB but only consume 1K of the
disk space.
Lew Pitcher
un...@wormhole.physics.ubc.ca wrote:
> On 2010-04-29, bzaman <bzaman...@gmail.com> wrote:
>> Hi ,
>>
>> Surprisingly , the du command in my desktop is showing more disk usage
>> than df . I came across systems where df command shows more disk usage
>> than du because df includes metadata information about partition and
>> also considers disk space consumed by files which are removed but not
>> yet released by some process .
>>
>>=====
>> $ sudo du -sh /
>> 13G /
>> $ sudo df -h /
>> Filesystem Size Used Avail Use% Mounted on
>> /dev/md0 9.5G 8.1G 903M 91% /
>> ---
>> $ sudo df -h /usr/
>> Filesystem Size Used Avail Use% Mounted on
>> /dev/sda5 8.6G 4.6G 3.6G 56% /usr
>> $ sudo du -sh /usr/
>> 5.0G /usr/
>
> Sparse files? du uses the size in the directories to add up the total
> used. A sparse file may show a size of 2GB but only consume 1K of the
> disk space.
Also
1) du includes filespace "reserved" for root, but df doesn't
2) du doesnt include the space in "deleted" files (files without inodes)
but df does
>>======
>>
>>
>> Can somebody explain why this discrepancy in df and du results .
>>
>>
>> Thanks in Advance
>> Badiuzzaman
>>
--
Lew Pitcher
Master Codewright & JOAT-in-training | Registered Linux User #112576
Me: http://pitcher.digitalfreehold.ca/ | Just Linux: http://justlinux.ca/
---------- Slackware - Because I know what I'm doing. ------
Lew Pitcher
wrote:
3) du accounts for used filespace at a byte-level (filesize), while df
accounts for used filespace at a block level
Robert Nichols
> Hi ,
>
> Surprisingly , the du command in my desktop is showing more disk usage
> than df . I came across systems where df command shows more disk usage
> than du because df includes metadata information about partition and
> also considers disk space consumed by files which are removed but not
> yet released by some process .
>
> =====
> $ sudo du -sh /
> 13G /
> $ sudo df -h /
> Filesystem Size Used Avail Use% Mounted on
> /dev/md0 9.5G 8.1G 903M 91% /
> ---
> $ sudo df -h /usr/
> Filesystem Size Used Avail Use% Mounted on
> /dev/sda5 8.6G 4.6G 3.6G 56% /usr
> $ sudo du -sh /usr/
> 5.0G /usr/
> ======
Try it again, this time passing the "-x" option to 'du' so that it
won't descend into mount points that it encounters.
--
Bob Nichols AT comcast.net I am "RNichols42"
bzaman
Thanks . Tried with the -x option , '/' directory space utilization
report looks almost same for both du and df .
=====
zamanb@shopfrobsoon-dr ~]$ sudo df -h /
Filesystem Size Used Avail Use% Mounted on
/dev/md0 9.5G 8.1G 902M 91% /
[zamanb@shopfrobsoon-dr ~]$ sudo du -shx /
8.0G /
====
For , /usr partition , the difference is much less now .
===
[zamanb@shopfrobsoon-dr ~]$ sudo du -shx /usr/
Password:
5.0G /usr/
[zamanb@shopfrobsoon-dr ~]$ sudo df -h /usr/
Filesystem Size Used Avail Use% Mounted on
/dev/sda5 8.6G 4.6G 3.6G 56% /usr
===
--Bz
Robert Nichols
That 0.4G difference for /usr sounds like the space allocated for the
resize inode (inode 7), which 'df' takes into account. Run
tune2fs -l /dev/sda5 | grep features
and see if "resize_inode" appears in the list. It would appear that
your root filesystem was built without that feature. You can also
run:
debugfs /dev/sda5
and type "stat <7>" at the "debugfs:" prompt. Near the top of
the output you should see something like:
Links: 1 Blockcount: 106088
Multiply that block count by the FS block size (probably 4096) to
find the space used.
Note: Running 'debugfs' that way is safe. As long as you don't run
it with the "-w" option it will never write to the filesystem.
Matt Giwer
Formatting defaults to reserving space for su use. Do not use sudo unless you
are interested in that reserved space. Why do you do that in the first place?
If the size of the disk is 1 and du is X then df is 1-X and with rounding
they cannot add up to 1 with the -h option. If you output sectors it should
only be off by one sector because of the X and 1-X results.
Practical matters.
If it is just a generic storage disc then what you add will exactly track a
decrease in df in a non-su case. I have never played with the sudo thing but
as root I get the same as a user. It is not clear exactly what you are getting
with your options. KISS. Use only du -h and df -h. They work and have only the
X and 1-X difference.
There are some marginal gains you can get if a file increased incrementally
over many write operations by copying it to another file. The copy-to file
should be slightly smaller. Then rm the old and mv the new to the old name.
Generally if you are down to playing games like this you need to buy a new
drive. They are CHEAP! these days. Do it.
Given the sizes you report you are nursing a 16bit system far beyond the time
it deserves to be put out of its misery and given a decent burial. Do the
honorable thing.
--
"The werewolf ate my homework," declared the young Dr. Frankenstein.
-- The Iron Webmaster, 4263
http://www.haaretz.com What is Israel really like? http://www.jpost.com a7
Tue May 11 05:51:17 EDT 2010
unruh
> On 04/29/2010 07:17 AM, bzaman wrote:
>> Hi ,
>
>> Surprisingly , the du command in my desktop is showing more disk usage
>> than df . I came across systems where df command shows more disk usage
>> than du because df includes metadata information about partition and
>> also considers disk space consumed by files which are removed but not
>> yet released by some process .
>
>> =====
>> $ sudo du -sh /
>> 13G /
>> $ sudo df -h /
>> Filesystem Size Used Avail Use% Mounted on
>> /dev/md0 9.5G 8.1G 903M 91% /
>> ---
>> $ sudo df -h /usr/
>> Filesystem Size Used Avail Use% Mounted on
>> /dev/sda5 8.6G 4.6G 3.6G 56% /usr
>> $ sudo du -sh /usr/
>> 5.0G /usr/
>> ======
>
>> Can somebody explain why this discrepancy in df and du results .
>
> Formatting defaults to reserving space for su use. Do not use sudo unless you
> are interested in that reserved space. Why do you do that in the first place?
No idea what you mean. He is asking for the ouput of du and df, not
actually using that file space. df reports the same for everyone. du
will have trouble if a directory does not read/search permission for the
user running du.
>
> If the size of the disk is 1 and du is X then df is 1-X and with rounding
> they cannot add up to 1 with the -h option. If you output sectors it should
> only be off by one sector because of the X and 1-X results.
No Not if there are sparse files.
From
http://administratosphere.wordpress.com/2008/05/23/sparse-files-what-why-and-how/
"Sparse files are basically just like any other file except that blocks
that only contain zeros (i.e., nothing) are not actually stored on disk.
This means you can have an apparently 16G file ? with 16G of ?data? ?
only taking up 1G of space on disk"
Thus du will report all 16GB, while df will report 1GB.
>
Robert Nichols
> From
> http://administratosphere.wordpress.com/2008/05/23/sparse-files-what-why-and-how/
> "Sparse files are basically just like any other file except that blocks
> that only contain zeros (i.e., nothing) are not actually stored on disk.
> This means you can have an apparently 16G file ? with 16G of ?data? ?
> only taking up 1G of space on disk"
>
> Thus du will report all 16GB, while df will report 1GB.
No, 'du' will report 1GB unless you use the "--apparent-size" option.
$ dd if=/dev/zero of=junk bs=1k seek=100000 count=1
1+0 records in
1+0 records out
1024 bytes (1.0 kB) copied, 0.000105715 s, 9.7 MB/s
$ ls -ls junk
12 -rw-rw-r--. 1 rnichols rnichols 102401024 2010-05-11 21:23 junk
$ du junk
12 junk
$ du --apparent-size junk
100001 junk
Matt Giwer
That might explain it. I don't claim to be an expert only experienced in
explaining the dumb things I do.
--
All squattertowns beyond the Green Line are considered by the world to be
criminal. Why are not those who live in them arrested as criminals?
-- The Iron Webmaster, 4253
http://www.giwersworld.org/00_files/zion-hit-points.phtml a16
Jews stole the land. The owners want it back. a16
Sun May 16 05:36:59 EDT 2010