"No space left on device" error, but df shows plenty of space

[Kynn Jones]

Hi everyone!

When I tried to install `ntp`, there was a "No space left on device" at the end of `apt-get`'s output, but `df` shows no shortage of space:

    # df
    Filesystem     1K-blocks      Used Available Use% Mounted on
    /dev/sda5      381993164 206410036 156155956  57% /
    udev               10240         0     10240   0% /dev
    tmpfs            6588000      9676   6578324   1% /run
    tmpfs           16469996       108  16469888   1% /dev/shm
    tmpfs               5120         4      5116   1% /run/lock
    tmpfs           16469996         0  16469996   0% /sys/fs/cgroup
    tmpfs            3294000         8   3293992   1% /run/user/1000

According to `df`, the maximum percent usage is 57%.

So I tried `dpgk-reconfigure ntp` (as root), and got the same error:

    # dpkg-reconfigure ntp
    Error: No space left on device
    update-rc.d: warning: start and stop actions are no longer supported; falling back to defaults
    Error: No space left on device

How can I troubleshoot this problem further?

    # uname -ar
    Linux luna 3.16.0-4-amd64 #1 SMP Debian 3.16.7-ckt20-1+deb8u3 (2016-01-17) x86_64 GNU/Linux

Thanks in advance!

kj

[Andy Smith]

Hi Kynn,

On Tue, Jan 31, 2017 at 08:01:00AM -0500, Kynn Jones wrote:
> Filesystem 1K-blocks Used Available Use% Mounted on
> /dev/sda5 381993164 206410036 156155956 57% /

[…]

> # dpkg-reconfigure ntp
> Error: No space left on device
>

> How can I troubleshoot this problem further?

Does /dev/sda5 have a btrfs filesystem on it? Type "mount" and look
for the "type" part. If so, these problems are common with btrfs;
see btrfs wiki and/or linux-btrfs mailing list:

https://btrfs.wiki.kernel.org/index.php/FAQ#Help.21_Btrfs_claims_I.27m_out_of_space.2C_but_it_looks_like_I_should_have_lots_left.21

If not, could be things like deleted files that are still open (see
"lsof" output, look for "(deleted)". Or maybe ran out of inodes. See
"df -i" out to check that.

Cheers,
Andy

--
https://bitfolk.com/ -- No-nonsense VPS hosting

[Tony Baldwin]

On 01/31/2017 08:03 AM, Eduardo M KALINOWSKI wrote:

> On Ter, 31 Jan 2017, Kynn Jones wrote:
>> Hi everyone!
>>
>> When I tried to install `ntp`, there was a "No space left on device"
>> at the
>> end of `apt-get`'s output, but `df` shows no shortage of space:
>

> Try 'df -i', you're probably out of inodes.

I guess I should have read your reply before sending one suggesting the
same precise thing.

Sorry,
Tony

--
http://tonybaldwin.me
all tony, all the time

[Brad Rogers]

On Tue, 31 Jan 2017 08:01:00 -0500
Kynn Jones <kyn...@gmail.com> wrote:

Hello Kynn,

>When I tried to install `ntp`, there was a "No space left on device" at
>the end of `apt-get`'s output, but `df` shows no shortage of space:

I've no idea whether this is relevant to your case, but it's possible to
be unable to use disk space if there are insufficient inodes left to
store the relevant directory/file info. AIUI, a "No space..." error is
the result.

--
Regards _
/ ) "The blindingly obvious is
/ _)rad never immediately apparent"
Just stop and take a second
U & Ur Hand - P!nk

[Tony Baldwin]

On 01/31/2017 08:01 AM, Kynn Jones wrote:
> Hi everyone!
>
> When I tried to install `ntp`, there was a "No space left on device" at
> the end of `apt-get`'s output, but `df` shows no shortage of space:
>
> # df
> Filesystem 1K-blocks Used Available Use% Mounted on
> /dev/sda5 381993164 206410036 156155956 57% /
> udev 10240 0 10240 0% /dev
> tmpfs 6588000 9676 6578324 1% /run
> tmpfs 16469996 108 16469888 1% /dev/shm
> tmpfs 5120 4 5116 1% /run/lock
> tmpfs 16469996 0 16469996 0% /sys/fs/cgroup
> tmpfs 3294000 8 3293992 1% /run/user/1000
>
> According to `df`, the maximum percent usage is 57%.
>
> So I tried `dpgk-reconfigure ntp` (as root), and got the same error:
>
> # dpkg-reconfigure ntp
> Error: No space left on device
> update-rc.d: warning: start and stop actions are no longer
> supported; falling back to defaults
> Error: No space left on device
>
> How can I troubleshoot this problem further?

try doing
$ df -i
This shows inodes.
I once had a problem where I could no longer write to a disk, because
all of the inodes were occupied due to not properly rotating some
log(s). Let us know if that reveals anything seemingly significant.

[Eduardo M KALINOWSKI]

On Ter, 31 Jan 2017, Kynn Jones wrote:

> Hi everyone!
>
> When I tried to install `ntp`, there was a "No space left on device" at the
> end of `apt-get`'s output, but `df` shows no shortage of space:

Try 'df -i', you're probably out of inodes.

--
Eduardo M KALINOWSKI
edu...@kalinowski.com.br

[Kynn Jones]

Hi everyone!  Thank you for your suggestions.  They taught some new tricks.

As it happened, I had to reboot the machine for an unrelated (?) reason (monitor would not wake from sleep).

After the machine rebooted, I was able to run `dpkg-reconfigure ntp` without error.

Unfortunately, I'll never know what the problem was.

Thank you all for your help.

kj

[Andy Smith]

Hi Kynn,

On Tue, Jan 31, 2017 at 06:42:39PM -0500, Kynn Jones wrote:
> After the machine rebooted, I was able to run `dpkg-reconfigure ntp`
> without error.
>
> Unfortunately, I'll never know what the problem was.

Do you use btrfs?

What does "df -i" report now, after your reboot when things are
working?

[Kynn Jones]

On Wed, Feb 1, 2017 at 7:24 AM, Andy Smith <an...@strugglers.net> wrote:

Hi Kynn,

On Tue, Jan 31, 2017 at 06:42:39PM -0500, Kynn Jones wrote:
> After the machine rebooted, I was able to run `dpkg-reconfigure ntp`
> without error.
>
> Unfortunately, I'll never know what the problem was.

Do you use btrfs?

Not that I'm aware of.  (FWIW, if I run `mount | grep -i btrfs` (as root), I get no output.)

What does "df -i" report now, after your reboot when things are
working?

    # df -i
    Filesystem       Inodes   IUsed    IFree IUse% Mounted on
    /dev/sda5      24264704 1464023 22800681    7% /
    udev            4115384     445  4114939    1% /dev
    tmpfs           4117499     735  4116764    1% /run
    tmpfs           4117499      14  4117485    1% /dev/shm
    tmpfs           4117499       5  4117494    1% /run/lock
    tmpfs           4117499      13  4117486    1% /sys/fs/cgroup
    tmpfs           4117499      10  4117489    1% /run/user/1000

Unfortunately, I don't have the output of `df -i` before I rebooted.

Best,

kj

[Andy Smith]

Hi Kynn,

On Wed, Feb 01, 2017 at 10:43:37AM -0500, Kynn Jones wrote:
> On Wed, Feb 1, 2017 at 7:24 AM, Andy Smith <an...@strugglers.net> wrote:
> > On Tue, Jan 31, 2017 at 06:42:39PM -0500, Kynn Jones wrote:
> > > Unfortunately, I'll never know what the problem was.
> >
> > Do you use btrfs?
>
> Not that I'm aware of. (FWIW, if I run `mount | grep -i btrfs` (as root),
> I get no output.)

Okay, so not a btrfs issue.

> > What does "df -i" report now, after your reboot when things are
> > working?
>
> # df -i
> Filesystem Inodes IUsed IFree IUse% Mounted on
> /dev/sda5 24264704 1464023 22800681 7% /

…and that doesn't show as being anywhere near full, so it most
likely wasn't that either.

Well I suppose it could have been processes holding open deleted
files, though it would have to have been some really big files in
that case, as your filesystem didn't show as being anywhere near
full.

[Eike Lantzsch]

let me guess:

facts:
/var is on the same partition as /
there is plenty of space and there are free i-nodes but anyway you got error
apt-get resulted in "no space left on device"
you rebooted and the error disappeared

=>

/ was mounted as r/o for some reason (unclean filesystem e.g.)