no space left on device -- but there is

[Barry Kaplan]

On one of our nodes we see:

2016/07/05 17:09:46 [ERR] serf: Failed to update snapshot: write /var/lib/consul/serf/local.snapshot: no space left on device

but:

$ df
Filesystem      Size  Used Avail Use% Mounted on
udev            7.9G   12K  7.9G   1% /dev
tmpfs           1.6G  620K  1.6G   1% /run
/dev/xvda1       16G  4.5G   11G  30% /
none            4.0K     0  4.0K   0% /sys/fs/cgroup
none            5.0M     0  5.0M   0% /run/lock
none            7.9G  260K  7.9G   1% /run/shm
none            100M     0  100M   0% /run/user
/dev/xvdf        99G  1.1G   93G   2% /data

What device could it be talking about?

[Brian Lalor]

Check your inode usage.  `df -i`.

--
This mailing list is governed under the HashiCorp Community Guidelines - https://www.hashicorp.com/community-guidelines.html. Behavior in violation of those guidelines may result in your removal from this mailing list.
 
GitHub Issues: https://github.com/hashicorp/consul/issues
IRC: #consul on Freenode
---
You received this message because you are subscribed to the Google Groups "Consul" group.
To unsubscribe from this group and stop receiving emails from it, send an email to consul-tool...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/consul-tool/3646d703-94f9-458b-916c-c074f7541928%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

— 

Brian Lalor

bla...@bravo5.org

[Barry Kaplan]

I restarted the client and the error stopped. I'm clueless.

[Michael Fischer]

Which version of Consul are you using?  Prior to 0.6 it would create a 16 or 32 GB memory mapped file and if your data filesystem didn't have enough space it would cause this error. 

The solution is to ensure you have at least that much free space or to upgrade to 0.6. 

On Tuesday, July 5, 2016, Barry Kaplan <mem...@gmail.com> wrote:

I restarted the client and the error stopped. I'm clueless.

--

This mailing list is governed under the HashiCorp Community Guidelines - https://www.hashicorp.com/community-guidelines.html. Behavior in violation of those guidelines may result in your removal from this mailing list.
 
GitHub Issues: https://github.com/hashicorp/consul/issues
IRC: #consul on Freenode
---
You received this message because you are subscribed to the Google Groups "Consul" group.
To unsubscribe from this group and stop receiving emails from it, send an email to consul-tool...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/consul-tool/56252363-ffd0-4cba-8289-534835052001%40googlegroups.com.

[Barry Kaplan]

This is version 0.6.3

[Barry Kaplan]

Just encountered this problem again with 0.6.3, on dozens of nodes. No evidence of disk shortage or inode shortage. I'm quite certain this is not a consul issue, but I cannot find the cause.

Failed to update snapshot: write /var/lib/consul/serf/local.snapshot: no space left on device

[s-mesos-slave-5 10.0.136.120]$ df

Filesystem      Size  Used Avail Use% Mounted on
udev            7.9G   12K  7.9G   1%

/dev
tmpfs           1.6G  968K  1.6G   1% /run
/dev/xvda1       16G  8.4G  6.6G  56% /

none            4.0K     0  4.0K   0% /

sys/fs/cgroup
none            5.0M     0  5.0M   0% /run/lock
none            7.9G  2.2M  7.9G   1% /run/shm
none            100M     0  100M   0% /run/user
/dev/xvdf        99G  4.6G   89G   5% /data
bindfs           16G  8.4G  6.6G  56% /log

[s-mesos-slave-5 10.0.136.120]$ df -i
Filesystem     Inodes IUsed IFree IUse% Mounted on
udev             2.0M   391  2.0M    1% /dev
tmpfs            2.0M   36K  2.0M    2% /run
/dev/xvda1       1.0M  198K  827K   20% /
none             2.0M    11  2.0M    1% /sys/fs/cgroup
none             2.0M     1  2.0M    1% /run/lock
none             2.0M    43  2.0M    1% /run/shm
none             2.0M     3  2.0M    1% /run/user
/dev/xvdf        6.3M  208K  6.1M    4% /data
bindfs           1.0M  198K  827K   20% /log

[Jonathan Ballet]

I faced this issue some weeks ago, due to a full disk at some point, but even after making some space, Consul couldn't recover.

There's an issue opened for this with a proposed fix, but I haven't tested yet, see https://github.com/hashicorp/consul/issues/1744

--

This mailing list is governed under the HashiCorp Community Guidelines - https://www.hashicorp.com/community-guidelines.html. Behavior in violation of those guidelines may result in your removal from this mailing list.
 
GitHub Issues: https://github.com/hashicorp/consul/issues
IRC: #consul on Freenode
---
You received this message because you are subscribed to the Google Groups "Consul" group.
To unsubscribe from this group and stop receiving emails from it, send an email to consul-tool...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/consul-tool/2ac211e7-2e6d-43f6-8c43-0854243ee115%40googlegroups.com.

[Barry Kaplan]

Hmm, that could have been the case for me. Thanks.