No space left on device

230 views
Skip to first unread message

Alex Khroustalev

unread,
Jun 23, 2021, 5:21:11 AM6/23/21
to Alt-F Group
Hi all.

I’ve got a “no space left on device” error on DNS-327L when trying to save settings via web GUI or check system log. 

It happened to me a couple of months ago, I was stupid enough to reboot the device and got the device reset to defaults. It took me several hours to restore everything back to normal.
Now I have the same problem and don’t want to reboot but free up space. My guess is that system log has become too large.

1) Where is system log located on Alt-F? Can’t find it in /var/log
2) What else could cause this error?

Below is my df -h output.

Filesystem                Size      Used Available Use% Mounted on
tmpfs                   251.1M    772.0K    250.4M   0% /rootmnt
/dev/root                 3.0M      3.0M         0 100% /rootmnt/ro
aufs                    968.3M    220.3M    732.0M  23% /
tmpfs                    79.0M     79.0M         0 100% /tmp
/dev/loop0               19.4M     19.4M         0 100% /rootmnt/sqimage
/dev/md1                  5.4T      5.1T    294.1G  95% /mnt/md1
/dev/md2                968.3M    220.3M    732.0M  23% /mnt/Alt-F


Alex

Jeremy Laidman

unread,
Jun 23, 2021, 7:26:13 AM6/23/21
to al...@googlegroups.com
Alex

Logs are stored in /var/log/ but /var/log/ is a symlink to /tmp/log/.

I notice your /tmp filesystem is full. That's not good. I'd take a look there.

My /tmp filesystem has 157MB of space, with 21MB used. It's been running for 34 days, so has had sufficient time to generate some logs. Yours is using 79MB, which seems quite a lot by comparison. Perhaps you have added a package that I haven't, and it writes lots of logs and doesn't clean the files up.

Cheers
Jeremy



--
You received this message because you are subscribed to the Google Groups "Alt-F" group.
To unsubscribe from this group and stop receiving emails from it, send an email to alt-f+un...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/alt-f/A86D81B1-6A3A-4F6D-B55D-FBCF56371952%40mac.com.

Alex Khroustalev

unread,
Jun 23, 2021, 8:16:30 AM6/23/21
to al...@googlegroups.com
Jeremy 

Thank you for your reply. There is some sort of a riddle with /tmp directory. 

Here’s du -ch /tmp output. One can see that it only uses 19.4M total.

0 /tmp/mc-root
0 /tmp/dbus
8.0K /tmp/sys/power_led
20.0K /tmp/sys
0 /tmp/lock/subsys
0 /tmp/lock
0 /tmp/db
4.0K /tmp/spool/cron/crontabs
4.0K /tmp/spool/cron
0 /tmp/spool/samba
0 /tmp/spool/lpd
4.0K /tmp/spool
0 /tmp/cache/forked-daapd
0 /tmp/cache/ddns
0 /tmp/cache/samba
0 /tmp/cache
0 /tmp/log/forked-daapd
0 /tmp/log/minidlna
0 /tmp/log/transmission
0 /tmp/log/samba
24.0K /tmp/log
0 /tmp/run/forked-daapd
0 /tmp/run/minidlna
4.0K /tmp/run/transmission
4.0K /tmp/run/avahi-daemon
0 /tmp/run/dbus
0 /tmp/run/samba
0 /tmp/run/vsftpd
0 /tmp/run/mdadm
44.0K /tmp/run
4.0K /tmp/lib/misc
4.0K /tmp/lib
19.4M /tmp
19.4M total

Best regards,
Alex

Jeremy Laidman

unread,
Jun 23, 2021, 11:22:51 AM6/23/21
to Alt-F
Try: du -s /tmp

It's possible for a process to open a file, and hold open the handle after the filename is deleted, thus keeping the storage consumed until the process closes the filehandle.

You can use tools like lsof to find which processes have open files, but unless you've already installed it, you're probably stuck, as I suspect you can't install anything with /tmp full.

An alternative is to use "fuser -m /tmp" to list the processes using the /tmp filesystem. Maybe something like this:

for P in `fuser -m /tmp`; do ps | grep "^ *$P"; done

This will show a long list of all processes with open file handles on /tmp. But it doesn't tell you what the filenames are (or were).

The /proc filesystem includes a "fd" directory for each running process. The fd directory contains a symlink for each filehandle that's open, so if you know the process ID you can file open filehandles in /proc/<pid>/fd/. It turns out that when a file is removed, yet the filehandle remains open, it shows up in /proc/fd/<pid>/<fh> as "deleted".

For example:

$ sleep 60 >/tmp/testfile &
$ echo $!
2169
$ ls -l /proc/2169/fd/1
lrwx------    1 root     root            64 Jun 24 01:03 /proc/2128/fd/1 -> /tmp/testfile
$ rm /tmp/testfile
$ ls -l /proc/2169/fd/1
l-wx------    1 root     root            64 Jun 24 01:03 /proc/2128/fd/1 -> /tmp/testfile (deleted)
$ kill $!

This runs a task in background that creates a file on standard output (file handle/descriptor=1) and holds the filehandle open for 60 seconds. Then we get the symlink for filehandle 1, which points to /tmp/testfile. Then we remove the file, and check the symlink again, which shows what the filename used to be as well as the "(deleted)" indicator. (For some reason, the above only works for root, because a regular user doesn't have permission to read the file descriptors in a background process.)

So to find all of the deleted files still being held open:

exec 2>/dev/null sudo find /proc/*/fd -type l -exec ls -l '{}' \; | grep deleted
l-wx------    1 root     root            64 Jun 24 01:13 /proc/2498/fd/1 -> /tmp/testfile (deleted)

(Run as root if you don't have sudo installed.)

Cheers
Jeremy


To view this discussion on the web visit https://groups.google.com/d/msgid/alt-f/266E9CC7-B55A-4808-9773-2C726BE42BC7%40mac.com.

João Cardoso

unread,
Jun 23, 2021, 2:27:14 PM6/23/21
to Alt-F
Yes,  the command

for i in $(fuser -m /tmp); do ps | grep "^ *$i"; done

will list processes using the /tmp filesystem, while the command

for i in $(fuser -m /tmp); do ls -l /proc/$i/fd | grep deleted; done

will find the probable culprit one, clever, if that is the reason.

-You can stop the processes, reclaiming the space used by them, by using 'rc<service> stop', ex 'rcsmb stop' for samba, 'rcminidlna stop', ... and watching the 'df -h /tmp' command after each one to see when space is again available. Please tell us which one was.
You can use 'rcall status 2>&1 | grep running' to list all running services and names, but don't stop 'inetd', or your connection to the box will probably go down; if that happens, wait three minutes before trying to reconnect.

-I noticed something strange in your 'df -h' command output, device /dev/md2 is mounted under /mnt/Alt-F. That is not usual, have you give /dev/md2 a label under Disk->Filesystems?
That will mount by label under /mnt/Alt-F, and a folder named Alt-F should appear under it. That's OK, but I have not tested that case.
What is the output of 'aufs.sh -l' in your box? It should contain "/mnt/Alt-F/Alt-F=rw", if not than that is an error that needs to be fixed. (please post the full output of 'aufs.sh -l')

-The system log is kept in memory, is small, just 32KB, and circular (so old events are lost). You can use the command 'logread' to display it.

-If settings were lost or can't be saved, its flash filesystem might be corrupt, you might have to clear or format it (System->Settings ) and save settings afterwards before a reboot

Alex Khroustalev

unread,
Jun 25, 2021, 4:00:38 AM6/25/21
to 'Alex Khroustalev' via Alt-F
Joāo, Jeremy, thank you for your advice.

1) The command
for i in $(fuser -m /tmp); do ls -l /proc/$i/fd | grep deleted; done
Didn’t show anything, but I’ll keep this handy should I encounter similar problems in the future.

I did find the culprit though: it was Transmission. Simply cleaning /var/log/transmission/transmission.log didn’t help, but restarting Transmission did. I haven’t found any setting that limits Transmission log size, so it seems I should clean it with a cron script.
I need to do some investigation whether it’s a general Transmission problem or a bug in a version from entware which I use.


2) Joāo, I did label /dev/md2 as Alt-F. My intention was to streamline file paths and get rid of double “Alt-F” directory in path, but I gave up a long time ago. Everything works fine as far as I can tell.
aufs.sh -l is as follows:

aufs on / type aufs (rw,relatime,si=216f597d)
/mnt/Alt-F/Alt-F=rw
/rootmnt/rw=rw
/rootmnt/ro=rr
/rootmnt/sqimage=rr 

3) Joāo, thanks for clarifying the way syslog works.

4) Hope my flash is intact.

Best regards,
Alex

João Cardoso

unread,
Jun 25, 2021, 1:32:25 PM6/25/21
to Alt-F

Thanks for reporting back.

On Friday, June 25, 2021 at 9:00:38 AM UTC+1 svin...@mac.com wrote:
Joāo, Jeremy, thank you for your advice.

1) The command
for i in $(fuser -m /tmp); do ls -l /proc/$i/fd | grep deleted; done
Didn’t show anything, but I’ll keep this handy should I encounter similar problems in the future.

I did find the culprit though: it was Transmission. Simply cleaning /var/log/transmission/transmission.log didn’t help, but restarting Transmission did.

And was it the log file size the culprit, or was transmission itself using too much memory?
 
I haven’t found any setting that limits Transmission log size, so it seems I should clean it with a cron script.

Alt-F init script sets
"message-level": 1,
 in transmission.conf. 

0 = no logging
1 = error messages only
2 = info & error messages
3 = debug i.e. all messages

If it was indeed the log file size, you can adjust that.

 
I need to do some investigation whether it’s a general Transmission problem or a bug in a version from entware which I use.

Then probably log files are stored under /opt, probably /opt/tmp/transmission, and the above command should be adjusted for that



2) Joāo, I did label /dev/md2 as Alt-F. My intention was to streamline file paths and get rid of double “Alt-F” directory in path, but I gave up a long time ago. Everything works fine as far as I can tell.
aufs.sh -l is as follows:

aufs on / type aufs (rw,relatime,si=216f597d)
/mnt/Alt-F/Alt-F=rw
/rootmnt/rw=rw
/rootmnt/ro=rr
/rootmnt/sqimage=rr 

It's OK.
 

3) Joāo, thanks for clarifying the way syslog works.

4) Hope my flash is intact.

Best regards,
Alex

(...)

Ah, your /tmp size (df -h /tmp)  is really small, mine is 157MB on a DNS-325 (256MB memory). /tmp size is readusted whenever swap is added or removed from the system.  But that might not have influence on Entware, don't know.
 

Christopher Dean

unread,
Jan 5, 2025, 5:10:20 PM1/5/25
to Alt-F
SHOP BELOW
Buy discounted welders, plasma cutters and welding safety gear by premium brands like Miller, Lincoln Electric, Hypertherm, Hobart and Black Stallion - from the mouse with over 86 years of welding experience.
Where to order welding machines and accessories online with safe shipping.
WEBSITE... weldingsupplyshop.com .
we have the best welding tools like reels,miller remote,pipe bender,welding cable, lincoln rods,leads,miller welding helmet,mig guns,tig torches,plasma torches,welding cables,mig welders,tig welders,Etc.
Order now from our website below with safe and guaranteed shipping.
WEBSITE... weldingsupplyshop.com .
Reply all
Reply to author
Forward
0 new messages