log message "inetd not running" and Alt-F not working on USB drive

[Jeremy Laidman]

Hi

I can't work out what's wrong with my DNS-320L-Ax. All of a sudden, dnsmasq had poor response, and when I logged in, load average was more than 100, and everything was very slow. Eventually I couldn't login, and a power cycle was the only option.

After that, the web interface came back, but some of the services weren't working. The status showed the log message "watch-inetd: inetd not running, restarting it with an alternative minimum safe configuration." I changed openssh to run as a daemon so I could login and get a shell.

The Alt-F package page showed that Alt-F was running from md0 instead of sdc. I switched it back to sdc, and tried to reboot. But reboot didn't work. I couldn't reboot from the command line either. The reboot and powerdown commands seemed to do nothing.

After another power cycle, same problems. I did a wipe/save on flash, switched Alt-F back to USB drive, and did another power cycle. Still didn't fix it.

I can't find anything relevant in the logs apart from that inetd message, and something in the openssh logfile saying that /var/run/empty must not be world-writeable (which it was, and I fixed).

Any ideas what might be wrong, or where I can look next.

Cheers

Jeremy

[Jeremy Laidman]

OK, even stranger things now.

* inetd was not starting because S41inetd did not have execute permission, so I did the chmod, but after a reboot, the perms had reverted

* openssh didn't start as a service, even when reconfigured in the GUI, because it created the file /var/run/empty as world-writeable, so I hacked the script  to do that

* I can't activate sdc1 for Alt-F packages because shutting down services doesn't work, and running "aufs.sh -u") gives an error saying aufs on / is busy

* the status page is now showing this message at the top: "'dumb': unknown terminal type. sh: bad number sh: bad number sh: bad number"

[João Cardoso]

On Wednesday, September 22, 2021 at 10:38:59 AM UTC+1 Jezzaaaa wrote:

OK, even stranger things now.

* inetd was not starting because S41inetd did not have execute permission, so I did the chmod, but after a reboot, the perms had reverted

And have you saved settings?  permissions are reapplied at reboot from the last saved settings. If you have saved settings then there might be a problem with the flash-saved settings filesystem.

* openssh didn't start as a service, even when reconfigured in the GUI, because it created the file /var/run/empty as world-writeable, so I hacked the script  to do that

OK 

* I can't activate sdc1 for Alt-F packages because shutting down services doesn't work, and running "aufs.sh -u") gives an error saying aufs on / is busy

Yes. the best is to "boot disable" all them and reboot; they will not be used afterwards. If both are boot enabled, the first to be discovered is used (the first discovered might not always be the same)

* the status page is now showing this message at the top: "'dumb': unknown terminal type. sh: bad number sh: bad number sh: bad number"

 

You might have one or two problems regarding settings and/or the on-disk package instalation (the Alt-F dir). When the Alt-F dir is in use, any file that is contains and also exists in the system takes over, so if, say, inetd.conf exists in /Alt-F/etc/ it will be used instead. Configuration are loaded from flash-saved settings, then if it also exists under Alt-F, the new one will be used.

So, you can do two things: disable all Alt-F and reboot -- if problems still occurs, then the issue is in settings, you have to clear (or even format) all saved settings on flash, then reboot. This will bring the system to a "factory" state -- only configuration will be lost, not disk data, so you will have to set it again.

If problems are with the Alt-F dir, (as it has incorrect or corrupted files that take over the saved settings) you have to delete it -- all on-disk packages will be removed, along with its configuration. Don't rely on the other Alt-F dir, it might have incorrect or corrupted configuration files.

A final note: Don't directly add/delete/edit anything under /Alt-F, instead edit the normal /, i.e., don't edit /Alt-F/whateverdir/whaterverfile, add/remove/edit  /whateverdir/whaterverfile instead. If you really need to change anything under /Alt-F, use 'aufs.sh -n' first, and 'aufs.sh -r' when you are done.

Notice also that if /Alt-F/somedir exists, and you edit or create /somedir/somefile, the edit will be stored in /Alt-F/somedir/somefile.

The 'fixup clean' command will erase all files under /Alt-F that also exists in the base system (remove if it exists under /Alt-F, is not a conf file nor a file from a disk-installed package). So, if there is a corrupted status.cgi file under /Alt-F/usr/www/cgi-bin, the command will remove it, making the normal /usr/www/cgi-bin/status.cgi to be used. This is complex, and the faster way it to delete the Alt-F dir, reinstall on-disk packages and start fresh.

[Jeremy Laidman]

João, I really appreciate your help with this.

On Thu, 23 Sept 2021 at 05:13, João Cardoso <whoami...@gmail.com> wrote:

On Wednesday, September 22, 2021 at 10:38:59 AM UTC+1 Jezzaaaa wrote:

OK, even stranger things now.

* inetd was not starting because S41inetd did not have execute permission, so I did the chmod, but after a reboot, the perms had reverted

And have you saved settings?  permissions are reapplied at reboot from the last saved settings. If you have saved settings then there might be a problem with the flash-saved settings filesystem.

 

Ah, OK, I didn't think of this. I have now fixed permissions and then saved settings. Now inetd is starting correctly.

* I can't activate sdc1 for Alt-F packages because shutting down services doesn't work, and running "aufs.sh -u") gives an error saying aufs on / is busy

Yes. the best is to "boot disable" all them and reboot; they will not be used afterwards. If both are boot enabled, the first to be discovered is used (the first discovered might not always be the same)

I'm not quite understanding this. If I boot disable both of them, will nothing boot anymore? Or where will Alt-F boot from?

 

* the status page is now showing this message at the top: "'dumb': unknown terminal type. sh: bad number sh: bad number sh: bad number"

This message has now gone away. But there is still one last obvious problem: shutdown and reboot don't work; from the GUI it counts to 60 and carries on without a reboot. And of course not being able to activate Alt-F on sdc1. These all might be related.

You might have one or two problems regarding settings and/or the on-disk package instalation (the Alt-F dir). When the Alt-F dir is in use, any file that is contains and also exists in the system takes over, so if, say, inetd.conf exists in /Alt-F/etc/ it will be used instead. Configuration are loaded from flash-saved settings, then if it also exists under Alt-F, the new one will be used.

Oh, I think I understand. Is this how aufs works? I should read up on this.

 

So, you can do two things: disable all Alt-F and reboot -- if problems still occurs, then the issue is in settings, you have to clear (or even format) all saved settings on flash, then reboot. This will bring the system to a "factory" state -- only configuration will be lost, not disk data, so you will have to set it again.

I have to set it again, meaning I cannot save and then factory state, and then restore saved settings?

 

If problems are with the Alt-F dir, (as it has incorrect or corrupted files that take over the saved settings) you have to delete it -- all on-disk packages will be removed, along with its configuration. Don't rely on the other Alt-F dir, it might have incorrect or corrupted configuration files.

A final note: Don't directly add/delete/edit anything under /Alt-F, instead edit the normal /, i.e., don't edit /Alt-F/whateverdir/whaterverfile, add/remove/edit  /whateverdir/whaterverfile instead. If you really need to change anything under /Alt-F, use 'aufs.sh -n' first, and 'aufs.sh -r' when you are done.

Notice also that if /Alt-F/somedir exists, and you edit or create /somedir/somefile, the edit will be stored in /Alt-F/somedir/somefile.

OK thanks.

 

The 'fixup clean' command will erase all files under /Alt-F that also exists in the base system (remove if it exists under /Alt-F, is not a conf file nor a file from a disk-installed package). So, if there is a corrupted status.cgi file under /Alt-F/usr/www/cgi-bin, the command will remove it, making the normal /usr/www/cgi-bin/status.cgi to be used. This is complex, and the faster way it to delete the Alt-F dir, reinstall on-disk packages and start fresh.

So I would do this if reverting factory settings didn't fix my problems?

Cheers

Jeremy

[João Cardoso]

On Sunday, September 26, 2021 at 5:08:20 PM UTC+1 Jezzaaaa wrote:

João, I really appreciate your help with this.

On Thu, 23 Sept 2021 at 05:13, João Cardoso <whoami...@gmail.com> wrote:

On Wednesday, September 22, 2021 at 10:38:59 AM UTC+1 Jezzaaaa wrote:

OK, even stranger things now.

* inetd was not starting because S41inetd did not have execute permission, so I did the chmod, but after a reboot, the perms had reverted

And have you saved settings?  permissions are reapplied at reboot from the last saved settings. If you have saved settings then there might be a problem with the flash-saved settings filesystem.

 

Ah, OK, I didn't think of this. I have now fixed permissions and then saved settings. Now inetd is starting correctly.

* I can't activate sdc1 for Alt-F packages because shutting down services doesn't work, and running "aufs.sh -u") gives an error saying aufs on / is busy

Yes. the best is to "boot disable" all them and reboot; they will not be used afterwards. If both are boot enabled, the first to be discovered is used (the first discovered might not always be the same)

I'm not quite understanding this. If I boot disable both of them, will nothing boot anymore? Or where will Alt-F boot from?

From the flashed, read-only, firmware, as it always do (like a toaster or a frigo). Disks are not needed at all; Alt-F will say "no disks", but you can still setup and "save settings" (on flash) non-disk data.

If disks exist and the Alt-F folder does not exists, the box and Alt-F will still operate normally, using the pre-installed (in the firmware) packages. Additional on-disk (Alt-F folder) installed packages or existing packages updates (on the Alt-F folder) will not be obviously available.

So,

1-the system boots from the firmware stored in flash and starts running with the pre-installed packages on flash.

2-if/when a disk is discovered and has a Alt-F folder with new packages that are "boot enabled", they will be started; if a running firmware package has an updated version on the Alt-F folder, the running one is stopped and the new one started.

How this relates to "settings"?

As the firmware is read-only, settings are always saved on a special flash partition (as small as 64KB for the DNS-323). Those settings are used only for the firmware pre-installed packages.

1-At boot, default settings from the firmware  are loaded;

2-afterwards, if saved settings exists they are loaded and overrides the default ones.

3-If later on the Alt-F folder is found and it contains a "setting", it will take-over the current one.

4-Additional packages installed on the Alt-F folder save its own settings to disk, not flash, so they don't need a "save settings". But pre-installed on the firmware but updated on the Alt-F folder  packages also save their settings to disk, but they require a "save settings".

So this is complex ;-) and your problem can't be diagnosed remotely, you have to do it by yourself.

I would start keeping the current settings, boot disabling all Alt-F folders and reboot.

-If the problem remains, it is due to saved settings, and I would clear all settings an reboot -- this requires a total reconfiguration of the box, like what you did when you first flashed Alt-F  or performed a hard reset (>20sec recessed reset button press when the system is running) (the disk data is not touched!).

-if the problem seem solved (with or without a "clear settings"), I would enable one Alt-F folder. If problems reappear, that Alt-F folder has issues, I would delete it and try the other (you seems to have two). When a good one is found, I would copy it over the other.

 

* the status page is now showing this message at the top: "'dumb': unknown terminal type. sh: bad number sh: bad number sh: bad number"

This message has now gone away. But there is still one last obvious problem: shutdown and reboot don't work;

Search for that issue in the forum, it is already covered and seems to happens when the firmware Alt-F-utils package is updated (on the Alt-F folder disk)

from the GUI it counts to 60 and carries on without a reboot. And of course not being able to activate Alt-F on sdc1. These all might be related.

You might have one or two problems regarding settings and/or the on-disk package instalation (the Alt-F dir). When the Alt-F dir is in use, any file that is contains and also exists in the system takes over, so if, say, inetd.conf exists in /Alt-F/etc/ it will be used instead. Configuration are loaded from flash-saved settings, then if it also exists under Alt-F, the new one will be used.

Oh, I think I understand. Is this how aufs works? I should read up on this.

on a dns-325, similar in this respect to yours dns-320l:

 / # aufs.sh -l 

aufs on / type aufs (rw,relatime,si=a93b475c) <- this is "/", the "merge" of all the bellow
/mnt/sdb3/Alt-F=rw <- first priority, the Alt-F folder, if a file is encountered there it is used
/rootmnt/rw=rw <- second priority, memory, used when the Alt-F folder is not used, if a file is encountered here it is used
/rootmnt/ro=rr <- third priority, the read-only firmware packages, if a file is encountered here it is used
/rootmnt/sqimage=rr <- extra packages, also on read-only flash, if a file is encountered here it is used

 

So, you can do two things: disable all Alt-F and reboot -- if problems still occurs, then the issue is in settings, you have to clear (or even format) all saved settings on flash, then reboot. This will bring the system to a "factory" state -- only configuration will be lost, not disk data, so you will have to set it again.

I have to set it again, meaning I cannot save and then factory state, and then restore saved settings?

You ca't just disable settings, you have to delete them.

If you restore a corrupted saved  settings, issues will reappear. What you could do it to backup settings to a computer, clear settings, reboot. If everything looks normal load the backup from computer and you know if the settings is corrupted. Note: after clear settings you have to logout to not run the "first login wizard".

 

If problems are with the Alt-F dir, (as it has incorrect or corrupted files that take over the saved settings) you have to delete it -- all on-disk packages will be removed, along with its configuration. Don't rely on the other Alt-F dir, it might have incorrect or corrupted configuration files.

A final note: Don't directly add/delete/edit anything under /Alt-F, instead edit the normal /, i.e., don't edit /Alt-F/whateverdir/whaterverfile, add/remove/edit  /whateverdir/whaterverfile instead. If you really need to change anything under /Alt-F, use 'aufs.sh -n' first, and 'aufs.sh -r' when you are done.

Notice also that if /Alt-F/somedir exists, and you edit or create /somedir/somefile, the edit will be stored in /Alt-F/somedir/somefile.

OK thanks.

 

The 'fixup clean' command will erase all files under /Alt-F that also exists in the base system (remove if it exists under /Alt-F, is not a conf file nor a file from a disk-installed package). So, if there is a corrupted status.cgi file under /Alt-F/usr/www/cgi-bin, the command will remove it, making the normal /usr/www/cgi-bin/status.cgi to be used. This is complex, and the faster way it to delete the Alt-F dir, reinstall on-disk packages and start fresh.

So I would do this if reverting factory settings didn't fix my problems?

What you mean, using 'fixup clean' or deleting the Alt-F folders?

'fixup clean' would clean all changes in the Alt-F folder, so I would try that first, even before clear settings. A the name implies, all "fixes" will also be removed (but not on-disk installed packages or updates)

Cheers

Jeremy