No ipkg instalation found every reboot. DNS-323 rev. B1, Alt-F-0.1RC4.1, Alt-F folder found

[Jan Pisa]

Hi Joao, at first I would like to apologize for repetitive question. I read several threads about this problem, but I don't understand them well. My problem started after I applied all available updates to Alt-F packages.Things were getting worse and worse, so I decided to do complete Alt-F plugins reinstall. But now I am in situation, where I "loose packages" with every reboot. Alt-F plugins page say "No ipkg instalation found, install ipkg in:", when I install Alt-F packages again to same place (flashdisk), some of them are doubled in list (older and newer version). So I uninstall them again, do clean installation, but on next reboot the same problem.

hot_aux log contains line "hot_aux: Alt-F directory found in flashdisk", don't know where to find other needed logs :-/

Tried to change flashdisk for another one, tried to install Alt-F packages to "md0"...nothing helped.

Everything worked before that damn packages update :-(

Thanks for "kicking" the right way...

jimmy

[Jan Pisa]

Finally solved by complete cleanup (formatted flashdisk,cleared settings) and complete re-set of settings. Magically works now, don't know why it didn't worked before...

edit: HERE WE GO AGAIN!! After some power failure (UPS batteries problem) the same behavior occurs. Cannot reboot box from SSH via standard reboot (must use reboot -F), after every reboot no ALT-F installation found... Tried to reload old settings...no help. I can reimage mu USB flashdisk (I did backup of working flashdisk). :(

edit2: problem caused by user script, looking for solution

Dne úterý 15. listopadu 2016 21:59:37 UTC+1 Jan Pisa napsal(a):

[Konrád Lőrinczi]

Hi,

I have the same problem, but I would not like to clear the settings & reinstall.

Best regards,
Konrad Lorinczi

On Nov 17, 2016 13:56, "Jan Pisa" <jan....@gmail.com> wrote:

Finally solved by complete cleanup (formatted flashdisk,cleared settings) and complete re-set of settings. Magically works now, don't know why it didn't worked before...

Dne úterý 15. listopadu 2016 21:59:37 UTC+1 Jan Pisa napsal(a):

Hi Joao, at first I would like to apologize for repetitive question. I read several threads about this problem, but I don't understand them well. My problem started after I applied all available updates to Alt-F packages.Things were getting worse and worse, so I decided to do complete Alt-F plugins reinstall. But now I am in situation, where I "loose packages" with every reboot. Alt-F plugins page say "No ipkg instalation found, install ipkg in:", when I install Alt-F packages again to same place (flashdisk), some of them are doubled in list (older and newer version). So I uninstall them again, do clean installation, but on next reboot the same problem.

hot_aux log contains line "hot_aux: Alt-F directory found in flashdisk", don't know where to find other needed logs :-/

Tried to change flashdisk for another one, tried to install Alt-F packages to "md0"...nothing helped.

Everything worked before that damn packages update :-(

Thanks for "kicking" the right way...

jimmy

--
You received this message because you are subscribed to the Google Groups "Alt-F" group.
To unsubscribe from this group and stop receiving emails from it, send an email to alt-f+unsubscribe@googlegroups.com.
Visit this group at https://groups.google.com/group/alt-f.
For more options, visit https://groups.google.com/d/optout.

[João Cardoso]

On Thursday, 17 November 2016 13:39:25 UTC, Konrád Lőrinczi wrote:

Hi,

I have the same problem, but I would not like to clear the settings & reinstall.

That's difficult to diagnose.

If you are in RC4.1 and the hot_aux log says "Alt-F directory found in xxx" but the Alt-F packages don't appear installed, use and post the following commands before re-installing the packages:

ls -l / /mnt/xxx/Alt-F /Alt-F # with xxx replaced with
aufs.sh -l
aufs.sh -m

 

Best regards,
Konrad Lorinczi

[Konrád Lőrinczi]

Hi,

I got the following results:

root@dns320]# ls -l /mnt/sda2/Alt-F/ /Alt-F/
/Alt-F/:
total 20
-rw-rw-rw-    1 root     root           112 Oct 29 23:48 README.txt
drwxr-xr-x   14 root     root          4096 Oct 29 15:32 etc
drwxr-xr-x    2 root     root          4096 Aug  1 18:11 opt
drwxr-xr-x   10 root     root          4096 Feb 21  2012 usr
drwxr-xr-x    4 root     root          4096 Mar 14  2016 var

/mnt/sda2/Alt-F/:
total 20
-rw-rw-rw-    1 root     root           112 Oct 29 23:48 README.txt
drwxr-xr-x   14 root     root          4096 Oct 29 15:32 etc
drwxr-xr-x    2 root     root          4096 Aug  1 18:11 opt
drwxr-xr-x   10 root     root          4096 Feb 21  2012 usr
drwxr-xr-x    4 root     root          4096 Mar 14  2016 var

[root@dns320]# aufs.sh -l
aufs on / type aufs (rw,relatime,si=a99e9ad8)
/rootmnt/rw=rw
/rootmnt/ro=rr
/rootmnt/sqimage=rr

[root@dns320]# aufs.sh -m

Still working after a few minutes on the last command.

Is this normal?

Thanks,
Konrad Lorinczi

[Konrád Lőrinczi]

What
aufs.sh -m
is doing?

Can I stop it with CTRL + C.

Thanks,
Konrad Lorinczi

[João Cardoso]

You missed the 'ls -l /' before the 'aufs.sh -m'

On Thursday, 17 November 2016 21:27:43 UTC, Konrád Lőrinczi wrote:

What
aufs.sh -m
is doing?

Not sure, probably trying to get a lock.

Does /tmp/.aufs-lock exists? What does 'ls -l /tmp/.aufs-lock' outputs? If/when you reboot, use that command before the 'aufs.sh -m' one.

Can I stop it with CTRL + C.

Try first, from another console, the commands 'ps; rmdir /tmp/.aufs-lock' to see if aufs.sh continues and displays an error. Please post those commands output.

Have you any "unusual" service starting? Have you a working internet connection active?

The underwoods:

At boot time, before boot enabled fw services are started, the file /tmp/.aufs-lock is created, and it is removed when all services are started.

If the Alt-F folder is discovered meanwhile, and to prevent simultaneous start of some on-disk Alt-F services, it waits for the  /tmp/.aufs-lock file to disappear.

That is *probably* the reason why 'aufs.sh -m' is never returning. But if that is the reason, than some fw services are not completing, and that needs to be investigated.

Thanks,
Konrad Lorinczi

[Konrád Lőrinczi]

Here it is:
[root@dns320]# ps; rmdir /tmp/.aufs-lock
PID USER COMMAND
1 root init
2 root [kthreadd]
3 root [ksoftirqd/0]
5 root [kworker/0:0H]
7 root [khelper]
8 root [writeback]
9 root [bioset]
10 root [kblockd]
11 root [ata_sff]
12 root [khubd]
13 root [md]
14 root [rpciod]
16 root [kswapd0]
17 root [fsnotify_mark]
18 root [nfsiod]
19 root [crypto]
32 root [scsi_eh_0]
33 root [scsi_eh_1]
34 root [kworker/u2:1]
36 root [kworker/u2:3]
41 root [deferwq]
55 root {rcS} /bin/sh /etc/init.d/rcS
101 root [loop0]
265 root [kworker/0:1H]
303 root {hot_aux.sh} /bin/sh /usr/sbin/hot_aux.sh fsck -p d
365 root [jbd2/sda2-8]
366 root [ext4-dio-unwrit]
432 root {aufs.sh} /bin/sh /usr/sbin/aufs.sh -m
502 root udhcpc -R -b -p /var/run/udhcpc.eth0.pid -i eth0 -r
573 root {rcall} /bin/sh /sbin/rcall start
577 root syslogd -C -m 0 -D
580 root klogd
598 root sysctrl
612 root crond
621 root smartd -i 1800
675 root inetd
723 root nmbd -D
728 root smbd -D
762 root {S98ffp} /bin/sh /etc/init.d/S98ffp start
763 root {rc} /ffp/bin/sh /ffp/etc/rc start
764 root smbd -D
768 root {sshd.sh} /ffp/bin/sh /ffp/start/sshd.sh start
772 root /ffp/bin/ssh-keygen -t ecdsa -f /ffp/etc/ssh/ssh_ho
1849 root dropbear -i
1854 root -sh
2295 root {aufs.sh} /bin/sh /usr/sbin/aufs.sh -m
2450 root [kworker/0:1]
2952 root dropbear -i
2973 root -sh
3011 root usleep 500000
3012 root usleep 500000
3013 root ps
3188 root [kworker/0:0]
[root@dns320]#

After removing the lock, the aufs.sh -m finished.
It seems we are in the finish of fixing this problem.

Ok, what's next?
Restarting the box?
Installing ALT-F plugins?

Thanks for your help!
Konrad

--

[Konrád Lőrinczi]

After rebooting the ALT-F packages are reported as not installed :-(

[João Cardoso]

On Friday, 18 November 2016 20:33:02 UTC, Konrád Lőrinczi wrote:

Here it is:

please see my comments to the 'ps' command output. It explains it all.

There is a filesystem check here (fsck)

 

365 root [jbd2/sda2-8]
366 root [ext4-dio-unwrit]
432 root {aufs.sh} /bin/sh /usr/sbin/aufs.sh -m
502 root udhcpc -R -b -p /var/run/udhcpc.eth0.pid -i eth0 -r
573 root {rcall} /bin/sh /sbin/rcall start

This is where all boot enabled  services are started (rcall). This is where the aufs lock is created (and removed when all services have been started)

577 root syslogd -C -m 0 -D
580 root klogd
598 root sysctrl
612 root crond
621 root smartd -i 1800
675 root inetd
723 root nmbd -D
728 root smbd -D
762 root {S98ffp} /bin/sh /etc/init.d/S98ffp start

This is one of the boot enabled services, ffp

 

763 root {rc} /ffp/bin/sh /ffp/etc/rc start
764 root smbd -D
768 root {sshd.sh} /ffp/bin/sh /ffp/start/sshd.sh start

and here a service that ffp starts. sshd, a ssh server.

Why have you enabled it? Alt-F already has a light ssh server running (dropbear), and, if needed, openssh.

 

772 root /ffp/bin/ssh-keygen -t ecdsa -f /ffp/etc/ssh/ssh_ho

here is  ffp-sshd generating its cryptographic keys. It takes a while and can block the system if not enough entropy is available.

This is where the system stops. the key generation doesn't finish, the 'ffp start' never returns, the rcall never returns, the aufs lock is not removed.

And it's your fault, you have enabled a conflicting (and unnecessary) ffp service.

1849 root dropbear -i
1854 root -sh
2295 root {aufs.sh} /bin/sh /usr/sbin/aufs.sh -m

waiting for the aufs lock removal that will never happens.

 

2450 root [kworker/0:1]
2952 root dropbear -i

This is Alt-F sshd server (already) running. Even if ffp sshd tries to start it will nmot suceed, as its port is already in use.

 

2973 root -sh
3011 root usleep 500000
3012 root usleep 500000

These "sleep" is the loop where aufs.sh waits for the lock to be removed

 

3013 root ps
3188 root [kworker/0:0]
[root@dns320]#

After removing the lock, the aufs.sh -m finished. 

It seems we are in the finish of fixing this problem.

Yes, don't start ffp sshd.

 

Ok, what's next?
Restarting the box?
Installing ALT-F plugins?

Thanks for your help!
Konrad

[Konrád Lőrinczi]

I don't really find sshd package.

I have the following ssh related alt-f packages installed:
dropbear2015.71
dropbear2014.63
openssh-sftp6.4p1
openssh-sftp7.1p2

I wanted SFTP, this is why openssh-sftp is installed.

Not installed:
openssh6.4p1
openssh7.1p2
Maybe it is a problem, that alt-f openssh is not installed.

ffp:
openssh-5.9p1-oarm-2

Attaching a screenshot.

Should I uninstall ffp openssh-5.9p1-oarm-2?

Thanks,
Konrad Lorinczi

[Konrád Lőrinczi]

Success!

Finally, I uninstalled the ffp
openssh-5.9p1-oarm-2 package,
and now the "No ipkg instalation found" error gone.

João Cardoso, thanks very much for your help and your patience. Thanks to you the problem solved successfully.

Just for future reference, the
"No ipkg instalation found" error
is caused by the conflict between two packages:
ALT-F / dropbear
and
FFP / openssh-5.9p1-oarm-2

The FFP / openssh-5.9p1-oarm-2 package is unnecessary, as ssh server feature is provided by default with ALT-F / dropbear.

SOLUTION: remove FFP / openssh-5.9p1-oarm-2 package.

SIDE EFFECT: the temperature / fan not reading problem also solved.

Have a nice day!

[Jan Pisa]

Just for future reference, the
"No ipkg instalation found" error
is caused by the conflict between two packages:
ALT-F / dropbear
and
FFP / openssh-5.9p1-oarm-2

It's not always true, I had this problem but never used  FFP plugins, just Alt-F. But it's great you succeeded and found solution (of course due to great help of Joao)

Have a nice day

jimmy

[Konrád Lőrinczi]

Yes, likely other conflicts can cause the same error.
The sshd daemon asked an interactive question, which of course remained unresponded, so the app did not exit and the /tmp/.aufs-lock was not removed.
I suppose this was the reason.

Any similar apps, which asks a question, will likely result, that lock remains open.

Of course all credits and thanks should go to João Cardoso, because I could not solve the problem without his instructions!

Thanks João!

--

[João Cardoso]

On Saturday, 19 November 2016 14:21:30 UTC, Konrád Lőrinczi wrote:

Yes, likely other conflicts can cause the same error.
The sshd daemon asked an interactive question, which of course remained unresponded, so the app did not exit and the /tmp/.aufs-lock was not removed.
I suppose this was the reason.

Yes, it looks like it was.

I have now added a 2 minutes timeout waiting for the lock to be released, displaying a message in the Status page (and turning on the orange leds). The exact reason why the lock has timed out isn't know, and when using ffp or Entware packages the user must be aware of that.

When ffp is installed using the Alt-F webUI, all its services are disabled, i.e., all /ffp/start/ scripts are execution-disabled, to prevent issues like this one. So you must have re-enabled ffp sshd either using the command line or by using Services->User, ffp, Configure.

 

Any similar apps, which asks a question, will likely result, that lock remains open.

Of course all credits and thanks should go to João Cardoso, because I could not solve the problem without his instructions!

Thanks João!

You are welcome :-)

Luckily you persevered, hunted the issue origin, had patience, and posted the conclusion, and that is *very* important to do, as it allows it to be fixed at its origin.

[Konrád Lőrinczi]

1) the fix will be available as a fix for RC4 or will be available in RC6?
Not for me, but for those users, who may meet this problem in the future.

2) could be possible to implement a warning for conflicting packages?
If dropbear is installed, then warn user for conflict, when he wants to install FFP  openssh package?
"The openssh package you want to install, conflicts with currently installed dropbear package!"

Opinion?

[João Cardoso]

On Saturday, 19 November 2016 20:17:58 UTC, Konrád Lőrinczi wrote:

1) the fix

There is no fix, the timeout in getting the lock generates a log warning that will be displayed on the Status page (and orange leds). The exact cause of the timeout is not know.

 

will be available as a fix for RC4 or will be available in RC6?

If it were available it would be for RC5, not for RC4 or RC4.1

 

Not for me, but for those users, who may meet this problem in the future.

2) could be possible to implement a warning for conflicting packages?

That is not possible. With ffp and Entware-ng there are now thousands of user-installable packages. Most of them have to be manually configured first, so it's supposed that the user knows what he is doing and able to undo it if something runs wrong. There are several recovery options available, just in case.

Even the existence of a user-written boot script has its dangers, but some people has to customize the box for his specific needs or preferences. That's the reason for the warning on it:

# Script to execute as the root user at boot time.

# You can loose your data or make the system inaccessible

# if using the wrong commands. You have been warned!

 

If dropbear is installed, then warn user for conflict, when he wants to install FFP  openssh package?

dropbear is always installed and active, it is in the base fw.

Alt-F openssh package can also be installed and activated, if needed, and has a webUI to avoid possible port conflicts with dropbear. But if the user decides to edit the configuration files itself, because he has special needs, there is possibly no way to avoid or predict problems.

As a rule of thumb, if users edit configuration files themselves and the system stops working as expected, that's their responsibility.

"The openssh package you want to install, conflicts with currently installed dropbear package!"

Opinion?