miniserver random reboots

[bartm]

Hi all,

Starting from today, my miniserver reboots at random times. (sometimes after running 1 hour, sometimes after 5mins) This is actually the first time the miniserver is letting me down after running stable for +1 year.

Very annoying ofcourse and I have no clue on how to debug this. I had the monitor running but didn't notice anything weird in it - although several 'missing lines' entries were present. Any idea on how/where to look further? FYI, my loxone installation is pretty simple, it consists of a miniserver, knx pushbuttons and  knx relays controlling lighting only. Plans are to extend in the future. That is, if this issue is solved quickly... :)

 

[Santeri / Finlandia Automation Ltd.]

One of our customers MS is having the same problem, started last night. Restart fixed it for 12 hours, but now it started again. Trying to figure something out just now. Chech your logs and see if there is something weird going on. Put this in your browser:

yourmsaddress/dev/fsget/log/def.log

 

We found that this lineup is out of the ordinary:

2016-11-02 22:43:10.000;New NTP time from 131.107.13.100 ( 0ms)

2016-11-02 22:43:10.931;weather error authentication exception: user is not activ

2016-11-02 20:43:33.423;Read network config: IP: 192.168.10.50, Mask: 255.255.255.0, Gateway: 192.168.10.1, NTP: 0.pool.ntp.org

2016-11-02 22:43:47.581;PRG Reboot 8.0.7.19

2016-11-02 22:43:48.637;Save network config: IP: 192.168.10.50, Mask: 255.255.255.0, Gateway: 192.168.10.1, NTP: 0.pool.ntp.org

2016-11-02 22:43:51.180;PRG start program

2016-11-02 22:43:51.418;RestoreRemanenceState /sys/rem/rem92.xml and /sys/rem/rem192.xml

2016-11-02 22:43:58.502;Unable to send start entry to crash log server

2016-11-02 22:43:58.502;Firewall enabled UseLocally: 0 max Syn/s: 20, statistics Blocked SYN: 0, Blocked Ports: 0, Bans: 0

2016-11-02 22:43:59.630;weather error authentication exception: user is not activ

 

This loop keeps on going until it revertrs back to version 7.4 of the config=all dmx lights turn on 1-wire sensor give false readings.

Santeri

Finlandia Automation Ltd.

[bartm]

Hi Santeri,

Thanks for pointing me to that log file. The number of entries in there is indeed large starting from 01/11/2016. They look similar to what you posted:

2016-11-01 23:21:27.000;New NTP time from 131.107.13.100 ( 0ms)
2016-11-02 00:00:00.000;Firewall enabled UseLocally: 0 max Syn/s: 20, statistics Blocked SYN: 0, Blocked Ports: 0, Bans: 0
2016-11-01 23:25:06.584;Read network config: IP: 192.168.0.21, Mask: 255.255.255.0, Gateway: 192.168.0.1, NTP: Undefined
2016-11-02 00:25:20.999;PRG Reboot 8.0.7.19
2016-11-02 00:25:21.890;Save network config: IP: 192.168.0.21, Mask: 255.255.255.0, Gateway: 192.168.0.1, NTP: Undefined
2016-11-02 00:25:23.568;PRG start program
2016-11-02 00:25:23.772;RestoreRemanenceState /sys/rem/rem67.xml and /sys/rem/rem167.xml
2016-11-02 00:25:29.499;Unable to send start entry to crash log server
2016-11-02 00:25:29.499;Firewall enabled UseLocally: 0 max Syn/s: 20, statistics Blocked SYN: 0, Blocked Ports: 0, Bans: 0
2016-11-02 00:25:34.297;weather error authentication exception: user is not activ
2016-11-01 23:25:57.602;Read network config: IP: 192.168.0.21, Mask: 255.255.255.0, Gateway: 192.168.0.1, NTP: Undefined
2016-11-02 00:26:12.268;PRG Reboot 8.0.7.19
2016-11-02 00:26:13.308;Save network config: IP: 192.168.0.21, Mask: 255.255.255.0, Gateway: 192.168.0.1, NTP: Undefined
2016-11-02 00:26:14.000;New NTP time from 131.107.13.100 ( 0ms)
2016-11-02 00:26:14.999;PRG start program
2016-11-02 00:26:15.137;RestoreRemanenceState /sys/rem/rem67.xml and /sys/rem/rem167.xml
2016-11-02 00:26:22.466;Unable to send start entry to crash log server
2016-11-02 00:26:22.466;Firewall enabled UseLocally: 0 max Syn/s: 20, statistics Blocked SYN: 0, Blocked Ports: 0, Bans: 0
2016-11-01 23:27:25.094;Read network config: IP: 192.168.0.21, Mask: 255.255.255.0, Gateway: 192.168.0.1, NTP: Undefined
2016-11-02 00:27:39.343;PRG Reboot 8.0.7.19

[Santeri / Finlandia Automation Ltd.]

Yea exactly the same. I´ll add this info to the ticket I´ve opened with Loxone and I´ll get back to you when I´ve heard something from Loxone´s support.

Santeri

[seb303]

I had exactly the same thing.  I happened to have the Loxone Config open at the time so I turned on the Monitor, and I saw lots of connections from the internet from random IPs.

As soon as I shut my firewall, the reboots stopped happened.

So seems that some network attack was going on and this was causing the Miniserver to crash and reboot.  I don't think the attack was intended for the Miniserver specifically, as I saw probes on other common ports too in my firewall logs.

Incidentally, the various IPs belonged to betting company William Hill.  Probably a load of hacked servers being used for further hacking attempts.

I saved the Monitor file, but haven't got round to opening a ticket with Loxone yet.

Seb

[Santeri / Finlandia Automation Ltd.]

Got a response from Loxone:

"We got some detailed information about a possible reason for the reboots yesterday evening. It could be a syn-flood attack on your local network on Port 80. Due to this the Miniserver crashed."

We used a different port than 80, but the port forwarding was also done via port 80, hence the crashes. This was the only customer with port 80 redirected, no other customers had this problem.

Santeri

Finlandia Automation Ltd. 

[seb303]

I got a similar response from Loxone:

Thanks for sending that though. Please try another port than port 80. It's not that you've got a port open that's causing the problem merely that port 80 is a common port.

We have seen recently there have been many Denial Of Service Attacks to IP Addresses. IP Addresses are attacked from anonymous Computers (could be Virus infected systems), these attacks are mostly sent to port 80. Miniservers who are reachable through an Internet address on Port 80 would likely get attacked. The Miniserver has a stability feature, when it stops working, it makes a reboot to become available again. When the Miniserver is attacked after this reboot again it makes a reboot again. Please use another port then 80 and 443 (perhaps a port bigger than 50000)

While it seems sensible to use a port other than 80, I'm not convinced this is a complete fix.  This was my response to Loxone:

Thanks for your reply. I understand your comments about using a different port to mitigate against generic attacks on web-servers, although of course the unusual port number could be easily discovered by an automated scan.

It is still concerning that it is possible, and seemingly trivial, for someone to crash my Miniserver remotely.

What I think you are saying is that the Miniservers have some kind of watchdog timer and if part of the system becomes unresponsive (due to a flood of connection attempts for example), then the whole system is rebooted.

Will this be fixed in a future firmware release, so the Miniserver uses a more appropriate countermeasure rather than simply rebooting? Seems like that would be a good idea for a system that is designed to be connected to the internet.

[seb303]

Here are Loxone's 2 responses.  I'm pleased to see that they will be addressing this issue in the next firmware release.

I would not say it is trivial to crash the Miniserver. The reboot is intended to close any connections that are being held open. The Miniserver already has an internal inbuilt firewall and will block denial of service attacks automatically by blocking IP addresses with failed login attempts, but a distributed denial of service attack is extremely hard to shut out since it emulates login requests from different sources, which look totally genuine. Unfortunately these are becoming more common and like a few weeks back actually took down major parts of the internet, see:

https://www.theguardian.com/technology/2016/oct/21...

We have had a few reports of this since the 1st November, but only systems using port 80 or 443 are affected, hence our recommendation.

We will explore other countermeasures in future releases, but as I already said it is actually very complicated to effectively shut down a DDOS attack.

....

There will be further firewall based countermeasures to prevent against this in the next release. This type of attack is a recent occurence in this industry. Leaving a Miniserver on port 80 (default port for http) is something we've been advising against for some time now. An open port attack by simplistic program such as I mentioned before is a very non targeted attack, simply spamming. If you were to change a port to something completely random as suggested this means that anything that could cause this would need to be a specific targeted attack to you. This is not likely.

If this is something that is a concern for you then it is possible to set up a VPN instead to get around this.

[bartm]

I had a similar response from Loxone. Changed port and I'm OK for now, although I agree that it should survive a targeted attack as well. At least I know now that i can plug out the network cable if the lights start blinking again...

[smartbusinesstools.be]

This is something that would apply to all 'servers' on your LAN, e.g. your door intercom, cameras, NAS, PBX, your router's UI, and should be handled for all by your network's firewall/router.

My soho/smb router has this DoS defense mechanism.

When needed, I can also block the WAN ports and connect via SSL VPN.

[seb303]

My router also has DOS protection, albeit without much configuration like your router.  But this didn't seem to help with the Miniserver rebooting.  Going by the firewall logs, there weren't a great many connections, so perhaps not enough to trigger the DOS rules.  IMO, the Miniserver should have been able to handle this on its own - and hopefully it will be able to after the next firmware update.

[Glenn A]

Hi all,

Is anyone still having this problem? My miniserver is randomly rebooting. (sometimes a few times a day) I have checked the firewall and changed the port to above 50000, but it's still happening. I have also changed the IP address of my miniserver in case that would help.

I have checked the log as stated here before, but this is not really helping me further. In loxone config I also don't see strange things.

Does anybody know where I need to be looking further into?

Thnx!

[Rob_in]

Not sure if this helps or is related but...

We recently change our ISP router and they gave us a separate TV box (it's how things work here in France). Cool, so I put the TV box on our LAN, it's just a custom Android device so should be no problem. Sometime shortly afterwards I noticed the Miniserver was going offline for short periods. Looking at our switch could see that the connectivity light on the Miniserver's port was going off for short periods. Very odd. Changed the network cable and port but had no effect.

The ISP's TV box being the only change, I put it on an isolated VLAN and... voila! Issue gone.

Everything else on our LAN was working just fine, but whatever traffic this TV box is spewing out, the Miniserver certainly didn't like it! It didn't cause a full reboot but seemed to screw up the Miniserver's IP stack/cause the stack to restart/whatever.

Ideally I should do some better LAN separation anyhow, but don't have a fancy enough switch for that. Next time will get a managed version for sure. Doh!

Robin

[Glenn A]

Hello Robin,

Thank you for your answer. The TV box is indeed also on my LAN. I did not immediately find a way to create a VLAN on the ISP router. This is a bbox3 router from Proximus here in Belgium and they don't give you all freedom I guess.

As a quick fix i have put a managed switch (that I had lying around here) in front of the miniserver. I hope this solves the problem for now.  (in the logs I saw again lot's of reboots today, but I did not notice it, as I was at work)

Anyhow I will be switching towards a Ubiquitti network here at home. The ISP router (that I cannot fully control) is controlling my whole LAN network which has already given me headaches.

Thanks for the help!

Best regards,

Glenn

[Rob_in]

On Monday, 21 November 2022 at 20:21:51 UTC+1 Glenn A wrote:

As a quick fix i have put a managed switch (that I had lying around here) in front of the miniserver.

Looking at your diagram, you have your managed/dumb switches the opposite way round to me.

I have the following:

ISP router -> managed switch/firewall (OpenWRT on a TP-Link box) -> 3 x VLANs (1 for ISP's TV box, 1 for traffic tied to an upstream VPN, 1 for everything else)

*Noting that our Miniserver is currently within 'everything else' which needs fixing but don't have enough VLAN ports right now to do things properly (also have some IP cameras to deal with).

 

The ISP router (that I cannot fully control) is controlling my whole LAN network which has already given me headaches.

Yeah, I quickly discovered one should not trust gear supplied by an ISP and put that on the public side of any firewall. See my recent mistake forgetting that and letting their TV box loose inside our LAN!

Robin