Sitewhere recovery

[Ajay Nagarkar]

Hello,

I have a sitewhere instance running on AWS. It was running fine for some time and our device was sending data properly. Today we saw that we were not able to login to sitewhere. After looking at the log files here is what I saw

2017-11-24 09:28:47,575 INFO  [ResponseTimerFilter]  Call for GET to 'http://XX.XXX.XXX.XXX:8080/sitewhere/api/assignments/d51785d4-9ccf-4252-a6a8-01cb6a259c8b/measurements/series' returned in 14 ms.

2017-11-25 15:53:40,028 INFO  [Http11NioProcessor]  Error parsing HTTP request header

 Note: further occurrences of HTTP header parsing errors will be logged at DEBUG level.

java.lang.IllegalArgumentException: Invalid character (CR or LF) found in method name

        at org.apache.coyote.http11.AbstractNioInputBuffer.parseRequestLine(AbstractNioInputBuffer.java:228)

        at org.apache.coyote.http11.AbstractHttp11Processor.process(AbstractHttp11Processor.java:1010)

        at org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:674)

        at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1500)

        at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.run(NioEndpoint.java:1456)

        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)

        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)

        at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)

        at java.lang.Thread.run(Thread.java:748)

2017-11-28 06:05:28,359 INFO  [Http11NioProcessor]  Error parsing HTTP request header

 Note: further occurrences of HTTP header parsing errors will be logged at DEBUG level.

java.lang.IllegalArgumentException: Invalid character (CR or LF) found in method name

        at org.apache.coyote.http11.AbstractNioInputBuffer.parseRequestLine(AbstractNioInputBuffer.java:228)

        at org.apache.coyote.http11.AbstractHttp11Processor.process(AbstractHttp11Processor.java:1010)

        at org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:674)

        at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1500)

        at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.run(NioEndpoint.java:1456)

        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)

        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)

        at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)

        at java.lang.Thread.run(Thread.java:748)

I restarted sitewhere and this resulted in the following logs

2017-11-29 05:45:44,812 INFO  [SiteWhereWebApplication]  Starting SiteWhereWebApplication on ip-XXX-XX-XX-XXX.us-west-2.compute.internal with PID 18811 (started by root in /opt/sitewhere/bin)

2017-11-29 05:45:44,820 INFO  [SiteWhereWebApplication]  No active profile set, falling back to default profiles: default

Now I had some data that got lost in the process. Is there a way to restore that data ? 

Any help would be appreciated. 

Ajay 

[Derek Adams]

Hi Ajay,

Which version of SiteWhere are you using? Also, are there any other exceptions in the log file? The exception above should not cause SiteWhere to become unresponsive. 

Regarding restoring missing data, is the data coming in via the REST services or via a transport such as MQTT? Data coming in via REST is stored immediately in the database, but some data coming in via event sources can be lost if not already processed when an instance runs out of memory or crashes. We have solved that problem in SiteWhere 2.0 by adding Apache Kafka into the mix to assure reliability for all message processing. We will be releasing an early access version of SiteWhere 2.0 in the next week, but the GA will not be available until early 2018. In the meantime, if the data did not reach the database, there is no way to recover it other than re-sending it from the device.

Thanks,

Derek