Fosstrak webadapter SQLite database

[djbur...@gmail.com]

Another query, sorry!

I've had a Fosstrak EPCIS demo system up and running on my laptop for a while now (as localhost). This uses Fosstrak 0.5.0 running on MySQL 5.6.17, MySQL Connector Java 5.1.35 & Tomcat 8.0.23, all on Java 1.8.0.60.

When I first set it up, I also got the webadapter working (see previous posts). This included setting the epcisurl (via http://localhost:8080/epcis-webadapter-0.1.0/rest/1/config/epcisurl ) to http://localhost:8080/epcis-repository-0.5.0/ (as the default was http://localhost:8080/epcis-repository-0.4.2/

At least, I thought I had. It all seemed to work, and I was getting proper results back.

I then ignored the webadapter for a while, and just used SOAPUI as my main query interface. Today, I went back to the webadapter, and have been getting errors trying to get into it. Trying to go to http://localhost:8080/epcis-webadapter-0.1.0 and then clicking on the EPCIS Web Adapter link opens the http://localhost:8080/epcis-webadapter-0.1.0/rest/1/ page which shows an error:

java.lang.NullPointerException
    org.fosstrak.webadapters.epcis.db.InternalDatabase.getConfigurationEntry(InternalDatabase.java:202)
at org.fosstrak.webadapters.epcis.db.InternalDatabase.initializeIfNotPresent(InternalDatabase.java:121)
at org.fosstrak.webadapters.epcis.db.InternalDatabase.<init>(InternalDatabase.java:73)
at org.fosstrak.webadapters.epcis.db.InternalDatabase.getInstance(InternalDatabase.java:84)
at org.fosstrak.webadapters.epcis.logic.AbstractEventFinderBusinessLogic.reload(AbstractEventFinderBusinessLogic.java:94)
[...]

Refreshing the page shows the normal EPCIS Browser page with links for Query Form, Browsable Event Finder, etc.
Clicking on the Query Form link goes to the normal query page, with a popup error "Error in HTTP GET: Cannot load Business Locations."  When I click OK to this error, I get another identical one, clicking OK on that I can then get to the query form properly.

I think these two errors are connected?

When I then enter some details for a query and click Submit, I get a page showing "The server sent HTTP status code 404: Not Found"

Additionally, if I go back to the EPCIS Browser page and click on Configuration, then "Configuration of the EPCIS URL", I get to the page http://localhost:8080/epcis-webadapter-0.1.0/rest/1/config/epcisurl which shows "The actual EPCIS URL: http://localhost:8080/epcis-repository-0.4.2" - which is clearly not what I originally set it up as!

If I try to change it to http://localhost:8080/epcis-repository-0.5.0 and click Update, I get a popup "Error in HTTP POST". In the log file this shows as:
java.lang.NullPointerException
at org.fosstrak.webadapters.epcis.db.InternalDatabase.getConfigurationEntry(InternalDatabase.java:202)
at org.fosstrak.webadapters.epcis.db.InternalDatabase.initializeIfNotPresent(InternalDatabase.java:121)
at org.fosstrak.webadapters.epcis.db.InternalDatabase.<init>(InternalDatabase.java:73)
at org.fosstrak.webadapters.epcis.db.InternalDatabase.getInstance(InternalDatabase.java:84)
at org.fosstrak.webadapters.epcis.config.Config.SET_EPCIS_REPOSITORY_URL(Config.java:225)
[...]

So I had a look at the underlying SQLite database ( <path>\apache-tomcat-8.0.23\webapps\epcis-webadapter-0.1.0\epcis-restadapter.db ) using the sqlite command line (and, as a further check, a couple of GUI frontends I found). All of them report the configuration table as:

key    value
lastInitialized    1433241728905
epcisurl    http://localhost:8080/epcis-repository-0.4.2

Note that 1433241728905 appears to translate to a date in June - which was when I got the webadapter working!

If I manually edit the epcisurl row to show http://localhost:8080/epcis-repository-0.5.0, the changes are saved, but restarting the webadapter does not change any of the error messages I am getting - even http://localhost:8080/epcis-webadapter-0.1.0/rest/1/config/epcisurl still shows the wrong (old 0.4.2) address!

Am I missing something really simple here??! I've tried looking into how to increase the logging level for the webadapter, but so far haven't been able (there aren't any .properties files in <path>\apache-tomcat-8.0.23\webapps\epcis-webadapter-0.1.0\WEB-INF\classes and adding a log4j.properties file doesn't seem to make any difference).

As usual, any help would be appreciated!

Thanks,
Dave

[djbur...@gmail.com]

I've just noticed - I also seem to have a file nullepcis-restadapter.db in the folder containing the .bat file I use to start Tomcat. The file has a last modification date of 30/10/2015, so it's been used recently. Browsing the database in the file shows that it has the correct epcisurl, but a lastInitialized value for around 1st June - so *before* the epcis-restadapter.db in <path>\apache-tomcat-8.0.23\webapps\epcis-webadapter-0.1.0\ !

So, more questions - what causes the file to be named "nullepcis-restadapter.db" rather than just "epcis-restadapter.db"? and why should it be created in the wrong place? and which one is the right one?!

Thanks...
Dave

[djbur...@gmail.com]

Aha - when all else fails, RTI:-)

From http://fosstrak.github.io/epcis/docs/webadapter-guide.html :

8<------------------------

"I get the following exception when deploying the EPCIS Webadapter"

java.lang.NullPointerException
        org.fosstrak.webadapters.epcis.db.InternalDatabase.getConfigurationEntry(InternalDatabase.java:202)
        org.fosstrak.webadapters.epcis.db.InternalDatabase.initializeIfNotPresent(InternalDatabase.java:121)
        org.fosstrak.webadapters.epcis.db.InternalDatabase.<init>(InternalDatabase.java:73)
        org.fosstrak.webadapters.epcis.db.InternalDatabase.getInstance(InternalDatabase.java:84)
...

This is a known bug due to the fact that the SQLite embedded database-driver the EPCIS Webadapter uses is based on JNI. Thus, in some containers (e.g., Tomcat) it can only be loaded once and is not released when undeploying the application. The only workaround we currently have is to restart the Tomcat instance. We are working on migrating the SQLite database to one that does not require the use of a JNI driver.

8<------------------------

Doesn't explain why it's just started happening though. Looking back in the logs, it seems that, sometimes, sqliteHome is correctly set to <path>\apache-tomcat-8.0.23\webapps\epcis-webadapter-0.1.0\ , but at other times it's set to null. But I can see that there's always an error "The web application [epcis-webadapter-0.1.0] registered the JDBC driver [org.sqlite.JDBC] but failed to unregister it when the web application was stopped. To prevent a memory leak, the JDBC Driver has been forcibly unregistered." whenever I close Tomcat.

Perhaps I need to step back to Tomcat 7 or even 6 after all?

D

On Wednesday, November 11, 2015 at 2:29:42 PM UTC, djbur...@gmail.com wrote:

[djbur...@gmail.com]

OK, I've tried with Tomcat 8028, 7065, 6044, and with Java 8u66, 8u60 and 7u55. I can't seem to get the webadapter working properly under any combination.

I've also tried following the hint about "some containers e.g. Tomcat" and had a go with Glassfish 4.1.1. Which also doesn't work - with that I get what seems to be a common problem trying to tell Glassfish that no, you really *don't* want to use the jdbc/__default, you actually want to use the MySQL Connector/J - only to have it ignore everything you tell it and try to connect to database port 1527 which isn't there, instead of MySQL's port 3306, which is...

Something somewhere is fundamentally broken!

D


On Wednesday, November 11, 2015 at 2:29:42 PM UTC, djbur...@gmail.com wrote:

[djbur...@gmail.com]

Brief update - I've now tried with Glassfish 3.1.2.2, which works a whole lot better than GF4.1.1. At least, I can install & run the epcis-repository war file, with a little bit of fiddling with the jdbc/EPCISDB settings in the GF admin console, and I can run SOAP queries against it.

I can also install the epcis-webadapter with no problem. However, I still get the same problem when trying to get into the Browsable Event Finder:

java.lang.NullPointerException
    at org.fosstrak.webadapters.epcis.db.InternalDatabase.getConfigurationEntry(InternalDatabase.java:202)
    at org.fosstrak.webadapters.epcis.db.InternalDatabase.initializeIfNotPresent(InternalDatabase.java:121)
    at org.fosstrak.webadapters.epcis.db.InternalDatabase.<init>(InternalDatabase.java:73)
    at org.fosstrak.webadapters.epcis.db.InternalDatabase.getInstance(InternalDatabase.java:84)

So whatever the problem is with using an SQLite database is not specific to Tomcat! One has to wonder why it uses an SQLite database rather than, say, an xml file for configuration...!

The epcisurl configuration seems to work (though the first time you click on the link you get an error), and the query form seems to work (though you get an error about loading business objects).

I think, at this point, unless I come up with any more ideas, I will have to draw a line and say that I just can't run certain parts of the webadapter for reasons unknown.

Hope this monologue is of use to someone else out there anyway...

D


On Wednesday, November 11, 2015 at 2:29:42 PM UTC, djbur...@gmail.com wrote:

[djbur...@gmail.com]

Not sure if anyone's following this or not. I've not found a more recent location for discussing the webadapter, so I'm still posting here:-)

I've obtained the latest code from github, and managed to work out how to use maven. I think. I've forced maven (via the pom.xml file) to use jdk1.6 (I think). I've also had to update it to allow for the non-existence of Codehaus (now pointing to http://repo.maven.apache.org/maven2/ ). I've also reverted the last 3 commits (from 2011) to get back to the code as on 27/12/2010 which is the date on the class files in the downloadable .war file at https://oss.sonatype.org/content/repositories/public/org/fosstrak/webadapters/epcis-webadapter/0.1.0/epcis-webadapter-0.1.0-bin-with-dependencies.zip .

However, I'm unable to generate identical .class files to what's in the publicly available .war file. Looks like some .jars are different in what maven's pulled off the various public repositories compared with what's in the downloadable .war from the Fosstrak site, despite being the same declared version. Below is a list of the variant .jars, and the dates of the class files in them:

activation-1.1.jar: public: 02/05/2006; webadapter: 12/09/2006 & 24/01/2007
jaxb-api-2.1.jar: public: 24/07/2007; webadapter: 23/01/2009
saaj-api-1.3.jar: public: 29/03/2006; webadapter: 14/12/2009
servlet-api-2.5.jar: public: 10/05/2006; webadapter: 18/10/2006

Perhaps with the exception of saaj-api (which may actually be version 1.3.1?), they look like they've been recompiled specifically for the webadapter project. But if they have, there's no longer any traceability to get the original source code, so any debugging of the webadapter is going to be rather "fraught".

It does of course explain why I'm having problems just adding extra logging and trying to reproduce my original issue...!

If anyone happens to be reading this that knows a bit more how maven & repositories work, maybe there's something obvious I'm doing wrong?

On Wednesday, November 11, 2015 at 2:29:42 PM UTC, djbur...@gmail.com wrote:

[djbur...@gmail.com]

OK, it appears that my main problem is that the EPCIS Webadapter is indeed fundamentally broken, and has a major design flaw, much more basic than the use of the sqlite database.

The main "home" page is at http://<yourhost>/epcis-webadapter/. The first of two links on this page is to "EPCIS Webadapter" ( http://localhost:8080/epcis-webadapter/rest/1/ ).
Clicking on this link apparently causes the webadapter to query the ***entire*** EPCIS repository for all events, (in order, object, aggregate, quantity, transaction) from 1970 to a month ahead of today's date.

Obviously, when I first played with it I had very little data, and didn't notice the performance problem. Now I've got 1.8M object events and 0.5M transaction events, I've got things timing out all over with associated memory leaks etc.

No idea why the original author thought this was necessary, but if anyone else wants to be able to access EPCIS events through a browser in the way the webadapter project originally intended, I suggest you don't start from here:-)

D

On Wednesday, November 11, 2015 at 2:29:42 PM UTC, djbur...@gmail.com wrote: