A couple of wiki pages

[Steve Fisher]

Hi,

I have added a couple of wiki pages to icat.server and ids.server on github. These pages are for developers and tell people how to develop and test. They are initially for Rolf so that he can test his new locking.

Steve

[Rolf Krahl]

Steve,

I just tried to follow your instructions in order to run the
ids.server test suite. I get the following error:

------------------------- start test output --------------------------

Running org.icatproject.ids.TestICATGetter
Tests run: 4, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 0.181 sec <<< FAILURE! - in org.icatproject.ids.TestICATGetter
testGetService(org.icatproject.ids.TestICATGetter) Time elapsed: 0.18 sec <<< ERROR!
org.icatproject.IcatException_Exception: Unable to connect to: https://smfisher:8181
at org.icatproject.ids.ICATGetter.throwSessionException(ICATGetter.java:106)
at org.icatproject.ids.ICATGetter.getService(ICATGetter.java:85)
at org.icatproject.ids.TestICATGetter.testGetService(TestICATGetter.java:27)

-------------------------- end test output ---------------------------


Indeed, "https://smfisher:8181" doesn't look much like a valid URL, so
failure to connect isn't surprising. Contrary to what the
instructions state, the tests does not seem to take the URL from
~/.m2/settings.xml. In the corresponding source code, I read:

------- src/test/java/org/icatproject/ids/TestICATGetter.java: -------

@Test
public void testGetService() throws Exception {
ICAT icat = ICATGetter.getService("https://smfisher:8181");
System.out.println(icat.getApiVersion());
}

----------------------------------------------------------------------

I tried the sources at the v1.7.0 tag as well as the HEAD of the
master branch. When I edit this code and set the URL to my test
server, I get another error:

------------------------- start test output --------------------------

Running org.icatproject.ids.TestICATGetter
Tests run: 4, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 0.219 sec <<< FAILURE! - in org.icatproject.ids.TestICATGetter
testGetService(org.icatproject.ids.TestICATGetter) Time elapsed: 0.216 sec <<< ERROR!
org.icatproject.IcatException_Exception: sun.security.validator.ValidatorException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target
at org.icatproject.ids.ICATGetter.throwSessionException(ICATGetter.java:106)
at org.icatproject.ids.ICATGetter.getService(ICATGetter.java:62)
at org.icatproject.ids.TestICATGetter.testGetService(TestICATGetter.java:27)

-------------------------- end test output ---------------------------

So it seems that the javax.net.ssl.trustStore setting in
~/.m2/settings.xml is also not taken into account.

Furthermore, if I look into the properties files in
src/test/resources, there are plenty of settings that certainly will
not work in my environment, such as:

icat.url = https://smfisher:8181
cache.dir = /home/fisher/ids/cache/
ids.url=https://smfisher:8181
glassfish=/home/fisher/pf/glassfish4

I have no idea whether these properties files are actually used in any
test though.

So, at the moment, it seems that I'm not able to get the tests running
here.

Best,
Rolf

--
Rolf Krahl <rolf....@helmholtz-berlin.de>
Helmholtz-Zentrum Berlin für Materialien und Energie (HZB)
Albert-Einstein-Str. 15, 12489 Berlin
Tel.: +49 30 8062 12122

[Steve Fisher]

I have checked in revised code which may go out as 1.7.1. It now uses the "new" configuration style and is free of hardcoded smfishers. I have updated the wiki page to match. Let me know how you get on.

Steve

Rolf Krahl <rolf.krahl@helmholtz-berlin.de>

[Rolf Krahl]

Steve,

Am Mittwoch, 5. April 2017, 07:34:43 schrieb Steve Fisher:
>
> I have checked in revised code which may go out as 1.7.1. It now uses the
> "new" configuration style and is free of hardcoded smfishers. I have
> updated the wiki page to match. Let me know how you get on.

I gave it another try. Here is what I did:

* I needed to install icat.utils from the sources (I used the current
HEAD of master from the icatproject github repository) first, as
ids.server now depends on 4.16.1-SNAPSHOT which is not in the
icatproject maven repository.
* I install a test GlassFish on my local machine as indicated in the
Wiki pages.
* I made sure to have GlassFish's bin directory with asadmin in the
PATH.
* I did edit ~/.m2/settings.xml and
src/test/resources/test.properties as indicated in the Wiki pages.
* I ran the tests from the command line with "mvn test".

The ids.server tests have been run successfully:

| Results :
|
| Tests run: 14, Failures: 0, Errors: 0, Skipped: 0

But the test did not seem to have even tried to deploy ids.server on
my test GlassFish. In fact, I don't see any trace of any attempt to
interact with the test GlassFish.

Is this what I should have expected?

Best,
Rolf

--
Rolf Krahl <rolf....@helmholtz-berlin.de>

[Rolf Krahl]

Am Montag, 10. April 2017, 14:15:00 schrieb Rolf Krahl:
>
> * I ran the tests from the command line with "mvn test".
>
> The ids.server tests have been run successfully:
>
> | Results :
> |
> | Tests run: 14, Failures: 0, Errors: 0, Skipped: 0
>
> But the test did not seem to have even tried to deploy ids.server on
> my test GlassFish. In fact, I don't see any trace of any attempt to
> interact with the test GlassFish.

Next thing I tried: I ran "mvn deploy" from the command line. This
yielded an error:

============================= mvn error: =============================

[INFO] --- exec-maven-plugin:1.4.0:exec (Deploy to glassfish) @ ids.server ---
remote failure: Error occurred during deployment: Exception while loading the app : javax.ejb.CreateException: Initialization failed for Singleton FileChecker. Please see server.log for more details.
Command deploy failed.
[INFO] ------------------------------------------------------------------------
[INFO] BUILD FAILURE
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 18.104 s
[INFO] Finished at: 2017-04-10T14:17:35+02:00
[INFO] Final Memory: 32M/388M
[INFO] ------------------------------------------------------------------------
[ERROR] Failed to execute goal org.codehaus.mojo:exec-maven-plugin:1.4.0:exec (Deploy to glassfish) on project ids.server: Command execution failed. Process exited with an error: 1 (Exit value: 1) -> [Help 1]

======================================================================

In the server.log I got:

============================ server.log: =============================

[2017-04-10T14:17:35.458+0200] [glassfish 4.0] [SEVERE] [] [javax.enterprise.system.tools.deployment.common] [tid: _ThreadID=29 _ThreadName=admin-listener(4)] [timeMillis: 1491826655458] [levelValue: 1000] [[
Exception while invoking class org.glassfish.ejb.startup.EjbApplication start method
javax.ejb.EJBException: javax.ejb.CreateException: Initialization failed for Singleton FileChecker
at com.sun.ejb.containers.AbstractSingletonContainer$SingletonContextFactory.create(AbstractSingletonContainer.java:656)
at com.sun.ejb.containers.AbstractSingletonContainer.instantiateSingletonInstance(AbstractSingletonContainer.java:396)
...
Caused by: javax.ejb.CreateException: Initialization failed for Singleton FileChecker
at com.sun.ejb.containers.AbstractSingletonContainer.createSingletonEJB(AbstractSingletonContainer.java:483)
at com.sun.ejb.containers.AbstractSingletonContainer.access$000(AbstractSingletonContainer.java:81)
at com.sun.ejb.containers.AbstractSingletonContainer$SingletonContextFactory.create(AbstractSingletonContainer.java:654)
... 78 more
Caused by: java.lang.IllegalStateException: Unable to locate resource run.properties
at org.icatproject.ids.PropertyHandler.abort(PropertyHandler.java:227)
at org.icatproject.ids.PropertyHandler.<init>(PropertyHandler.java:220)
at org.icatproject.ids.PropertyHandler.getInstance(PropertyHandler.java:41)
at org.icatproject.ids.FileChecker.init(FileChecker.java:305)
...
]]

======================================================================


Since it did create a ids.server-1.7.1-SNAPSHOT-distro.zip file, next
thing to try was to deploy that manually. After configuring the
properties files, I got an error from setup:

============================ setup error: ============================

$ ./setup install
Administered object jms/IDS/log created.
Command create-jms-resource executed successfully.
remote failure: Error occurred during deployment: Exception while loading the app : javax.ejb.CreateException: Initialization failed for Singleton Tidier. Please see server.log for more details.

Deployment failed
Setup is not complete

======================================================================

With the same issue in the server.log: "Unable to locate resource
run.properties". But the run.properties files was embedded in the war
file as WEB-INF/classes/run.properties.

Any ideas?

[Steve Fisher]

Your original problem was that mvn test only runs those tests that can be run without installing. To run the other tests you need at least mvn verify. mvn install is OK as well and will store a copy of the mvn artifacts below ~/.m2 The command mvn deploy should be avoided as it puts the artifacts in the public repository.

I am not interested in your attempt to install manually but would like to know what is inside your src/test/install directory = which should contain a run.properties suitable for the test being run.

Steve

Rolf

--
Rolf Krahl <rolf.krahl@helmholtz-berlin.de>

Helmholtz-Zentrum Berlin für Materialien und Energie (HZB)
Albert-Einstein-Str. 15, 12489 Berlin
Tel.: +49 30 8062 12122

--
You received this message because you are subscribed to the Google Groups "icatgroup" group.
To unsubscribe from this group and stop receiving emails from it, send an email to icatgroup+unsubscribe@googlegroups.com.
To post to this group, send email to icat...@googlegroups.com.
Visit this group at https://groups.google.com/group/icatgroup.
For more options, visit https://groups.google.com/d/optout.

[Rolf Krahl]

Steve,

Am Montag, 10. April 2017, 14:38:02 schrieb Steve Fisher:
> Your original problem was that mvn test only runs those tests that can be
> run without installing. To run the other tests you need at least mvn
> verify. mvn install is OK as well and will store a copy of the mvn
> artifacts below ~/.m2 The command mvn deploy should be avoided as it puts
> the artifacts in the public repository.

Since there have been some development in ids.server recently, I just
tried it once again using the latest version from GitHub. But I got
the same results as reported in my mail from 25.04.2017.

The setting and test procedure was the following:

* local GlassFish 4.0, having authn.db-1.2.1, icat.server-4.8.0, and
ids.storage_test installed.
* Starting with a fresh clone of ids.server from GitHub
(commit b1be13d).
* ~/.m2/settings.xml edited as indicated in the wiki and
src/test/resources/test.properties adapted from
test.properties.examples.
* mvn install

The default-tests run by the surefire-plugin were all successful. But
deploying to GlassFish failed:

| [INFO] --- exec-maven-plugin:1.4.0:exec (Deploy to glassfish) @ ids.server ---
| remote failure: Error occurred during deployment: Exception while loading the app : javax.ejb.CreateException: Initialization failed for Singleton FileChecker. Please see server.log for more details.
| Command deploy failed.

The server.log shows that the initialization for Singleton FileChecker
failed with the error "Unable to locate resource run.properties". See
the attachment for the output from maven and the complete server.log
from GlassFish.


I also tried the same with Payara (having authn.db-2.0.0,
icat.lucene-1.0.0, icat.server-4.9.0, and ids.storage_test installed)
with the very same result.

> I am not interested in your attempt to install manually but would like to
> know what is inside your src/test/install directory = which should contain
> a run.properties suitable for the test being run.

There must be clearly something missing in the test setup. There are
no files in src/test/install (besides the .gitignore which came with
the sources), in particular no run.properties. The file modification
time of the directory is not changed, which means apparently the test
did not try to put anything there.

There is however a run.properties in target/classes with the
${serverUrl} substituted from the ~/.m2/settings.xml (${HOME} is left
unsubstituted in the file).

Best,

Rolf

--
Rolf Krahl <rolf....@helmholtz-berlin.de>

[Rolf Krahl]

Dear all,

I believe, I found the troublemaker (although I still don't understand
all the details).

Am Mittwoch, 14. Juni 2017, 17:18:04 schrieb Rolf Krahl:
>
> Since there have been some development in ids.server recently, I just
> tried it once again using the latest version from GitHub. But I got
> the same results as reported in my mail from 25.04.2017.
>
> The setting and test procedure was the following:
>
> * local GlassFish 4.0, having authn.db-1.2.1, icat.server-4.8.0, and
> ids.storage_test installed.
> * Starting with a fresh clone of ids.server from GitHub
> (commit b1be13d).
> * ~/.m2/settings.xml edited as indicated in the wiki and
> src/test/resources/test.properties adapted from
> test.properties.examples.
> * mvn install

I tried the following: I can build an ids.server distribution from the
sources as usual with "mvn package -DskipTests". I can manually
deploy this distribution with ids.storage_file as plugin and it works
fine. Now, if I try to manually deploy the very same distribution
with ids.storage_test as plugin, I get the same "Unable to locate
resource run.properties" error that I also got from trying to run the
tests.

It seems that ids.storage_test (as taken from github at commit
73a70d9) is not compatible with recent versions of ids.server.
Indeed, looking into the sources, the way the plugins get initialized
has been changed in ids.server and ids.storage_test is not yet adapted
to this change.


There are however a few details that I still don't understand:
+ the exact location that this error is triggered, in line 88 of
PropertyHandler.java, in the props.loadFromResource() call. At
this point, the PropertyHandler did not try to instantiate the
plugin classes yet.
+ why has the directory src/test/install not been populated when
trying to run the test? From which directory did the test try to
deploy ids.server?
But these are just a side remark, I guess.

Regards,

[Steve Fisher]

You are quite right. The storage_test had not been pushed to github. I have now done so. I need to update the wiki a bit and for anybody writing their own plugin include the following information:

Updating a plugin for ids 1.8.0

An ids plugin must now have a constructor that accepts a java.util.Properties argument rather than reading in its own properties from a file.

 

As before:

• the plugin must include all its own dependencies

• It must not using any libraries that are also used by the IDS itself

• It must not rely upon the CLASSPATH

 

Because of the above constraints:

1. I have not found a good way to do logging so am no longer using it within plugins.

2. It will have a dependency upon icat.utils which drags in some unwanted classes. The maven shade plugin must be used to include only org.icatproject:ids.plugin and any extra libraries used by the plugin but NOT by the IDS

 

Properties should have names starting “plugin.archive.” or “plugin.main.”

 

Tests must be updated to match of course

 

The setup script can be greatly simplified

Rolf

--
Rolf Krahl <rolf.krahl@helmholtz-berlin.de>

Helmholtz-Zentrum Berlin für Materialien und Energie (HZB)
Albert-Einstein-Str. 15, 12489 Berlin
Tel.: +49 30 8062 12122