Testing Ozark

[Gregor Tudan]

Hi!

While ozark has quiet a few integration-tests in the travis build, it would be great to get faster feedback if change broke anything.

Manually deploying the test-app to a few application servers works, but get's a little annoying after a while.

I would like to offer porting the tests to arquillian, so testing gets a little easier. Or are there other plans/ideas on how to approach this?

Thanks,

Gregor

[Christian Kaltepoth]

Hey Gregor,

Sorry for my delayed reply. I had a busy week and didn't find much time to watch the mailing lists.

I basically agree with you. I also don't like that the Ozark integration tests are actually individual Maven modules which contain full applications. This is a very "heavyweight" approach and makes running the tests locally difficult. I created some scripts for that, but in most cases I simply push code that I want to test to temporary branches in my private GitHub fork so that Travis does this work for me. But that's a less-than-ideal solution in my opinion.

I would also prefer real Arquillian tests instead. A few thoughts about how we could achieve that:

• I guess we need a separate Maven module for the integration tests. Especially because the exact dependencies we need to add to the integration test archives depend on the target container. But maybe we can add the 3rd party view engine tests directly to the corresponding ext module?
• Eventually it would be great to be able to run the tests against different containers. However, this could be a subsequent step. At first we should make sure that the new Arquillian tests against Glassfish. For the TCK I basically created a utils class is able to create a "base archive" depending on the target environment. And it is working great.

Any other thoughts?

Perhaps we could create some kind of POC by migrating a first test webapp and see how good it works?

Christian

--
You received this message because you are subscribed to the Google Groups "MVC 1.0 Users Mailing List" group.
To unsubscribe from this group and stop receiving emails from it, send an email to jsr371-users...@googlegroups.com.
To post to this group, send email to jsr371...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/jsr371-users/218dec8a-ccfa-4fc3-9b73-c005bd4738af%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--

Christian Kaltepoth

Blog: http://blog.kaltepoth.de/

Twitter: http://twitter.com/chkal

GitHub: https://github.com/chkal

[Gregor Tudan]

Hi Christian,

no problem, I'm on vaccation myself, so I don't mind the delay. :-) I hope you enjoyed yours?

I'm not yet sure on how to approach the TCK - it might make sense to keep it the way it is to have it running against an untampered server.

I aggree with you that the test apps would be a good place to start. So I went ahead and gave the thymeleaf app a first shot. It's not running yet, but I wanted to share the progress made. Please bare in mind that his is really just some POC and I'm not even thinking about opening a PR. ;-)

https://github.com/gtudan/ozark/commit/3307e92a3e40311bf4197cfa911f795ca833a5f4

- I added the arquillian dependencies to the test parent module and created a profile for wildfly and glassfish

- maven downloads and unpacks the application server. This is done for every test app, but we could easily change this to a common directory.

- There's now a seperate ThymeleafArquillianIT.java that tries to do the same things as the original one

- for functional testing I used arquillian drone instead of htmlunit - htmlunit is still used under the hood, but drone takes care of the setup. Drone didn't like your htmlunit-version, so I updated it. Some methods were renamed, so that's where the many changes came from.

The test can be run with: "mvn install verify -Parquillian-wildfly-managed -Dtest=ThymeleafArquillianIT"

This fails on wildfly and I'm not really sure why. I couldn't get the current master of the thymeleaf test app to run on wildfly 12 and 13, so it might be a different problem. I'll have to look deeper into this. Glassfish has trouble as well, but it's compaining about an expired certificate on glassish.org, so it might be a temporary issue.

So no big news yet, but I just wanted to let you know that I'm still on it.

- Gregor

[Christian Kaltepoth]

Thanks a lot for sharing your progress.

I'll have a deeper look at you first draft and give some feedback. This is a difficult topic, so I needs some more time to think about it. :-)

Christian

To view this discussion on the web visit https://groups.google.com/d/msgid/jsr371-users/f18595e1-df28-4df9-b5ff-2fbe26ca02f6%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

[Christian Kaltepoth]

Hi Gregor,

I'm sorry for coming back to you so late. I just had a quick look at the commit you provided. A few thoughts:

• Updating HTMLUnit is fine of course. I had a look at Arquillina Drone quite some time ago (a few years actually). I'm not completely sure if it brings much benefit over plain HTMLUnit for our use case, but we can give it a try.
• I'm not sure what the best way is to get and setup the container. For the TCK I basically use remote adapters if possible which require a running container which is much faster in my experience when running many tests. But I'm fine with also trying different approaches.
• Maybe we should think about the overall structure of the tests. Having one Maven module per tests looks like an overkill to me. Maybe we should change the module structure to get single "integration test" module? This would simplify the whole build.

What I was thinking of is something like this:

• First we could rename the "test" directory to something like "samples" or "demos". Because that's what they actually are. They are full Maven projects containing sample apps (with a test verifying that they work correctly)
• Then we could create a single new Maven module in a directory called "tests". This module would then contain all the Arquillian tests. The pom could look similar to the one in the "tck" directory with profiles for each container, like you did it in your draft.
• Then we could incrementally migrate the sample apps one by one to real Arquillian integration tests in the new "tests" module and remove them from the "samples" directory.

What do you and others think?

Christian

[Gregor Tudan]

Hi Christian,

I made good progress today. I renamed the tests to „examples“ and created a new Testsuite-module like you suggested. It makes sense to have a single module for most tests. The only exception could be ext - adding the extra jars via shrink-wrap could be painful, but let’s start with the big test suite module for now.

Then I migrated the thymeleaf-module into it. Getting the maven setup right, so container specific modules like "ozark-resteasy“ only get included when needed was the hardest part, but I think it’s working pretty well now. 

You’re right, the managed-mode for the containers is not ideal. I figured out how to get travis to run docker images in the background and used the wildfly image as container. That makes it pretty easy test against different application servers and versions. The only downside of the remote approach is that we don’t get logs from the server - this could make debugging more difficult. But I guess we could dump the logs via „docker logs“ if we needed them.

Here’s the first green travis build: https://travis-ci.org/gtudan/ozark/jobs/431838384

The code is on GitHub: https://github.com/gtudan/ozark/tree/arquillian-test

- Gregor

The next step would be to get more application servers running. Then we can get 

To view this discussion on the web visit https://groups.google.com/d/msgid/jsr371-users/CAEXeC6z0-kTwqJnh7ZFCfirpHNiRkCup%2B3ymW6VwjAsSEbvKpw%40mail.gmail.com.

[Christian Kaltepoth]

Hi Gregor,

Awesome! That sounds very promising. I just had a quick look at your fork and the basic structure looks good.

I guess because this is a rather huge "refactoring" of the project structure, we should try to split it up into a few steps. The first one would be to renamed the old "test" directory to "examples". I'll check if there are any open pull requests or other work-in-progress which should be merged first, so we don't end up with merge conflicts. I'll try to find some time today to do so and then do the renaming.

After this step you could create a first pull request for the basic Arquillian setup and the new module with a first migrated test (Thymeleaf?). We could discuss all the details in the pull request. And I guess that adding support for more containers should be straight forward from there. Especially, because we already have as similar setup for the TCK runner.

Thanks a lot for your help with this :-)

Christian

To view this discussion on the web visit https://groups.google.com/d/msgid/jsr371-users/9D1AC4A7-75C7-42A9-95D2-E8A2A82B3D1C%40tudan.de.

For more options, visit https://groups.google.com/d/optout.

[Christian Kaltepoth]

Hi Gregor,

I just renamed the folder! So feel free to create a pull request for a first minimal Arquillian test module! :-)

Christian

[Gregor Tudan]

Hi,

we got the first tests up and running and I think I might have found a first regression: https://travis-ci.org/gtudan/ozark/builds/462686714

The testNoOverrideMvc-Test in the Annotations-Example runs fine on glassfish, but fails on Wildfly 14. It throws a 500 on Glassfish, but wildfly returns the error.jsp with status 200.

Before I start running around shouting "Bugs!!1" I'd like to know how trustworthy the current test suite is. Aren't those examples deployed to a container and tested by the current TCK-Tests? Sure, it might be due to a newer wildfly version, so results are not 100% comparable.

If this really is a bug, how should we deal with it in the test suite? Should we disable the test for wildfly with "assume" and add a comment pointing to the issue for it?

Thanks,

Gregor

[Christian Kaltepoth]

Hi Gregor,

see my response inline:

we got the first tests up and running and I think I might have found a first regression: https://travis-ci.org/gtudan/ozark/builds/462686714

The testNoOverrideMvc-Test in the Annotations-Example runs fine on glassfish, but fails on Wildfly 14. It throws a 500 on Glassfish, but wildfly returns the error.jsp with status 200.

Before I start running around shouting "Bugs!!1" I'd like to know how trustworthy the current test suite is. Aren't those examples deployed to a container and tested by the current TCK-Tests? Sure, it might be due to a newer wildfly version, so results are not 100% comparable.

All the examples were only run against Glassfish in the past. The TCK is a completely separate set of tests (which isn't complete yet by the way). So it may be possible that this is really a bug.

If this really is a bug, how should we deal with it in the test suite? Should we disable the test for wildfly with "assume" and add a comment pointing to the issue for it?

I would prefer to keep the test as it is and simply configure Travis to allow failures for this pipeline step. This is simple to do and doesn't "hide" the tests by disabling them. I do it the same way with the TCK profile, because there are some known failures which will be addressed later which should not prevent the snapshots from being deployed.

Christian