Need help with running unit tests on an extension (using pipelines)

[Labian Gashi]

Hi everyone,

I am currently building some extensions on geospatial data support for OpenRefine and OpenRefine integration with OSM using Overpass.

I have these extensions hosted on GitLab and as you might know, GitLab offers CI/CD pipelines for your code. I am trying to create a pipeline that runs the unit tests for each of my extensions everytime a new push is made. The issue however, is that I keep getting this error as soon as the build is finished and I try to run the tests:

`[ERROR] Failed to execute goal on project geojson: Could not resolve dependencies for project org.openrefine:geojson:jar:3.6-SNAPSHOT: Could not find artifact org.openrefine:main:jar:3.6-SNAPSHOT in snapshots (https://oss.sonatype.org/content/repositories/snapshots/) -> [Help 1]`

I am building the project by running: mvn clean compile test-compile dependency:build-classpath

I am trying to run the tests by running: mvn test -f extensions/geojson where geojson is the name of my extension.

On my local machine this works and the test suite is run smoothly, but this is not the case on the pipeline. I was wondering if someone encountered a similar issue with the above error and knew how to fix it?

Thanks for your attention,

Labian Gashi.

P.S. If anyone has experience with GitLab's CI/CD file, here's what my file looks like: 

stages:

  - build

  - test

image: maven:3.6.3-jdk-11

build:

  stage: build

  script: 

    - mvn clean compile test-compile dependency:build-classpath

test:

  stage: test

  needs: [build]

  script:

    - mvn test -f extensions/geojson

[Antonin Delpeuch (lists)]

Hi Labian,

I am far from being a Maven expert so maybe there is a good way to do what you want here, but generally I avoid trying to run Maven for some submodules only.

So instead of `mvn test -f extensions/geojson` I would just run `mvn test`… That would run tests on all other modules but perhaps it's not such a big deal? Backend tests are relatively quick.

The proper solution, I think, would be that you don't even have to clone OpenRefine's repository to work on your own extension: you should be able to do it independently, in your own Maven project. For that, I am planning to publish Maven artifacts for OpenRefine's modules: https://github.com/OpenRefine/OpenRefine/issues/2254

But as long as our artifacts are not published on Maven, your approach seems to be the easiest way to go.

Best,

Antonin

--
You received this message because you are subscribed to the Google Groups "OpenRefine Development" group.
To unsubscribe from this group and stop receiving emails from it, send an email to openrefine-de...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/openrefine-dev/a20b2cff-3a32-42ff-be9e-299e9feb54a6n%40googlegroups.com.

[Labian Gashi]

Hi Antonin,

I already tried that but there are some errors that I get before reaching my extensions (the database extension's tests fail) and the whole process is skipped.

I guess I'll skip the CI/CD approach for now though until I find some other solution.

Best,

Labian.

[Thad Guidry]

Hi Labian,

Most likely your maven config or context is not correct inside of that image you are using " maven:3.6.3-jdk-11 ".

I notice you are not modifying any of the environment variables for the image, so everything is default as baked into that image.

Like using oss.sonatype.org as the central repository.

That's fine and indeed OpenRefine's dependencies are there as you can see:
https://oss.sonatype.org/#nexus-search;quick~openrefine

But not at that particular path that the error line mentions.

(Sonatype hosts a mirror of many OSS - open source software, hence that subdomain configured as a <repository> in that maven image.

Anyways...

On GitHub, since you are given hundreds and hundreds of free hours for virtual environments, you can simply use their own provided available environments as shown here:
https://github.com/actions/virtual-environments

That way you get all the pre-installed tools you might need for debugging.

Here is GitHub's own tutorial on building and testing Java projects with Maven:
https://docs.github.com/en/actions/guides/building-and-testing-java-with-maven?learn=continuous_integration

I don't think you want to publish any Java packages to a Maven repository, but in case you do later:
https://docs.github.com/en/actions/guides/publishing-java-packages-with-maven

Thad

https://www.linkedin.com/in/thadguidry/

https://calendly.com/thadguidry/

To view this discussion on the web visit https://groups.google.com/d/msgid/openrefine-dev/c8f03a28-e5eb-406e-bf5b-2b72721ef213n%40googlegroups.com.

[Antonin Delpeuch (lists)]

Hi Labian,

Yes I have also noticed that the database extension tests have started
failing locally, I will push a fix for this asap.

Best,
Antonin

On 23/06/2021 14:43, Labian Gashi wrote:
> Hi Antonin,
>
> I already tried that but there are some errors that I get before

> reaching my extensions (the *database *extension's tests fail) and the

>> *`[ERROR] Failed to execute goal on project geojson: Could not

>> resolve dependencies for project
>> org.openrefine:geojson:jar:3.6-SNAPSHOT: Could not find artifact
>> org.openrefine:main:jar:3.6-SNAPSHOT in snapshots
>> (https://oss.sonatype.org/content/repositories/snapshots/)

>> <https://oss.sonatype.org/content/repositories/snapshots/)> ->
>> [Help 1]`*
>> *
>> *

>> I am building the project by running:

>> *mvn clean compile test-compile dependency:build-classpath*
>> *
>> *

>> I am trying to run the tests by running:

>> *mvn test -f extensions/geojson *where geojson is the name of my

>> extension.
>>
>> On my local machine this works and the test suite is run smoothly,
>> but this is not the case on the pipeline. I was wondering if
>> someone encountered a similar issue with the above error and knew
>> how to fix it?
>>
>> Thanks for your attention,
>> Labian Gashi.
>>
>> P.S. If anyone has experience with GitLab's CI/CD file, here's
>> what my file looks like: 

>> *stages:*
>> *  - build*
>> *  - test*
>> *
>> *
>> *image: maven:3.6.3-jdk-11*
>> *
>> *
>> *build:*
>> *  stage: build*
>> *  script: *
>> *    - mvn clean compile test-compile dependency:build-classpath*
>> *
>> *
>> *test:*
>> *  stage: test*
>> *  needs: [build]*
>> *  script:*
>> *    - mvn test -f extensions/geojson*

>>
>> --
>> You received this message because you are subscribed to the Google
>> Groups "OpenRefine Development" group.
>> To unsubscribe from this group and stop receiving emails from it,
>> send an email to openrefine-de...@googlegroups.com.
>> To view this discussion on the web visit
>> https://groups.google.com/d/msgid/openrefine-dev/a20b2cff-3a32-42ff-be9e-299e9feb54a6n%40googlegroups.com

>> <https://groups.google.com/d/msgid/openrefine-dev/a20b2cff-3a32-42ff-be9e-299e9feb54a6n%40googlegroups.com?utm_medium=email&utm_source=footer>.

>
> --
> You received this message because you are subscribed to the Google
> Groups "OpenRefine Development" group.
> To unsubscribe from this group and stop receiving emails from it, send
> an email to openrefine-de...@googlegroups.com

> <mailto:openrefine-de...@googlegroups.com>.

> To view this discussion on the web visit

> https://groups.google.com/d/msgid/openrefine-dev/c8f03a28-e5eb-406e-bf5b-2b72721ef213n%40googlegroups.com
> <https://groups.google.com/d/msgid/openrefine-dev/c8f03a28-e5eb-406e-bf5b-2b72721ef213n%40googlegroups.com?utm_medium=email&utm_source=footer>.