Dataverse Automated Testing

[Durand, Gustavo]

Hi all,

We're planning on updating our developer guides with a testing statement of what we expect / prefer to be included in incoming PRs related to Automated testing.

I would appreciate if you could respond (either directly to me or to the list) with any feedback you have on the below:

Automated tests (Unit tests vs Integration tests):

Overall expectation: when adding new functionality, please provide integration (REST assured) tests that demonstrate both a successful usage of the functionality as well as any common failures (for example, for functionality to change a username, two common errors would be the old username does not exist and the new username is already in use). Additionally a test that confirms the permissions to run that functionality are correct should be included.

Integration tests are more effective than unit tests because they can fully test a piece of functionality from validation of the user input to the db queries that are run. (for example, unit tests do not test that queries written as Named Statements are written correctly. They also provide a straightforward way to call the commands with different types of users to test permissions.

Writing integration tests instead of unit tests should also help us get in the practice of running the integration test suite.

For a particular complex area of code, it can be useful to know if anything is broken at build time (the advantage of unit test). For those cases, unit tests can be written *in addition* to integration tests. In particular this approach may be worthwhile, if a self contained method has complicated logic that could easily break if changed slightly. However, as a best practice, this type of method should be avoided, generally, and either be simplified, broken out into several methods, or both.

_________________

Thank you,

Gustavo

[Oliver Bertuch]

Hey Gustavo,
let me add my 2 cents to this.

I am all in when it comes to testing, extending tests and raise the
coverage. I fully agree that integration tests are an absolute necessity
and are a great addition in making development, deployment and usage
more stable, robust and mature.

What I am not so sure about is, whether we share the same mindset about
the intention of tests.
For me, unit tests are a necessity, too. I follow the rules of "unit
tests for business logic", which is indeed easy to test at build time.
In many cases, that will need mocks and more, but it is a vital thing.
Integration tests on the other hand for me are as important as unit
tests. You cannot unit test things like "is my S3 integration working"
except on a very low level, like the generation of identifiers, etc.

Many things IMHO cannot be integration tested right now due to a lack of
infrastructure. Like how do you test the S3 integration? And there needs
to be a discussion about how much details you want, as testing more
details normally come at the price of long runs and costs. I miss a
clear strategy and rules for orientation.

I also wonder if more or less relying on RESTassured only is a good
approach when trying to do integration tests for stuff which is not
exposed through an REST API. Like integration testing complex setups,
e.g. external integrations, storage, ... Before adding such a statement
it would IMHO be a necessity to provide examples what you expect right
within your codebase.

I also miss an explanation what happens, when you don't provide those
tests. Will it be declined? Will you guys do sth. about it? What should
I do? What may I expect from QA, etc?

Again: just my 2 cents as a community member. Happy to discuss more on
this very important topic.
Cheers,
Oliver

Am 24.07.19 um 22:27 schrieb Durand, Gustavo:

> Hi all,
>
> We're planning on updating our developer guides with a testing
> statement of what we expect / prefer to be included in incoming PRs
> related to Automated testing.
>
> I would appreciate if you could respond (either directly to me or to
> the list) with any feedback you have on the below:
>

> *Automated tests (Unit tests vs Integration tests):*

> --
> You received this message because you are subscribed to the Google
> Groups "Dataverse Dev" group.
> To unsubscribe from this group and stop receiving emails from it, send
> an email to dataverse-de...@googlegroups.com
> <mailto:dataverse-de...@googlegroups.com>.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/dataverse-dev/CAF2sSedXGEsidCspdYn%2B1_Pnc-ndbiLQ5XJid-vknHMo2zkskw%40mail.gmail.com
> <https://groups.google.com/d/msgid/dataverse-dev/CAF2sSedXGEsidCspdYn%2B1_Pnc-ndbiLQ5XJid-vknHMo2zkskw%40mail.gmail.com?utm_medium=email&utm_source=footer>.

--
-------------------------------------------------------------------------------------
Forschungszentrum Juelich GmbH
Zentralbibliothek - Forschungsdatenmanagement
Oliver Bertuch
52425 Juelich
+49 2461 - 61 85 370
http://www.fz-juelich.de

Sitz der Gesellschaft: Juelich
Eingetragen im Handelsregister des Amtsgerichts Dueren Nr. HR B 3498
Vorsitzender des Aufsichtsrats: MinDir Volker Rieke
Geschaeftsfuehrung: Prof. Dr.-Ing. Wolfgang Marquardt (Vorsitzender),
Karsten Beneke (stellv. Vorsitzender), Prof. Dr.-Ing. Harald Bolt,
Prof. Dr. Sebastian M. Schmidt
-------------------------------------------------------------------------------------