Poll on your dev environment & queries on dev environment
Eunice Soh
Oliver Bertuch
Hi Eunice,
I'm using IntelliJ IDEA as my IDE, as I don't like Eclipse nor Netbeans.
1. netbeans/payara/postgres/solr installed on a local Mac/Linux machine
This seems like the most straightforward e.g. https://guides.dataverse.org/en/latest/developers/dev-environment.html
Most devs at IQSS seem to use this option. (as I recently
learned, most of it became muscle memory, so it's quick to use for
'em...)
2. vagrant/payara/postgres/solr installed on a virtual machine, using local Windows machine
This seems like an option for those using Windows e.g. https://guides.dataverse.org/en/latest/developers/windows.html
Q: How's development done here? Using netbeans as well and how?
Phil uses this sometimes for ephemeral setups, but it seems to be
rarely used for dev environments.
These days, it's not the only option on Windows, though. There is also WSL, which becomes more and more stable
If you are going to do development on Windows, please remember to
use the correct line ending setting within your IDE/editors/...!
3. docker, using a local Linux machine
Recently read that Docker can also be used in a dev environment: https://docs.docker.com/desktop/dev-environments/
Q: Can this be used for development, and how?
Not sure DANS is using this for development.
4. kubernetes, using a local Linux machine
Q: Can this be used for development, and how?
It could, but I'd advise not to for now (it's outdated). As the maintainer of this repo, I am trying hard to get away from creating the images in there, so it can focus on Kubernetes usage only.
For easy installations of Docker on Windows, Mac (even Linux) the
Minikube project is still a great option to avoid "Docker for
Desktop" license costs (although its a great product worth the
money).
Another upcoming option is my feature branch at https://github.com/gdcc/dataverse, allowing container usage right from the Maven CLI. Please note this is WIP, so it's not advertised anywhere yet and it may break any time. With the recent pickup of Maven modules within upstream, I am working on making at least the Solr image an upstream thing. (https://github.com/IQSS/dataverse/pull/8320) Documentation for using this stuff will be added to the development guide if it gets merged.
I am always using containers for my dev environment, as it's very
fast to setup and run all the things you need in clean, ephemeral
ways.
Other questions
1. Which types of tests must be run or good to run prior to pull request on the local dev environment? unit tests, integration tests (API test/ testcontainers test)? Aall units tests or just associated code that was modified? Where should these be run (e.g. docker on a cloud instance, or can it run without docker)?
IMHO the Dataverse codebase suffers from a low test coverage.
Many features aren't tested in an automated fashion.
Currently, there are no Testcontainers based tests. Again, I am working on making integration testing with this possible, which is based on the container efforts I am working on. (This can be tried within the https://github.com/poikilotherm/dataverse/tree/testcontainers feature branch. Again this is WIP, expect things to be broken and out of date.)
Please note there is also the docker-aio option to run API tests
locally. (I hope to replace this with TC...)
2. What's run on the continuous integration tests using jenkins and github action e.g. integration tests? The jenkins link seems broken, is it actively being used?
What we have in terms of automated tests are JUnit based unit tests (may be executed from the IDE or CLI) and API tests (our best and only kind of integration testing). Unit tests are executed for each Pull Request and have to pass. The API tests are executed on Jenkins setting up EC2 instances to run these tests against, also for every PR. The Jenkins instance is prone to be targeted by bots, which is why it's not really accessible for outsiders. Ask Don for details (here, in Matrix or Community Slack).
My aim is to run API and future integration tests inside Github
Actions with Testcontainers, making this stuff more accessible.
If you wanna chat, you're welcome to join us at
https://chat.dataverse.org or Community Slack. Phil, Don and I are
usually around (beware of timezone mushrooms...).
Best,
Oliver
-- ------------------------------------------------------------------------------------- Oliver Bertuch Forschungszentrum Jülich GmbH Zentralbibliothek / Central Library Forschungsdatenmanagement / Research Data Management Entwicklung von Forschungssoftware / Research Software Engineering 52425 Jülich +49 2461 61-85370 https://www.fz-juelich.de/zb Sitz der Gesellschaft: Jülich Eingetragen im Handelsregister des Amtsgerichts Düren Nr. HR B 3498 Vorsitzender des Aufsichtsrats: MinDir Volker Rieke Geschaeftsfuehrung: Prof. Dr.-Ing. Wolfgang Marquardt (Vorsitzender), Karsten Beneke (stellv. Vorsitzender), Prof. Dr. Astrid Lambrecht, Prof. Dr. Frauke Melchior -------------------------------------------------------------------------------------
Eunice Soh
Thank you Oliver for sharing about the dev environment & the testing strategy... This is giving me a better picture now of the dev environment.
Péter Király
I use IntelliJ and local servers in Linux with some small changes. As
far as I remember the dev scripts had some specific path hardcoded for
e.g. maven, which did not fit my enviroment (which is Ubuntu, while
the official Dataverse suggests Red Hat, and I guess Harvard developer
use that for local development as well.
The dataverse-docker and the dev Docker are two different things. The
first one is for running services (either for testing the service, or
for production), not for development purpose. It uses a particular
stable release, which the dev docker uses the actual state of the code
on your machine.
Best,
Péter
> You received this message because you are subscribed to the Google Groups "Dataverse Dev" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to dataverse-de...@googlegroups.com.
> To view this discussion on the web visit https://groups.google.com/d/msgid/dataverse-dev/1c513552-931b-4793-8685-6fee87051484n%40googlegroups.com.
--
Péter Király
software developer
GWDG, Göttingen - Europeana - eXtensible Catalog - The Code4Lib Journal
http://linkedin.com/in/peterkiraly
Philip Durbin
> 2. vagrant/payara/postgres/solr installed on a virtual machine, using local Windows machine
> Q: How's development done here? Using netbeans as well and how?
When I think about using Dataverse in Docker I always turn to datavese-aio (all in one), which we document here: https://guides.dataverse.org/en/5.9/developers/testing.html#running-the-full-api-test-suite-using-docker
Like the Vagrant environment, we try to keep dataverse-aio working, including when we update dependencies such as Payara, PostgreSQL, or Solr.
> Recently read that Docker can also be used in a dev environment: https://docs.docker.com/desktop/dev-environments/ Q: Can this be used for development, and how?
> 4. kubernetes, using a local Linux machine https://github.com/gdcc/dataverse-kubernetes Q: Can this be used for development, and how?
> 1. Which types of tests must be run or good to run prior to pull request on the local dev environment? unit tests, integration tests (API test/ testcontainers test)? Aall units tests or just associated code that was modified? Where should these be run (e.g. docker on a cloud instance, or can it run without docker)?
> 2. What's run on the continuous integration tests using jenkins and github action e.g. integration tests? The jenkins link seems broken, is it actively being used?
To view this discussion on the web visit https://groups.google.com/d/msgid/dataverse-dev/CABFhGt%3Djn-bSM0G8N2BssU%3DYGGGpBP-VBcUmifwHQVq6BiOigw%40mail.gmail.com.
--
Software Developer for http://dataverse.org
http://www.iq.harvard.edu/people/philip-durbin
James Myers
FWIW: I use Eclipse on Windows. For deployment, I usually use dataverse-ansible to spin up a test machine and then just repeatedly replace the war file (using ec2 stop/start instance to keep the db state etc. without running the instance all the time). I use Ubuntu on Windows for local tasks and as a way to log into the remote EC2 instances.
-- Jim
--
You received this message because you are subscribed to the Google Groups "Dataverse Dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to
dataverse-de...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/dataverse-dev/0e723e86-836b-4c25-9e53-bf5e2d1157e6n%40googlegroups.com.
Andreev, Leonid
To view this discussion on the web visit https://groups.google.com/d/msgid/dataverse-dev/MN2PR07MB7343DEEC525FF1A945A1FF61BF0F9%40MN2PR07MB7343.namprd07.prod.outlook.com.
Durand, Gustavo
I do most of my development in a most boring, by the book (by the guide) environment - Netbeans/everything local on MacOS. I spin up EC2 branches when I need to test building everything up from scratch.
On Mon, Mar 14, 2022 at 10:51 AM James Myers <qqm...@hotmail.com> wrote:
To view this discussion on the web visit https://groups.google.com/d/msgid/dataverse-dev/MN2PR07MB7343DEEC525FF1A945A1FF61BF0F9%40MN2PR07MB7343.namprd07.prod.outlook.com.
--
You received this message because you are subscribed to the Google Groups "Dataverse Dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to dataverse-de...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/dataverse-dev/CAGSsyste1a5m9ZQhDhF9GDiFj_Kxi9bsU5Dyk-zRVbUtQR-f%3DQ%40mail.gmail.com.
Jan Mansum, van
2. vagrant/payara/postgres/solr installed on a virtual machine, using local Windows machineThis seems like an option for those using Windows e.g. https://guides.dataverse.org/en/latest/developers/windows.htmlQ: How's development done here? Using netbeans as well and how?
Jan
Eunice Soh
- Unit tests can be run locally on IDE (IntelliJ, Eclipse, Netbeans) or maven.
- API integration tests may be run locally with docker-aio (https://github.com/IQSS/dataverse/tree/develop/conf/docker-aio). Oliver hopes to replace with Testcontainers
- Testcontainers integration tests are WIP.
- Tests should be added where appropriate & run when code is changed.
- Unit tests should be run minimally during development, and integration tests can also be run locally but if not, on Jenkins (remotely, on pull request). Jenkins pass/fail can be viewed on pull requests. Jenkins logs not publicly viewable.
- Unit tests (Github Actions) and API integration tests (Jenkins) are also run remotely on pull request.
- IQSS/Phil, Leonid & Gustavo and Péter use IDE/solr/postgres/payara (on Mac/Unix-based)
- Oliver uses IntelliJ (IDE)/maven plugins/containers (feature branch at https://github.com/gdcc/dataverse)
- QDR/Jim uses Eclipse (IDE)/Ubuntu WSL (on Windows) & EC2 Docker/Ansible (https://github.com/GlobalDataverseCommunityConsortium/dataverse-ansible) for dev deployment by replacing war files
- DANS/Jan uses vagrant with IntelliJ (IDE) attached for remote debugging with Payara
Philip Durbin
--
You received this message because you are subscribed to the Google Groups "Dataverse Dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to dataverse-de...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/dataverse-dev/f21eadcd-860f-4618-a512-f25517491036n%40googlegroups.com.