error in building DV docker version

64 views
Skip to first unread message

domingo...@unimib.it

unread,
Feb 28, 2018, 6:51:46 AM2/28/18
to Dataverse Users Community
Hi all,
I'm trying to build a docker image following this guide: https://github.com/IQSS/dataverse/tree/master/conf/docker-aio 
I'm using in a brand new Linux VM.
I got an error running 1prep.sh script, regarding the test suite "edu.harvard.iq.dataverse.ingest.tabulardata.impl.plugins.csv.CSVFileReaderTest", test "testVariableUNFs". Here's the error:
-------------------------------------------------------------------------------
Test set: edu.harvard.iq.dataverse.ingest.tabulardata.impl.plugins.csv.CSVFileReaderTest
-------------------------------------------------------------------------------
Tests run: 5, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 0.387 sec <<< FAILURE! - in edu.harvard.iq.dataverse.ingest.tabulardata.impl.plugins.csv.CSVFileReaderTest
testVariableUNFs(edu.harvard.iq.dataverse.ingest.tabulardata.impl.plugins.csv.CSVFileReaderTest)  Time elapsed: 0.138 sec  <<< ERROR!
java.lang.ArrayIndexOutOfBoundsException: 1
at org.dataverse.unf.RoundRoutines.calcMantissa(RoundRoutines.java:399)
at org.dataverse.unf.RoundRoutines.Genround(RoundRoutines.java:322)
at org.dataverse.unf.UnfNumber.UNF5(UnfNumber.java:272)
at org.dataverse.unf.UnfNumber.RUNF5(UnfNumber.java:215)
at org.dataverse.unf.UnfDigest.unfV(UnfDigest.java:376)
at org.dataverse.unf.UnfDigest.unf(UnfDigest.java:236)
at org.dataverse.unf.UNFUtil.calculateUNF(UNFUtil.java:426)
at edu.harvard.iq.dataverse.ingest.tabulardata.impl.plugins.csv.CSVFileReaderTest.testVariableUNFs(CSVFileReaderTest.java:350)

Does anybody idea how to resolve it?

Thank you for your help!
Domingo

Domingo Scisci (PhD)

Centro Interdipartimentale UniData
Università degli Studi di Milano-Bicocca
Via Bicocca degli Arcimboldi 8
20126 - Milano

Philip Durbin

unread,
Feb 28, 2018, 7:47:47 AM2/28/18
to dataverse...@googlegroups.com
Hi Domingo,

Thanks for the bug report. You mentioned the "master" branch and I'm wondering which commit you are on. As of this writing the latest commit on "master" is 64cb1ef and according to Travis, there were no test failures for that commit: https://travis-ci.org/IQSS/dataverse/builds/330059891

Can you please switch to the "develop" branch and see if you get the same result? When you report back, please indicate which commit you are on since we are always merging pull requests into "develop". (We use the "git flow" model, described here: http://guides.dataverse.org/en/4.8.5/developers/version-control.html#branching-strategy .) For what it's worth, the tests also passed on the most recent commit on "develop", which was 6dd1fcb: https://travis-ci.org/IQSS/dataverse/builds/340715624

Also, I'm not sure if this will help or not but I recently stepped through the instructions in conf/docker-aio/readme.txt myself and made a few tweaks to clarify a few thing. That commit hasn't been merged yet but it's here if you'd like to take a look: https://github.com/IQSS/dataverse/commit/e796935

Please feel free to pop into http://chat.dataverse.org (#dataverse on freenode IRC) if you need help. This "docker-aio" code is fairly new and I'm glad you're trying it out. If you feel like it, please back up and tell us a little more about what you're up to in general. The "docker-aio" code is oriented toward developers running the API test suite.

Thanks,

Phil

--
You received this message because you are subscribed to the Google Groups "Dataverse Users Community" group.
To unsubscribe from this group and stop receiving emails from it, send an email to dataverse-community+unsub...@googlegroups.com.
To post to this group, send email to dataverse-community@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/dataverse-community/61dd5c5c-105f-4402-a2ed-50abefb2b414%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.



--

domingo...@unimib.it

unread,
Feb 28, 2018, 10:35:56 AM2/28/18
to Dataverse Users Community
Dear Phil,
thank you for your quick reply. I've cloned the "develop" version just now, commit  6dd1fcb. I run the 1prep.sh but I got the same error. Perhaps is it a problem with my VM's configuration?
Let me know if you need more info.
Thank you,
Domingo
To post to this group, send email to dataverse...@googlegroups.com.

Philip Durbin

unread,
Feb 28, 2018, 11:56:12 AM2/28/18
to dataverse...@googlegroups.com
The failure is in the unit test suite (JUnit). In the "testVariableUNFs" test a file called "election_precincts.csv"[1] is being fed into the CSV parser in Dataverse and UNFs[2] are attempting to be calculated.

The ArrayIndexOutOfBoundsException seems to be coming from `if (!dec.equalsIgnoreCase(canon[1]))`[3] in the UNF library. `canon` comes from `atom.split("\\" + sep)` and `sep` is called "the decimal point" in a comment so I'm wondering if this has something to do with decimals being commas instead of periods in Italy[4], where you seem to be from.

I know you're using Docker but I'd be curious to hear if running `vagrant up` works for you. If the war file hasn't been compiled (the unit tests are run as a compilation step), it will be built within the Vagrant VM. You can read more about this at http://guides.dataverse.org/en/4.8.5/developers/tools.html#vagrant

We are obviously getting in the weeds for this general dataverse-community list, so could you please open a GitHub issue at https://github.com/IQSS/dataverse/issues so we can track this bug you found? We can continue the conversation there and summarize for the list after we figure it out.

Thanks!

Phil

To unsubscribe from this group and stop receiving emails from it, send an email to dataverse-community+unsubscribe...@googlegroups.com.

--
You received this message because you are subscribed to the Google Groups "Dataverse Users Community" group.
To unsubscribe from this group and stop receiving emails from it, send an email to dataverse-community+unsub...@googlegroups.com.
To post to this group, send email to dataverse-community@googlegroups.com.
 To view this discussion on the web visit https://groups.google.com/d/msgid/dataverse-community/a8436d9e-c0b1-457d-8430-1e693a22b7e1%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

Leonid Andreev

unread,
Feb 28, 2018, 12:20:29 PM2/28/18
to Dataverse Users Community
If you just need to build the .war file, the quickest way to do it would be to just add the "@Ignore" annotation to the test (and any other UNF-related test methods that are failing).
In the meantime, we'll try to figure out why it is failing for you.

I believe Phil is correct that this is somehow related to the locale. That would indicate that we are doing something wrong in that test; because the whole purpose of the UNF is to create a *normalized* data signature, that's not affected by any local formatting variations.

Pete Meyer

unread,
Mar 1, 2018, 12:26:18 PM3/1/18
to Dataverse Users Community
Hi Domingo,

This looks to me to be a possible localization issue as well; I'd expect that `mvn test` (or `mvn package`) from within your VM would fail the same way.

The output of `printenv` from your VM might be helpful, and potentially the distribution and version of linux.

Best,
Pete

domingo...@unimib.it

unread,
Mar 2, 2018, 5:29:07 AM3/2/18
to Dataverse Users Community
Hi all,
thank you for all your reply. I confirm it's a issue regarding the localization. I set the locale to en_GB and now I'm able to build the docker image. I opened a Github issue with all information, you can find here: https://github.com/IQSS/dataverse/issues/4482
Now I'm going ahead following the guide:
  • sudo docker build -t dv0 -f c7.dockerfile . - OK
  • sudo docker run -d -p 8083:8080 --name dv dv0 - OK
  • sudo docker exec -it dv /opt/dv/setupIT.bash - OK
  • sudo docker exec -it dv /opt/dv/install.bash: is it necessary? it asks me if I want yo replace some files (i.e. glassfish-setup.sh), and tries to create user dvnapp that already exists. at the end I got several errors.
  • sudo docker exec -it dv /usr/local/glassfish4/bin/asadmin create-jvm-options "-Ddataverse.siteUrl=http\://localhost\:8083" - OK
  • sudo mvn test -Dtest=DataversesIT,DatasetsIT,SwordIT,AdminIT,BuiltinUsersIT,UsersIT,UtilIT,ConfirmEmailIT,FileMetadataIT -Ddataverse.test.baseurl='http://localhost:8083' - FAIL (is it important?)
After all the commands I got a DV session running on localhost:8083. I tried to reboot the system and now at localhost:8083 I just have the index page of glassfish server. Maybe I miss something?

Thank you,
Domingo

Philip Durbin

unread,
Mar 2, 2018, 8:01:10 AM3/2/18
to dataverse...@googlegroups.com
Hi Domingo,

Thank you very much for confirming that the unit test isn't working in your locale and for opening that GitHub issue about it. I'm happy that this bug is being tracked now.

I would still love to hear more about what you are trying to do at a high level. I'm glad you found our "all in one" Docker config and that you got a working Dataverse installation out of it, but please be advised that the primary purpose of that "all in one" Docker image is for developers to run run the integration test suite (as opposed to the unit test suite which runs automatically on every build) as described in the "improve ability of developers to run integration tests" issue at https://github.com/IQSS/dataverse/issues/4369 . You asked if it's important if the "IT" (integration test) tests are failing and I would say that as a developer it's certainly important to me if they are failing. I open GitHub issues if I notice any of theses tests failing and include in the issue the details of which tests are failing so we can fix them.

(Since you also asked if running "/opt/dv/install.bash" is necessary, I would say that it is not necessary and in an a commit that has not yet been merged at https://github.com/IQSS/dataverse/commit/e796935 I clarify that developers should not normally need to run that "install.bash" script.)

You're also asking about rebooting the system. To be perfectly honest, when I'm making use of the "all in one" Docker environment, I never reboot the system because I treat is as ephemeral. I always delete the Docker container once I can tell that the test are passing. Again, this is because the purpose of the "all in one" Docker environment is to be up long enough to run the integration test suite. Afterwards, it can be deleted.

Can you please tell us more about what your goals are? Are you a developer who wants to run the integration test suite before making a pull request? Are you a sysadmin who is evaluating Dataverse and considering a production deployment in the future? Do you have some other goal? We would like to provide you advice based on your needs.

Thanks!

Phil

To unsubscribe from this group and stop receiving emails from it, send an email to dataverse-community+unsubscribe...@googlegroups.com.

--
You received this message because you are subscribed to the Google Groups "Dataverse Users Community" group.
To unsubscribe from this group and stop receiving emails from it, send an email to dataverse-community+unsub...@googlegroups.com.
To post to this group, send email to dataverse-community@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/dataverse-community/8b35ec11-9199-46e8-bc65-077e1fecfe50%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

domingo...@unimib.it

unread,
Mar 5, 2018, 3:44:46 AM3/5/18
to Dataverse Users Community
Hi Phil,
thank you for your reply and sorry for delay in my reply. I work in a Social Science Data Archive here in Italy and we want to test Dataverse as a possible platform for data discovery and distribution. Now we use R for the data archiving process and Wordpress for data discovery, so what I need is a fresh installation of DV on my PC to test some procedures: populate (data and metadata) DV through R, send metadata to DV through XML (we use the DDI standard), modify metadata fields in DV, etc. I focused on Docker because I thought it was the right way to test DV on my machine in a simple way but, well, maybe I made the wrong decision... In the last days I was able to build a working vagrant image, so it seems the right tool for me (I hope!). I just have a little silly question about vagrant setup: which is the account to log in? I cannot find it in any place.
So, again, thank you all and fell free to advise me guides, tutorials or any kind of documentation that can help me in reaching my goals.

Bests,
Domingo
To unsubscribe from this group and stop receiving emails from it, send an email to dataverse-community+unsub...@googlegroups.com.
To post to this group, send email to dataverse...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/dataverse-community/61dd5c5c-105f-4402-a2ed-50abefb2b414%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "Dataverse Users Community" group.
To unsubscribe from this group and stop receiving emails from it, send an email to dataverse-community+unsub...@googlegroups.com.
To post to this group, send email to dataverse...@googlegroups.com.

Vyacheslav Tikhonov

unread,
Mar 5, 2018, 6:04:24 AM3/5/18
to Dataverse Users Community
Hi Domingo,

That's great news that you've managed to install Dataverse with Docker! Sorry, I was on holidays last week and didn't managed to provide support but this forum is definitely right place to ask questions.

To all: Domingo is also participating in DataverseEU project and will deliver Italian translations of Dataverse web interface and metadata schema.

Best,
Slava

Philip Durbin

unread,
Mar 5, 2018, 9:14:59 AM3/5/18
to dataverse...@googlegroups.com
Slava, thanks for the context and Domingo, thanks for letting us know that you're interested in a running Dataverse installation on your laptop for development purposes. You asked, which account to use to log in when using Vagrant. Are you talking about how to ssh into the Linux box running inside of Vagrant? If so, you should type `vagrant ssh` and you will be ssh'ed in as the user "vagrant" (and you can type `sudo -i` to become root if you need to). I should mention this at http://guides.dataverse.org/en/4.8.5/developers/tools.html#vagrant . If you are talking about how to log into the Dataverse web interface, you should use "dataverseAdmin" as the username and "admin" as the password. This is explained at http://guides.dataverse.org/en/4.8.5/installation/config.html#changing-the-superuser-password

I hope this helps,

Phil

Slava
To unsubscribe from this group and stop receiving emails from it, send an email to dataverse-community+unsubscribe...@googlegroups.com.

--
You received this message because you are subscribed to the Google Groups "Dataverse Users Community" group.
To unsubscribe from this group and stop receiving emails from it, send an email to dataverse-community+unsubscribe...@googlegroups.com.

To post to this group, send email to dataverse...@googlegroups.com.

--
You received this message because you are subscribed to the Google Groups "Dataverse Users Community" group.
To unsubscribe from this group and stop receiving emails from it, send an email to dataverse-community+unsub...@googlegroups.com.
To post to this group, send email to dataverse-community@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/dataverse-community/bdb5c2cc-2bad-42e2-b693-143cf41cda1c%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

domingo...@unimib.it

unread,
Mar 5, 2018, 9:28:05 AM3/5/18
to Dataverse Users Community
Thank you Phil for the information, that's what I was looking for.
Thank Slava for your hints, I will contact you soon for more help :)
And again thank all the people that replied to my questions, I really appreciated!

Best,
Domingo
Slava
To unsubscribe from this group and stop receiving emails from it, send an email to dataverse-community+unsub...@googlegroups.com.

--
You received this message because you are subscribed to the Google Groups "Dataverse Users Community" group.
To unsubscribe from this group and stop receiving emails from it, send an email to dataverse-community+unsub...@googlegroups.com.
To post to this group, send email to dataverse...@googlegroups.com.

--
You received this message because you are subscribed to the Google Groups "Dataverse Users Community" group.
To unsubscribe from this group and stop receiving emails from it, send an email to dataverse-community+unsub...@googlegroups.com.
To post to this group, send email to dataverse...@googlegroups.com.
Reply all
Reply to author
Forward
0 new messages