"Explore button" for tabular files in dataverse

165 views
Skip to first unread message

Michel Bamouni

unread,
Feb 16, 2017, 4:40:53 AM2/16/17
to Dataverse Users Community


Hello,

I am trying to set up tabular files exploring in dataverse 4.5.1 with tworavens.
So I install tworavens but when I execute the curl command
"curl -X PUT -d true http://localhost:8080/api/admin/settings/:TwoRavensTabularView",
I dont see the "Explore" button on the ".tab" files of a dataset.
In the setting table of the dataverse database, the ":TwoRavensTabularView" is set to "true"  and ":TwoRavensUrl" contains the URL of the tworavens I installed.

Can some one help me please?

Best regards ,

Michel

Philip Durbin

unread,
Feb 16, 2017, 7:23:59 AM2/16/17
to dataverse...@googlegroups.com
Hmm, as a sanity check can you please uploading a file that we know is expected to be successfully ingested to see if the "Explore" button shows up? Please try this one: https://github.com/IQSS/dataverse/raw/v4.6/scripts/search/data/tabular/50by1000.dta

Thanks!

Phil

p.s. You're not alone. It takes some effort to get TwoRavens and friends set up properly.

--
You received this message because you are subscribed to the Google Groups "Dataverse Users Community" group.
To unsubscribe from this group and stop receiving emails from it, send an email to dataverse-community+unsub...@googlegroups.com.
To post to this group, send email to dataverse-community@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/dataverse-community/1b3c0686-0e76-4c91-b0fc-c37ed584d7ff%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.



--

Donald Sizemore II

unread,
Feb 16, 2017, 10:16:33 AM2/16/17
to Dataverse Users Community
Hello,

As Phil suggests, it sounds like an ingest issue. At the time of the file upload, do you see server.log entries similar to

[#|2017-02-13T10:00:36.248-0500|INFO|glassfish 4.1|edu.harvard.iq.dataverse.ingest.IngestServiceBean|_ThreadID=63;_ThreadName=p: thread-pool-1; w: 2;_TimeMillis=1486998036248;_LevelValue=800;|
  Tabular data successfully ingested; DataTable with 41 variables produced.|#]

[#|2017-02-13T10:00:36.249-0500|INFO|glassfish 4.1|edu.harvard.iq.dataverse.ingest.IngestServiceBean|_ThreadID=63;_ThreadName=p: thread-pool-1; w: 2;_TimeMillis=1486998036249;_LevelValue=800;|
  Tab-delimited file produced: /tmp/tempTabfile.7152155681761864330.tab|#]

Donald

Michel Bamouni

unread,
Feb 20, 2017, 4:04:31 AM2/20/17
to Dataverse Users Community

Hello,

First of all, thanks Donald and Philip.
When , i try to upload the file https://github.com/IQSS/dataverse/raw/v4.6/scripts/search/data/tabular/50by1000.dta,
in the glassfish logs, I have this stack trace :
"
[2017-02-20T09:45:49.575+0100] [glassfish 4.1] [INFO] [] [edu.harvard.iq.dataverse.ingest.tabulardata.impl.plugins.dta] [tid: _ThreadID=63 _ThreadName=p: thread-pool-1; w: 5] [timeMillis: 1487580349575] [levelValue: 800] [[
  ***** DTAFileReader: read() end *****]]

[2017-02-20T09:45:49.576+0100] [glassfish 4.1] [INFO] [] [edu.harvard.iq.dataverse.ingest.IngestServiceBean] [tid: _ThreadID=63 _ThreadName=p: thread-pool-1; w: 5] [timeMillis: 1487580349576] [levelValue: 800] [[
  Tabular data successfully ingested; DataTable with 50 variables produced.]]

[2017-02-20T09:45:49.576+0100] [glassfish 4.1] [INFO] [] [edu.harvard.iq.dataverse.ingest.IngestServiceBean] [tid: _ThreadID=63 _ThreadName=p: thread-pool-1; w: 5] [timeMillis: 1487580349576] [levelValue: 800] [[
  Tab-delimited file produced: /tmp/tempTabfile.1274224583344179449.tab]]

[2017-02-20T09:45:50.381+0100] [glassfish 4.1] [SEVERE] [] [org.dataverse.unf.RoundRoutines] [tid: _ThreadID=63 _ThreadName=p: thread-pool-1; w: 5] [timeMillis: 1487580350381] [levelValue: 1000] [[
  RoundRoutines:decimal separator no in right place]]

[2017-02-20T09:45:50.382+0100] [glassfish 4.1] [INFO] [] [edu.harvard.iq.dataverse.ingest.IngestServiceBean] [tid: _ThreadID=63 _ThreadName=p: thread-pool-1; w: 5] [timeMillis: 1487580350382] [levelValue: 800] [[
  Ingest failure: Failed to save tabular data (datatable, datavariables, etc.) in the database. Clearing the datafile object.]]

[2017-02-20T09:45:50.396+0100] [glassfish 4.1] [INFO] [] [edu.harvard.iq.dataverse.ingest.IngestServiceBean] [tid: _ThreadID=63 _ThreadName=p: thread-pool-1; w: 5] [timeMillis: 1487580350396] [levelValue: 800] [[
  Unknown excepton saving ingested file; Sent push notification to the page.]]

[2017-02-20T09:45:50.406+0100] [glassfish 4.1] [INFO] [] [edu.harvard.iq.dataverse.ingest.IngestMessageBean] [tid: _ThreadID=63 _ThreadName=p: thread-pool-1; w: 5] [timeMillis: 1487580350406] [levelValue: 800] [[
  Error occurred during ingest job!]]
"
In the dataverse web interface, I have the message which is join to this post named "50by1000IngestingError.png".

It seems that the ingesting is Ok but error occur when trying to save data un the database.


Michel

Leonid Andreev

unread,
Feb 21, 2017, 2:07:16 PM2/21/17
to Dataverse Users Community
Hmm. Yes, it does look like the ingest process works properly - but then something bad happens during the database write.
And there's not much to work with - the error message there just says "something went wrong".
In case this is some weird state issue,  could you please just try to delete that file, restart glassfish (for good measure) and try to ingest it again?

The other file in that screenshot did not get ingested as tabular data either  (otherwise it would be showing the information about variables and observations under the file name; in the spot where the MD5 is shown now). So that's the reason why there's no TwoRavens link - it's only shown when a file is properly ingested, and the extra information about the data variables is stored in the database.

Leonid Andreev

unread,
Feb 21, 2017, 8:12:08 PM2/21/17
to Dataverse Users Community
Also, can you download the files?
- if you click on the "download button" for 50by1000.tab, do you get the file? (or an error of some kind?)

And if you get the file, what format is it in? - the original Stata file you uploaded... or a plain text, tab-delimited version of it?
Thank you! We'll figure it out...

Michel Bamouni

unread,
Feb 22, 2017, 4:36:15 AM2/22/17
to Dataverse Users Community

Hello,

I can download and delete the 50by1000.dta file.
When I open the download file, it's a tabular file but the content is not same as the original file.
I Join two images which display the content of the original and the downloaded file.

I also deleting the file and restart glassfish server but I git the same errors.

Michel

Michel Bamouni

unread,
Feb 22, 2017, 4:50:50 AM2/22/17
to Dataverse Users Community
Hello,

In addition to my previous post, in what tables , ingesting data are stored inthe dataverse database please?

Leonid Andreev

unread,
Feb 22, 2017, 1:02:01 PM2/22/17
to Dataverse Users Community
When a tabular ingest is successful, you should see a new entry in the "datatable" SQL table, and several entries in the "datavariable" table. For some variables there will be extra entries in the "summarystatistic" and "variablecategory" tables.

Leonid Andreev

unread,
Feb 22, 2017, 1:11:06 PM2/22/17
to Dataverse Users Community
The original (binary) file is in Stata format. The plain text, tab-delimited file you get when you click "download" is the data values extracted from the Stata file during ingest.
This is what the ingest is supposed to be doing. So this part seems to be working correctly. So it is the database save that is failing.
OK, let me think of what else to try, in order to diagnose this.
It's a strange condition though... A bit difficult for me to think of what can possibly be failing there, at that stage.

Could you please let me know what version of PostgresQL you are running?

Michel Bamouni

unread,
Feb 23, 2017, 2:34:49 AM2/23/17
to Dataverse Users Community
Hello Leonid,

I use postgresql 9.2


Best regards,

Michel

Leonid Andreev

unread,
Feb 23, 2017, 10:48:57 PM2/23/17
to Dataverse Users Community
Hello Michel,

What is the address of your Dataverse server?
thank you,
-Leonid

Michel Bamouni

unread,
Feb 27, 2017, 3:51:53 AM2/27/17
to Dataverse Users Community
Hi,

My dataverse can be access only in my organization intranet network. It was not open to internet.
Do you know if tworavens work with dataverse 4.5.1 and if ingesting tabular files work with this version of dataverse?

Best regards,

Philip Durbin

unread,
Feb 27, 2017, 10:42:16 AM2/27/17
to dataverse...@googlegroups.com
I think TwoRavens should work fine even if your Dataverse installation isn't open to the Internet.

I noticed you mentioned earlier* that you were seeing "RoundRoutines:decimal separator no in right place" which reminded me of https://github.com/IQSS/dataverse/issues/2620 but I don't know if the workaround in the issue, changing the "LANG" environment before starting Glassfish, will help: https://github.com/IQSS/dataverse/issues/2620

Phil

* https://groups.google.com/d/msg/dataverse-community/g5pHge-rtdE/NQXgm0WHBwAJ

--
You received this message because you are subscribed to the Google Groups "Dataverse Users Community" group.
To unsubscribe from this group and stop receiving emails from it, send an email to dataverse-community+unsub...@googlegroups.com.
To post to this group, send email to dataverse-community@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/dataverse-community/a0a51a6d-68ee-406f-bb86-5c957bbd38fe%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

Michel Bamouni

unread,
Mar 1, 2017, 10:10:52 AM3/1/17
to Dataverse Users Community, philip...@harvard.edu

Hi Phil,
It was effectively a decimal separator problem.
After set "LANG=us_US.UTF-8", I see the "Explore" button when uploading the 50by1000.dta file but I don't this button for other dta files as you can in the joined image.
What is the best "Lang" to set have every .tab files having "Explore" button please?

cdlt,

Michel
To post to this group, send email to dataverse...@googlegroups.com.

Philip Durbin

unread,
Mar 2, 2017, 9:07:38 AM3/2/17
to dataverse...@googlegroups.com
That's great! So setting the LANG environment variable allowed the file to be ingested. I don't really understand why, but it does seems like you've run into the same thing Leonhard Maylein ran into a while back so I left a comment at https://github.com/IQSS/dataverse/issues/2620#issuecomment-283434497

Next you seem to be asking, "What about all the files that failed ingest... can an attempt to re-ingest them be made?" The standard work around for this problem is to delete the file and then add it again. For files that have already been saved to a dataset, there is no button to click to attempt to re-ingest them. Maybe there should be. That's sort of what these issues are about:

- https://github.com/IQSS/dataverse/issues/2218
- https://github.com/IQSS/dataverse/issues/2105
- https://github.com/IQSS/dataverse/issues/3288

That said, none of these issues quite capture your problem so you should feel free to open a fresh one.

Phil

To unsubscribe from this group and stop receiving emails from it, send an email to dataverse-community+unsubscribe...@googlegroups.com.
To post to this group, send email to dataverse...@googlegroups.com.

--
You received this message because you are subscribed to the Google Groups "Dataverse Users Community" group.
To unsubscribe from this group and stop receiving emails from it, send an email to dataverse-community+unsub...@googlegroups.com.
To post to this group, send email to dataverse-community@googlegroups.com.
 To view this discussion on the web visit https://groups.google.com/d/msgid/dataverse-community/df8bf6ce-a3d5-4ebd-a542-4b71c4bffcef%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

Michel Bamouni

unread,
Mar 2, 2017, 11:48:20 AM3/2/17
to Dataverse Users Community, philip...@harvard.edu


Hello,

Thanks for your answer.
After clicking on the "Explore" button , I got an empty page on tworavens like the joinned picture.
In "/var/www/html/custom/log_dir", I can't any log files that can help to diagnose why nothinh appear in tworavens.
What must I do to got a result in tworavens?
Where can I see the logs of tworavens?

best regards,

Michel

To unsubscribe from this group and stop receiving emails from it, send an email to dataverse-community+unsub...@googlegroups.com.
To post to this group, send email to dataverse...@googlegroups.com.

--
You received this message because you are subscribed to the Google Groups "Dataverse Users Community" group.
To unsubscribe from this group and stop receiving emails from it, send an email to dataverse-community+unsub...@googlegroups.com.
To post to this group, send email to dataverse...@googlegroups.com.

Philip Durbin

unread,
Mar 2, 2017, 11:56:13 AM3/2/17
to dataverse...@googlegroups.com
Hmm, the "missing pebbles" problem. I've seen it myself but I'm not sure how to fix it. I don't think I've every actually gotten TwoRavens working completely. I'm hoping someone who has can jump in with some suggestions.

To unsubscribe from this group and stop receiving emails from it, send an email to dataverse-community+unsubscribe...@googlegroups.com.
To post to this group, send email to dataverse...@googlegroups.com.

--
You received this message because you are subscribed to the Google Groups "Dataverse Users Community" group.
To unsubscribe from this group and stop receiving emails from it, send an email to dataverse-community+unsubscribe...@googlegroups.com.

To post to this group, send email to dataverse...@googlegroups.com.

--
You received this message because you are subscribed to the Google Groups "Dataverse Users Community" group.
To unsubscribe from this group and stop receiving emails from it, send an email to dataverse-community+unsub...@googlegroups.com.
To post to this group, send email to dataverse-community@googlegroups.com.
 To view this discussion on the web visit https://groups.google.com/d/msgid/dataverse-community/ad4a8d9d-8814-4417-82fd-8cdc3280e87a%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

Michel Bamouni

unread,
Mar 3, 2017, 4:34:25 AM3/3/17
to Dataverse Users Community, philip...@harvard.edu

Hi,

Thanks for answer. Can someone who have tworavens works help me please?

Michel
To unsubscribe from this group and stop receiving emails from it, send an email to dataverse-community+unsub...@googlegroups.com.
To post to this group, send email to dataverse...@googlegroups.com.

--
You received this message because you are subscribed to the Google Groups "Dataverse Users Community" group.
To unsubscribe from this group and stop receiving emails from it, send an email to dataverse-community+unsub...@googlegroups.com.
To post to this group, send email to dataverse...@googlegroups.com.

--
You received this message because you are subscribed to the Google Groups "Dataverse Users Community" group.
To unsubscribe from this group and stop receiving emails from it, send an email to dataverse-community+unsub...@googlegroups.com.
To post to this group, send email to dataverse...@googlegroups.com.
Reply all
Reply to author
Forward
0 new messages