Data conversion is not working after upgrading from 4.7 to 4.10.1

57 views
Skip to first unread message

Ahmed Mahdy

unread,
Jun 4, 2019, 3:15:17 PM6/4/19
to Dataverse Users Community
I am dataverse newbie, I upgraded the server from 4.7 to 4.10.1. I am trying to upload an xlsx file to a dataset that I created. It gets ingestedcorrectly. First I can't find the file "Tab-delimited file produced" as per the server log, and when I try to browse it with Two Ravens I get an empty page


[2019-06-04T15:18:27.425-0300] [glassfish 4.1] [SEVERE] [] [edu.harvard.hul.ois.jhove] [tid: _ThreadID=47 _ThreadName=jk-connector(1)] [timeMillis: 1559672307425] [levelValue: 1000] [[
  Testing SEVERE level]]

[2019-06-04T15:18:29.442-0300] [glassfish 4.1] [INFO] [] [edu.harvard.iq.dataverse.ingest.IngestServiceBean] [tid: _ThreadID=49 _ThreadName=jk-connector(3)] [timeMillis: 1559672309442] [levelValue: 800] [[
  Ingest of 1 tabular data file(s) is in progress.]]

[2019-06-04T15:18:29.611-0300] [glassfish 4.1] [INFO] [] [edu.harvard.iq.dataverse.ingest.tabulardata.impl.plugins.xlsx] [tid: _ThreadID=66 _ThreadName=p: thread-pool-1; w: 5] [timeMillis: 1559672309611] [levelValue: 800] [[
  entering processSheet]]

[2019-06-04T15:18:29.921-0300] [glassfish 4.1] [WARNING] [] [edu.harvard.iq.dataverse.ingest.tabulardata.impl.plugins.xlsx] [tid: _ThreadID=66 _ThreadName=p: thread-pool-1; w: 5] [timeMillis: 1559672309921] [levelValue: 900] [[
  Null r attribute in the first row element!]]

[2019-06-04T15:18:29.922-0300] [glassfish 4.1] [INFO] [] [edu.harvard.iq.dataverse.ingest.tabulardata.impl.plugins.xlsx] [tid: _ThreadID=66 _ThreadName=p: thread-pool-1; w: 5] [timeMillis: 1559672309922] [levelValue: 800] [[
  Established variable (column) count: 3]]

[2019-06-04T15:18:30.147-0300] [glassfish 4.1] [INFO] [] [edu.harvard.iq.dataverse.ingest.IngestServiceBean] [tid: _ThreadID=66 _ThreadName=p: thread-pool-1; w: 5] [timeMillis: 1559672310147] [levelValue: 800] [[
  Tabular data successfully ingested; DataTable with 3 variables produced.]]

[2019-06-04T15:18:30.147-0300] [glassfish 4.1] [INFO] [] [edu.harvard.iq.dataverse.ingest.IngestServiceBean] [tid: _ThreadID=66 _ThreadName=p: thread-pool-1; w: 5] [timeMillis: 1559672310147] [levelValue: 800] [[
  Tab-delimited file produced: /tmp/data-7178841449422335317.tab]]

TwoRavens.JPG

and I get the following error in the log file.

[2019-06-04T15:47:46.897-0300] [glassfish 4.1] [SEVERE] [] [] [tid: _ThreadID=50 _ThreadName=Thread-9] [timeMillis: 1559674066897] [levelValue: 1000] [[
  org.rosuda.REngine.Rserve.RserveException: voidEval failed, request status: R parser: syntax error
        at org.rosuda.REngine.Rserve.RConnection.voidEval(RConnection.java:181)
        at edu.harvard.iq.dataverse.rserve.RemoteDataFrameService.runDataPreprocessing(RemoteDataFrameService.java:594)
        at edu.harvard.iq.dataverse.dataaccess.DataConverter.runFormatConversion(DataConverter.java:277)
        at edu.harvard.iq.dataverse.dataaccess.DataConverter.performFormatConversion(DataConverter.java:112)
        at edu.harvard.iq.dataverse.api.DownloadInstanceWriter.writeTo(DownloadInstanceWriter.java:138)
        at edu.harvard.iq.dataverse.api.DownloadInstanceWriter.writeTo(DownloadInstanceWriter.java:45)
        at org.glassfish.jersey.message.internal.WriterInterceptorExecutor$TerminalWriterInterceptor.invokeWriteTo(WriterInterceptorExecutor.java:263)
        at org.glassfish.jersey.message.internal.WriterInterceptorExecutor$TerminalWriterInterceptor.aroundWriteTo(WriterInterceptorExecutor.java:250)
        at org.glassfish.jersey.message.internal.WriterInterceptorExecutor.proceed(WriterInterceptorExecutor.java:162)
        at org.glassfish.jersey.server.internal.JsonWithPaddingInterceptor.aroundWriteTo(JsonWithPaddingInterceptor.java:106)
        at org.glassfish.jersey.message.internal.WriterInterceptorExecutor.proceed(WriterInterceptorExecutor.java:162)
        at org.glassfish.jersey.server.internal.MappableExceptionWrapperInterceptor.aroundWriteTo(MappableExceptionWrapperInterceptor.java:89)
        at org.glassfish.jersey.message.internal.WriterInterceptorExecutor.proceed(WriterInterceptorExecutor.java:162)
        at org.glassfish.jersey.message.internal.MessageBodyFactory.writeTo(MessageBodyFactory.java:1154)
        at org.glassfish.jersey.server.ServerRuntime$Responder.writeResponse(ServerRuntime.java:621)
        at org.glassfish.jersey.server.ServerRuntime$Responder.processResponse(ServerRuntime.java:377)
        at org.glassfish.jersey.server.ServerRuntime$Responder.process(ServerRuntime.java:367)
        at org.glassfish.jersey.server.ServerRuntime$1.run(ServerRuntime.java:274)
        at org.glassfish.jersey.internal.Errors$1.call(Errors.java:271)
        at org.glassfish.jersey.internal.Errors$1.call(Errors.java:267)
        at org.glassfish.jersey.internal.Errors.process(Errors.java:315)
        at org.glassfish.jersey.internal.Errors.process(Errors.java:297)
        at org.glassfish.jersey.internal.Errors.process(Errors.java:267)
        at org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:297)
        at org.glassfish.jersey.server.ServerRuntime.process(ServerRuntime.java:254)
        at org.glassfish.jersey.server.ApplicationHandler.handle(ApplicationHandler.java:1028)
        at org.glassfish.jersey.servlet.WebComponent.service(WebComponent.java:372)
        at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:381)
        at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:344)
        at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:221)
        at org.apache.catalina.core.StandardWrapper.service(StandardWrapper.java:1682)
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:344)
        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:214)
        at org.ocpsoft.rewrite.servlet.RewriteFilter.doFilter(RewriteFilter.java:226)
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:256)
        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:214)
        at edu.harvard.iq.dataverse.api.ApiBlockingFilter.doFilter(ApiBlockingFilter.java:162)
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:256)
        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:214)
        at edu.harvard.iq.dataverse.api.ApiRouter.doFilter(ApiRouter.java:30)
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:256)
        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:214)
        at org.apache.catalina.core.ApplicationDispatcher.doInvoke(ApplicationDispatcher.java:873)
        at org.apache.catalina.core.ApplicationDispatcher.invoke(ApplicationDispatcher.java:739)
        at org.apache.catalina.core.ApplicationDispatcher.processRequest(ApplicationDispatcher.java:575)
        at org.apache.catalina.core.ApplicationDispatcher.doDispatch(ApplicationDispatcher.java:546)
        at org.apache.catalina.core.ApplicationDispatcher.dispatch(ApplicationDispatcher.java:428)
        at org.apache.catalina.core.ApplicationDispatcher.forward(ApplicationDispatcher.java:378)
        at edu.harvard.iq.dataverse.api.ApiRouter.doFilter(ApiRouter.java:34)
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:256)
        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:214)
        at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:316)
        at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:160)
        at org.apache.catalina.core.StandardPipeline.doInvoke(StandardPipeline.java:734)
        at org.apache.catalina.core.StandardPipeline.invoke(StandardPipeline.java:673)
        at com.sun.enterprise.web.WebPipeline.invoke(WebPipeline.java:99)
        at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:174)
        at org.apache.catalina.connector.CoyoteAdapter.doService(CoyoteAdapter.java:415)
        at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:282)
        at com.sun.enterprise.v3.services.impl.ContainerMapper$HttpHandlerCallable.call(ContainerMapper.java:459)
        at com.sun.enterprise.v3.services.impl.ContainerMapper.service(ContainerMapper.java:167)
        at org.glassfish.grizzly.http.server.HttpHandler.runService(HttpHandler.java:201)
        at org.glassfish.grizzly.http.server.HttpHandler.doHandle(HttpHandler.java:175)
        at org.glassfish.grizzly.http.server.HttpServerFilter.handleRead(HttpServerFilter.java:235)
        at org.glassfish.grizzly.filterchain.ExecutorResolver$9.execute(ExecutorResolver.java:119)
        at org.glassfish.grizzly.filterchain.DefaultFilterChain.executeFilter(DefaultFilterChain.java:284)
        at org.glassfish.grizzly.filterchain.DefaultFilterChain.executeChainPart(DefaultFilterChain.java:201)
        at org.glassfish.grizzly.filterchain.DefaultFilterChain.execute(DefaultFilterChain.java:133)
        at org.glassfish.grizzly.filterchain.DefaultFilterChain.process(DefaultFilterChain.java:112)
        at org.glassfish.grizzly.ProcessorExecutor.execute(ProcessorExecutor.java:77)
        at org.glassfish.grizzly.nio.transport.TCPNIOTransport.fireIOEvent(TCPNIOTransport.java:561)
        at org.glassfish.grizzly.strategies.AbstractIOStrategy.fireIOEvent(AbstractIOStrategy.java:112)
        at org.glassfish.grizzly.strategies.WorkerThreadIOStrategy.run0(WorkerThreadIOStrategy.java:117)
        at org.glassfish.grizzly.strategies.WorkerThreadIOStrategy.access$100(WorkerThreadIOStrategy.java:56)
        at org.glassfish.grizzly.strategies.WorkerThreadIOStrategy$WorkerThreadRunnable.run(WorkerThreadIOStrategy.java:137)
        at org.glassfish.grizzly.threadpool.AbstractThreadPool$Worker.doWork(AbstractThreadPool.java:565)
        at org.glassfish.grizzly.threadpool.AbstractThreadPool$Worker.run(AbstractThreadPool.java:545)
        at java.lang.Thread.run(Thread.java:748)]]

[2019-06-04T15:47:46.916-0300] [glassfish 4.1] [WARNING] [] [edu.harvard.iq.dataverse.dataaccess] [tid: _ThreadID=50 _ThreadName=jk-connector(4)] [timeMillis: 1559674066916] [levelValue: 900] [[
  Format-converted file was not properly created.]]


Any direction about debugging this issue will be appreciated.

Ahmed Mahdy

unread,
Jun 4, 2019, 3:29:11 PM6/4/19
to Dataverse Users Community
an update, the tabular file gets created in the same path "/tmp" but with a different name

Philip Durbin

unread,
Jun 4, 2019, 3:30:49 PM6/4/19
to dataverse...@googlegroups.com
Hi! My first thought is for you to try uploading your xlsx file to https://demo.dataverse.org to see if it ingests properly or not.

--
You received this message because you are subscribed to the Google Groups "Dataverse Users Community" group.
To unsubscribe from this group and stop receiving emails from it, send an email to dataverse-commu...@googlegroups.com.
To post to this group, send email to dataverse...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/dataverse-community/e7b12781-9c50-42b1-bdb5-1025904e7a0a%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.


--

Ahmed Mahdy

unread,
Jun 5, 2019, 8:38:35 AM6/5/19
to Dataverse Users Community
I just tried on https://demo.dataverse.org and it works fine. any other thoughts? 
To unsubscribe from this group and stop receiving emails from it, send an email to dataverse-community+unsub...@googlegroups.com.

Don Sizemore

unread,
Jun 5, 2019, 8:41:41 AM6/5/19
to dataverse...@googlegroups.com
Ahmed,

Additional R libraries are required as of Dataverse 4.9.mumble. Just to verify you've installed these?

Don

To unsubscribe from this group and stop receiving emails from it, send an email to dataverse-commu...@googlegroups.com.

--
You received this message because you are subscribed to the Google Groups "Dataverse Users Community" group.
To unsubscribe from this group and stop receiving emails from it, send an email to dataverse-commu...@googlegroups.com.

To post to this group, send email to dataverse...@googlegroups.com.

Ahmed Mahdy

unread,
Jun 5, 2019, 8:54:27 AM6/5/19
to Dataverse Users Community
Hi Don,

            I just checked they are installed, Will the versions of them make any difference? and during the upgrade, I only installed haven during the upgrade from v4.9.1 to v4.9.2

these are the versions of my packages

R2HTML        "R2HTML"        "/usr/lib64/R/library" "2.3.2"
rjson              "rjson"               "/usr/lib64/R/library" "0.2.15"
DescTools     "DescTools"     "/usr/lib64/R/library" "0.99.19"
Rserve          "Rserve"        "/usr/lib64/R/library" "1.7-3"
haven           "haven"         "/usr/lib64/R/library" "2.1.0"

Thanks,
Ahmed
To unsubscribe from this group and stop receiving emails from it, send an email to dataverse-community+unsub...@googlegroups.com.

--
You received this message because you are subscribed to the Google Groups "Dataverse Users Community" group.
To unsubscribe from this group and stop receiving emails from it, send an email to dataverse-community+unsub...@googlegroups.com.

Ahmed Mahdy

unread,
Jun 6, 2019, 9:42:42 AM6/6/19
to Dataverse Users Community
Any other thoughts on this issue?

Philip Durbin

unread,
Jun 6, 2019, 2:28:03 PM6/6/19
to dataverse...@googlegroups.com
Hi Ahmed,

I guess I have two thoughts on this.

One thought I have is that if it's ok for your file to be public, you could upload it to https://github.com/IQSS/dataverse/issues/5896 so we can test with it in the future when we work on adding tests for Excel .xlsx files. (I have no idea when that would happen, though.)

My other thought is that I'm wondering if you could test a .xlsx file on went through successful ingest on Dataverse such as https://dataverse.harvard.edu/file.xhtml?persistentId=doi:10.7910/DVN/W3NJYY/RFIMVA on your own server. I just spun up an installation of Dataverse using https://github.com/IQSS/dataverse-ansible and uploaded that "unchangedoriginaldata.xlsx" Excel file. Ingest was successful, as shown in the attached screenshot.

I hope this helps,

Phil



On Thu, Jun 6, 2019 at 9:42 AM Ahmed Mahdy <ahmedsam...@gmail.com> wrote:
Any other thoughts on this issue?

--
You received this message because you are subscribed to the Google Groups "Dataverse Users Community" group.
To unsubscribe from this group and stop receiving emails from it, send an email to dataverse-commu...@googlegroups.com.

To post to this group, send email to dataverse...@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.
Screen Shot 2019-06-06 at 2.22.14 PM.png

Ahmed Mahdy

unread,
Jun 6, 2019, 2:48:56 PM6/6/19
to Dataverse Users Community
Hi Philip,

              Thanks for the reply, my xlsx file is a test file and I can upload it no problem, I downloaded the file you used and uploaded it to my server but I still have the same problem. here is a screenshot of my upload. I can upload and download the files with no issues, the errors I have in my original message happens when I click on the explore button. I believe there is a problem in my installation/upgrade and I am trying to figure a way to debug the issue but I don't know how.


Files.jpg



On Thursday, 6 June 2019 15:28:03 UTC-3, Philip Durbin wrote:
Hi Ahmed,

I guess I have two thoughts on this.

One thought I have is that if it's ok for your file to be public, you could upload it to https://github.com/IQSS/dataverse/issues/5896 so we can test with it in the future when we work on adding tests for Excel .xlsx files. (I have no idea when that would happen, though.)

My other thought is that I'm wondering if you could test a .xlsx file on went through successful ingest on Dataverse such as https://dataverse.harvard.edu/file.xhtml?persistentId=doi:10.7910/DVN/W3NJYY/RFIMVA on your own server. I just spun up an installation of Dataverse using https://github.com/IQSS/dataverse-ansible and uploaded that "unchangedoriginaldata.xlsx" Excel file. Ingest was successful, as shown in the attached screenshot.

I hope this helps,

Phil



On Thu, Jun 6, 2019 at 9:42 AM Ahmed Mahdy <ahmedsam...@gmail.com> wrote:
Any other thoughts on this issue?

--
You received this message because you are subscribed to the Google Groups "Dataverse Users Community" group.
To unsubscribe from this group and stop receiving emails from it, send an email to dataverse-community+unsub...@googlegroups.com.

Ahmed Mahdy

unread,
Jun 6, 2019, 3:27:04 PM6/6/19
to Dataverse Users Community
I found something else, When I upload the file to the dataset, there are two files created under /tmp, the owner of one of them is root. Is that a problem?

-rw-r--r--     1 root   root             64427 Jun  6 16:18 firstpass-6481046310518105467.tab
drwxr-xr-x. 5 rserve rserve        4096   Jun  6 16:18 Rserv
-rw-rw-rw-  1 rserve rserve        69313 Jun  6 16:18 dataverseTabData_.733601.tab

Philip Durbin

unread,
Jun 6, 2019, 3:42:37 PM6/6/19
to dataverse...@googlegroups.com
I think I got confused. I thought ingest of Excel .xlsx files was failing, which is done by Dataverse. Ingest seems to be working fine for you. Great.

You're having a TwoRavens problem, right? I think I'm going to echo what Danny said at https://groups.google.com/d/msg/dataverse-community/vYfYqsL1DTU/a2HVkqhOBAAJ which is "You may get an answer here, but the TwoRavens support is provided by a different group. I'd suggest reaching out to the group here: http://2ra.vn/community/index.html ."

Over at http://guides.dataverse.org/en/4.14/installation/r-rapache-tworavens.html we try to explain (in bold) that "support for TwoRavens is somewhat limited at the moment."

Perhaps someone can help here but you might want to ask at https://groups.google.com/forum/#!forum/2ravn which is the group linked on that "community" page. The also have an issue tracker at https://github.com/TwoRavens/TwoRavens

Have you considered using Data Explorer instead of TwoRavens? Since Data Explorer is listed at http://guides.dataverse.org/en/4.10.1/installation/external-tools.html I assume it works with Dataverse 4.10.1.

I hope this helps,

Phil
On Thu, Jun 6, 2019 at 3:27 PM Ahmed Mahdy <ahmedsam...@gmail.com> wrote:
I found something else, When I upload the file to the dataset, there are two files created under /tmp, the owner of one of them is root. Is that a problem?

-rw-r--r--     1 root   root             64427 Jun  6 16:18 firstpass-6481046310518105467.tab
drwxr-xr-x. 5 rserve rserve        4096   Jun  6 16:18 Rserv
-rw-rw-rw-  1 rserve rserve        69313 Jun  6 16:18 dataverseTabData_.733601.tab

On Thursday, 6 June 2019 15:48:56 UTC-3, Ahmed Mahdy wrote:
Hi Philip,

              Thanks for the reply, my xlsx file is a test file and I can upload it no problem, I downloaded the file you used and uploaded it to my server but I still have the same problem. here is a screenshot of my upload. I can upload and download the files with no issues, the errors I have in my original message happens when I click on the explore button. I believe there is a problem in my installation/upgrade and I am trying to figure a way to debug the issue but I don't know how.


Files.jpg



On Thursday, 6 June 2019 15:28:03 UTC-3, Philip Durbin wrote:
Hi Ahmed,

I guess I have two thoughts on this.

One thought I have is that if it's ok for your file to be public, you could upload it to https://github.com/IQSS/dataverse/issues/5896 so we can test with it in the future when we work on adding tests for Excel .xlsx files. (I have no idea when that would happen, though.)

My other thought is that I'm wondering if you could test a .xlsx file on went through successful ingest on Dataverse such as https://dataverse.harvard.edu/file.xhtml?persistentId=doi:10.7910/DVN/W3NJYY/RFIMVA on your own server. I just spun up an installation of Dataverse using https://github.com/IQSS/dataverse-ansible and uploaded that "unchangedoriginaldata.xlsx" Excel file. Ingest was successful, as shown in the attached screenshot.

I hope this helps,

Phil



On Thu, Jun 6, 2019 at 9:42 AM Ahmed Mahdy <ahmedsam...@gmail.com> wrote:
Any other thoughts on this issue?

--
You received this message because you are subscribed to the Google Groups "Dataverse Users Community" group.
To unsubscribe from this group and stop receiving emails from it, send an email to dataverse-commu...@googlegroups.com.

--
You received this message because you are subscribed to the Google Groups "Dataverse Users Community" group.
To unsubscribe from this group and stop receiving emails from it, send an email to dataverse-commu...@googlegroups.com.

To post to this group, send email to dataverse...@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

Ahmed Mahdy

unread,
Jun 6, 2019, 5:01:17 PM6/6/19
to Dataverse Users Community
Thanks, Philip.

I tried dataexplorer. it gave me an error in the beginning -----> {"status":"ERROR","message":"Unknown reserved word: {localeCode}"}
so I removed the parameter ""dvLocale": "{localeCode}"" from the json file and it worked but only for the newly ingested file, the files that were ingested before the upgrade are not working. Thanks for your help and sorry for the confusion.
To unsubscribe from this group and stop receiving emails from it, send an email to dataverse-community+unsub...@googlegroups.com.

--
You received this message because you are subscribed to the Google Groups "Dataverse Users Community" group.
To unsubscribe from this group and stop receiving emails from it, send an email to dataverse-community+unsub...@googlegroups.com.

Philip Durbin

unread,
Jun 6, 2019, 5:19:01 PM6/6/19
to dataverse...@googlegroups.com
Ok, I reported a localeCode issue at https://github.com/scholarsportal/Dataverse-Data-Explorer/issues/14 . I would suggest leaving a comment on that issue.

Also, you might be interested in these for your files that were ingested before you upgrade:


Phil

To unsubscribe from this group and stop receiving emails from it, send an email to dataverse-commu...@googlegroups.com.

--
You received this message because you are subscribed to the Google Groups "Dataverse Users Community" group.
To unsubscribe from this group and stop receiving emails from it, send an email to dataverse-commu...@googlegroups.com.

--
You received this message because you are subscribed to the Google Groups "Dataverse Users Community" group.
To unsubscribe from this group and stop receiving emails from it, send an email to dataverse-commu...@googlegroups.com.

To post to this group, send email to dataverse...@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.
Reply all
Reply to author
Forward
0 new messages