RData Error

[Santiago Florez]

Hi everyone, me again.

I got an error when i try to download a Rdata file. This is the error"{"status":"ERROR","code":404,"message":"datafile access error: requested optional service (image scaling, format conversion, etc.) could not be performed on this datafile."}". And this is the log message .

"[2023-03-28T18:56:32.108+0000] [Payara 5.2022.3] [SEVERE] [] [edu.harvard.iq.dataverse.rserve] [tid: _ThreadID=77 _ThreadName=http-thread-pool::http-listener-1(1)] [timeMillis: 1680029792108] [levelValue: 1000] [[
  login failed, request status: authorization failed]]"

Maybe if i change the password or something like that. What you think that i can to do?

[Philip Durbin]

Huh. Yeah, maybe the password? As you probably know, there are some docs here: https://guides.dataverse.org/en/5.13/installation/prerequisites.html#rserve

Was it working before?

--
You received this message because you are subscribed to the Google Groups "Dataverse Users Community" group.
To unsubscribe from this group and stop receiving emails from it, send an email to dataverse-commu...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/dataverse-community/a8082733-88c5-4d5b-863f-87dd6be58f9fn%40googlegroups.com.

--

Philip Durbin
Software Developer for http://dataverse.org
http://www.iq.harvard.edu/people/philip-durbin

[Santiago Florez]

Hi Philip, when I did the installation I leaved the user and the password with the default values. 

[Santiago Florez]

Hello I have been trying to correct the problem with RData, but I have not found a way to do it.
I've seen this documentation about R and Dataverse "https://cran.r-project.org/web/packages/dataverse/readme/README.html" and thought about writing a script in R and using it with JavaScript.
Or what other way can I use to make RData work?

I really need that RData work

[Don Sizemore]

Hello,

Have you double-checked the password specified in /etc/Rserv.pwd?

Thank you,

Don

To view this discussion on the web visit https://groups.google.com/d/msgid/dataverse-community/acf49abb-ea23-444c-8544-d274d3a95f36n%40googlegroups.com.

[Philip Durbin]

Hi Santiago,

Leonid posted a reply but something happened to it, I think. Here it is:

"But did you leave the default password unchanged on the R side as well, when you installed Rserve?
It's easy to check, the username and password are saved in /etc/Rserv.pwd.

Please note, that this Rserve subsystem is really old and really obsolete by now. We are seriously considering dropping it from the application. In all honesty, that "download as RData" option does not provide that much useful value - it should be almost just as easy for any R user to download the tab-delimited version of the data file and import it into R.

best,
-L.A."

I hope this helps!

Phil

To view this discussion on the web visit https://groups.google.com/d/msgid/dataverse-community/acf49abb-ea23-444c-8544-d274d3a95f36n%40googlegroups.com.

[Santiago Florez]

Hi Don and Phil

Yeah I checked the file and both are by the default value when I installed dataverse 

[Vaidas Morkevičius]

Hello,

> In all honesty, that "download as RData" option does not provide that much useful value - it should be almost just as easy for any R user to download the tab-delimited version of the data file and import it into R.

I would not agree as saving plain text tab-delimited file you loose variable/value labels (in case they exist in the original file), and saving in R format they are preserved. This is important for analysis of the data, especially, survey data. Possibility to save a file (with variable and value labels) in a tab-delimited format with info about variable/value labels would be a welcome option, as it would be "most software independent".

--
Vaidas Morkevičius

On Tue, Mar 28, 2023 at 11:36 PM leo...g.harvard.edu <leo...@g.harvard.edu> wrote:

But did you leave the default password unchanged on the R side as well, when you installed Rserve?

It's easy to check, the username and password are saved in /etc/Rserv.pwd.

Please note, that this Rserve subsystem is really old and really obsolete by now. We are seriously considering dropping it from the application. In all honesty, that "download as RData" option does not provide that much useful value - it should be almost just as easy for any R user to download the tab-delimited version of the data file and import it into R. 

best,

-L.A. 

To view this discussion on the web visit https://groups.google.com/d/msgid/dataverse-community/559aea12-5b09-4209-8ecd-53330a71bc07n%40googlegroups.com.

[Leonid Andreev]

Hello, 

From Vaidas Morkevičius:

> I would not agree as saving plain text tab-delimited file you loose variable/value labels (in case they exist in the original file), and saving in R format they are preserved. This is important for analysis of the data, especially, survey data. Possibility to save a file (with variable and value labels) in a tab-delimited format with info about variable/value labels would be a welcome option, as it would be "most software independent".

This is of course correct. But, only with some qualifiers. For starters, this extra information, such as variable and/or value labels, only exists in the variable-level metadata on the Dataverse side for *some* tabular files, depending on the format of the original ingested file. For example, here at our own production Dataverse, something like 70% of all the recently created tab. files were ingested from CSV. Therefore none of these have any labels or categorical values - or anything really except for the variable names and guessed types; so there's nothing extra to preserve in an RData frame.

If the original was Stata (our second most popular format here) - the R user can have all the extra labels and such preserved by downloading the .dta original and importing it into R directly. R has excellent Stata import support via the package "foreign". (As a matter of fact, when Dataverse serves download-as-RData for a tab. file with a Stata original, that's exactly what it does behind the scenes - it makes R import the .dta and save it as an R frame!). 

If the original was RData itself... then the whole point is moot. 

So this only leaves SPSS as a rich ingestable format that could benefit from having any extra metadata preserved in an RData frame - and how many people are still using SPSS, really? (that was a non-rhetorical question btw; I know we have fewer and fewer SPSS files uploaded here - but I'm assuming it could be different at other installations - ?).  

But then, whatever value this potentially provides, having these SPSS labels and categories preserved in an R file, it is a bit outweighed by the fact that the whole subsystem is just unreliable and flaky, and works on some files but not on others, unpredictably. We use some rather messy R code, to try to convert categorical variables into R factors with proper labels... but messy it is and none of the current core developers are brave enough to attempt to work on it...

All that said, I personally didn't feel it was necessary to retire this service until recently. I felt like flaky as it was, it wasn't hurting really, and it was providing something useful for at least a few files. But then this has been subject of some lively debate (in GitHub issues and elsewhere), and we've had some users explicitly ask us to drop it. 

I would happily agree that a good alternative to retiring this service would be improving it, making it more reliable, updating the R components behind it and making it produce better and more useful R data frames... (and for those who actively dislike the idea of their Dataverse instances serving RData, for whatever reason, we could just make it easy to disable, optionally). But, this effort would need to come from the community. Because the core team has no resources to invest into it. 

Best,

-L.A.

On Friday, March 31, 2023 at 10:00:11 AM UTC-4 vaidas.mo...@gmail.com wrote:

Hello,

> In all honesty, that "download as RData" option does not provide that much useful value - it should be almost just as easy for any R user to download the tab-delimited version of the data file and import it into R.

I would not agree as saving plain text tab-delimited file you loose variable/value labels (in case they exist in the original file), and saving in R format they are preserved. This is important for analysis of the data, especially, survey data. Possibility to save a file (with variable and value labels) in a tab-delimited format with info about variable/value labels would be a welcome option, as it would be "most software independent".

--
Vaidas Morkevičius

On Tue, Mar 28, 2023 at 11:36 PM leo...g.harvard.edu <leo...@g.harvard.edu> wrote:

But did you leave the default password unchanged on the R side as well, when you installed Rserve?

It's easy to check, the username and password are saved in /etc/Rserv.pwd.

Please note, that this Rserve subsystem is really old and really obsolete by now. We are seriously considering dropping it from the application. In all honesty, that "download as RData" option does not provide that much useful value - it should be almost just as easy for any R user to download the tab-delimited version of the data file and import it into R. 

best,

-L.A. 

On Tuesday, March 28, 2023 at 3:26:15 PM UTC-4 florezsa...@gmail.com wrote:

Hi Philip, when I did the installation I leaved the user and the password with the default values. 

...

El martes, 28 de marzo de 2023 a la(s) 14:20:06 UTC-5, Philip Durbin escribió:

Huh. Yeah, maybe the password? As you probably know, there are some docs here: https://guides.dataverse.org/en/5.13/installation/prerequisites.html#rserve

Was it working before?

On Tue, Mar 28, 2023 at 3:03 PM Santiago Florez <florezsa...@gmail.com> wrote:

Hi everyone, me again.

I got an error when i try to download a Rdata file. This is the error"{"status":"ERROR","code":404,"message":"datafile access error: requested optional service (image scaling, format conversion, etc.) could not be performed on this datafile."}". And this is the log message .

"[2023-03-28T18:56:32.108+0000] [Payara 5.2022.3] [SEVERE] [] [edu.harvard.iq.dataverse.rserve] [tid: _ThreadID=77 _ThreadName=http-thread-pool::http-listener-1(1)] [timeMillis: 1680029792108] [levelValue: 1000] [[
  login failed, request status: authorization failed]]"

Maybe if i change the password or something like that. What you think that i can to do?

--
You received this message because you are subscribed to the Google Groups "Dataverse Users Community" group.
To unsubscribe from this group and stop receiving emails from it, send an email to dataverse-commu...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/dataverse-community/a8082733-88c5-4d5b-863f-87dd6be58f9fn%40googlegroups.com.

--

Philip Durbin
Software Developer for http://dataverse.org
http://www.iq.harvard.edu/people/philip-durbin

--
You received this message because you are subscribed to the Google Groups "Dataverse Users Community" group.
To unsubscribe from this group and stop receiving emails from it, send an email to dataverse-commu...@googlegroups.com.

[Leonid Andreev]

Also, this may not be super helpful (and/or it may already be obvious to everybody), but I want to point out that if there is a situation where it is absolutely important to have downloadable R data frames for some specific tab. files, there is always a dumb workaround for achieving that: as long as it is possible to generate that RData file outside of Dataverse somehow, you can create it and simply drop it in storage, and the application will proceed to serve it for downloads (regardless of whether R and Rserve are installed and working). 

The file needs to be saved under the same name as the main file with the .RData extension. For example, if the physical tabular file is /usr/local/payara5/glassfish/domains/domain1/files/10.11111/FK2/XXXXXX/yyyyy-zzzzz, the cached file will need to be placed in that directory as yyyyy-zzzzz.RData. 

[Victoria Lubitch]

Hi Leonid, after moving from Nesstar to Dataverse for some of our collections, we lost the ability that Nesstar had, i.e for users to download in preferred statistical format irrespective what original file was deposited. Most of our users are really missing this feature and would benefit from having this available in Dataverse. Many of our users are not R users.

We were thinking that R can convert these formats to each other (SPSS to STATA, STATA to SPSS, to SAS , xslx, R, csv, txt and versions) and Dataverse can handle these conversions through already existing Rserve. But it seems that it needs improvements as you suggested above.

Yes, our researches are still using SPSS, STATA, SAS and R.The University of Toronto and many big institutions are licensing SPSS for use. Hence the reason we would want these converters to exists in Dataverse and in particular R would be important for these kind of conversions to support the community with more open tools for converters . 

DDI and R community already built tools for converting different formats that can be reused. For example  DDIwR package   https://github.com/dusadrian/DDIwR Therefore we were thinking adding this functionality to dataverse, but we want to do it in collaboration and insuring that R is still available and maintained, since R is so good with statistical data and in future more analysis tools can be added to dataverse.Our tools such as Data Explorer could leverage of R integration for statistical analysis. 

Regards,

Victoria

Scholars Portal

Borealis team