packrat restore windows (always problems)

[Adam Robinson]

I work in a company where we use R and R Studio on windows desktop and windows server environments. We try to use packrat for all projects to ensure that when collaborating or re-running old analyses we have the same versions. We generally use packrat with its defaults where we keep the srcs and git ignore the libs. 

The more we use packrat I consistently see that the main problems are around restoring from source. Particularly packages such as XML, stringi and stringr tend to always fail in restoring from source. This results in errors in the packrat process and forces us to eventually ignore these libraries from packrat tracking (via external packages options). 

Question is, what is the ideal course of action in this scenario? I think the ideal would be that the lock file would suffice and packrat would find the right version of the binaries online to install. From what I can tell though this is not how cran works. Another alternative would be to not git ignore the lib folders. I'm hesitant to do that though as it would bloat the project size and I'm not clear how it would work cross machine time.

Appreciate any advice on best ways of working here. I'm also considering docker but shorter term would like to solve the windows problem.

Adam 

[Kevin Ushey]

Hi Adam,

This is indeed one of the main pain points of working with Packrat. Because CRAN does not maintain older binary copies of packages, we instead attempt to install older copies of packages from source. This can fail if the system pre-requisites for compiling a particular package (Rtools; external libraries) are not available.

If we could go back in time, we'd likely make Packrat download and save binary copies of packages by default, so that Packrat could always restore with a local binary copy of the package if needed.

In theory, we could use MRAN to find older copies of binary packages as needed, but I'm not sure if there's a natural way to do this. (MRAN provides snapshots of CRAN at particular points in time, but I'm not sure if it's possible to discover what particular timepoint(s) of MRAN provide what particular version(s) of packages.)

For what it's worth, on Windows, binary copies of packages tend to be quite compatible across systems, so you could likely share your libraries across systems. (Packrat doesn't provide any helpers to make this easier, unfortunately)

Best,

Kevin

--
You received this message because you are subscribed to the Google Groups "packrat-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to packrat-discu...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Adam Robinson]

Hi Kevin,

Thanks for your reply. What do you think this means for a reproducible R programming pattern on windows generally? It seems to me like using packrat with these limitations on windows means that the core use of restoring an environment is not really possible if you use packrat like it is sold (and you'd expect to use on linux).

To get something working on Windows I guess the solution could be locking down to a particular MRAN snapshot where binaries are available. Currently don't see a clear pattern for this - would it make sense for windows users if there was something in the packrat opts to force snapshotted libraries to be from a set MRAN snapshot? The ideal would be some index of all the MRAN snapshots like you said.

The other solution like I said is docker containers or a linux r studio server but I wouldn't want to give up on a windows solution just yet,

Regards,

Adam

[Joe Cheng]

Are they failing to install from source simply because Rtools isn't installed? Or are there some other random system dependencies involved? A lot of packages these days seem to do a good job of installing from source on Windows even though they have random hard to obtain system dependencies, thanks to 

https://github.com/rwinlib.

[Adam Robinson]

Hi Joe,

This could actually be down to r tools (hopefully)! R tools has been installed and devtools::find_rtools() finds it as such. That said when i follow this link for some tests to see if its actually working: https://github.com/stan-dev/rstan/wiki/Install-Rtools-for-Windows i get errors on both  system('g++ -v') and system('where make'). 

Part of the problem here may be that i dont think we can edit the system path on our devices. We instead have a user path. Having checked the both i found that C:\Rtools\mingw_32 and C:\Rtools\mingw_64 were not in there even when i check add to path in the installer. Having added them to the user path the above command now works. I'll do some reading on what is actually meant to be in the mingw directories.

Hopefully this solves the problem with the packages above - i will keep a watch out. It might be an idea to have the devtools::find_rtools do a couple more checks such as those above as i think i've gone about a year thinking r tools was working fine :/. 

Thanks both for your help,

Adam 

[Adam Robinson]

Spoke too soon - get an error with openssl. It seems as though even though i've installed r tools on the machine it needs to go off to github to get some more libraries for openssl. The machine this is on doesnt is offline so its unable to reach github. I wouldve thought that rtools included all of these libraries already? Potentially another path issue or is it expected that you may need to install other libraries? 

Error: Command failed (1)

Failed to run system command:

"C:/PROGRA~1/R/R-34~1.2/bin/x64/R" --vanilla CMD INSTALL "C:\Users\AROBINSON4\AppData\Local\Temp\16\Rtmp2nMqHF\openssl" --library="C:/Users/AROBINSON4/Desktop/R Projects/test/packrat/lib/x86_64-w64-mingw32/3.4.2" --install-tests --no-docs --no-multiarch --no-demo 

The command failed with output:

* installing *source* package 'openssl' ...

** package 'openssl' successfully unpacked and MD5 sums checked

** libs

rm -f openssl.dll aes.o base64.o bignum.o cert.o compatibility.o diffie.o envelope.o error.o hash.o info.o keygen.o keys.o onload.o openssh.o password.o pem.o pkcs12.o pkcs7.o rand.o rsa.o signing.o ssl.o stream.o write.o win32/ipv6.o

"C:/PROGRA~1/R/R-34~1.2/bin/x64/Rscript.exe" "../tools/winlibs.R" 1.1.0f

Error in download.file(sprintf("https://github.com/rwinlib/openssl/archive/v%s.zip",  : 

  cannot open URL 'https://github.com/rwinlib/openssl/archive/v1.1.0f.zip'

In addition: Warning message:

In do

[Kevin Ushey]

Rtools only contains the libraries necessary to build R itself (and the compiler toolchain used for compiling C / C++ / Fortran sources in general); R packages often depend on libraries external to Rtools and need separate installation / configuration.

[Joe Cheng]

Ah, combined with offline it becomes a harder problem. You might consider the (commercially licensed) RStudio Package Manager, which is designed for these kinds of scenarios; if it isn't able to meet your needs I'm sure the team would be very interested to hear about it.

https://www.rstudio.com/products/package-manager/

[Adam Robinson]

Hi Both, 

Following on from this question I've noticed that there is a new way to install packages being developed via r-lib called pak https://github.com/r-lib/pak and this depends on something called pkgbuild https://github.com/r-lib/pkgbuild which looks to be trying to solve this problem. 

Kevin is this something you'll be considering working into packrat or have I misunderstood that this will solve the compiling on windows problem?

Regards,

Adam