Don't second-guess 'utils::download.file' in 'packrat:::download'

[Ken Williams]

I have a setup where I can successfully download & install packages when not using Packrat, but Packrat can't download anything.  It looks like this is because `packrat:::download` has its own logic for figuring out how to download files, but then it calls `utils::download.file`, which then reassesses how to do things, potentially producing a bad mishmash of methods.

In my case, `getOption("download.file.method")` is set to "curl", and `getOption("download.file.extra")` is set to "-L -f".  Which works fine.

However, `packrat:::download` sets the method to "wget", but leaves `getOption("download.file.extra")` set to "-L -f", and 'wget' doesn't understand these command-line flags, so it dies.

I think it would be best to eliminate the `packrat:::download` method and use `utils::download.file` directly, or perhaps leave `packrat:::download` in place but make it a much thinner wrapper that doesn't try to set/change any options.  Would that work?

 -Ken

[Ken Williams]

Incidentally: I'm using Packrat version 0.4.4, with R version 3.2.2.

 -Ken

[Kevin Ushey]

Hi Ken,

The primary motivation here was to help ensure users attempting to use 'https' repositories (e.g. with RStudio) would not run into problems when used with internal download methods. We're apparently not doing a great job because we certainly don't want to stomp on valid user download methods!

I'll file a bug -- if you're using the development version of packrat, you should also be able to work around this by setting:

    options(packrat.download.method = "curl")

Also, it might be relevant -- are you running within RStudio? I believe RStudio also attempts to set the download method + extras to help assist in this case.

Thanks,

Kevin

--
You received this message because you are subscribed to the Google Groups "packrat-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to packrat-discu...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Kevin Ushey]

Hi Ken,

I've just pushed a fix here: https://github.com/rstudio/packrat/commit/57bc3ee2fefda643d35d7695ff4347acaf7f4ff9; can you give it a shot and let me know if it works?

You can install the development version of packrat with:

    if (!require("devtools")) install.packages("devtools")

    devtools::install_github('rstudio/packrat')

Thanks,

Kevin

[Ken Williams]

Hi Kevin,

I just tried setting `options(packrat.download.method = "curl")`, (and my options("download.file.extra"="-L -f") is still in place) but now it fails like so:

Installing MASS (7.3-43) ... Error in if (!file.exists(file.path(pkg_path, "DESCRIPTION"))) { : 

  argument is of length zero

In addition: Warning messages:

1: In packifyStart:packifyEnd :

  numerical expression has 2 elements: only the first used

2: In FUN(X[[i]], ...) :

  Failed to infer source for package 'LinearizedSVR'; using latest available version on CRAN instead

I suspect there are several things going on there, but the "MASS" one seems to be the fatal one.  Is it maybe getting confused because "MASS" is a core package distributed with R?

 -Ken

[Ken Williams]

I think I'm hitting a couple additional problems when I try the latest on GitHub:

> packrat::init(options = list(external.packages = c("AMGUtils")))

Initializing packrat project in directory:

- "/Users/kwilliams/git/fpdc/src/main/R/FPDC"

Error in results[sapply(results, Negate(isTRUE))] : 

  invalid subscript type 'list'

In addition: Warning messages:

1: In packifyStart:packifyEnd :

  numerical expression has 2 elements: only the first used

2: In FUN(X[[i]], ...) :

  Package 'AMGUtils' not available in repository or locally

3: In symlinkExternalPackages(project = project) :

  The following external packages could not be located:

- 'AMGUtils'

4: In packifyStart:packifyEnd :

  numerical expression has 2 elements: only the first used

5: In FUN(X[[i]], ...) :

  Package 'AMGUtils' not available in repository or locally

6: In symlinkExternalPackages(project = project) :

  The following external packages could not be located:

- 'AMGUtils'

The "AMGUtils" is a local package developed by our team, and available on a full-fledged CPAN-like repository within our firewall (it's not just a directory, it's a web server with a PACKAGES index file & the normal CRAN directory structure).  I thought I could put the following in the packrat-created '.Rprofile' file, but it gets blown away when I do 'packrat::init(...)'.

#### -- Packrat Autoloader (version 0.4.4) -- ####

options(repos=c(WARP="http://cranalike.ourcompany.local/cran/",

                CRAN="http://cran.revolutionanalytics.com"))

source("packrat/init.R")

#### -- End Packrat Autoloader -- ####

In the meantime I'll see if I can use the "local.repos" feature to get around that.

 -Ken

[Ken Williams]

A bit more progress - I opened a fresh R session and told it about my local repo (it would be nice if packrat's "local.repos" could point to a real web server rather than just a local directory) like so:

options(repos=c(WARP="http://cranalike.ourcompany.local/cran/",

                CRAN="http://cran.revolutionanalytics.com"))

Then I just did a normal 'packrat::init(options = list(use.cache=TRUE))' and it seems to find every package except "MASS" .  For that, it still seems to be using "wget" and the Curl options, even though I've done `options(packrat.download.method = "curl")`.  I pasted the errors here:

    https://gist.github.com/kenahoo/689f69d9282242f43971

It appears that R isn't loading the version of Packrat from GitHub, it's loading the latest from CRAN, which doesn't honor the "packrat.download.method" setting.  Feels like a bit of a catch-22 here that I'm not sure how to get out of.

Forgot to answer your previous question - yes, this is all happening inside RStudio, on a Mac.  I can jump to a shell session of R if need be.

 -Ken

[Kevin Ushey]

Is it possible that the CRAN version of packrat is installed in the user library, while the development version of packrat (from GitHub) is instead installed in a separate private library? (What is the output of 'packageVersion("packrat")'?)

On initialization, the version of packrat in the user library is used, so that could be the culprit.

[Kevin Ushey]

And, FWIW, packrat 'should' be able to understand regular CRAN-like repositories. If you have a CRAN-like mirror set up and have it registered within `options("repos")`, packrat 'should' be able to fetch packages from there as it would a 'regular' CRAN repository. I think the only requirement that we have (e.g. in your case) is that each package's DESCRIPTION has `Repository: WARP`, so we know which repository it came from.

We're trying to push the 'use a CRAN-like repository' and avoid the 'local repos' bit in the future; hopefully packrat will also bundle some tools for easily creating local CRAN-like repositories.

[Ken Williams]

Well, I've already got a .Rprofile in this directory, so it's skipping my global ~/.Rprofile , where I set up my user library paths.  In a normal shell (not in this project's directory) I have these 3 libraries:

> .libPaths()

[1] "/Users/kwilliams/R/library/3.2"        # Contains packrat '0.4.4-28' from GitHub

[2] "/usr/local/lib/R/3.2/site-library"     # Contains packrat '0.4.4' from CRAN

[3] "/usr/local/Cellar/r/3.2.2_1/R.framework/Versions/3.2/Resources/library"  # Contains no packrat

When I start up R in the package's directory, that first directory isn't part of .libPaths(), so indeed it looks like it's using 0.4.4 from the second path.

So I should probably blow away the 0.4.4, force an install to there from GitHub, and try again?

 -Ken

[Ken Williams]

On Tuesday, October 20, 2015 at 4:56:46 PM UTC-5, Kevin Ushey wrote:

And, FWIW, packrat 'should' be able to understand regular CRAN-like repositories. If you have a CRAN-like mirror set up and have it registered within `options("repos")`, packrat 'should' be able to fetch packages from there as it would a 'regular' CRAN repository. I think the only requirement that we have (e.g. in your case) is that each package's DESCRIPTION has `Repository: WARP`, so we know which repository it came from.

Oh, okay - might want to add that to the docs then, it says it's only for local directories.  And I'm not sure where I'll set up my repository settings, because I usually do that in my ~/.Rprofile and that won't get read when packrat creates a .Rprofile in the local directory.  Any suggestions?

 -Ken 

[Kevin Ushey]

Packrat saves the active repositories as part of the lockfile, and applies them whenever an R session is initialized. You can check what repositories are in use by looking at the header of the `packrat/packrat.lock` file within your project.

These repositories should also get updated on `packrat::snapshot()` calls, so if you do the following:

    1. Use `options("repos")` to update the repositories used, and then

    2. use `packrat::snapshot()` to 'save the state' of your library,

you should see the repositories saved and reapplied for future R sessions.

--

You received this message because you are subscribed to the Google Groups "packrat-discuss" group.

[Ken Williams]

That seems to be working - I blew away the first packrat in "/usr/local/lib/R/3.2/site-library" and the ones in packrat/src/packrat/ and packrat/lib/x86_64-apple-darwin14.5.0/3.2.2/packrat/ ,  copied the one from "/Users/kwilliams/R/library/3.2" to "/usr/local/lib/R/3.2/site-library", and then it bootstrapped from that.  Now it's installing to packrat/lib, we'll see whether that completes.  

I can pick up this thread again tomorrow, time to head home now.

 -Ken