Error when using a conda environment file containing packages sourced from non-default channels

[Christopher Tomkins-Tinch]

I've been following the documentation page on exporting and using an environment file and have run into a problem. I have an active conda environment with dependencies from the bioconda channel, the default conda channel, as well as pip dependencies, and I'm trying to create a conda environment file that encapsulates all installed packages, both pip and (bio)conda. Exporting an environment file lists everything I have installed. 

When I try to create a new environment from the environment file, I receive an error that the bioconda-sourced packages are missing. Are environment files not intended to install packages from other channels? I assumed because of the bioconda-prefix "namespace notation" that the packages would be sourced from the appropriate channel.

A bit more about what I did:

An environment file was exported via:

conda env export > environment.yml

The file looks like this (prefix changed to be more generic):

name: project-test-env
channels: !!python/object/new:ruamel_yaml.comments.CommentedSeq
  listitems:
  - bioconda
  - defaults
  state:
    _yaml_format: !!python/object:ruamel_yaml.comments.Format
      _flow_style: false
    _yaml_line_col: !!python/object:ruamel_yaml.comments.LineCol
      col: 2
      data:
        0:
        - 1
        - 4
        1:
        - 2
        - 4
      line: 1
dependencies:
- bioconda::bcftools=1.3.1=0
- bioconda::curl=7.45.0=1
- bioconda::htslib=1.3.1=0
- bioconda::peewee=2.8.0=py35_1
- bioconda::pysam=0.9.1=py35_0
- bioconda::python-dateutil=2.3=py35_0
- bioconda::samtools=1.3.1=1
- bioconda::selscan=1.1.0b=0
- biopython=1.67=np111py35_0
- ecdsa=0.13=py35_0
- libgcc=4.8.5=1
- mkl=11.3.3=0
- ncurses=5.9=0
- numpy=1.11.0=py35_1
- openssl=1.0.2h=1
- pandas=0.18.1=np111py35_0
- paramiko=1.16.0=py35_0
- pip=8.1.2=py35_0
- pycrypto=2.6.1=py35_4
- python=3.5.1=0
- pytz=2016.4=py35_0
- readline=6.2=2
- setuptools=23.0.0=py35_0
- six=1.10.0=py35_0
- sqlite=3.13.0=0
- tk=8.5.18=0
- wheel=0.29.0=py35_0
- xz=5.0.5=1
- zlib=1.2.8=3
- pip:
  - apipkg==1.4
  - boltons==16.4.1
  - coverage==4.1
  - coveralls==1.1
  - docopt==0.6.2
  - execnet==1.4.1
  - flake8==2.6.0
  - mccabe==0.5.0
  - mock==2.0.0
  - pbr==1.10.0
  - peewee==2.8.0
  - py==1.4.31
  - pycodestyle==2.0.0
  - pyflakes==1.2.3
  - pysam==0.9.1
  - pytest==2.9.2
  - pytest-cov==2.2.1
  - pytest-xdist==1.14
  - python-dateutil==2.3
  - requests==2.10.0
prefix: /path/to/my-env

Next I tried to use the environment file to setup a new environment:

conda env create -f=environment.yml -n duplicate-env

I receive the following error:

Using Anaconda Cloud api site https://api.anaconda.org
Using Anaconda Cloud api site https://api.anaconda.org
Using Anaconda Cloud api site https://api.anaconda.org
Fetching package metadata .........
Solving package specifications: .
Error: Packages missing in current osx-64 channels:
  - bioconda::bcftools 1.3.1 0
  - bioconda::curl 7.45.0 1
  - bioconda::htslib 1.3.1 0
  - bioconda::peewee 2.8.0 py35_1
  - bioconda::pysam 0.9.1 py35_0
  - bioconda::python-dateutil 2.3 py35_0
  - bioconda::samtools 1.3.1 1
  - bioconda::selscan 1.1.0b 0

In the environment file, the "missing" packages specify the channel they should be coming from, so is it expected behavior that they are not being found? I'm not sure if it makes a difference, but I did place bioconda among the channels in my global config via "conda config --add channels bioconda" (before doing any of the above).

Here is the output from "conda info" (Miniconda3 install):

Using Anaconda Cloud api site https://api.anaconda.org

Current conda install:


             platform : osx-64
        conda version : 4.1.2
  conda-build version : 1.21.0
       python version : 3.5.1.final.0
     requests version : 2.9.1
     root environment : /Users/username/miniconda3  (writable)
  default environment : /Users/username/miniconda3
     envs directories : /Users/username/miniconda3/envs
        package cache : /Users/username/miniconda3/pkgs
         channel URLs : https://conda.anaconda.org/t/<TOKEN>/bioconda/osx-64/
                        https://conda.anaconda.org/t/<TOKEN>/bioconda/noarch/
                        https://repo.continuum.io/pkgs/free/osx-64/
                        https://repo.continuum.io/pkgs/free/noarch/
                        https://repo.continuum.io/pkgs/pro/osx-64/
                        https://repo.continuum.io/pkgs/pro/noarch/
          config file : /Users/tomkinsc/.condarc
    is foreign system : False

Thank you to any who can help shed some light on whether this is an internal conda issue, or whether I've made incorrect assumptions about how environment files are intended to work.

Unrelated to the above, but is there a way to hide the "Using Anaconda Cloud api site https://api.anaconda.org" messages that are printed with each conda command? They appear even if I specify "--json" which means that anything interpreting stdout has to strip them out before parsing the output as json.

Thanks,

Chris

[Ian Stokes Rees]

Hi Chris,

Please bug report this here: https://github.com/conda/conda/issues/new

Just cut-and-paste your original email.

The work-around may be to edit your environment.yml file and remove the bioconda:: prefix from those first dependency packages.

I just tried it and it worked for me.

The effect is that conda env will search all channels and pull the dependency packages from wherever it deems best, instead of from the specific channels (which I’m sure is what you’d prefer, and certainly seems to be what conda env export was thinking would work). So this is a bug in conda env, but I think the workaround should get you by for now.

Regards,

Ian

--
You received this message because you are subscribed to the Google Groups "conda - Public" group.
To unsubscribe from this group and stop receiving emails from it, send an email to conda+un...@continuum.io.
To post to this group, send email to co...@continuum.io.
Visit this group at https://groups.google.com/a/continuum.io/group/conda/.
To view this discussion on the web visit https://groups.google.com/a/continuum.io/d/msgid/conda/912fee55-cf1f-45ba-8a21-4c8c7b928fbd%40continuum.io.
For more options, visit https://groups.google.com/a/continuum.io/d/optout.

---

Ian Stokes-Rees
Computational Scientist

@ijstokes 617.942.0218

​

[Christopher Tomkins-Tinch]

Thanks, Ian.

I've confirmed that for my install of conda, removing the channel prefixes from the environment file allows me to install the packages. Ideally in the future the channel namespaces would be honored to disambiguate package origins.

I've also opened an issue for this on GitHub:

https://github.com/conda/conda/issues/2800