Executing Snakemake in Conda environments on Windows

[Torben Zeller]

Hello everyone,

I try to do the PyPSA-Eur Tutorial on Windows 10, but can't get to run the Snakemake command snakemake -j 1 results/networks/elec_s_6_ec_lcopt_Co2L-24H.nc.

When executing this command, I get the error that the command mv  zenodo.org/record/4709858/files/europe-2013-era5-tutorial.nc cutouts/europe-2013-era5-tutorial.nc cannot be executed because mv is not known. This is a command in Snakemakefile. So, I changed mv to move in Snakemakefile because move is the Windows equivalent of mv in Linux's Bash. 

When executing the same command after changes, I get the error that path zenodo.org/record/4709858/files/europe-2013-era5-tutorial.nc cutouts/europe-2013-era5-tutorial.nc cannot be found. After investigation, I found out that this is due to slashes which are used for path description in Bash, whereas cmd on Windows asks for backslashes.

After fixing this error, I got another one which I have not understood yet, but it is probably another problem with Windows's cmd. I deduce that there is a general problem by executing Snakemake on Windows. I use the Conda environment and found the issue Executing snakemake rules in conda environments on Windows - Stack Overflow on Stack Overflow. This person has got the same problem. So, it seems that Snakemake used in Conda environment has no support on Windows.

My question is: how do you get to run Snakemake on Windows?

Thanks,

Torben

[Fabian Neumann]

Hi Torben,



We are all Linux users in the development team, so we are no experts but
we have a Windows CI running for PyPSA-Eur that passes and I know of
people who have been running PyPSA-Eur on Windows.



You could try to find out what shell the Github Action is running by
default and apply that to your local system:



https://github.com/PyPSA/pypsa-eur/blob/master/.github/workflows/ci.yaml



You could also try Windows Subsystem for Linux(WSL)?



Best wishes,



Fabian

(sorry Torben for the duplicate, it should just go to the mailing list
as well)

On 21.10.21 10:33, 'Torben Zeller' via pypsa wrote:
> Hello everyone,
>
> I try to do the PyPSA-Eur Tutorial on Windows 10, but can't get to run

> the Snakemake command /snakemake -j 1
> results/networks/elec_s_6_ec_lcopt_Co2L-24H.nc./
> /
> /
> When executing this command, I get the error that the command /mv
> zenodo.org/record/4709858/files/europe-2013-era5-tutorial.nc
> cutouts/europe-2013-era5-tutorial.nc /cannot be executed because /mv /is
> not known. This is a command in /Snakemakefile/. So, I changed /mv /to
> move in /Snakemakefile /because move is the Windows equivalent of mv in

> Linux's Bash.
>
> When executing the same command after changes, I get the error that path

> /zenodo.org/record/4709858/files/europe-2013-era5-tutorial.nc
> cutouts/europe-2013-era5-tutorial.nc /cannot be found. After

> investigation, I found out that this is due to slashes which are used
> for path description in Bash, whereas cmd on Windows asks for backslashes.
>
> After fixing this error, I got another one which I have not understood
> yet, but it is probably another problem with Windows's cmd. I deduce
> that there is a general problem by executing Snakemake on Windows. I use
> the Conda environment and found the issue Executing snakemake rules in
> conda environments on Windows - Stack Overflow

> <https://stackoverflow.com/questions/62720926/executing-snakemake-rules-in-conda-environments-on-windows> on

> Stack Overflow. This person has got the same problem. So, it seems that
> Snakemake used in Conda environment has no support on Windows.
>
> My question is: how do you get to run Snakemake on Windows?
>
> Thanks,
> Torben
>

> --
> You received this message because you are subscribed to the Google
> Groups "pypsa" group.
> To unsubscribe from this group and stop receiving emails from it, send
> an email to pypsa+un...@googlegroups.com
> <mailto:pypsa+un...@googlegroups.com>.
> To view this discussion on the web, visit
> https://groups.google.com/d/msgid/pypsa/0d6943a5-9539-4cbd-98d1-a7495e625997n%40googlegroups.com
> <https://groups.google.com/d/msgid/pypsa/0d6943a5-9539-4cbd-98d1-a7495e625997n%40googlegroups.com?utm_medium=email&utm_source=footer>.

[Adrian Odenweller]

Hi Torben and Fabian,

I just started with PyPSA (going through the tutorial for now) and I'm encountering the same problem - that mv is not recognised. Also, due to IT restrictions on my laptop, it looks like I cannot install WSL. How did you fix this issue in the end? Is there another way apart from using a Linux VM?

Thanks a lot,

Adrian

[Fabian Neumann]

Hi Adrian,

I see, you could try replacing

shell: "mv {input} {output}"

with

run:
import shutil
shutil.move(input[0], output[0])

(There should be three rules affected by this)

If that works, I'll propose that as permanent solution.

Best wishes,

Fabian N

On 22.02.22 17:43, Adrian Odenweller wrote:
> Hi Torben and Fabian,
>
> I just started with PyPSA (going through the tutorial for now) and I'm

> encountering the same problem - that /mv/ is not recognised. Also, due

> <http://zenodo.org/record/4709858/files/europe-2013-era5-tutorial.nc>
> > cutouts/europe-2013-era5-tutorial.nc
> <http://europe-2013-era5-tutorial.nc> /cannot be executed because

> /mv /is
> > not known. This is a command in /Snakemakefile/. So, I changed
> /mv /to
> > move in /Snakemakefile /because move is the Windows equivalent of
> mv in
> > Linux's Bash.
> >
> > When executing the same command after changes, I get the error
> that path
> > /zenodo.org/record/4709858/files/europe-2013-era5-tutorial.nc

> <http://zenodo.org/record/4709858/files/europe-2013-era5-tutorial.nc>
> > cutouts/europe-2013-era5-tutorial.nc
> <http://europe-2013-era5-tutorial.nc> /cannot be found. After

> <https://groups.google.com/d/msgid/pypsa/0d6943a5-9539-4cbd-98d1-a7495e625997n%40googlegroups.com?utm_medium=email&utm_source=footer
> <https://groups.google.com/d/msgid/pypsa/0d6943a5-9539-4cbd-98d1-a7495e625997n%40googlegroups.com?utm_medium=email&utm_source=footer>>.

>
>
> --
> You received this message because you are subscribed to the Google
> Groups "pypsa" group.
> To unsubscribe from this group and stop receiving emails from it, send
> an email to pypsa+un...@googlegroups.com
> <mailto:pypsa+un...@googlegroups.com>.
> To view this discussion on the web, visit

> https://groups.google.com/d/msgid/pypsa/94c5ad29-d330-47ed-bf4d-8056fd98fd71n%40googlegroups.com
> <https://groups.google.com/d/msgid/pypsa/94c5ad29-d330-47ed-bf4d-8056fd98fd71n%40googlegroups.com?utm_medium=email&utm_source=footer>.

--
Dr.-Ing.
Fabian Neumann (he/him)
Postdoctoral Researcher

Department of Digital Transformation in Energy Systems
Institute of Energy Technology
Technische Universität Berlin

Group website: https://tub-ensys.github.io
Personal website: https://neumann.fyi

Zoom: https://tu-berlin.zoom.us/my/fneum
Schedule Meeting: https://calendly.com/fneum

Visitor Address:
Einsteinufer 25 (TA 8)
10587 Berlin

My working hours may not be your working hours. Do not feel pressure to
reply to this email outside your working hours.

[Adrian Odenweller]

Thanks Fabian, that worked well!

I encountered two more issues that I was able to resolve. I'll just add them here for the record so that other Windows users  might find them using the search function.

First, installing Gurobi via conda and then using solver: name: gurobi_cl somehow didn't work due to some issues with pyomo I suspect. Instead, I used the full msi installer and then solver: name: gurobi, which worked fine.

Second, I got a Windows 1314 error code somewhere due to missing privileges for creating symbolic links. Executing the shell as administrator could have solved that, but I'm not sure if that works well with conda environments, which were set up as a different user. So I went with the solution posted here.

Best wishes,
Adrian

[Fabian Neumann]

Hi Adrian,

great! thanks!

Just one correction: PyPSA-Eur doesn't use pyomo as backend solver
interface anymore, but PyPSA's custom interface.

Regarding the symbolic links: Removing the "shadow: 'minimal'"
directives in the solve_network rules might also cause trouble on Windows.

Best wishes,

Fabian

On 23.02.22 12:11, Adrian Odenweller wrote:
> Thanks Fabian, that worked well!
>
> I encountered two more issues that I was able to resolve. I'll just add
> them here for the record so that other Windows users  might find them
> using the search function.

> First, installing Gurobi via conda and then using /solver: name:
> gurobi_cl/ somehow didn't work due to some issues with pyomo I suspect.
> Instead, I used the full msi installer and then /solver: name: gurobi/,

> which worked fine.
> Second, I got a Windows 1314 error code somewhere due to missing
> privileges for creating symbolic links. Executing the shell as
> administrator could have solved that, but I'm not sure if that works
> well with conda environments, which were set up as a different user. So
> I went with the solution posted here

> <https://stackoverflow.com/a/65504258>.

> <http://zenodo.org/record/4709858/files/europe-2013-era5-tutorial.nc

> <http://zenodo.org/record/4709858/files/europe-2013-era5-tutorial.nc>>
> > > cutouts/europe-2013-era5-tutorial.nc
> <http://europe-2013-era5-tutorial.nc>

> > <http://europe-2013-era5-tutorial.nc

> <http://europe-2013-era5-tutorial.nc>> /cannot be executed because
> > /mv /is
> > > not known. This is a command in /Snakemakefile/. So, I changed
> > /mv /to
> > > move in /Snakemakefile /because move is the Windows equivalent of
> > mv in
> > > Linux's Bash.
> > >
> > > When executing the same command after changes, I get the error
> > that path
> > > /zenodo.org/record/4709858/files/europe-2013-era5-tutorial.nc
> <http://zenodo.org/record/4709858/files/europe-2013-era5-tutorial.nc>
> >

> <http://zenodo.org/record/4709858/files/europe-2013-era5-tutorial.nc

> <http://zenodo.org/record/4709858/files/europe-2013-era5-tutorial.nc>>
> > > cutouts/europe-2013-era5-tutorial.nc
> <http://europe-2013-era5-tutorial.nc>

> > <http://europe-2013-era5-tutorial.nc

> <https://groups.google.com/d/msgid/pypsa/94c5ad29-d330-47ed-bf4d-8056fd98fd71n%40googlegroups.com?utm_medium=email&utm_source=footer

> <https://groups.google.com/d/msgid/pypsa/94c5ad29-d330-47ed-bf4d-8056fd98fd71n%40googlegroups.com?utm_medium=email&utm_source=footer>>.
>
>
> --
> Dr.-Ing.
> Fabian Neumann (he/him)
> Postdoctoral Researcher
>
> Department of Digital Transformation in Energy Systems
> Institute of Energy Technology
> Technische Universität Berlin
>
> Group website: https://tub-ensys.github.io
> <https://tub-ensys.github.io>

> Personal website: https://neumann.fyi <https://neumann.fyi>

>
> Zoom: https://tu-berlin.zoom.us/my/fneum
> <https://tu-berlin.zoom.us/my/fneum>
> Schedule Meeting: https://calendly.com/fneum
> <https://calendly.com/fneum>
>
> Visitor Address:
> Einsteinufer 25 (TA 8)
> 10587 Berlin
>
> My working hours may not be your working hours. Do not feel pressure to
> reply to this email outside your working hours.
>

> --
> You received this message because you are subscribed to the Google
> Groups "pypsa" group.
> To unsubscribe from this group and stop receiving emails from it, send
> an email to pypsa+un...@googlegroups.com
> <mailto:pypsa+un...@googlegroups.com>.
> To view this discussion on the web, visit

> https://groups.google.com/d/msgid/pypsa/76649f7e-35cf-4ad0-9f66-7d988e0f58c8n%40googlegroups.com
> <https://groups.google.com/d/msgid/pypsa/76649f7e-35cf-4ad0-9f66-7d988e0f58c8n%40googlegroups.com?utm_medium=email&utm_source=footer>.