The file does not seem to exist remotely

Samira Vogt

unread,

Nov 18, 2021, 9:40:28 AM11/18/21

to pypsa

Hello,

since yesterday I have a problem running the model due to an error directly in the beginning, when downloading files. It happened with the 'europe-2013-era5.nc' file as well as the 'natura.tiff' file.

When I open the links in the browser, die files download. Any ideas why the model cant download it?

Downloading from remote: zenodo.org/record/4709858/files/europe-2013-era5.nc

HTTPFileException:

The file does not seem to exist remotely: https://zenodo.org/record/4709858/files/europe-2013-era5.nc

I tried downloading them manually and putting them in the right folder. In that case the following error occurs:

(pypsa-eur) samira@Samiras-MacBook-Pro pypsa-eur % snakemake -j 1 results/networks/elec_s_37_ec_lcopt_Co2L0.244-3H.nc

Academic license - for non-commercial use only - expires 2022-09-24

Using license file /Users/samira/gurobi.lic

No parameters matching '_test' found

Building DAG of jobs...

IncompleteFilesException:

The files below seem to be incomplete. If you are sure that certain files are not incomplete, mark them as complete with

snakemake --cleanup-metadata <filenames>

To re-generate the files rerun your command with the --rerun-incomplete flag.

Incomplete files:

cutouts/europe-2013-era5.nc

cutouts/europe-2013-sarah.nc

(pypsa-eur) samira@Samiras-MacBook-Pro pypsa-eur % snakemake --cleanup-metadata cutouts/europe-2013-era5.nc

Academic license - for non-commercial use only - expires 2022-09-24

Using license file /Users/samira/gurobi.lic

No parameters matching '_test' found

Error: you need to specify the maximum number of CPU cores to be used at the same time. If you want to use N cores, say --cores N or -cN. For all cores on your system (be sure that this is appropriate) use --cores all. For no parallelization use --cores 1 or -c1.

(pypsa-eur) samira@Samiras-MacBook-Pro pypsa-eur % snakemake --cleanup-metadata cutouts/europe-2013-era5.nc cutouts/europe-2013-sarah.nc

Academic license - for non-commercial use only - expires 2022-09-24

Using license file /Users/samira/gurobi.lic

No parameters matching '_test' found

Error: you need to specify the maximum number of CPU cores to be used at the same time. If you want to use N cores, say --cores N or -cN. For all cores on your system (be sure that this is appropriate) use --cores all. For no parallelization use --cores 1 or -c1.

Thanks for any help!

Best,

Samira

Fabian Hofmann

unread,

Nov 18, 2021, 10:41:15 AM11/18/21

to py...@googlegroups.com

Hey Samira,

the first error is weird. Could you send the full error again?

The error in the cleanup already tells you what is missing:

"Error: you need to specify the maximum number of CPU cores to be used at the same time. If you want to use N cores, say --cores N or -cN. For all cores on your system (be sure that this is appropriate) use --cores all. For no parallelization use --cores 1 or -c1."

So just add a "-c 2" (for example) to your snakemake command.

Best

Fabian H

--
You received this message because you are subscribed to the Google Groups "pypsa" group.
To unsubscribe from this group and stop receiving emails from it, send an email to pypsa+un...@googlegroups.com.
To view this discussion on the web, visit https://groups.google.com/d/msgid/pypsa/ad5759a0-950c-4b12-883f-2d60ef4005f2n%40googlegroups.com.

Samira Vogt

unread,

Nov 18, 2021, 11:08:41 AM11/18/21

to pypsa

Hi Fabian,

this is the first error:

(pypsa-eur) samira@Samiras-MacBook-Pro pypsa-eur % snakemake -j 1 results/networks/elec_s_37_ec_lcopt_Co2L0.244-3H.nc

Building DAG of jobs...

Using shell: /bin/bash

Provided cores: 1 (use --cores to define parallelism)

Rules claiming more threads will be scaled down.

Job stats:

job count min threads max threads

------------------------ ------- ------------- -------------

add_electricity 1 1 1

add_extra_components 1 1 1

base_network 1 1 1

build_bus_regions 1 1 1

build_hydro_profile 1 1 1

build_load_data 1 1 1

build_powerplants 1 1 1

build_renewable_profiles 4 1 1

build_shapes 1 1 1

cluster_network 1 1 1

prepare_network 1 1 1

retrieve_cutout 2 1 1

retrieve_databundle 1 1 1

retrieve_load_data 1 1 1

retrieve_natura_raster 1 1 1

simplify_network 1 1 1

solve_network 1 1 1

total 21 1 1

Select jobs to execute...

[Thu Nov 18 17:01:39 2021]

rule retrieve_cutout:

input: zenodo.org/record/4709858/files/europe-2013-sarah.nc

output: cutouts/europe-2013-sarah.nc

jobid: 19

wildcards: cutout=europe-2013-sarah

resources: tmpdir=/var/folders/gn/gbwc08zs72gfh9kj_v4jddt40000gn/T

Downloading from remote: zenodo.org/record/4709858/files/europe-2013-sarah.nc

HTTPFileException:

The file does not seem to exist remotely: https://zenodo.org/record/4709858/files/europe-2013-sarah.nc

I added the -c2, but it doesn't continue the run afterwards. So I add "--cleanup-metadata cutouts/europe-2013-sarah.nc -c2" to the snakemake command of the run and nothing happens. And if I try to start the run normally again afterwards the "IncompleteFilesException" occurs again. I attached the whole terminal output as a txt file.

Best,

Samira

IncompleteFilesException.txt

energyLS

unread,

Nov 23, 2021, 12:18:43 PM11/23/21

to pypsa

Hi Samira and Fabian,

regarding the first error: The same has occured to me yesterday:

Downloading from remote: zenodo.org/record/4709858/files/europe-2013-era5.nc

HTTPFileException:

The file does not seem to exist remotely: https://zenodo.org/record/4709858/files/europe-2013-era5.nc

I then downloaded the files manually and instead of "putting them in the right folder" I downloaded the file (in this case "natura.tiff") in the pypsa-eur main directory and modified the snakefile to use the local file instead of downloading it:

OLD:

if config['enable'].get('retrieve_natura_raster', True):
    rule retrieve_natura_raster:
        input: HTTP.remote("zenodo.org/record/4706686/files/natura.tiff", keep_local=True, static=True)
        output: "resources/natura.tiff"
        shell: "mv {input} {output}"

NEW:

if config['enable'].get('retrieve_natura_raster', True):
    rule retrieve_natura_raster:
        input: "natura.tiff"
        output: "resources/natura.tiff"
        shell: "mv {input} {output}"

This is certainly not the solution to handle the issue appropiately but this workaround might help until a better fix comes up.