Pypsa-Eur: network optimization clustering sensitivities

[Eduardo Díaz]

Dear pypsa community,

I am reading Frysztacki, Martha Maria's thesis: The Role of Spatial Scale in Electricity System Optimisation M... (kit.edu)

I am finding in it extremely interesting because I am conducting some sensitivity studies on a Spanish grid regarding the number of nodes considered.

In the thesis, there appears to be a way perform 2 different clusterization techniques:

1. Keeping the number of generators and nodes the same (coupled option)

2. De-coupling generators or transmission from nodes, so you can keep x nodes and have y resolution of generators (ideal for me because no information on optimal sitting for generators is lost)

Is there a way to do this in pypsa-eur? So far, I know every bus accepts one generator so I would not know how we can keep a high resolution of generators/transmission with low resolution of nodes. 

Kind regards,

Eduardo.

[Fabian Neumann]

Hi Eduardo,

You can exclude certain carriers from aggregation in the config file using the `exclude_carriers` block for simplify and cluster:

https://pypsa-eur.readthedocs.io/en/latest/configuration.html#clustering

Best wishes,

Fabian N

[Eduardo Díaz]

Hi Fabian, 

Thank you for your answer. 

I am applying your advice for the cluster network rule. Prior to that, my goal is to cluster the network in simplify rule to around 100 generators and then applying the carrier exclusion in cluster network rule to maintain those 100 gens against less nodes.

I however get an error in simplify_network, which i think has to do with the symbol attribute of n.buses, which is later dropeed.

There is a function that asserts that all the buses have the same symbol attribute, but my network appears to have two different ones: "Substation" and "joint". Please find below the complete error log.

rule simplify_network:
    input: resources/sp2022-1y-FormEnergy/networks/elec.nc, data/costs_2030_mod.csv, resources/sp2022-1y-FormEnergy/regions_onshore.geojson, resources/sp2022-1y-FormEnergy/regions_offshore.geojson
    output: resources/sp2022-1y-FormEnergy/networks/elec_s100.nc, resources/sp2022-1y-FormEnergy/regions_onshore_elec_s100.geojson, resources/sp2022-1y-FormEnergy/regions_offshore_elec_s100.geojson, resources/sp2022-1y-FormEnergy/busmap_elec_s100.csv, resources/sp2022-1y-FormEnergy/connection_costs_s100.csv
    log: logs/sp2022-1y-FormEnergy/simplify_network/elec_s100.log
    jobid: 5
    benchmark: benchmarks/sp2022-1y-FormEnergy/simplify_network/elec_s100
    reason: Forced execution
    wildcards: simpl=100
    resources: tmpdir=C:\Users\SE91723\AppData\Local\Temp, mem_mb=12000, mem_mib=11445

INFO:pypsa.io:Imported network elec.nc has buses, carriers, generators, lines, links, loads, storage_units, transformers
INFO:__main__:Mapping all network lines onto a single 380kV layer
INFO:__main__:Simplifying connected link components
INFO:__main__:Removing stubs
INFO:__main__:Clustering to 100 buses
WARNING:cluster_network:Keyword argument feature is only valid for algorithm `hac`. Given feature `solar+onwind-time` will be ignored.
Traceback (most recent call last):
  File "C:\Users\SE91723\Modelos\pypsa\pypsa-eur\.snakemake\scripts\tmpxdu7j0o2.simplify_network.py", line 600, in <module>
    n, cluster_map = cluster(
  File "C:\Users\SE91723\Modelos\pypsa\pypsa-eur\.snakemake\scripts\tmpxdu7j0o2.simplify_network.py", line 518, in cluster
    clustering = clustering_for_n_clusters(
  File "C:\Users\SE91723\Modelos\pypsa\pypsa-eur\rules\..\scripts\cluster_network.py", line 410, in clustering_for_n_clusters
    clustering = get_clustering_from_busmap(
  File "C:\Users\SE91723\Anaconda3\envs\pypsa-eur\lib\site-packages\pypsa\clustering\spatial.py", line 534, in get_clustering_from_busmap
    buses = aggregatebuses(n, busmap, custom_strategies=bus_strategies)
  File "C:\Users\SE91723\Anaconda3\envs\pypsa-eur\lib\site-packages\pypsa\clustering\spatial.py", line 341, in aggregatebuses
    aggregated = n.buses.groupby(busmap).agg(strategies)
  File "C:\Users\SE91723\Anaconda3\envs\pypsa-eur\lib\site-packages\pandas\core\groupby\generic.py", line 1269, in aggregate
    result = op.agg()
  File "C:\Users\SE91723\Anaconda3\envs\pypsa-eur\lib\site-packages\pandas\core\apply.py", line 163, in agg
    return self.agg_dict_like()
  File "C:\Users\SE91723\Anaconda3\envs\pypsa-eur\lib\site-packages\pandas\core\apply.py", line 420, in agg_dict_like
    results = {
  File "C:\Users\SE91723\Anaconda3\envs\pypsa-eur\lib\site-packages\pandas\core\apply.py", line 421, in <dictcomp>
    key: obj._gotitem(key, ndim=1).agg(how) for key, how in arg.items()
  File "C:\Users\SE91723\Anaconda3\envs\pypsa-eur\lib\site-packages\pandas\core\groupby\generic.py", line 269, in aggregate
    return self._python_agg_general(func, *args, **kwargs)
  File "C:\Users\SE91723\Anaconda3\envs\pypsa-eur\lib\site-packages\pandas\core\groupby\generic.py", line 288, in _python_agg_general
    result = self.grouper.agg_series(obj, f)
  File "C:\Users\SE91723\Anaconda3\envs\pypsa-eur\lib\site-packages\pandas\core\groupby\ops.py", line 994, in agg_series
    result = self._aggregate_series_pure_python(obj, func)
  File "C:\Users\SE91723\Anaconda3\envs\pypsa-eur\lib\site-packages\pandas\core\groupby\ops.py", line 1015, in _aggregate_series_pure_python
    res = func(group)
  File "C:\Users\SE91723\Anaconda3\envs\pypsa-eur\lib\site-packages\pandas\core\groupby\generic.py", line 285, in <lambda>
    f = lambda x: func(x, *args, **kwargs)
  File "C:\Users\SE91723\Anaconda3\envs\pypsa-eur\lib\site-packages\pypsa\clustering\spatial.py", line 136, in consense
    assert (x == v).all() or x.isnull().all(), (
AssertionError: In Bus cluster symbol, the values of attribute symbol do not agree:
Bus
2787    Substation
2789    Substation
2791    Substation
7388         joint
7308         joint
7705         joint
7704         joint
7309         joint
8609         joint
7244         joint
Name: symbol, dtype: object
[Mon Nov 13 16:12:19 2023]
Error in rule simplify_network:
    jobid: 5
    input: resources/sp2022-1y-FormEnergy/networks/elec.nc, data/costs_2030_mod.csv, resources/sp2022-1y-FormEnergy/regions_onshore.geojson, resources/sp2022-1y-FormEnergy/regions_offshore.geojson
    output: resources/sp2022-1y-FormEnergy/networks/elec_s100.nc, resources/sp2022-1y-FormEnergy/regions_onshore_elec_s100.geojson, resources/sp2022-1y-FormEnergy/regions_offshore_elec_s100.geojson, resources/sp2022-1y-FormEnergy/busmap_elec_s100.csv, resources/sp2022-1y-FormEnergy/connection_costs_s100.csv
    log: logs/sp2022-1y-FormEnergy/simplify_network/elec_s100.log (check log file(s) for error details)
    conda-env: C:\Users\SE91723\Modelos\pypsa\pypsa-eur\.snakemake\conda\8d21106675d914e6a68cd9a9a9e842f4_

Removing output files of failed job simplify_network since they might be corrupted:
resources/sp2022-1y-FormEnergy/connection_costs_s100.csv
Shutting down, this might take some time.
Exiting because a job execution failed. Look above for error message
Complete log: .snakemake\log\2023-11-13T160711.159309.snakemake.log

[Eduardo Díaz]

This error was avoided by, in simplify_network.py, swapping the order of:

    # some entries in n.buses are not updated in previous functions, therefore can be wrong. as they are not needed

    # and are lost when clustering (for example with the simpl wildcard), we remove them for consistency:

    remove = [

        "symbol",

        "tags",

        "under_construction",

        "substation_lv",

        "substation_off",

        "geometry",

        "underground",

    ]

    n.buses.drop(remove, axis=1, inplace=True, errors="ignore")

    n.lines.drop(remove, axis=1, errors="ignore", inplace=True)

which are below :

    if snakemake.wildcards.simpl:

        n, cluster_map = cluster(

            n,

            int(snakemake.wildcards.simpl),

            params.focus_weights,

            solver_name,

            params.simplify_network["algorithm"],

            params.simplify_network["feature"],

            params.aggregation_strategies,

        )

        busmaps.append(cluster_map)

The dropping should be done prior to the cluster call, as inside that call there is a function that asserts that some of the attributes of n.buses must be the same. In particular, symbol attribute usually gives an error because it can be either joint or substation. I am not sure this is the intended behaviour, but as a hot fix, simply dropping those attributes (which are not needed later) before clustering circumvents it.