Error with optimize_with_rolling_horizon: KeyError: "not all values found in index 'snapshot'. Try setting the `method` keyword argument (example: method='nearest')."

[gktho...@gmail.com]

Dear group,

I am encountering the above error when I run a rolling horizon with linearized unit commitment. It appears to be something with the ramp rate definition. I checked of course that all units have entries for these parameters. 

Do you have any idea what could be the issue?

Best,

Georg 

KeyError Traceback (most recent call last) File ~/micromamba/envs/open_eraa/lib/python3.12/site-packages/xarray/core/indexes.py:859, in PandasIndex.sel(self, labels, method, tolerance) 858 try: --> 859 indexer = self.index.get_loc(label_value) 860 except KeyError as e: File ~/micromamba/envs/open_eraa/lib/python3.12/site-packages/pandas/core/indexes/datetimes.py:627, in DatetimeIndex.get_loc(self, key) 625 else: 626 # unrecognized type --> 627 raise KeyError(key) 629 try: KeyError: 1262664000000000000 The above exception was the direct cause of the following exception: KeyError Traceback (most recent call last) Cell In[63], line 1 ----> 1 n.optimize.optimize_with_rolling_horizon(solver_name="cplex", **solver_options, linearized_unit_commitment=True) File ~/micromamba/envs/open_eraa/lib/python3.12/site-packages/pypsa/optimization/abstract.py:543, in OptimizationAbstractMixin.optimize_with_rolling_horizon(self, snapshots, horizon, overlap, **kwargs) 536 if not n.c.storage_units.static.empty: 537 n.c.storage_units.static.state_of_charge_initial = ( 538 n.c.storage_units.dynamic.state_of_charge.loc[ 539 snapshots[start - 1] 540 ] 541 ) --> 543 status, condition = n.optimize(sns, **kwargs) 544 if status != "ok": 545 logger.warning( 546 "Optimization failed with status %s and condition %s", 547 status, 548 condition, 549 ) File ~/micromamba/envs/open_eraa/lib/python3.12/site-packages/pypsa/optimization/optimize.py:464, in OptimizationAccessor.__call__(self, snapshots, multi_investment_periods, transmission_losses, linearized_unit_commitment, model_kwargs, extra_functionality, assign_all_duals, solver_name, solver_options, compute_infeasibilities, **kwargs) 461 n._linearized_uc = linearized_unit_commitment 463 n.consistency_check(strict=["unknown_buses"]) --> 464 m = n.optimize.create_model( 465 sns, 466 multi_investment_periods, 467 transmission_losses, 468 linearized_unit_commitment, 469 consistency_check=False, 470 **model_kwargs, 471 ) 472 if extra_functionality: 473 extra_functionality(n, sns) File ~/micromamba/envs/open_eraa/lib/python3.12/site-packages/pypsa/optimization/optimize.py:572, in OptimizationAccessor.create_model(self, snapshots, multi_investment_periods, transmission_losses, linearized_unit_commitment, consistency_check, **kwargs) 568 define_operational_constraints_for_extendables( 569 n, sns, c, attr, transmission_losses 570 ) 571 define_operational_constraints_for_committables(n, sns, c) --> 572 define_ramp_limit_constraints(n, sns, c, attr) 573 define_fixed_operation_constraints(n, sns, c, attr) 575 meshed_threshold = kwargs.get("meshed_threshold", 45) File ~/micromamba/envs/open_eraa/lib/python3.12/site-packages/pypsa/optimization/constraints.py:786, in define_ramp_limit_constraints(n, sns, component, attr) 784 status_start = c.dynamic["status"].iloc[start_i] 785 limit_diff = (limit_up - limit_start).isel(snapshot=0) --> 786 rhs.loc[{"snapshot": rhs.coords["snapshot"].item(0)}] += ( 787 limit_diff * status_start 788 ) 790 mask = active_com & non_null_up 791 m.add_constraints( 792 lhs, "<=", rhs, name=f"{c.name}-com-{attr}-ramp_limit_up", mask=mask 793 ) File ~/micromamba/envs/open_eraa/lib/python3.12/site-packages/xarray/core/dataarray.py:239, in _LocIndexer.__getitem__(self, key) 237 labels = indexing.expanded_indexer(key, self.data_array.ndim) 238 key = dict(zip(self.data_array.dims, labels, strict=True)) --> 239 return self.data_array.sel(key) File ~/micromamba/envs/open_eraa/lib/python3.12/site-packages/xarray/core/dataarray.py:1664, in DataArray.sel(self, indexers, method, tolerance, drop, **indexers_kwargs) 1548 def sel( 1549 self, 1550 indexers: Mapping[Any, Any] | None = None, (...) 1554 **indexers_kwargs: Any, 1555 ) -> Self: 1556 """Return a new DataArray whose data is given by selecting index 1557 labels along the specified dimension(s). 1558 (...) 1662 Dimensions without coordinates: points 1663 """ -> 1664 ds = self._to_temp_dataset().sel( 1665 indexers=indexers, 1666 drop=drop, 1667 method=method, 1668 tolerance=tolerance, 1669 **indexers_kwargs, 1670 ) 1671 return self._from_temp_dataset(ds) File ~/micromamba/envs/open_eraa/lib/python3.12/site-packages/xarray/core/dataset.py:2912, in Dataset.sel(self, indexers, method, tolerance, drop, **indexers_kwargs) 2844 """Returns a new dataset with each array indexed by tick labels 2845 along the specified dimension(s). 2846 (...) 2909 2910 """ 2911 indexers = either_dict_or_kwargs(indexers, indexers_kwargs, "sel") -> 2912 query_results = map_index_queries( 2913 self, indexers=indexers, method=method, tolerance=tolerance 2914 ) 2916 if drop: 2917 no_scalar_variables = {} File ~/micromamba/envs/open_eraa/lib/python3.12/site-packages/xarray/core/indexing.py:197, in map_index_queries(obj, indexers, method, tolerance, **indexers_kwargs) 195 results.append(IndexSelResult(labels)) 196 else: --> 197 results.append(index.sel(labels, **options)) 199 merged = merge_sel_results(results) 201 # drop dimension coordinates found in dimension indexers 202 # (also drop multi-index if any) 203 # (.sel() already ensures alignment) File ~/micromamba/envs/open_eraa/lib/python3.12/site-packages/xarray/core/indexes.py:861, in PandasIndex.sel(self, labels, method, tolerance) 859 indexer = self.index.get_loc(label_value) 860 except KeyError as e: --> 861 raise KeyError( 862 f"not all values found in index {coord_name!r}. " 863 "Try setting the `method` keyword argument (example: method='nearest')." 864 ) from e 866 elif label_array.dtype.kind == "b": 867 indexer = label_array KeyError: "not all values found in index 'snapshot'. Try setting the `method` keyword argument (example: method='nearest')."

[gktho...@gmail.com]

I forgot to mention, the issue somehow only occurs once the second horizon should get optimized. The first horizon runs smoothly. 

Best,

Georg 

[Iegor Riepin]

Dear Georg,

Could you please open PyPSA issue with this problem? and if possible add a minimalistic example to reproduce it. You can tag me (@Irieo) there. I will take care of this problem, and will share with you later what it was.

Kindly Iegor