Issue with Pypsa optimization parallelization

[Santiago De Felipe García]

Dear Pypsa members:

I am studying diferent electric system panorams based as in the onshore wind and solar PV installed capacity as in the storage units located in my country with Pypsa, in order to detrmine the optimal solution to supply the electrtic demand in a specific periodo of time (a year with hourly resolution, currently). However, every optimization process take plenty amount of time (around 13-14 hours) and the complete simulation of all of the posible panorams take mora than a week, so I tried a code that runs Pypsa simultaneously for as cases as my computer memory let me with a parallelization method.

However, after visualizating the results and comparing them with what I obtained running Pypsa serially, I have noticed that the results aren't the same, that is to say, the total technologies generation supply the demand in both methods, but the generation of some technologies and others fluctuates with each other and they are not the same with one method as with another.

Said all of this, I write you to warn about this little issue, because it seems Pypsa don't let to parallelize in the optimization without fliuctuations in the results. Please let me know if I'm wrong in my conclusion and, in that case, how could I run the oiptimizations simultaneously in order to save time without this issue.

Regards,

Santiago.

[Fabian Neumann]

Hi Santiago,

It is a bit difficult to tell without knowing how exactly you tried to parallelise the optimisation of different scenarios.

If you make copies of the networks passed to the spawn processes, I would be surprised about unwanted interactions.

It may also make sense to set a random number seed for your solver.

Best wishes,

Fabian N

[Santiago De Felipe García]

Hi, Fabian:

I only have a computer with a script that takes use of an Ubuntu library called "parallel", this library runs every case of my electric system in diferente cores of the same computer simultaneously, so I achieve to parallelize my electric system simulations and save calculation time; every case is executed individually with its own network copy and doesn't influence on the input data of the other ones, so there aren't problems of overlap in calculations. However, as I commented previously, it seems this method has some kind of issue and doesn't generate the same optimization outputs that the method that runs the different situations in a queue, one behind the other. I suppose the parallel library doesn't work properly with Pypsa, please let me know if there is some detail I'm not having in mind.

Regards,

Santiago

--
You received this message because you are subscribed to a topic in the Google Groups "pypsa" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/pypsa/4iCMAFQYX_c/unsubscribe.
To unsubscribe from this group and all its topics, send an email to pypsa+un...@googlegroups.com.
To view this discussion on the web, visit https://groups.google.com/d/msgid/pypsa/04c23dc9-1d6c-4ef8-892a-c07fe6ba7ec4n%40googlegroups.com.

[Santiago De Felipe García]

Hi:

Ok, I think I will try other parallelization methods in order to save time in the simulations. Thank you for your time, Fabian.

Regards,

Santiago

El mar, 12 mar 2024 a las 14:12, Fabian Neumann (<f.ne...@tu-berlin.de>) escribió:

Hi Santiago,

I have made good experiences with the multiprocessing library within the
Python scripts.

Can't help with the "parallel" library.

Best wishes,

Fabian

Dr.-Ing.
Fabian Neumann (he/him)
Postdoctoral Researcher

Department of Digital Transformation in Energy Systems
Institute of Energy Technology
Technische Universität Berlin

Group website: https://www.tu.berlin/ensys
Personal website: https://neumann.fyi

Einsteinufer 25 (TA 8)
10587 Berlin

On 3/12/24 12:49, Santiago De Felipe García wrote:
> Hi, Fabian:
>
> I only have a computer with a script that takes use of an Ubuntu library
> called "parallel", this library runs every case of my electric system in
> diferente cores of the same computer simultaneously, so I achieve to
> parallelize my electric system simulations and save calculation time;
> every case is executed individually with its own network copy and
> doesn't influence on the input data of the other ones, so there aren't
> problems of overlap in calculations. However, as I commented previously,
> it seems this method has some kind of issue and doesn't generate the
> same optimization outputs that the method that runs the different
> situations in a queue, one behind the other. I suppose the parallel
> library doesn't work properly with Pypsa, please let me know if there is
> some detail I'm not having in mind.
>
> Regards,
>
> Santiago
>
> El vie, 8 mar 2024 a las 19:00, Fabian Neumann (<fn...@mail.tu-berlin.de

> <mailto:fn...@mail.tu-berlin.de>>) escribió:

>
>     Hi Santiago,
>
>     It is a bit difficult to tell without knowing how exactly you tried
>     to parallelise the optimisation of different scenarios.
>
>     If you make copies of the networks passed to the spawn processes, I
>     would be surprised about unwanted interactions.
>
>     It may also make sense to set a random number seed for your solver.
>
>     Best wishes,
>
>     Fabian N
>
>     On Friday 1 March 2024 at 13:01:28 UTC+1 sfe...@ujaen.es

>     <https://groups.google.com/d/topic/pypsa/4iCMAFQYX_c/unsubscribe>.

>     To unsubscribe from this group and all its topics, send an email to
>     pypsa+un...@googlegroups.com

>     <mailto:pypsa+un...@googlegroups.com>.

>     To view this discussion on the web, visit

>     https://groups.google.com/d/msgid/pypsa/04c23dc9-1d6c-4ef8-892a-c07fe6ba7ec4n%40googlegroups.com <https://groups.google.com/d/msgid/pypsa/04c23dc9-1d6c-4ef8-892a-c07fe6ba7ec4n%40googlegroups.com?utm_medium=email&utm_source=footer>.
>