Using pypsa with google colab?

[Barry McMullin]

Dear all -

Just wondering if anyone has managed to get pypsa to run inside google collaboratory (colab)? (colab offer the ability to run a .ipynb notebook directly from google drive, in browser, using a cloud-hosted runtime - no local install needed.)  I've tried a variety of approaches (pip, conda and some variations/combinations), but every time I hit some package that can't be installed or can't be found (even if nominally installed). The underlying problem seems to be that even though one can deploy miniconda (say) and use it to install packages, one can't force colab to actually run the miniconda python image (in place of its default). I've tried to work around this by manually tweaking sys.path (and other more radical surgery - installing miniconda so that it overwrites the default install) but have not hit on anything that gets to a working configuration.

So: if anyone happens to have already got through this pain, and has a recipe that works, I would really appreciate seeing how it can be done... (Probably I need to find a colab support list instead, but since pypsa is specifically what I want to run, I thought I would try this venue first...) 

Thanks - Barry.

[Fabian Neumann]

Hi Barry,

I have managed to create a running version of the minimal lopf example
(https://www.pypsa.org/examples/minimal_example_lopf.html) on colab:

https://colab.research.google.com/drive/1P_3xJjgoZlWWZsCtM1wyLM38Pr7ma9GU

I had difficulties with the solver 'glpk' at first, but alternatively
'cbc' works just fine.

```
# install solver
!apt-get install coinor-cbc

# adapted from
https://stackoverflow.com/questions/49202649/how-to-build-libraries-via-conda-on-colab-research

!wget -c https://repo.continuum.io/archive/Anaconda3-5.1.0-Linux-x86_64.sh
!chmod +x Anaconda3-5.1.0-Linux-x86_64.sh
!bash ./Anaconda3-5.1.0-Linux-x86_64.sh -b -f -p /usr/local
!conda install -y --prefix /usr/local -c conda-forge pypsa

import sys
sys.path.append('/usr/local/lib/python3.6/site-packages/')
```

The installation of anaconda, cbc and pypsa (+dependencies) takes about
90 seconds.

I hope this solves your problem!

Best wishes,
Fabian

On 25/06/2019 23:20, Barry McMullin wrote:
> Dear all -
>
> Just wondering if anyone has managed to get pypsa to run inside google
> collaboratory

> <https://colab.research.google.com/notebooks/welcome.ipynb> (colab)?

> /
>
> *Séanadh Ríomhphoist/Email Disclaimer*
>
> *Tá an ríomhphost seo agus aon chomhad a sheoltar leis faoi rún agus is
> lena úsáid ag an seolaí agus sin amháin é. Is féidir tuilleadh a léamh
> anseo. <https://sites.google.com/view/seanadh-riomhphoist>*
>
> *This e-mail and any files transmitted with it are confidential and are
> intended solely for use by the addressee. Read more here.
> <https://sites.google.com/view/dcu-email-disclaimer>*
>
>
>
> /
>
>
>
> <https://www.facebook.com/DCU/><https://twitter.com/DublinCityUni><https://www.linkedin.com/company/dublin-city-university><https://www.instagram.com/dublincityuniversity/?hl=en><https://www.youtube.com/user/DublinCityUniversity>
>
> --
> You received this message because you are subscribed to the Google
> Groups "pypsa" group.
> To unsubscribe from this group and stop receiving emails from it, send
> an email to pypsa+un...@googlegroups.com
> <mailto:pypsa+un...@googlegroups.com>.
> To post to this group, send email to py...@googlegroups.com
> <mailto:py...@googlegroups.com>.
> To view this discussion on the web, visit
> https://groups.google.com/d/msgid/pypsa/ffad791c-4f8a-4933-902b-35b415ac2831%40googlegroups.com
> <https://groups.google.com/d/msgid/pypsa/ffad791c-4f8a-4933-902b-35b415ac2831%40googlegroups.com?utm_medium=email&utm_source=footer>.
> For more options, visit https://groups.google.com/d/optout.

--
Karlsruhe Institute of Technology (KIT)
Institute for Automation and Applied Informatics (IAI)

Fabian Neumann, M.Sc.
PhD Student in Energy System Modelling

Phone: +49 721 608 25707
Mobile: +49 171 2943831
Fax: +49 721 608 22602
Website: https://www.iai.kit.edu/ESM.php
Github: fneum
LinkedIn: fabian-neumann-renewables
ORCID: 0000 0001 8551 1480

Building 445, Office 304
Campus North
Hermann-von-Helmholtz-Platz 1
76344 Eggenstein-Leopoldshafen

KIT – the Research University in the Helmholtz Association

[Barry McMullin]

Hi Fabian -

Many thanks for that: your example also works fine for me, so that was very helpful! (I had defaulted to working with a python2.7 configuration: that may or may not have been causing some of the extra pain for me. But in any case, it it great to now have a functioning example of pypsa in google collab...)

Kind regards - Barry.

Séanadh Ríomhphoist/Email Disclaimer

Tá an ríomhphost seo agus aon chomhad a sheoltar leis faoi rún agus is lena úsáid ag an seolaí agus sin amháin é. Is féidir tuilleadh a léamh anseo. 

This e-mail and any files transmitted with it are confidential and are intended solely for use by the addressee. Read more here.

[Barry McMullin]

For the record, using Fabian's example recipe for everything else, glpk also seemed to work fine for me:

###

!apt-get install glpk-utils
!conda install -y --prefix /usr/local -c conda-forge glpk

network.lopf(solver_name='glpk')

###

Regards - Barry.

[Ameen Sarhan]

Thanks  a lot for for such helpful example, I run it first time it worked but when I reseted the runtime and run it again I am stuck as shown in the attached screenshot

May I know how to verify PYTHONPATH ? 

> an email to py...@googlegroups.com

[Barry McMullin]

I have also had this experience of unreliable behaviour with colab+conda+pypsa.

I think the failure is probably unrelated to the specific warning message you mention (I think this will always appear when trying to use conda in the colab framework?). Rather, the issue is more likely with the limited backend resources that google allocates to any given colab session; something like that the google session times out or exhausts some allocated resource (memory? disk? network bandwidth?) before the pypsa+dependencies install can be completed. But trying another time, exactly the same thing might work fine!

For what it is worth, my overall conclusion is that while colab is intriguing, it is not yet ready for serious work; certainly not with something, such as pypsa, that requires substantial package installation (including conda): both because that appears to be unreliable and because, even if reliable, it is not persistent across spontaneous crashing of the colab session (on the google side), so that one then would have to go back and start the whole installation all over again.

So, for the moment at least, I would recommend reverting to local (ana)conda installation, and local pypsa installation, which will then be persistent; and running jupyter notebooks in the "conventional" way, interacting with a local jupyter notebook server. (It is possible to ask the google colab browser front end to connect with such a local jupter notebook server also, instead of a remote google server: I have not actually tried that, but it is then unclear what, if any, advantage it might offer over the conventional jupyter notebook browser experience).

Hope that helps,

- Barry.

To unsubscribe from this group and stop receiving emails from it, send an email to pypsa+un...@googlegroups.com.
To post to this group, send email to py...@googlegroups.com.
To view this discussion on the web, visit https://groups.google.com/d/msgid/pypsa/617cb6bc-9872-481f-8a10-cfa0297f1207%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

Séanadh Ríomhphoist/Email Disclaimer

Tá an ríomhphost seo agus aon chomhad a sheoltar leis faoi rún agus is lena úsáid ag an seolaí agus sin amháin é. Is féidir tuilleadh a léamh anseo. 

This e-mail and any files transmitted with it are confidential and are intended solely for use by the addressee. Read more here.

[Ameen Sarhan]

Thank you very much Barry for such helpful insights

To unsubscribe from this group and stop receiving emails from it, send an email to py...@googlegroups.com.

To post to this group, send email to py...@googlegroups.com.
To view this discussion on the web, visit https://groups.google.com/d/msgid/pypsa/617cb6bc-9872-481f-8a10-cfa0297f1207%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.