Go/Python interop

[Sebastien Binet]

Hi there,

In this week "compiler minutes" [1], one can read:

"""
- Go on future platforms (RAM efficiency. NUMA?)
- (maybe) Go-Python interop for AI-powered applications
- [David]: is it a good idea to use cgo for Go-Python interop?
- [Michael]: no. better with pipe or RPC
"""

Would it be possible to have a bit more informations ?
What kind of interop is it ? Exchanging binary data ? On disk ? Establishing a protobuf-based-like standard ?
Something else ?

Cheers,
-s

[1] https://github.com/golang/go/issues/43930#issuecomment-1821586955

[Eli Bendersky]

On Wed, Nov 22, 2023 at 11:31 AM Sebastien Binet <sebasti...@cern.ch> wrote:

Hi there,

In this week "compiler minutes" [1], one can read:

"""
- Go on future platforms (RAM efficiency. NUMA?)
- (maybe) Go-Python interop for AI-powered applications
- [David]: is it a good idea to use cgo for Go-Python interop?
- [Michael]: no. better with pipe or RPC
"""

Would it be possible to have a bit more informations ?
What kind of interop is it ? Exchanging binary data ? On disk ? Establishing a protobuf-based-like standard ?
Something else ?

All of it, maybe :-)

We're just exploring the issue, throwing ideas around. There are many potential options, each with its own tradeoffs in terms of performance vs. effort.

Eli

 

Cheers,
-s

[1] https://github.com/golang/go/issues/43930#issuecomment-1821586955

--
You received this message because you are subscribed to the Google Groups "golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/golang-nuts/f685b40c-00d1-4065-b423-ba945bebda88%40cern.ch.

[Sebastien Binet]

On Thu Nov 23, 2023 at 01:24 CET, Eli Bendersky wrote:
> On Wed, Nov 22, 2023 at 11:31 AM Sebastien Binet
> <sebasti...@cern.ch>
> wrote:
>
> > Hi there,
> >
> > In this week "compiler minutes" [1], one can read:
> >
> > """
> > - Go on future platforms (RAM efficiency. NUMA?)
> > - (maybe) Go-Python interop for AI-powered applications
> > - [David]: is it a good idea to use cgo for Go-Python interop?
> > - [Michael]: no. better with pipe or RPC
> > """
> >
> > Would it be possible to have a bit more informations ?
> > What kind of interop is it ? Exchanging binary data ? On disk ?
> > Establishing a protobuf-based-like standard ?
> > Something else ?
> >
>
> All of it, maybe :-)
> We're just exploring the issue, throwing ideas around. There are many
> potential options, each with its own tradeoffs in terms of performance
> vs. effort.

depending on the timescale, one could also imagine having a Go-based python VM (e.g. github.com/go-python/gpython) that can run a limited set of python modules (even the C-based ones, like pypy did at some point ?).
alternatively, "just" rely on pickle-based, Apache Arrow-based or numpy array-based exchanged data.

for Arrow and numpy, there are already packages that do offer a fair amount of interop:

- https://github.com/apache/arrow/tree/main/go
- https://github.com/sbinet/npyio/ (shameless plug.)

(we could imagine also adding some buffer protocol implementation à la PEP-3118)

for pickle, gpython has support up to protocol=3, and gopickle seems to have support for up to 5:
- https://github.com/nlpodyssey/gopickle
(adding support for array.array to gopickle was relatively straightforward)

-s

[Michael Knyszek]

Also, I'd like to clarify:

- [David]: is it a good idea to use cgo for Go-Python interop?
- [Michael]: no. better with pipe or RPC

I'm wrong about this. A conversation after the meeting clarified a misunderstanding I had about Go->Python calls specifically. Both Go->Python and Python->Go with cgo may very well be preferable to pipe/RPC in many circumstances. :)

[Sebastien Binet]

On Thu Nov 23, 2023 at 15:23 CET, 'Michael Knyszek' via golang-nuts wrote:
> Also, I'd like to clarify:
>
> - [David]: is it a good idea to use cgo for Go-Python interop?
> - [Michael]: no. better with pipe or RPC
>
> I'm wrong about this. A conversation after the meeting clarified a
> misunderstanding I had about Go->Python calls specifically. Both
> Go->Python
> and Python->Go with cgo may very well be preferable to pipe/RPC in many
> circumstances. :)

I think that, at least for the ML/AI use cases (where the API is python-based), tackling this via the cgo dance will be cumbersome:

to access the data or call the methods/functions, one will need to go
through the C.PyXYZ calls (even though most of the ML/AI modules are implemented in C for performances)

this means a world of C.PyIncref/Decref, C-callbacks, ...
(having done something along these lines with go-python/gopy (a set of tools to automatically expose a Go package as a CPython module), I speak from "some" experience)

so, I'd tend to agree with your previous self :)
what did change your mind ?

-s

> --
> You received this message because you are subscribed to the Google
> Groups "golang-nuts" group.
> To unsubscribe from this group and stop receiving emails from it, send
> an email to golang-nuts...@googlegroups.com.
> To view this discussion on the web visit

> https://groups.google.com/d/msgid/golang-nuts/cf461932-52bb-4ac1-9690-053bec7e432an%40googlegroups.com.

[Michael Knyszek]

On Thu, Nov 23, 2023 at 11:52 AM Sebastien Binet <bi...@cern.ch> wrote:

On Thu Nov 23, 2023 at 15:23 CET, 'Michael Knyszek' via golang-nuts wrote:
> Also, I'd like to clarify:
>
> - [David]: is it a good idea to use cgo for Go-Python interop?
> - [Michael]: no. better with pipe or RPC
>
> I'm wrong about this. A conversation after the meeting clarified a
> misunderstanding I had about Go->Python calls specifically. Both
> Go->Python
> and Python->Go with cgo may very well be preferable to pipe/RPC in many
> circumstances. :)

I think that, at least for the ML/AI use cases (where the API is python-based), tackling this via the cgo dance will be cumbersome:

to access the data or call the methods/functions, one will need to go
through the C.PyXYZ calls (even though most of the ML/AI modules are implemented in C for performances)

this means a world of C.PyIncref/Decref, C-callbacks, ...
(having done something along these lines with go-python/gopy (a set of tools to automatically expose a Go package as a CPython module), I speak from "some" experience)

so, I'd tend to agree with your previous self :)
what did change your mind ?

Sure, I recognize that it's still cumbersome and it may be better to use some language-agnostic protocol to communicate in some cases. I don't have very much experience with this specific boundary, so maybe I'm now wrong again for a different reason. :)

My misunderstanding was regarding a concern about concurrent Go->Python calls locking up the Python interpreter. I've come to understand that this generally isn't a problem in practice (or no worse than Python execution already mostly being serialized).