Proposal to add Index operator methods

[Raanan Hadar]

I have written a proposal to add index operator methods to Go2 here [1]. I would love to hear your commends, thank you.

[1] https://medium.com/@rhadar/go2-proposal-index-operator-methods-edc621cfefca

[jake...@gmail.com]

I stopped reading at the " Show example code before and after the change " because many of the examples of "before the change" code seem to be invalid in go, and the others do not do what you seem to imply they do.  Am I misinterpreting this section in some way?

[Raanan Hadar]

So I may have jumped too far ahead in that section in order to keep the proposal short.

The examples are typical of gonum and gorgonia/tensor. I have also implemented in my experience report a small library

in that spirit to make  sure that everything worked as well.

So you are right, its not vanilla Go and probably assumes a lot of familiarity.

I think its best that I will add the implementation as an appendix.

Thank you for the feedback.

[Dan]

On Sun, 2020-10-04 at 09:06 -0700, jake...@gmail.com wrote:
> I stopped reading at the " Show example code before and after the
> change " because many of the examples of "before the change" code
> seem to be invalid in go, and the others do not do what you seem to
> imply they do. Am I misinterpreting this section in some way?
>

I'm curious as to which examples of "before the change" you think are
invalid. From what I can see there, one is borderline (in the last
example there is no package selector, but it could have been a dot
import and is likely this way for brevity). The rest look like normal
method invocations, which are the standard way for this kind of thing
to be done in the current implementations of matrix and tensor
operations in Go.

Dan

[jake...@gmail.com]

The examples I was looking at are gone now. That section has been completely rewritten. So its kind of moot. Its possible that I was confusing the 'before' and 'after' examples, since they were not clearly labeled. In any case the rewritten version seems to make sense now.

[Raanan Hadar]

It actually didn't change that much, I just applied your valuable feedback. Thanks alot!

Its critical to communicate clearly.

so I changed the before and after part of the proposal to be more approachable to people who

are less familiar with numerical libraries in Go.

Its not a trivial proposal in that the main audience will probably use it in a library and not in vanilla Go.

But it was still a very valid point.

thanks again.

[Jon Reiter]

i think this looks nice but realistically only cleans up v simple cases.  a lot of the very-verbose-and-not-so-elegant-looking multidimensional math stuff looks "worse" than other languages because go doesn't have operator overloading.  it is not clear to me this improves syntax too much in practice on it's own.  (this is not a request to add operator overloading but merely the non-controversial statement that operator overloading can help math-heavy code look more "hand drawn" sometimes).

the gorgonia example is indicative of the issue: nil may be less aesthetically pleasing than ":" (or not) but it's unlikely the surrounding code looks anything like the math as expressed on paper by hand either.

gonum is quite convenient even with this...code that uses it can look a bit more 1960s than other languages but it is simple and super-clear.  i like that i am never (read: rarely) confused by () or , or [] etc characters in go.  if you could find a way to cover more ground the change would look more compelling to me.

--
You received this message because you are subscribed to the Google Groups "golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/golang-nuts/f49975aa-5a76-45b7-a158-b1067b1dc580n%40googlegroups.com.

[Raanan Hadar]

Lets start with saying that I think this is a valid criticism.

Although it does not solve things 100%, it does solve the heart of the problem for me at least:

There is a mixture of index operators with functional operators in the syntax:

The [] operator gives the reader the 'where' context and the () gives the reader the 'what' context. If you are using all your arguments via () causes the readability problem. So while not having full operator overloading, you still have to use .Add() instead of '+', the grammar is now consistent and concise and separates the 'where' from the 'what'.

Also, please consider that different users have a different 'value scale' to gain from this proposal:

To a gonum user, this proposal is about a 20% improvement and nothing to run home about.

To a matlab/python programmer, this solves about 80% of their problems and combining this with the other benefits of Go is enough to strongly consider switching to Go.

Realistically, I don't think that I can convince the Go core dev team to add full operator overloading and that is

probably the only way to achieve what you are suggesting... if you have another way to achieve this, I am open

In the end of the day, the question to ask is whether or not we

are better off in implementing this.

[Wojciech S. Czarnecki]

Dnia 2020-10-06, o godz. 00:56:38
Raanan Hadar <rha...@gmail.com> napisał(a):

> To a gonum user, this proposal is about a 20% improvement and nothing to run home about.
> To a matlab/python programmer, this solves about 80% of their problems and combining this
> with the other benefits of Go is enough to strongly consider switching to Go.

> if you have another way to achieve this, I am open In the end of the day,
> the question to ask is whether or not we are better off in implementing this.

A "better notation" for the minority of language users can be achieved using go generate.

Your OP problem* calls for a transpiling tool - one that from the python/matlab input produces boring Go readable also by general Go audience.

*[that matlab/numpy accustomed programmers would be more productive using known or similar to known to them syntax]

Hope this helps,

--
Wojciech S. Czarnecki
<< ^oo^ >> OHIR-RIPE

[Raanan Hadar]

A transpiler that will implement index operator methods will be very appealing at first. But it won't be long before general operator overloading will be added to its feature set and the result is splitting Go to a different language...

The question is, what is best to the go community in the long run? To add the minimal viable set of features that solves the crux of the problem (I don't think this proposal degrades the experience for the "general Go audience"), or to throw in the towel and spin this off to a different language? Personally, I would rather have index operator methods and forego full operator overloading because of the huge long term engineering costs that a transpiled language introduces.

[Randall O'Reilly]

Jon's points probably sink this proposal. Go is a very different language, and making it less self-consistent to solve some fraction of syntactic readability is almost certainly not going to get over the very high threshold for changing the language.

If you're going to switch over to Go, you'll have to learn a LOT of new concepts and change the way you approach things. In the end, it is better that Go remain "pure and simple" -- the simpler it remains, the easier it is for anyone to learn it, no matter where they are coming from.

Another strategy, which I'm pursuing, is to support Python as a scripting language on top of Go backend code that does all the heavy lifting. That is how Python is used in most cases already, except with C / C++ backends. So, keep Python Python, and keep Go Go, but allow people to interoperate more easily. This way, people can get some exposure to the virtues of Go, and maybe decide to switch over at some point..

This tool makes it relatively easy: https://github.com/go-python/gopy I've also written a GoToPy converter https://github.com/goki/gotopy (will be moving over to go-python soon -- no time right now), so you can write 'main' programs in Go, then easily convert them over to Python (Go -> Python is very easy, the other way not so much), which provides a point of departure for Python folks to start using your code..

- Randy

> To view this discussion on the web visit https://groups.google.com/d/msgid/golang-nuts/ee94ef0a-c461-4262-99fe-25a52ee11646n%40googlegroups.com.

[Raanan Hadar]

So we respectfully disagree here, so the best thing I can do is add some quality arguments. I think this is an important discussion that might cement Go as a broad DSL in the sense that it does a lot of things exceptionally well, but this issue will remain a blind spot.

I fully understand the need to keep Go "pure and simple". Its the key to its success. But this is an important area where Go does not have enough features to be effective. Secondly, I think that solving this issue can be achieved while maintaining orthogonality and not hurting experience of the majority. Third, as I answered Jon, I think that looking at this proposal, or more generally, this problem, from the Go programmer's perspective, is like trying to optimize a local optima. Go programmers don't have much to gain from this. But from a global perspective that seeks to optimize software development as a whole, there's a huge benefit that is overlooked here. Go was not designed to cater the needs of the programmer but software engineering as an end to end process (https://talks.golang.org/2012/splash.article).

With regards to Python, I've detailed common examples in my proposal, where Python for numerical computing leaves a lot to be desired. Fundamental issues which Go solves by design and are not addressed effectively by putting the two together as you suggested. Secondly, I am not the only one saying that there is a real need for a better language than Python. To name two good examples: Yann LeCun said this last year: https://venturebeat.com/2019/02/18/facebooks-chief-ai-scientist-deep-learning-may-need-a-new-programming-language/ and Jeremy Howard, former president and chief scientist of Kaggle and the author of the fast.ai, strongly advocates that Python is not the way to go in long run: https://www.youtube.com/watch?v=c-KAf2g6YN4 

[Randall O'Reilly]

I think we all agree that Python has some major issues.. :)

But the point of Jon's argument which struck me as very compelling is that Go would really need full operator overloading to serve as a serious competitor given the standards to which people have become accustomed. I didn't see in your replies a clear response to that, but maybe I missed it?

The really tricky issue is that everyone wants something like Python *as a starting point* when just messing around, because it is so easy and flexible to just write simple expressions etc. But then once things start to scale up, all the "issues" start to outweigh those initial advantages. And for a huge number of Python users, they never scale up and don't care about anything more than getting their immediate needs solved, mostly by copy / paste / modify programming..

That's why I think the Python / Go hybrid approach could be good -- end-user type people can mess around in Python, but the same functionality is available in Go, and as stuff starts to get scaled up to the point of requiring more libraries, those can hopefully be done in Go, so that Python always remains on the "messing around" scale of things, while the deeper infrastructure gets built out in Go.. This probably requires a fair bit of extra work and motivation on the part of the Go-side developers to keep up with things and support the developing needs of their users, so it may not be plausible for a larger-scale community, but it is working pretty well for my small-scale biological neural network modeling community..

Functionally, it is sort of similar to having a transpiler, in that Python is just providing a simple front-end syntax for accessing the deeper Go functionality.

Anyway, wasn't Julia supposed to solve all these issues? I don't know much about it in detail, but I googled just to see what the latest discussion there is relative to Python -- this thread is informative and current:

https://discourse.julialang.org/t/why-is-python-not-julia-still-used-for-most-state-of-the-art-ai-research/45896/17

The basic thrust is that people don't care and just go with what everyone else is doing. This was the best quote IMO:

> There are countries that still do not use the metric system, you figure it out.

Also this is informative:

> For my subjective idea I would also say Python, as someone mentioned here, is much easier language than Julia. Coming from MATLAB looking at packages of Julia seems to me as an act of wizard.

Python hits a sweet spot of simplicity and power.. Julia apparently does not. Go hits a different such sweet spot, but not with the same kind of transparent syntactic expressions needed for data science / ML etc.

- Randy

> To view this discussion on the web visit https://groups.google.com/d/msgid/golang-nuts/c475d7e6-3724-409d-9532-6b933b67cf2dn%40googlegroups.com.

[Raanan Hadar]

So the first paragraph in my answer to Jon addresses this. But I think I could have been clearer: I think 80% of the readability problem is addressed by index operator methods, which is the critical part. Full operator overloading is nice to have in my opinion and I argue people will gladly live without it given all the other benefits Go brings to the table in software engineering. I think its a very similar approach to what Go did with pointers: Is full pointer arithmetic beneficial? yes, but it would no longer be in the spirit of Go.

I think that using languages such as Matlab and Python as a starting point, or "prototyping languages" is becoming less and less relevant today. This might have been the way we developed algorithms ten years ago but today its just like any software development. The reality many data scientists and engineers face today is that data pipelines are constantly changing and increasing in complexity. They need to be maintainable and allow for quick iteration, integration and deployment. These are the qualities that Go excels at. A language that can effectively achieve a prototype in 80% the syntax efficiency but can sustain quality over time  is much more important than having the richest syntax possible. Another point is that even while in the "prototyping" phase, big data challenges which data scientist face today, unlike ten years ago, will require the data scientist to do some heavy lifting to get efficient compute, network and IO to get that initial result. So in many cases the "prototype" is often a complex pipeline that is deployed in k8s for example.

I think that a Go-Python integration is excellent but as I noted, it leaves a lot to be desired. A lot has been written about the "infinite loop of sadness" (here is a good link: https://medium.com/pachyderm-data/sustainable-machine-learning-workflows-8c617dd5506d). I think this is a big problem with companies like Google having an excellent understanding of challenges of sustaining data pipelines at scale. Its a lot of what they do in their day to day. This is why, I think the synergy in a single language will be amazing.

As for Julia, I was also excited when I first learned about it and I hope I am not offending anyone using Julia here. Because on paper its marketed as the killer language for this, but Julia has two big problems (I sample Julia regularly):

1. Julia is nowhere near ready. Julia has a few more years before it gets its basics down. I think it has made a lot of progress in the last few years but someone trying Julia out in 2020 is guaranteed to hit some really annoying hurdles from day one. I'll give you a trivial example, lets say you want an IDE for Julia. Your best bet today is probably a text editor like vim or emacs. You have JUNO for atom which is nice but no longer developed or the vscode IDE, which might be amazing in a few years but is far from stable (https://github.com/julia-vscode/julia-vscode/issues). And this is just one critical issue out of many.

2. My second issue with Julia is more troublesome because unlike the first, I don't think it will ever be solved. Julia is a language that was designed to be very rich syntax and feature wise. But its toolchain was completely neglected. Go has an amazing toolchain, its not the sexiest part of the language, but its critical and it was designed to encourage good software engineering. As Rob Pike said: "Although tools like these are rarely mentioned in the context of language design, they are an integral part of a language's ecosystem and the fact that Go was designed with tooling in mind has a huge effect on the development of the language, its libraries, and its community". Julia has poor package management and tooling which make it a pain to deploy in production compared to Go.

In summary, If Go were to implement this, I really believe that it will run circles around Julia even in a few years from now when it reaches puberty. Because Go is a better language when it comes to software engineering.

[Randall O'Reilly]

Those are good points and FWIW I would be excited to use the proposed indexing functions and agree that they are a big part of the issue, independent of the operator overloading. There has been a lot of discussion about supporting multidimensional indexing etc so this has been a major issue over the years, so maybe this will go forward!

- Randy

> To view this discussion on the web visit https://groups.google.com/d/msgid/golang-nuts/82947d3c-e7ad-4f3b-8358-50d6809c040fn%40googlegroups.com.

[Raanan Hadar]

Thank you for your supportive words. I do hope that this will get the ball rolling.

[Raanan Hadar]

Hi everyone,

The proposal was closed about a week ago with the conclusion that: "There was further discussion, but no change in consensus on this particular proposal."

First of all, let me begin that I am always humbled by the wisdom and passion of the Go community and that this whole process has taught me a lot.

Since most Go2 proposals fail to be accepted, I am not surprised by the outcome. As I said in my experience report, my main goal was to raise awareness of the problem addressed in the proposal, and not realistically solve it in one Go. As I have put a lot of effort into writing the experience report and later, the proposal, I would really appreciate to hear some feedback about what the community thinks, and if a core dev team can share a bit more specifics, this will allow me to learn and improve.

With this introduction over, I have 3 simple question. You DON'T need to answer them all. Any answer will be appreciated.

1. What did you think about the problem the proposal addresses? Do you think its significant?

2. Do you think this is a problem that Go should solve in a future version?

3. Lastly, what are the strengths and weaknesses of the proposal?

Many thanks.