Translate/Localize Go, documentation, and identifiers

[matthe...@gmail.com]

The Go 2 blog post mentions Go in multiple languages:

"I'm not counting minor housekeeping changes like maybe allowing identifiers in more spoken languages…”

There’s an open proposal for a localization library: https://groups.google.com/forum/#!msg/golang-dev/hVRxxt1deoo/g6az0NduBAAJ

Today there was an issue added to the tracker for adding documentation to a Go tool. Reading that, I was wondering if there’s been discussion about localizing Go and Go documentation. Requiring translation is a significant requirement for documentation changes.

Discussion on golang-nuts had me wondering about hardware manufacturers around the world that I don’t know about. My guess is that not everybody is looking at English assembly and English compilers.

A tool to convert between keywords in different languages seems straightforward, but I know for me trying to work with identifiers in Chinese characters would be tough. I imagine working with English is difficult for many potential Go programmers around the world that use languages other than English. Perhaps Google Translate could help with this problem. Is there a vim that works from right to left?

Is there discussion or a consensus on freeing Go and the Go community from written language constraints?

Matt

[Andrew Bonventre]

Hi Matt,

On Thu, Jan 18, 2018 at 11:29 AM <matthe...@gmail.com> wrote:

The Go 2 blog post mentions Go in multiple languages:

"I'm not counting minor housekeeping changes like maybe allowing identifiers in more spoken languages…”

There’s an open proposal for a localization library: https://groups.google.com/forum/#!msg/golang-dev/hVRxxt1deoo/g6az0NduBAAJ

Today there was an issue added to the tracker for adding documentation to a Go tool. Reading that, I was wondering if there’s been discussion about localizing Go and Go documentation. Requiring translation is a significant requirement for documentation changes.

There has been discussion about an improved experience for our non-English speakers, yes. How that will manifest itself has not been determined to my knowledge.

 

Discussion on golang-nuts had me wondering about hardware manufacturers around the world that I don’t know about. My guess is that not everybody is looking at English assembly and English compilers.

A tool to convert between keywords in different languages seems straightforward, but I know for me trying to work with identifiers in Chinese characters would be tough. I imagine working with English is difficult for many potential Go programmers around the world that use languages other than English. Perhaps Google Translate could help with this problem. Is there a vim that works from right to left?

Is there discussion or a consensus on freeing Go and the Go community from written language constraints?

Perhaps I'm being pedantic, but it's not possible to do away with written language (whether it be documentation, identifiers, etc.). Could you be more specific on what you mean by "freeing [us] from written language constraints"?

Thanks,

A

Matt

--
You received this message because you are subscribed to the Google Groups "golang-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to golang-dev+...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[matthe...@gmail.com]

Could you be more specific on what you mean by "freeing [us] from written language constraints"?

Sure. I mean communicating code and code documentation without sharing a spoken/written language. For example, somebody that speaks and writes only German could give me a code review for my English-only code without knowing any English including Go English keywords.

Today every community separated by spoken/written language reinvents the wheel or are constrained to learning English. Given unicode and services like Google Translate I think we're within reach of such a utopian concept. And maybe developers in another spoken/written language will reach that goal first and provide a planetary programming language. I like Go though.

Matt

[matthe...@gmail.com]

Since this effort would take many people to realize I haven’t opened a proposal, but if somebody here thinks this is worthwhile to formally discuss then I would add it to the tracker. Here’s a rough draft section that may help illustrate the idea:

A component is a tool to convert between keywords in source files of different languages. These are the Go keywords (https://golang.org/ref/spec#Keywords):

break default func interface select case defer go map struct chan else goto package switch const fallthrough if range type continue for import return var

Predeclared identifiers would also be translated by this tool (https://golang.org/ref/spec#Predeclared_identifiers):

bool byte complex64 complex128 error float32 float64 int int8 int16 int32 int64 rune string uint uint8 uint16 uint32 uint64 uintptr true false iota nil append cap close complex copy delete imag len make new panic print println real recover

The predeclared identifiers are a special case because they are defined in the universe block and can be redefined by the programmer. These identifiers are only translated if in universe scope.

The tool would include all translations with the words in order:

var (
 English = [...]string{"break", "default", "func", "interface", "select", "case", "defer", "go", "map", "struct", "chan", "else", "goto", "package", "switch", "const", "fallthrough", "if", "range", "type", "continue", "for", "import", "return", "var", "bool", "byte", "complex64", "complex128", "error", "float32", "float64", "int", "int8", "int16", "int32", "int64", "rune", "string", "uint", "uint8", "uint16", "uint32", "uint64", "uintptr", "true", "false", "iota", "nil", "append", "cap", "close", "complex", "copy", "delete", "imag", "len", "make", "new", "panic", "print", "println", "real", "recover"}

 // initial Google Translate version to be edited by somebody that writes Latin
 Latin = [...]string{"abscindo", "ordinarius", "usus", "ora", "desumo", "conditio", "differo", "vado", "tabula", "struct", "canalis", "alium"

The text for English encodes into about 600 bytes. Including 50 languages would only take ~30KB, so including all keyword and predeclared identifier translations in a regular tool seems reasonable.

gotrans [flags] [path …]

The tool operates on files like gofmt. Target language can be specified, or if not then the set operating system locale is used. The tool guesses the current source file language by matching to these language lists. Part of go build would be to translate from another language into English for input into the compiler.

Good translations are key to the usefulness of the tool. Perhaps a web service can be developed to allow people writing in a language to propose and approve words, or expert translators can be recruited from the community. The word list would then be included in a tool update for each major Go release.

Changing Go to allow more written languages for identifiers is a necessary condition for this to be useful. Constraints like writing in different directions or representing numbers differently are not considered here.

Matt

[Andrew Bonventre]

+Aditya who has been working on this type of stuff for a bit.

[Brad Fitzpatrick]

Didn't Visual Basic try this back in the day and decided it was a mistake?

But I can't find a source for that now. Maybe I'm thinking of the wrong thing.

To unsubscribe from this group and stop receiving emails from it, send an email to golang-dev+unsubscribe@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "golang-dev" group.

To unsubscribe from this group and stop receiving emails from it, send an email to golang-dev+unsubscribe@googlegroups.com.

[Rob Pike]

It's not the right thing, I'll say that. I understand the intentions but I do not believe it fixes any real problem would fracture the community in irreparable ways.

-rob

[Andy Balholm]

I don’t know if Visual Basic did it. But HyperTalk and AppleScript both had foreign-language versions.

Andy

[matthe...@gmail.com]

While looking for Visual Basic information I found this article stating that there are a lot of programming languages in many languages: https://en.wikipedia.org/wiki/Non-English-based_programming_languages

If Go has merit for foreign language communities then perhaps their own downstream fork would be a better approach than adding features to English Go.

For example, the capitalization for identifier package visibility is a great feature in English. Maybe foreign language specifications would have similar features.

Matt

[Ralph Corderoy]

Hi Brad,

> Didn't Visual Basic try this back in the day and decided it was a
> mistake?

Seems so.

In the early 1990s Rodrigo [Miranda] worked in software translation.
At some point a call came: Microsoft needed someone in Brazil to
translate Excel's Visual Basic into Portuguese. Rodrigo could
barely contain his laughter when he was telling me the story.
`Yes', he said, `they literally wanted to translate all the
keywords. They wanted to make it "se" instead of "if", for
example.' Despite the prospect of making good money quickly,
Rodrigo told me that he tried to dissuade Microsoft from doing this.
When they decided to go ahead with the project, however, he agreed
to do it -- the money was too good to pass up. The Portuguese
version of Visual Basic failed miserably, much as Rodrigo expected
it to. (The money he got for it, though, paid for a new computer.)
— Coding Places: Software Practice in a South American City.
https://books.google.co.uk/books?id=YV-7wLbZKcgC&pg=PA53&lpg=PA53&dq=visual+basic+language+translations+keywords&source=bl&ots=I8FLkD51HH&sig=J9sWiTX7Go6-C9Z6zZ3JYC5SKco&hl=en&sa=X&ved=0ahUKEwichOqt5-zYAhVGIMAKHS14COcQ6AEIMDAG#v=onepage&q&f=false

--
Cheers, Ralph.
https://plus.google.com/+RalphCorderoy

[Aditya Mukerjee]

Thanks for the ping - and Matt, thanks for bringing this up. I've done some work in this area, and I've talked about it extensively (most recently at Go conferences in London and Brazil).

First, why this solves a really pressing problem: people who don't speak English as their first language consistently cite natural-language barriers as their top challenge when learning to program. And that's for people who already do have some degree of fluency in English. 89% of the world (6.7 billion people) don't. For those of us who grew up reading the Latin script, it's easy to forget, but even reading identifiers in a foreign script can be an additional burden.

And as far as burdens go, it's a completely unnecessary one, because computers can solve it for us. We don't require everyone to use the same text editor, or to use the same width for indenting code. Everyone uses the editor they want, with the indentation width that they want, and then gofmt takes care of transforming the code into a common, standardized form which we share. It's trivial to extend that same logic to localizing the Go language itself.

To demonstrate what localizing Go would look like, I translated Go into Bengali. Koro (https://github.com/ChimeraCoder/koro) is a translation of Go into Bengali, fully interoperable with the Go code we already know and love. For Bengali developers, it's much easier to be able to read and write code in the script that they're the most familiar with. When they're done, their code is automatically translated (via korofmt) into English.

As a language, Go lends itself incredibly well to localization - far more than any other language out there. Go made some wise decisions very early on which make this easy;; for example: the use of UTF-8 as a native encoding, a compiler bootstrapped in the language itself, a very simple grammar, and well-supported static analysis tools.

Because everyone's still able to communicate about the same code, there's no fragmentation involved here, any more than having non-native English speakers fragments the Go community. And I think we can all agree we're much better off for the contributions that non-native speakers have made to Go already.

Bridging natural-language barriers takes some work, but it's a fundamentally tractable problem. And IMO, there's no language better-suited for it than Go.


Aditya

[Rob Pike]

I find it hard to understand how this can make much difference. Even for native English speakers, the words in a program must be learned: the keyword "if" does not mean the same as "if"; ditto for "while" and "for" and "case"; let alone non-words like "func" and "const". And that leaves aside the libraries, the library documentation (doc strings), the function names, the variables, the idioms..... on and on. The keywords are a tiny part of what needs to be learned and understood.

Now, I am not saying that English's centrality is a universal good. I believe I am very privileged and lucky to speak the language of nerds, and -- I do think my record speaks for me here -- I have worked hard through much of my career to broaden the availability of non-English text to computing. (I recently learned, for instance, that Sam had wide usage in Greece in the 1980s because it was one of the few editors that could handle Greek text). We need to do much more to support speakers of other languages.

But I don't believe cosmetic changes as you propose are near the top of the list, or even all that helpful. We need better support for error messages in other languages, formats for times, numbers, and dates that support local customs. We need manuals that explain things in non-English languages (and I could tell stories about that too, from long ago). Those, and similar things that really support non-English languages, seem far more important that the relatively small things you propose. Your ideas may make people happy, which is not a small thing, but do they really help? And I am terrified of fracturing the community at any level.

As I said, though, I am lucky. Perhaps my perspective is wrong.

-rob

To unsubscribe from this group and stop receiving emails from it, send an email to golang-dev+unsubscribe@googlegroups.com.

[Henrik Johansson]

So wait would this result in actually translated keywords like this:

συνεχής (
    ταυτότητα_χρήστη ακέραιος_αριθμός 
)

instead of:

const (
   ταυτότητα_χρήστη int
)

assuming Google Translate does the right thing here...

That means we would have to learn one Go for each language or am I missing something?

I can say right away that it would make it impossible to use in a multi language team without enforcing any one particular language on a project and then we are back to the original problem.

I think Robs fears are well motivated even using this tiny example. 

[Volker Dobler]

On Tuesday, 23 January 2018 06:21:41 UTC+1, Aditya Mukerjee wrote:

[...] Because everyone's still able to communicate about the same code [...]

How should that be possible if they do not speak the same language?

If I want to communicate _about_ code I need some language, typically natural

language and both parties have to speak/understand enough to communicate.

I do not want to communicate _with_ code to other people (at least not

always and only with code). Explaining what code does and why and how

is done (and I think should be done) in natural language.

Maybe understanding "range" can be a problem in learning to write, read

and understand Go code if you do not speak any English, but I doubt

it will help if you offer a German, Swiss-German, Austrian-German,

Bavarian and an Allgäuerisch translation of "range".

V.   

 

[matthe...@gmail.com]

My thought was that this tool (like korofmt), as one part of the proposal, automates fast conversion between keywords/unique identifiers in varying natural languages, so somebody getting code with “συνεχής” could automatically convert it back to “const” at the speed of gofmt. The compiler only sees English keywords and unique identifiers. User identifiers in other natural languages have more use this way but translating those is another problem, as is translating documentation, most of which is necessary to use code.

This enables people to write in their own character system. While English letters are very readable to me, trying to recognize Chinese characters would be hard. I assumed this difficulty is the case for many people reading English code. I’ll probably never write code in another natural language, and as time moves on I think less people will have to. Only a third of programming languages are in English.

Even so, I don’t think forcing and directly supporting translation is the right thing for the Go project like proposed here. Go is tied to English and translating efforts will not be first class compared to a foreign community admiring the features and doing their own port (assuming government and Google regulations let them).

Go would be better served by focusing on communicating with foreign programming communities through bilingual individuals and tools like Google Translate and taking in novel code changes, not decreeing programming style as a foreigner.

Matt

[Hyang-Ah Hana Kim]

As a person who learned English-based programming languages before learning English,

I want to say that dealing with English-like keywords was not an issue at all. I (a primary

school student back then) and my classmates already accepted them as something 

similar to mathematical notation. 

Moreover, Go does not have too many keywords to remember - fewer than the number

of variables to memorize when reading a usual program.

Contrary to what some said, I still turn to 'code' when I can't communicate effectively 

in English because I know that my peers and I will interpret the code in the same way.

So, I am not sure whether I will like code with a lot of Korean or Chinese characters.

The challenge was more in finding resources like good books, manuals, or articles.

Participating in discussion in SNS, mailing lists, community(?), or conferences is still

challenging. I hope, as Go becomes more popular, we build more active community 

and have resources available in my home country and one can learn Go without 

reading anything written in English. I learned C/C++/Java (happily) without referencing 

any English article or documents. I hope Go is in the same state soon.

I also hope some syntax rules that assume Latin-based language (Upper case letter for 

exported names), error message handling, or style/naming guidelines and formatting 

rules that don't work outside English are rethought when developing future versions of Go.

Translation is more than word-by-word mapping, which I often consider bad translation. 

Bad translation is much more harmful than documentations available only in one language.

- Hana

--

You received this message because you are subscribed to the Google Groups "golang-dev" group.

To unsubscribe from this group and stop receiving emails from it, send an email to golang-dev+unsubscribe@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--

__

[Florian Weimer]

On 01/22/2018 10:30 PM, Brad Fitzpatrick wrote:
> Didn't Visual Basic try this back in the day and decided it was a mistake?

Visual Basic for Applications translated the keywords, yes.

This wasn't as crazy as it may sound today because the run-time library
of both Word and Excel had already been localized, probably following
Lotus 1-2-3's lead in this area. The Excel formula language is
localized up to the current day:

<https://support.office.com/de-de/article/ist-funktionen-0f2d7971-6019-40a0-a171-f2d869135665>

One side effect is that you need to know the English-language and
localized version of the environment if you want to reuse example code
(which generally is written in English). There are also numerous
technical issues, such as supporting applications which generate source
code, irrespective of the user-chosen locale.

On the Go side, I really don't see how the run-time library can be
translated in a semantics-preserving way. You basically would have to
have a 1:1 mapping for method names, so that interface compatibility is
preserved, and translate everything to English for reflect (something
that Excel did, IIRC0. That's going to be rather awkward.

Thanks,
Florian

[matthe...@gmail.com]

As a person who learned English-based programming languages before learning English,
I want to say that dealing with English-like keywords was not an issue at all. I (a primary
school student back then) and my classmates already accepted them as something 
similar to mathematical notation. 

After thinking about these issues and the responses here this past week a thought is that natural language is an important component of writing programs. While Go can be written without knowing English, good Go programs that can be reviewed by the Go community have carefully selected English words for identifiers.

Not every natural language will have a programming language as mature as Java, C, C++, or Go though. But the ideal solution is a programming language designed for the natural language and set of target computing applications.

I still haven’t read about non-English processor manufacturers, although assembly specification must be simpler to translate than a higher level programming language.

I also hope some syntax rules that assume Latin-based language (Upper case letter for 
exported names), error message handling, or style/naming guidelines and formatting 
rules that don't work outside English are rethought when developing future versions of Go.

I don’t agree with this now. The English nature of Go maximizes effective expression of a program in English.

Matt

[Lucio De Re]

I have been thinking that we do have it all backwards, so let me start
from the distant past and refer to something that seems to have been
totally overlooked in this discussion, namely the fable of the Tower
of Babel (I seem to recall you'll find it in Genesis, in the Bible's
Old Testament - there's always Wikipedia, I have no doubt).

Here at the Southern tip of Africa, we are cursed with eleven official
"languages" although all conversations around language (and there have
been some significant, in my opinion quite misguided decisions in the
recent past) of course occur in English.

Now, my thinking went like this: we write code for computers in a
notation (the use of the word "language" adds to the confusion)
intended to be a middle ground between our own natural language and
the instruction set of a particular target computer. In our efforts to
address as wide a computer selection as possible (portability), we
have made great strides in developing the notation, but we have had
minimal success in persuading the engineers designing computers to
approach an understanding of computation needs (I could go off on a
tangent here and rant about Intel, but I clearly don't need to).

What counts is that the problem should be tackled differently: our
notation needs a different perspective to evolve into a useful tool
and our "compilers" also need a different target.

Let me offer this to start the conversation: we need compilers that
translate code into natural language. And we need what our very own
Rob Pike named "idioms"
(https://en.wikipedia.org/wiki/The_Practice_of_Programming) to make
this possible.

As more than just an example, we need to express the Go runtime
library as an "implementation" of Donald Knuth's "Art of Computer
Programming" (and obviously considerably more) so that the compiler
(perhaps by some other name) can translate any invocation of, say, a
sort function into a localised idiom (freshly minted, if necessary)
for the reading benefit of the human (or android) that is reading,
analysing, the code.

Once we have inverted the role of the "compiler" like this, the
problem of a multitude of natural languages is solved: after all, Go
shows clearly that portability is neither difficult nor onerous (that,
to me, is the most important aspect of Go: I develop on my Plan 9
system and deploy more or less where I need to: Linux, NetBSD, Window
and OS-X, not quite IOS or Android, yet, sadly).

So, because I know I don't express myself clearly, do bear with me: my
wish for Go2 (more probably a later version of Go or not Go at all -
and I know that I share my admiration with "significant" others here
for APL - another story, relevant indeed) is that it should be
intended to be as agnostic as possible both towards natural language
and towards instruction sets and the algorithms implemented in such
instruction sets and that it should be possible to represent (rather
than translate) computer programs and even only code snippets into
accepted natural language descriptions, drawing on a dictionary of
idioms.

Once we have a notation that is sufficiently advanced to be able to do
this and we have cooperation in the creation and acceptance of a
growing encyclopedia of idioms readily expressed in such a notation,
the relevance of a CPU or even GPU instruction set becomes purely an
engineering task, rather than the political one that lead to Meltdown
and Spectre.

Of course, I do not have a sufficiently deep educational background in
Computer Science to support my suggestions, but I would certainly like
to have a reasonable conversations with anyone who feels this subject
can be taken further.

I apologise to anybody who may arrive at this point and feel I have
merely teased them without delivering any interesting insights.
Programming notations have been my pet subject since computing entered
my life officially in 1974, but I have mixed my hobby up with my
career as an IT general practitioner and I have been both unwilling
and unable to pursue this at an academic level. Please be patient with
my lack of a deeper understanding.

Lucio.

> --
> You received this message because you are subscribed to the Google Groups
> "golang-dev" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to golang-dev+...@googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.
>


--

Lucio De Re
2 Piet Retief St
Kestell (Eastern Free State)
9860 South Africa

Ph.: +27 58 653 1433
Cell: +27 83 251 5824
FAX: +27 58 653 1435

[matthe...@gmail.com]

In our efforts to address as wide a computer selection as possible (portability), we 
have made great strides in developing the notation, but we have had 
minimal success in persuading the engineers designing computers to 
approach an understanding of computation needs (I could go off on a 
tangent here and rant about Intel, but I clearly don't need to). 

Can you explain this more? I don’t see how instruction set architects could help here.

Once we have inverted the role of the "compiler" like this, the 
problem of a multitude of natural languages is solved

But then we’re back to programming in assembly. I’m not sure I understand how this adds when we’re talking about algorithms and systems tailored for all uses of the standard library and then we need to fix something hard fast.

it should be intended to be as agnostic as possible both towards natural language 
and towards instruction sets and the algorithms implemented in such 
instruction sets and that it should be possible to represent (rather 
than translate) computer programs and even only code snippets into 
accepted natural language descriptions, drawing on a dictionary of 
idioms. 

Does the reverse compiler present the terms “classes” and “objects”? Programming as-is already has many tradeoffs and disagreements.

I think looking ahead is an important part of this discussion. My opinion today is this reverse compiler would be best served here by focused development on Go with English features.

Matt

[Dan Kortschak]

This is not a tractable problem as it requires that there be an
algorithm that can determine the function of an arbitrary selection of
code. It seems to me that this is a restatement of the halting problem.

[Lucio De Re]

On 1/29/18, Dan Kortschak <dan.ko...@adelaide.edu.au> wrote:
> This is not a tractable problem as it requires that there be an
> algorithm that can determine the function of an arbitrary selection of
> code. It seems to me that this is a restatement of the halting problem.
>

It becomes (more) tractable if the programming language itself
highlights certain "constructs" as "deplored" or even deplorable. I
appreciate that that in itself is not a complete option, but nor is
the need for a computer language to encode all possible algorithms.

My gut feel is that certain patterns that "disturb" the conversion of
coded algorithms into a natural language representation could at
minimum be identified, if not deprecated.

Leaving aside the academic concept that an algorithm is defined as a
finite (halting) computation (from my memories of a long time ago), I
think it is still possible to express all computation in more or less
all languages with a scientific bent. If we can dedicate so much
effort to produce micro-optimisers, is it not possible, at least
asymptotically, to aim to define as many computations as possible as a
growing vocabulary of algorithms, at least, that can greatly simplify
the ability to exchange code with those who may speak, as computers
do, a different language?

And would that not be a valuable objective in bringing the very
intractability of certain classes of problems under some control?

Isn't it indeed a worthy objective to be able to identify and
potentially isolate instances of the halting problem, or is this
identification itself a halting problem? I fear my understanding is
limited in this respect.

Lucio.

PS: I'm out of my depth here and it shows. But I am keenly interested
in understandable explanations of where my thinking may be at fault.
Even analysing the discussion seems fruitful to me: the "halting
problem" is itself an "idiom" that exceeds representation in a
programming notation, is it not? Are there any examples of the
reverse: an encoded operation that cannot usefully be represented in
one or more natural languages?

[matthe...@gmail.com]

Hi Lucio,

I had the thought that what you are describing is a disassembler. While a program can be understood this way, and perhaps having a dictionary of idioms would help (when compiler optimizations don’t make this impossible), I don’t see how this would be better than translating the original source.

Matt

[Andy Balholm]

This idiom-recognition reminds me of the “sufficiently-smart compiler” problem: http://prog21.dadgum.com/40.html

Andy

[as....@gmail.com]

I imagine a Go where everyone uses keywords in their preferred language. Then I imagine stdlib, symbols, pull requests, merge conflicts, line/column numbers.

[Lucio De Re]

On 2/10/18, as....@gmail.com <as....@gmail.com> wrote:
> I imagine a Go where everyone uses keywords in their preferred language.
> Then I imagine stdlib, symbols, pull requests, merge conflicts, line/column
>
> numbers.
>

One could mandate the use of Esperanto for the keywords, etc.

There is a fair bit of APL love/hate among the Go developers and
aficionados and I think eventually the idea to turn everything into
precedence-free operators will win. Go 3 or 4, is my bet :-).

Why have UTF at all, if that isn't the final outcome?

(None of the above is meant to be taken seriously: but I do miss a
sane approach to operator overloading and everything said follows from
there.)

Lucio.