[tools] gopls/doc: Document semantic tokens

Peter Weinberger (Gerrit)

unread,

Jun 8, 2021, 9:31:50 AM6/8/21

to goph...@pubsubhelper.golang.org, golang-co...@googlegroups.com

Peter Weinberger has uploaded this change for review.

gopls/doc: Document semantic tokens

WORK IN PROGRESS, DO NOT SUBMIT!
Describe how gopls handles semantic tokens. Comments welcome.

Change-Id: I7af056690c26cc219fd703ca4069d239afada939
---
A gopls/doc/semantictokens.md
M internal/lsp/testdata/semantic/a.go
M internal/lsp/testdata/semantic/a.go.golden
3 files changed, 114 insertions(+), 0 deletions(-)

diff --git a/gopls/doc/semantictokens.md b/gopls/doc/semantictokens.md
new file mode 100644
index 0000000..901acda
--- /dev/null
+++ b/gopls/doc/semantictokens.md
@@ -0,0 +1,108 @@
+# Semantic Tokens
+
+The [LSP](https://microsoft.github.io/language-server-protocol/specifications/specification-3-17/#textDocument_semanticTokens)
+specifies semantic tokens as a way of telling clients about language-specific
+properties in a file being edited.
+
+The client asks for a set of semantic tokens and modifiers. This note describe which ones
+gopls will return, and under what circumstances. Gopls has no control over how the client
+converts semantic tokens into colors (or some other visible indication). In vscode it
+is possible to modify the color a theme uses by setting the `editor.semanticTokenColorCustomizations`
+object. We provide an [example](#Colors) later.
+
+There are 22 semantic tokens, with 10 possible modifiers. In principle, each semantic
+token can be used with any of the 1024 subsets of possible modifiers.
+
+The 22 semantic tokens are `namespace`, `type`, `class`, `enum`, `interface`,
+		`struct`, `typeParameter`, `parameter`, `variable`, `property`, `enumMember`,
+		`event`, `function`, `member`, `macro`, `keyword`, `modifier`, `comment`,
+		`string`, `number`, `regexp`, `operator`.
+
+The 10 modifiers are `declaration`, `definition`, `readonly`, `static`,
+		`deprecated`, `abstract`, `async`, `modification`, `documentation`, `defaultLibrary`.
+
+For the implementation to work correctly the client and server have to agree on the ordering
+of the tokens and of the modifiers. Gopls, therefore, will only send tokens and modifiers
+that the client has asked for. This document says what gopls would send if the client
+asked for everything. By default, vscode asks for everything.
+
+Gopls sends 11 tokens for `.go` files and 1 for `.*tmpl` files.
+Nothing is sent for any other kind of file.
+This list could change.
+
+For `.*tmpl` files gopls sends `macro`, and no modifiers, for each `{{`...`}}` scope.
+
+## Semantic tokens for Go files
+
+There are two contrasting guiding principles that might be used to decide what to mark
+with semantic tokens. All clients already do some kind of syntax marking. E.g., vscode
+uses a TextMate grammar. The minimal principle would send semantic tokens only for those
+language features that cannot be reliably found without parsing Go and looking at types.
+The maximal principle would attempt to convey as much as possible about the Go code,
+using all available parsing and type information.
+
+There is much to be said for returning minimal information, but the minimal principle is
+not well-specified. Gopls has no way of knowing what the clients know about the Go program
+being edited. Even in vscode the TextMate grammars can be more or less elaborate
+and change over time. (A minimal implementation probably would never return `keyword`,
+`number`, `comment`, or `string`.)
+
+The maximal position isn't particularly well-specified either. To chose one example, a
+format string might have formatting codes (`%[4]-3.6f`), escape sequences (`\U00010604`), and regular
+characters. Should these all be distinguished? One could even imagine distinguishing
+different runes by their Unicode language assignment, or some other Unicode property.
+
+Gopls does not come close to either of these principles.  Semantic tokens are returned for
+identifiers, keywords, operators, comments, and literals. (Sematic tokens do not
+cover the file. They are not returned for
+white space or punctuation, and there is no semantic token for labels.)
+The following describes more precisely what it
+does, with a few notes on alternative choices.
+The references to *object* refer to the
+`types.Object` returned by the type checker. The references to *nodes* refer to the
+`ast.Node` from the parser.
+
+1. __`keyword`__ All Go keywords are marked `keyword`.
+1. __`namespace`__ All package names are marked `namespace`. In an import, if there is an
+alias, it would be marked. Otherwise the last component of the import path is marked.
+1. __`type`__ Objects of type `types.TypeName` are marked `type`. If they are also `types.Basic`
+the modifier is `defaultLibrary`. (And in `type B struct{C}`, `B` has modifier `definition`.) <span style="color:red">CHECK THIS, semantic.go:522</span>.
+1. __`parameter`__ The formal arguments in `ast.FuncDecl` nodes are marked `parameter`.
+1. __`variable`__ Identifiers (node `ast.Ident`) with no type are modified with `definition`.
+(Give more examples!) Identifiers in the
+scope of `const` are modified with `readonly`. `nil` is a `variable` modified with both
+`readonly` and `defaultLibrary`. (`nil` is a predefined identifier; the user can redefine it,
+in which case it would just be a variable, or whatever.) Identifiers of type `types.Variable` are,
+not surprisingly, marked `variable`. Identifiers being defined (node `ast.GenDecl`) are modified
+by `definition` and, if appropriate, `readonly`. Receivers (in method declarations) are
+`variable` <span style="color:red">what's happening at line 535?</span>
+1. __`member`__ Members are marked at their definition (```func (x foo) bar() {}```) or declaration
+in an `interface`. Members are not marked where they are used. In `x.y()`, `x` will be marked
+either as a `namespace` if it is a package name, or as a `variable` if it is an interface value,
+so distinguishing `y` seemed superfluous.
+1. __`function`__ Bultins (`types.Builtin`) are modified with `defaultLibrary`
+(e.g., `make`, `len`, `copy`). Identifiers whose
+object is `types.Func` or whose node is `ast.FuncDecl` are `function`.
+1. __`comment`__ Comments and struct tags. (Perhaps struct tags should be `property`?)
+1. __`string`__ Strings. Could add modifiers for e.g., escapes or format codes.
+1. __`number`__ Numbers. Should the `i` in `23i` be handled specially?
+1. __`operator`__ Assignment operators, binary operators, ellipses (`...`), increment/decrement
+operators, sends (`<-`), and unary operators.
+
+Gopls will send the modifier `deprecated` if it finds a comment
+`// deprecated` in the godoc.
+
+The unused tokens for Go code are `class`, `enum`, `interface`,
+		`struct`, `typeParameter`, `property`, `enumMember`,
+		`event`, `macro`, `modifier`,
+		`regexp`
+
+## Colors
+
+These comments are about vscode.
+
+The documentation has a [helpful](https://code.visualstudio.com/api/language-extensions/semantic-highlight-guide#custom-textmate-scope-mappings)
+description of which semantic tokens correspond to scopes in TextMate grammars. Themes seem
+to use the TextMate scopes to decide on colors.
+
+Some examples of color customizations are [here](https://medium.com/@danromans/how-to-customize-semantic-token-colorization-with-visual-studio-code-ac3eab96141b).
\ No newline at end of file
diff --git a/internal/lsp/testdata/semantic/a.go b/internal/lsp/testdata/semantic/a.go
index 756c56e..54d6c8a 100644
--- a/internal/lsp/testdata/semantic/a.go
+++ b/internal/lsp/testdata/semantic/a.go
@@ -55,6 +55,8 @@
 	w := b[4:]
 	j := len(x)
 	j--
+	q := []interface{}{j, 23i, &y}
+	g(q...)
 	return true
 }
 
@@ -74,5 +76,6 @@
 	if !ok {
 		switch x := vv[0].(type) {
 		}
+		goto Never
 	}
 }
diff --git a/internal/lsp/testdata/semantic/a.go.golden b/internal/lsp/testdata/semantic/a.go.golden
index 512a83e..55e2de1 100644
--- a/internal/lsp/testdata/semantic/a.go.golden
+++ b/internal/lsp/testdata/semantic/a.go.golden
@@ -56,6 +56,8 @@
 	/*⇒1,variable,[definition]*/w /*⇒2,operator,[]*/:= /*⇒1,variable,[]*/b[/*⇒1,number,[]*/4:]
 	/*⇒1,variable,[definition]*/j /*⇒2,operator,[]*/:= /*⇒3,function,[defaultLibrary]*/len(/*⇒1,variable,[]*/x)
 	/*⇒1,variable,[]*/j/*⇒2,operator,[]*/--
+	/*⇒1,variable,[definition]*/q /*⇒2,operator,[]*/:= []/*⇒9,keyword,[]*/interface{}{/*⇒1,variable,[]*/j, /*⇒3,number,[]*/23i, /*⇒1,operator,[]*/&/*⇒1,variable,[]*/y}
+	/*⇒1,function,[]*/g(/*⇒1,variable,[]*/q/*⇒3,operator,[]*/...)
 	/*⇒6,keyword,[]*/return /*⇒4,variable,[readonly]*/true
 }
 
@@ -75,6 +77,7 @@
 	/*⇒2,keyword,[]*/if /*⇒1,operator,[]*/!/*⇒2,variable,[]*/ok {
 		/*⇒6,keyword,[]*/switch /*⇒1,variable,[definition]*/x /*⇒2,operator,[]*/:= /*⇒2,variable,[]*/vv[/*⇒1,number,[]*/0].(/*⇒4,keyword,[]*/type) {
 		}
+		/*⇒4,keyword,[]*/goto Never
 	}
 }

To view, visit change 326049. To unsubscribe, or for help writing mail filters, visit settings.

kokoro (Gerrit)

unread,

Jun 8, 2021, 9:37:26 AM6/8/21

to Peter Weinberger, goph...@pubsubhelper.golang.org, Go Bot, golang-co...@googlegroups.com

Kokoro presubmit build finished with status: SUCCESS
Logs at: https://source.cloud.google.com/results/invocations/3a570a51-d3f4-4617-93ae-f01dd1c67748

Patch set 1:gopls-CI +1

View Change

To view, visit change 326049. To unsubscribe, or for help writing mail filters, visit settings.

Peter Weinberger (Gerrit)

unread,

Jun 9, 2021, 8:20:29 AM6/9/21

to goph...@pubsubhelper.golang.org, golang-co...@googlegroups.com

Attention is currently required from: Rebecca Stambler.

Peter Weinberger uploaded patch set #2 to this change.

View Change

gopls/doc: Document semantic tokens

WORK IN PROGRESS, DO NOT SUBMIT!
Describe how gopls handles semantic tokens. Comments welcome.

Also fixes some errors in generating semantic tokens.



Change-Id: I7af056690c26cc219fd703ca4069d239afada939
---
A gopls/doc/semantictokens.md
M internal/lsp/testdata/semantic/a.go
M internal/lsp/testdata/semantic/a.go.golden

3 files changed, 125 insertions(+), 1 deletion(-)

To view, visit change 326049. To unsubscribe, or for help writing mail filters, visit settings.

Peter Weinberger (Gerrit)

unread,

Jun 9, 2021, 8:29:30 AM6/9/21

to goph...@pubsubhelper.golang.org, golang-co...@googlegroups.com

Attention is currently required from: Peter Weinberger, Rebecca Stambler.

Peter Weinberger uploaded patch set #3 to this change.

View Change

gopls/doc: Document semantic tokens

WORK IN PROGRESS, DO NOT SUBMIT!
Describe how gopls handles semantic tokens. Comments welcome.

Also fixes some errors in generating semantic tokens.

Change-Id: I7af056690c26cc219fd703ca4069d239afada939
---
A gopls/doc/semantictokens.md

M internal/lsp/semantic.go
M internal/lsp/testdata/semantic/a.go
M internal/lsp/testdata/semantic/a.go.golden
4 files changed, 153 insertions(+), 3 deletions(-)

To view, visit change 326049. To unsubscribe, or for help writing mail filters, visit settings.

Peter Weinberger (Gerrit)

unread,

Jul 6, 2021, 11:38:43 AM7/6/21

to goph...@pubsubhelper.golang.org, Go Bot, kokoro, Rebecca Stambler, golang-co...@googlegroups.com

Peter Weinberger abandoned this change.

View Change

Abandoned

new revisions, and semantic.go changes in a separate CL

To view, visit change 326049. To unsubscribe, or for help writing mail filters, visit settings.

Reply all

Reply to author

Forward