[tools] gopls/internal/protocol/semtok: fix delta encoding bug

[Alan Donovan (Gerrit)]

Alan Donovan submitted the change

Change information

Commit message:

gopls/internal/protocol/semtok: fix delta encoding bug

In the previous implementation, the delta encoding logic
assumed that i==0 when indexing tokens for the line number,
but this is false if the first token was skipped.
This CL rewrites the encoder more simply to avoid the bug.
(+ test)

Also, fix a TODO in Editor.interpretTokens that causes
it to mishandle non-BMP characters, since the Token.Length
field is expressed in UTF-16 code units.
(+ test)

Fixes golang/go#78269

Change-Id: Ia122fe0ed2529ceca17428beb29537f82ab972bd

Reviewed-on: https://go-review.googlesource.com/c/tools/+/758100

Reviewed-by: Madeline Kalil <mka...@google.com>

LUCI-TryBot-Result: Go LUCI <golang...@luci-project-accounts.iam.gserviceaccount.com>

Files:

• M gopls/internal/protocol/semtok/semtok.go
• M gopls/internal/test/integration/fake/editor.go
• M gopls/internal/test/integration/misc/semantictokens_test.go

Change size: M

Delta: 3 files changed, 172 insertions(+), 62 deletions(-)

Branch: refs/heads/master

Submit Requirements:

• Code-Review: +2 by Madeline Kalil
• TryBots-Pass: LUCI-TryBot-Result+1 by Go LUCI

Open in Gerrit

Inspect html for hidden footers to help with email filtering. To unsubscribe visit settings.

Gerrit

Gerrit-MessageType: merged

Gerrit-Project: tools

Gerrit-Branch: master

Gerrit-Change-Id: Ia122fe0ed2529ceca17428beb29537f82ab972bd

Gerrit-Change-Number: 758100

Gerrit-PatchSet: 7

Gerrit-Owner: Alan Donovan <adon...@google.com>

Gerrit-Reviewer: Alan Donovan <adon...@google.com>

Gerrit-Reviewer: Go LUCI <golang...@luci-project-accounts.iam.gserviceaccount.com>

Gerrit-Reviewer: Madeline Kalil <mka...@google.com>

Gerrit-CC: Ingo Oeser <night...@googlemail.com>