[go] encoding/json/v2: report wrapped io.ErrUnexpectedEOF
5 views
Skip to first unread message
Joseph Tsai (Gerrit)
Jul 9, 2025, 7:59:16 PMJul 9
to goph...@pubsubhelper.golang.org, golang-co...@googlegroups.com
Joseph Tsai has uploaded the change for review![]( "Open in Gerrit")
Commit message
encoding/json/v2: report wrapped io.ErrUnexpectedEOF
In the event that the input is just JSON whitespace,
the underlying jsontext.Decoder treats this as an empty stream
and reports io.EOF.
The logic in unmarshalFull simply casted io.EOF as io.ErrUnexpectedEOF,
which is inconsistent with how all other io.ErrUnexpectedEOF are reported,
which are wrapped within a jsontext.SyntacticError.
Do the same thing for consistency.
We add a v1 test (without goexperiment.jsonv2) to verify that
the behavior is identical to how v1 has always behaved.
We add a v1in2 test (with goexperiment.jsonv2) to verify that
the v1in2 behavior correctly replicates historical v1 behavior.
Fixes #74548
Change-Id: Ibca52c3699ff3c09141e081c85f853781a86ec8e
Change diff
diff --git a/src/encoding/json/decode_test.go b/src/encoding/json/decode_test.go
index 473fd02..d6568c8 100644
--- a/src/encoding/json/decode_test.go
+++ b/src/encoding/json/decode_test.go
@@ -12,6 +12,7 @@
"errors"
"fmt"
"image"
+ "io"
"maps"
"math"
"math/big"
@@ -469,6 +470,8 @@
{CaseName: Name(""), in: `{"alphabet": "xyz"}`, ptr: new(U), err: fmt.Errorf("json: unknown field \"alphabet\""), disallowUnknownFields: true},
// syntax errors
+ {CaseName: Name(""), in: ``, ptr: new(bool), err: &SyntaxError{"unexpected end of JSON input", 0}},
+ {CaseName: Name(""), in: " \n\r\t", ptr: new(bool), err: &SyntaxError{"unexpected end of JSON input", 4}},
{CaseName: Name(""), in: `{"X": "foo", "Y"}`, err: &SyntaxError{"invalid character '}' after object key", 17}},
{CaseName: Name(""), in: `[1, 2, 3+]`, err: &SyntaxError{"invalid character '+' after array element", 9}},
{CaseName: Name(""), in: `{"X":12x}`, err: &SyntaxError{"invalid character 'x' after object key:value pair", 8}, useNumber: true},
@@ -1377,6 +1380,9 @@
if tt.disallowUnknownFields {
dec.DisallowUnknownFields()
}
+ if strings.TrimSpace(tt.in) == "" && tt.err != nil {
+ tt.err = io.EOF // in streaming mode, we expect EOF instead
+ }
if err := dec.Decode(v.Interface()); !equalError(err, tt.err) {
t.Fatalf("%s: Decode error:\n\tgot: %v\n\twant: %v\n\n\tgot: %#v\n\twant: %#v", tt.Where, err, tt.err, err, tt.err)
} else if err != nil && tt.out == nil {
diff --git a/src/encoding/json/v2/arshal.go b/src/encoding/json/v2/arshal.go
index 10b16ef..5cd2106 100644
--- a/src/encoding/json/v2/arshal.go
+++ b/src/encoding/json/v2/arshal.go
@@ -438,7 +438,8 @@
case nil:
return export.Decoder(in).CheckEOF()
case io.EOF:
- return io.ErrUnexpectedEOF
+ offset := in.InputOffset() + int64(len(in.UnreadBuffer()))
+ return &jsontext.SyntacticError{ByteOffset: offset, Err: io.ErrUnexpectedEOF}
default:
return err
}
diff --git a/src/encoding/json/v2/arshal_test.go b/src/encoding/json/v2/arshal_test.go
index 8494deed..879a2f3 100644
--- a/src/encoding/json/v2/arshal_test.go
+++ b/src/encoding/json/v2/arshal_test.go
@@ -7138,7 +7138,13 @@
inBuf: ``,
inVal: addr(structAll{}),
want: addr(structAll{}),
- wantErr: io.ErrUnexpectedEOF,
+ wantErr: &jsontext.SyntacticError{Err: io.ErrUnexpectedEOF},
+ }, {
+ name: jsontest.Name("Structs/Invalid/ErrUnexpectedEOF"),
+ inBuf: " \n\r\t",
+ inVal: addr(structAll{}),
+ want: addr(structAll{}),
+ wantErr: &jsontext.SyntacticError{Err: io.ErrUnexpectedEOF, ByteOffset: len64(" \n\r\t")},
}, {
name: jsontest.Name("Structs/Invalid/NestedErrUnexpectedEOF"),
inBuf: `{"Pointer":`,
diff --git a/src/encoding/json/v2_decode_test.go b/src/encoding/json/v2_decode_test.go
index 3ab20e2..f140467 100644
--- a/src/encoding/json/v2_decode_test.go
+++ b/src/encoding/json/v2_decode_test.go
@@ -12,6 +12,7 @@
"errors"
"fmt"
"image"
+ "io"
"maps"
"math"
"math/big"
@@ -473,6 +474,8 @@
{CaseName: Name(""), in: `{"alphabet": "xyz"}`, ptr: new(U), err: fmt.Errorf("json: unknown field \"alphabet\""), disallowUnknownFields: true},
// syntax errors
+ {CaseName: Name(""), in: ``, ptr: new(bool), err: &SyntaxError{"unexpected end of JSON input", 0}},
+ {CaseName: Name(""), in: " \n\r\t", ptr: new(bool), err: &SyntaxError{"unexpected end of JSON input", 4}},
{CaseName: Name(""), in: `{"X": "foo", "Y"}`, err: &SyntaxError{"invalid character '}' after object key", len64(`{"X": "foo", "Y"`)}},
{CaseName: Name(""), in: `[1, 2, 3+]`, err: &SyntaxError{"invalid character '+' after array element", len64(`[1, 2, 3`)}},
{CaseName: Name(""), in: `{"X":12x}`, err: &SyntaxError{"invalid character 'x' after object key:value pair", len64(`{"X":12`)}, useNumber: true},
@@ -1382,6 +1385,9 @@
if tt.disallowUnknownFields {
dec.DisallowUnknownFields()
}
+ if strings.TrimSpace(tt.in) == "" && tt.err != nil {
+ tt.err = io.EOF // in streaming mode, we expect EOF instead
+ }
if err := dec.Decode(v.Interface()); !equalError(err, tt.err) {
t.Fatalf("%s: Decode error:\n\tgot: %v\n\twant: %v\n\n\tgot: %#v\n\twant: %#v", tt.Where, err, tt.err, err, tt.err)
} else if err != nil && tt.out == nil {
diff --git a/src/encoding/json/v2_scanner.go b/src/encoding/json/v2_scanner.go
index 475bf58..aef045f 100644
--- a/src/encoding/json/v2_scanner.go
+++ b/src/encoding/json/v2_scanner.go
@@ -30,6 +30,10 @@
xd := export.Decoder(d)
xd.Struct.Flags.Set(jsonflags.AllowDuplicateNames | jsonflags.AllowInvalidUTF8 | 1)
if _, err := d.ReadValue(); err != nil {
+ if err == io.EOF {
+ offset := d.InputOffset() + int64(len(d.UnreadBuffer()))
+ err = &jsontext.SyntacticError{ByteOffset: offset, Err: io.ErrUnexpectedEOF}
+ }
return transformSyntacticError(err)
}
if err := xd.CheckEOF(); err != nil {
Change information
Files:
- M src/encoding/json/decode_test.go
- M src/encoding/json/v2/arshal.go
- M src/encoding/json/v2/arshal_test.go
- M src/encoding/json/v2_decode_test.go
- M src/encoding/json/v2_scanner.go
Change size: S
Delta: 5 files changed, 25 insertions(+), 2 deletions(-)
Related details
Attention set is empty
Submit Requirements:
Code-Review
No-Unresolved-Comments
Review-Enforcement
TryBots-Pass
| Inspect html for hidden footers to help with email filtering. To unsubscribe visit settings. |
Joseph Tsai (Gerrit)
Jul 9, 2025, 8:01:48 PMJul 9
to goph...@pubsubhelper.golang.org, Damien Neil, Dmitri Shuralyov, golang-co...@googlegroups.com
Attention needed from Damien Neil and Dmitri Shuralyov
Joseph Tsai voted![]( "Open in Gerrit")
| Auto-Submit | +1 |
| Commit-Queue | +1 |
Related details
Attention is currently required from:
- Damien Neil
- Dmitri Shuralyov
Submit Requirements:
Code-Review
No-Unresolved-Comments
Review-Enforcement
TryBots-Pass
| Inspect html for hidden footers to help with email filtering. To unsubscribe visit settings. |
Damien Neil (Gerrit)
Jul 9, 2025, 8:23:43 PMJul 9
to Joseph Tsai, goph...@pubsubhelper.golang.org, Go LUCI, Dmitri Shuralyov, golang-co...@googlegroups.com
Attention needed from Dmitri Shuralyov and Joseph Tsai
Damien Neil voted and added 1 comment![]( "Open in Gerrit")
Votes added by Damien Neil
| Code-Review | +2 |
1 comment
Patchset-level comments
Related details
Attention is currently required from:
- Dmitri Shuralyov
- Joseph Tsai
Submit Requirements:
Code-Review
No-Unresolved-Comments
Review-Enforcement
TryBots-Pass
| Inspect html for hidden footers to help with email filtering. To unsubscribe visit settings. |
Joseph Tsai (Gerrit)
Jul 9, 2025, 8:24:30 PMJul 9
to goph...@pubsubhelper.golang.org, Damien Neil, Go LUCI, Dmitri Shuralyov, golang-co...@googlegroups.com
Attention needed from Dmitri Shuralyov
Joseph Tsai added 1 comment![]( "Open in Gerrit")
Patchset-level comments
Damien Neil . resolved
Thanks.
Joseph Tsai
I apologize the delay. I vaguely recall you mentioning this in an email, but lost track of it.
Related details
Attention is currently required from:
- Dmitri Shuralyov
Submit Requirements:
Code-Review
No-Unresolved-Comments
Review-Enforcement
TryBots-Pass
| Inspect html for hidden footers to help with email filtering. To unsubscribe visit settings. |
Damien Neil (Gerrit)
Jul 11, 2025, 12:37:56 PMJul 11
to Joseph Tsai, goph...@pubsubhelper.golang.org, Go LUCI, Dmitri Shuralyov, golang-co...@googlegroups.com
Attention needed from Dmitri Shuralyov and Joseph Tsai
Damien Neil added 1 comment![]( "Open in Gerrit")
Patchset-level comments
Damien Neil . unresolved
It turns out that `json.Unmarshal` returns "unexpected end of JSON input" on a truncated input, but `json.Decoder.Decode` returns `io.EOF` or `io.UnexpectedEOF`.
And, of course, I've found at least some code that depends on this, at least in tests.
Not sure if we should have Decoder behave differently or what.
Related details
Attention is currently required from:
- Dmitri Shuralyov
- Joseph Tsai
Submit Requirements:
Code-Review
No-Unresolved-Comments
Review-Enforcement
TryBots-Pass
| Inspect html for hidden footers to help with email filtering. To unsubscribe visit settings. |
Joseph Tsai (Gerrit)
Jul 11, 2025, 2:48:50 PMJul 11
to goph...@pubsubhelper.golang.org, golang-co...@googlegroups.com
Attention needed from Damien Neil, Dmitri Shuralyov and Joseph Tsai
Joseph Tsai uploaded new patchset![]( "Open in Gerrit")
Joseph Tsai uploaded patch set #2 to this change.
Following approvals got outdated and were removed:
- Code-Review: +2 by Damien Neil
- TryBots-Pass: LUCI-TryBot-Result+1 by Go LUCI
Related details
Attention is currently required from:
- Damien Neil
- Dmitri Shuralyov
- Joseph Tsai
Submit Requirements:
Code-Review
No-Unresolved-Comments
Review-Enforcement
TryBots-Pass
| Inspect html for hidden footers to help with email filtering. To unsubscribe visit settings. |
Joseph Tsai (Gerrit)
Jul 11, 2025, 2:49:58 PMJul 11
to goph...@pubsubhelper.golang.org, Damien Neil, Go LUCI, Dmitri Shuralyov, golang-co...@googlegroups.com
Attention needed from Damien Neil and Dmitri Shuralyov
Joseph Tsai voted and added 1 comment![]( "Open in Gerrit")
Votes added by Joseph Tsai
| Auto-Submit | +1 |
| Commit-Queue | +1 |
It turns out that `json.Unmarshal` returns "unexpected end of JSON input" on a truncated input, but `json.Decoder.Decode` returns `io.EOF` or `io.UnexpectedEOF`.
And, of course, I've found at least some code that depends on this, at least in tests.
Not sure if we should have Decoder behave differently or what.
Joseph Tsai
Wow... gotta love the inconsistency between the two...
Apparently there's a bug (https://go.dev/issue/25956) about this discrepancy. I decided to preserve historical v1 behavior to reduce probable churn of when v1 is implemented in terms of v2.
Related details
Attention is currently required from:
- Damien Neil
- Dmitri Shuralyov
Submit Requirements:
Code-Review
No-Unresolved-Comments
Review-Enforcement
TryBots-Pass
| Inspect html for hidden footers to help with email filtering. To unsubscribe visit settings. |
Damien Neil (Gerrit)
Jul 11, 2025, 4:37:53 PMJul 11
to Joseph Tsai, goph...@pubsubhelper.golang.org, Go LUCI, Dmitri Shuralyov, golang-co...@googlegroups.com
Attention needed from Dmitri Shuralyov and Joseph Tsai
Damien Neil voted and added 1 comment![]( "Open in Gerrit")
Votes added by Damien Neil
| Code-Review | +2 |
1 comment
File src/encoding/json/decode_test.go
Line 1389, Patchset 2 (Latest):
}Damien Neil . resolved
Thanks, I hate it.
(But I hate it less than fixing the code that depends on the current behavior.)
Related details
Attention is currently required from:
- Dmitri Shuralyov
- Joseph Tsai
Submit Requirements:
Code-Review
No-Unresolved-Comments
Review-Enforcement
TryBots-Pass
| Inspect html for hidden footers to help with email filtering. To unsubscribe visit settings. |
Carlos Amedee (Gerrit)
Jul 11, 2025, 5:26:40 PMJul 11
to Joseph Tsai, goph...@pubsubhelper.golang.org, Damien Neil, Go LUCI, Dmitri Shuralyov, golang-co...@googlegroups.com
Attention needed from Dmitri Shuralyov and Joseph Tsai
Carlos Amedee voted Code-Review+1![]( "Open in Gerrit")
| Code-Review | +1 |
Related details
Attention is currently required from:
- Dmitri Shuralyov
- Joseph Tsai
Submit Requirements:
Code-Review
No-Unresolved-Comments
Review-Enforcement
TryBots-Pass
| Inspect html for hidden footers to help with email filtering. To unsubscribe visit settings. |
Gopher Robot (Gerrit)
Jul 11, 2025, 5:27:22 PMJul 11
to Joseph Tsai, goph...@pubsubhelper.golang.org, golang-...@googlegroups.com, Carlos Amedee, Damien Neil, Go LUCI, Dmitri Shuralyov, golang-co...@googlegroups.com
Gopher Robot submitted the change![]( "Open in Gerrit")
Change information
Commit message:
encoding/json/v2: report wrapped io.ErrUnexpectedEOF
In the event that the input is just JSON whitespace,
the underlying jsontext.Decoder treats this as an empty stream
and reports io.EOF.
The logic in unmarshalFull simply casted io.EOF as io.ErrUnexpectedEOF,
which is inconsistent with how all other io.ErrUnexpectedEOF are reported,
which are wrapped within a jsontext.SyntacticError.
Do the same thing for consistency.
We add a v1 test (without goexperiment.jsonv2) to verify that
the behavior is identical to how v1 has always behaved.
We add a v1in2 test (with goexperiment.jsonv2) to verify that
the v1in2 behavior correctly replicates historical v1 behavior.
We also fix a faulty check in v1 Decoder.Decode,
where it tried to detect errUnexpectedEnd and
return an unwrapped io.ErrUnexpectedEOF error.
This is the exact semantic that v1 has always done
in streaming Decoder.Decode (but not non-streaming Unmarshal).
There is a prior bug reported in #25956 about this inconsistency,
but we aim to preserve historical v1 behavior to reduce
the probability of churn when v1 is re-implemented in terms of v2.
Fixes #74548
Change-Id: Ibca52c3699ff3c09141e081c85f853781a86ec8e
Reviewed-on: https://go-review.googlesource.com/c/go/+/687115
Auto-Submit: Joseph Tsai <joe...@digital-static.net>
Reviewed-by: Carlos Amedee <car...@golang.org>
LUCI-TryBot-Result: Go LUCI <golang...@luci-project-accounts.iam.gserviceaccount.com>
Reviewed-by: Damien Neil <dn...@google.com>
Files:
- M src/encoding/json/decode_test.go
- M src/encoding/json/v2/arshal.go
- M src/encoding/json/v2/arshal_test.go
- M src/encoding/json/v2_decode_test.go
- M src/encoding/json/v2_scanner.go
- M src/encoding/json/v2_stream.go
Change size: S
Delta: 6 files changed, 40 insertions(+), 7 deletions(-)
Branch: refs/heads/master
Submit Requirements:
Code-Review: +2 by Damien Neil, +1 by Carlos Amedee
TryBots-Pass: LUCI-TryBot-Result+1 by Go LUCI
| Inspect html for hidden footers to help with email filtering. To unsubscribe visit settings. |
Reply all
Reply to author
Forward
0 new messages