f3f3270f70 (master): [ruby/json] Fix ResumableParser losing tokens before a feed-boundary suspension

[Masataka Pocke Kuwabara]

Masataka Pocke Kuwabara 2026-07-06 04:16:42 +0000 (Mon, 06 Jul 2026)

New Revision: f3f3270f70

https://github.com/ruby/ruby/commit/f3f3270f70

Log:
[ruby/json] Fix ResumableParser losing tokens before a feed-boundary suspension

When a feed boundary fell right after a consumed token -- a `,` or an opening
`[` / `{` -- whose effect was not yet committed to the parser's persistent
state, the token was dropped on resume, producing parse errors or silently
wrong values.

## Reproduction

```ruby
require "json"

def parse_in_two(first, second, **opts)
parser = JSON::ResumableParser.new(**opts)
parser << first
parser.parse
parser << second
parser.parse
parser.value
rescue JSON::ParserError => e
e.message
end

# a ',' and its closing bracket split across the boundary
p parse_in_two("[1,", "]", allow_trailing_comma: true)
# a comment right after '[' split across the boundary
p parse_in_two("[/*", "*/1]", allow_comments: true)
```

## Expected Behavior

```
[1]
[1]
```

Feeding `[1,]` or `[/**/1]` in a single chunk parses to `[1]`, so splitting it
at the comma or inside the comment should make no difference.

## Actual Behavior

```
"unexpected character: ']'"
1
```

For the comma, the closing bracket that arrived in the later chunk was rejected
because the comma had been forgotten. For the bracket, the array's first
element leaked out as a bare top-level document (`1` instead of `[1]`), and a
following `parse` would raise `unexpected character: ']'` on the orphaned
closing bracket. The same shapes affect objects (`{"a":1,` + `}`, `{/*` +
`*/"a":1}`), line comments (`[//` + `"x\n1]"`), and comments right after a
comma. It happens with the default configuration too, since comments are
accepted (with a deprecation warning) unless disabled.

## Description

Both bugs share one root cause: the parser consumed a token, then hit a
suspension point (end of buffer, possibly inside a comment) before recording
the token's effect in the persistent frame/phase state, so resuming had no
memory of the token. They differ in where that record lives, so the fix has
two parts.

On `,`, the comma phases used to eat whitespace and comments before
committing the frame's phase, in order to peek for a trailing-comma close.
The fix advances the phase to `JSON_PHASE_VALUE` (array) or
`JSON_PHASE_OBJECT_KEY` (object) immediately after consuming the comma.
Trailing-comma detection moves into those phases: an element/key position
that finds `]` / `}` (only reachable after a comma, since an empty container
closes inline) hands off to the comma phase to close.

On `[` / `{`, the bracket's frame is only pushed after the empty-container
check, and pushing it earlier is not an option: an empty container is decoded
inline without a frame, which is what keeps `[]` from consuming a nesting
level (`[[]]` parses with `max_nesting: 1`). Committing a frame first would
break that property or require reworking nesting accounting, affecting the
non-resumable parser as well. Instead, widen the rewind: the suspension
already rewinds the cursor, it just rewound to the start of the comment,
which is after the bracket. `json_eat_comments` gains a `resume_pos` that,
in resumable mode only, overrides the rewind target for all three suspension
sites (unterminated block comment, unterminated line comment, and a comment
marker split at the boundary), and the `[` / `{` cases pass the bracket's own
position via the new `json_eat_whitespace_resume_at` variant. Re-reading the
bracket and the comment on resume mirrors how strings and numbers already
rewind to `value_start` when they hit the end of the buffer. Non-resumable
parsing is unaffected in both parts: the same tokens are consumed in the same
order, and error positions still point at the comment.

Co-Authored-By: Claude Opus 4.8 <nor...@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PpeQSEFkF1X1Uuzjx9BsYe

https://github.com/ruby/json/commit/cc010c4688

Modified files:
ext/json/parser/parser.c
test/json/resumable_parser_test.rb