e30b5a06dc (master): ZJIT: Add diff tool for running benchmarks between git refs (#16160)

[Jeff Zhang]

Jeff Zhang 2026-03-03 16:19:28 +0000 (Tue, 03 Mar 2026)

New Revision: e30b5a06dc

https://github.com/ruby/ruby/commit/e30b5a06dc

Log:
ZJIT: Add diff tool for running benchmarks between git refs (#16160)

For https://github.com/Shopify/ruby/issues/822

Adds a small script for automatically running benchmarks between two git refs.

Example output (doubled an increment to `inline_cfunc_optimized_send_count` to illustrate stat diffs):
```
$ ./zjit_diff --before master
I, [2026-03-03T10:24:52.691660 #48554] INFO -- : Existing worktree found at /var/folders/l_/p20zlp5j3wx76vr9ggzgzcmr0000gn/T/ruby-zjit-before
HEAD is now at 7a3940e8b6 Unify rb_node_list_new and rb_node_list_new2
I, [2026-03-03T10:24:52.729815 #48554] INFO -- : Found existing build for master, skipping build
I, [2026-03-03T10:24:52.729881 #48554] INFO -- : Existing worktree found at /var/folders/l_/p20zlp5j3wx76vr9ggzgzcmr0000gn/T/ruby-zjit-after
HEAD is now at 3859f12966 dup
I, [2026-03-03T10:24:52.801287 #48554] INFO -- : Found existing build for HEAD, skipping build
I, [2026-03-03T10:24:52.801354 #48554] INFO -- : Running benchmarks
I, [2026-03-03T10:24:52.801404 #48554] INFO -- : ruby-bench already cloned, pulling from upstream

<... benchmark run logs ...>

before: ruby 4.1.0dev (2026-03-03T14:05:00Z master 7a3940e8b6) +ZJIT dev +PRISM [arm64-darwin25]
last_commit=Unify rb_node_list_new and rb_node_list_new2
after: ruby 4.1.0dev (2026-03-03T15:17:20Z :detached: 3859f12966) +ZJIT dev +PRISM [arm64-darwin25]

---------- ------------- ------------- ------------- ------------
bench before (ms) after (ms) after 1st itr before/after
lobsters 747.9 ± 2.9% 757.7 ± 2.4% 0.983 0.987
railsbench 1183.4 ± 1.9% 1214.5 ± 2.4% 1.001 0.974 (*)
---------- ------------- ------------- ------------- ------------

Legend:
- after 1st itr: ratio of before/after time for the first benchmarking iteration.
- before/after: ratio of before/after time. Higher is better for after. Above 1 represents a speedup.
- ***: p < 0.001, **: p < 0.01, *: p < 0.05 (Welch's t-test)

Output:
data/zjit_diff.json
================================================================================
ZJIT Stats Comparison
================================================================================

before (baseline):
ruby 4.1.0dev (2026-03-03T14:05:00Z master 7a3940e8b6) +ZJIT dev +PRISM [arm64-darwin25]
last_commit=Unify rb_node_list_new and rb_node_list_new2
after:
ruby 4.1.0dev (2026-03-03T15:17:20Z :detached: 3859f12966) +ZJIT dev +PRISM [arm64-darwin25]

BENCHMARK TIMINGS (lower is better)
--------------------------------------------------------------------------------
lobsters:
before avg: 0.748s min: 0.706s ★ (baseline)
after avg: 0.758s min: 0.735s +1.3% (slower)
railsbench:
before avg: 1.183s min: 1.158s ★ (baseline)
after avg: 1.214s min: 1.182s +2.6% (slower)

MEMORY USAGE
--------------------------------------------------------------------------------
lobsters:
before maxrss: 548.3MB zjit_mem: 68.6MB
after maxrss: 520.8MB zjit_mem: 68.3MB
railsbench:
before maxrss: 208.3MB zjit_mem: 29.3MB
after maxrss: 205.7MB zjit_mem: 29.3MB

SEND COUNTERS (showing differences > 5.0%)
--------------------------------------------------------------------------------
lobsters:
inline_cfunc_optimized_send_count 64,066,387 ( 37.3%) → 105,498,619 ( 61.4%) ▲ +64.7%
optimized_send_count 136,557,743 ( 79.5%) → 177,989,922 (103.6%) ▲ +30.3%
send_count 171,791,578 (100.0%) → 213,223,697 (124.1%) ▲ +24.1%
railsbench:
inline_cfunc_optimized_send_count 130,638,873 ( 36.3%) → 225,643,322 ( 62.8%) ▲ +72.7%
optimized_send_count 306,690,230 ( 85.3%) → 401,694,689 (111.8%) ▲ +31.0%
send_count 359,393,477 (100.0%) → 454,397,936 (126.4%) ▲ +26.4%

SUMMARY COUNTERS (showing differences > 5.0%)
--------------------------------------------------------------------------------
railsbench:
invalidation_time 7ms → 7ms ▲ +5.3%

COMPILE HIR PASS TIMINGS (showing differences > 5.0%)
--------------------------------------------------------------------------------
lobsters:
eliminate_dead_code_time 245ms → 232ms ▼ -5.3%
```

Added files:
tool/zjit_diff.rb