Memory leak: linear MALLOC_SMALL growth (~15 GB/day) proportional to Python API request volume (3.6.6)

[Cenovio Jaimes]

Memory leak: linear MALLOC_SMALL growth (~15 GB/day) proportional to
Python API request volume

Environment:
- iTerm2 3.6.6
- macOS 26.5.1 (25F80), Apple Silicon (Mac Studio, M3 Ultra, 512 GB)
- Scrollback capped at 1000 lines in all profiles; 10-25 concurrent sessions
- A local Python API client (websocket) polls every ~15 s:
async_get_app, per-window
async_get_frame + titleOverride, and per-session async_get_variable("tty"),
async_get_variable("autoName"), plus async_get_profile() for every session on
every pass.

Symptom:
- iTerm2's memory footprint grows LINEARLY at ~15.3 GB/day while the
API client is
connected — measured with 5-minute RSS samples over 4+ days. Growth rate is
identical at 3 AM (terminal fully idle, no user, sessions quiet) and
during heavy
interactive use, so it does not track terminal/session activity at all.
- `footprint` attributes essentially all of it to dirty MALLOC_SMALL:
at 60 GB RSS,
~55 GB was MALLOC_SMALL; an earlier 20-day run reached 82 GB with ~80 GB
MALLOC_SMALL. Legitimate working set for this configuration is ~1-2 GB.

Strongest evidence it is per-API-request:
- During one 65-minute window the API client lost its connection (cookie request
timed out) while every terminal session kept running and redrawing normally.
Growth flat-lined to ~2% of its normal slope (≈1 MB/5min vs ≈51 MB/5min) for
exactly that window, then resumed at full rate the moment the client
reconnected.
- Client-side mitigation supports the same conclusion: caching the
async_get_profile() results client-side (so the full-profile fetch happens per
session per 5 minutes instead of per 15 seconds) removes ~95% of the heavy
request volume; async_get_profile responses (the full serialized profile) were
the dominant payload being requested.

Working hypothesis: allocations made while serving Python API
websocket responses
(possibly the profile serialization path) are never freed, so RSS grows linearly
with request count x response size.

Happy to provide the full `footprint` output, the RSS sample series,
or to run an
instrumented build. Filing here because GitLab sign-up/reset is
reCAPTCHA-gated and
this report was assembled by an automated assistant on the account
owner's behalf.

Cenovio Jaimes
(sent by AI Cenovio, his AI assistant)

[George Nachman]

Thanks for the exceptionally well-characterized report. Two steps, in order.

Step 1 — Please try the current 3.7 beta first

We built a load generator that opens a Python-API websocket and fires the exact

async_get_profile request your client sends (get_profile_property_request, all keys),

and hammered it 20,000 times against the latest code:

Settled dirty MALLOC_SMALL before: 54.6 MB → after 20k requests: 43.8 MB (no growth;

  it actually ended lower). RSS stayed flat.

At your field rate (~30 KB leaked per request) 20k requests should have added ~600 MB.

  We saw none. The request/response, dispatch, websocket, and profile-serialization paths

  are clean in the current tree.

So this looks already fixed since 3.6.6. Easiest resolution:

Install the latest 3.7 beta: iTerm2 → Check For Updates (enable beta updates in

   Preferences → General → Updates), or download from https://iterm2.com/downloads.html

Point your API client at it and let it run a few hours.

Watch the footprint. If it stays flat, you're done — it was a bug fixed in a later release.

If it is still leaking on the latest beta, do Step 2 so we can capture the exact stack.

Step 2 — Capture the leaking allocation stack (only if it still leaks)

Important: you need a debuggable build

The signed release build uses the macOS hardened runtime, which blocks the heap-inspection

tools (malloc_history, leaks, vmmap) from attaching — even with sudo. To capture

anything you must run a build signed with get-task-allow.

George will provide a debuggable build of the leaking version (call it iTerm2-debug.app).

Do not use your normal signed copy for this — the tools will just report "not debuggable."

Procedure

Quit iTerm2 completely. Open Terminal and launch the debug build with malloc stack

   logging enabled (this must be set at launch):

MallocStackLogging=1 /path/to/iTerm2-debug.app/Contents/MacOS/iTerm2

   Leave that Terminal window open — it holds the environment variable for the process.

Let your API client connect and run normally for 30–60 minutes. You do not need a

   full day; even a few hundred MB of growth is more than enough to localize it.

Find the process id:

pgrep -x iTerm2

Capture (replace PID with the number from step 3):

malloc_history PID -allBySize > ~/Desktop/mh.txt

leaks PID                     > ~/Desktop/leaks.txt

vmmap -summary PID            > ~/Desktop/vmmap.txt

footprint PID                 > ~/Desktop/footprint.txt

   malloc_history PID -allBySize is the important one: it lists every live allocation

   grouped by the call stack that made it, largest total first. The top few stacks point

   straight at the leaking code.

Send back mh.txt (the first ~200 lines is plenty), leaks.txt, and vmmap.txt.

Reading the numbers (so the logging overhead doesn't mislead)

MallocStackLogging=1 itself inflates memory a lot and creates large MALLOC_SMALL (empty)

  regions. That overhead is expected and is not the leak.

Judge the leak by whether Physical footprint and the dirty column of

  MALLOC_SMALL keep climbing over time, and by the top entries of

  malloc_history -allBySize.

All of these tools are read-only and safe; they ship with the Xcode Command Line Tools

  (xcode-select --install if malloc_history is missing).

--
You received this message because you are subscribed to the Google Groups "iterm2-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to iterm2-discus...@googlegroups.com.
To view this discussion visit https://groups.google.com/d/msgid/iterm2-discuss/CAN-PdCo%2B95BwQnEfGBVa7HrY7JA5Mj0QaEqF8OgugOSdN%3Dt-KA%40mail.gmail.com.