[ANN] tragach — eBPF observability for Firebird (1.0.0-beta)
40 views
Skip to first unread message
Zlatan Omerović
May 12, 2026, 10:21:09 PM (13 days ago) May 12
to firebird-general
Hello everyone,
Following the Firebird Web Client thread on a different angle: observability.
I've just published tragach (Bosnian: "tracer / one who follows tracks") — eBPF tracing for Firebird v5 SuperServer on Linux. Two small CLI tools that watch a running Firebird from outside. No client code change, no Firebird recompile, no Trace API plugin to configure.
tragach-slowquery — every DSQL statement the engine runs, with separate
prepare and execute timings, attachment ID, and the SQL
text. Cursor-based SELECTs are tracked through their full
openCursor -> fetchNext -> EOF lifecycle, not just the
open.
tragach-iowait — kernel-level off-CPU profiling of Firebird's threads.
Tells you WHERE a thread is blocked (block I/O, futex,
scheduler delay) and for how long, with representative
kernel-stack attribution per bucket.
Why this matters. isql's SET STATS ON gives you one wall-clock number per statement, but that number includes the TCP round-trip and the time isql spends formatting and printing rows. tragach probes DSQL_prepare / DSQL_execute / openCursor->EOF directly inside libEngine13.so, so its timings are *only* what the engine actually did. On localhost, isql can overstate engine cost by ~1 ms per statement — that's more than 50% measurement error for sub-millisecond queries. tragach also sees every attachment on the server at once, not just the one you happen to be typing into.
One request to Firebird upstream — USDT probes.
Today tragach has to resolve mangled C++ offsets from .debug/libEngine13.so.debug, filter .cold clones, and know the SysV AMD64 calling convention to read function arguments. It works (it's what 1.0.0-beta ships), but it's fragile: every Firebird release means re-validating a symbols artifact, and the symbols aren't ABI so they can shift on recompile.
If Firebird upstream were to add USDT probes — the Linux DTrace successor that PostgreSQL, MySQL, Node, Ruby, the JVM, and OpenSSL all ship — at the event boundaries the Trace API already exposes (event_dsql_prepare, event_dsql_execute, the cursor open/fetch lifecycle), tragach could attach by stable names like firebird:dsql_prepare__entry, drop the .debug dependency entirely, stop maintaining per-version symbol artifacts, and survive Firebird recompiles automatically. The probes the Trace API already calls into are the natural locations; the cost upstream is a handful of STAP_PROBE macros and a one-time <sys/sdt.h> build dep. I would be very happy to follow up with a focused RFC and patch once a few months of real-world tragach usage have shown which probe points carry their weight.
Companion project — tragach-forge.
To stress-test tragach I built a peer project, tragach-forge: an adversarial Firebird workload generator. 15 scenarios from steady OLTP and hot-row contention to cold scans, complex multi-table joins, correlated subqueries, recursive CTEs, window functions, and SUSPEND-style PSQL streaming. Mixed-mode loader, seed-deterministic NDJSON intent recording, plus a correlate.py that joins forge's recorded intents against tragach's observation stream and reports coverage / accuracy / false positives.
Forge is rough around the edges by design — its job is to give tragach interesting work, not to be a benchmark — and there's plenty of room to improve. It also runs standalone as a Firebird stress harness.
Where it's headed.
The current CLI binaries are the foundation. Reasonable next steps:
* Prometheus / OpenTelemetry / Grafana exporters so tragach plugs into
whatever observability stack your shop already uses, instead of being
one more thing to learn.
* Daemon mode hosting multiple probes simultaneously, with a control plane
over Unix socket or HTTP.
* The USDT upstream collaboration above. If that lands, the per-version
fragility goes away and tragach becomes meaningfully better for everyone.
* Classic / SuperClassic and Firebird v4 / v6 support, once v5
foundations are exercised in the wild.
Status. 1.0.0-beta — first public pre-release. Both probes implemented, validated against a live Firebird, overhead measured (~10% wall-clock at 3,300 stmts/sec for slowquery, ~0.6% for iowait on a Firebird workload). CI matrix builds and releases on Ubuntu 22.04 / 24.04 and Debian 12 / 13.
Following the Firebird Web Client thread on a different angle: observability.
I've just published tragach (Bosnian: "tracer / one who follows tracks") — eBPF tracing for Firebird v5 SuperServer on Linux. Two small CLI tools that watch a running Firebird from outside. No client code change, no Firebird recompile, no Trace API plugin to configure.
tragach-slowquery — every DSQL statement the engine runs, with separate
prepare and execute timings, attachment ID, and the SQL
text. Cursor-based SELECTs are tracked through their full
openCursor -> fetchNext -> EOF lifecycle, not just the
open.
tragach-iowait — kernel-level off-CPU profiling of Firebird's threads.
Tells you WHERE a thread is blocked (block I/O, futex,
scheduler delay) and for how long, with representative
kernel-stack attribution per bucket.
Why this matters. isql's SET STATS ON gives you one wall-clock number per statement, but that number includes the TCP round-trip and the time isql spends formatting and printing rows. tragach probes DSQL_prepare / DSQL_execute / openCursor->EOF directly inside libEngine13.so, so its timings are *only* what the engine actually did. On localhost, isql can overstate engine cost by ~1 ms per statement — that's more than 50% measurement error for sub-millisecond queries. tragach also sees every attachment on the server at once, not just the one you happen to be typing into.
One request to Firebird upstream — USDT probes.
Today tragach has to resolve mangled C++ offsets from .debug/libEngine13.so.debug, filter .cold clones, and know the SysV AMD64 calling convention to read function arguments. It works (it's what 1.0.0-beta ships), but it's fragile: every Firebird release means re-validating a symbols artifact, and the symbols aren't ABI so they can shift on recompile.
If Firebird upstream were to add USDT probes — the Linux DTrace successor that PostgreSQL, MySQL, Node, Ruby, the JVM, and OpenSSL all ship — at the event boundaries the Trace API already exposes (event_dsql_prepare, event_dsql_execute, the cursor open/fetch lifecycle), tragach could attach by stable names like firebird:dsql_prepare__entry, drop the .debug dependency entirely, stop maintaining per-version symbol artifacts, and survive Firebird recompiles automatically. The probes the Trace API already calls into are the natural locations; the cost upstream is a handful of STAP_PROBE macros and a one-time <sys/sdt.h> build dep. I would be very happy to follow up with a focused RFC and patch once a few months of real-world tragach usage have shown which probe points carry their weight.
Companion project — tragach-forge.
To stress-test tragach I built a peer project, tragach-forge: an adversarial Firebird workload generator. 15 scenarios from steady OLTP and hot-row contention to cold scans, complex multi-table joins, correlated subqueries, recursive CTEs, window functions, and SUSPEND-style PSQL streaming. Mixed-mode loader, seed-deterministic NDJSON intent recording, plus a correlate.py that joins forge's recorded intents against tragach's observation stream and reports coverage / accuracy / false positives.
Forge is rough around the edges by design — its job is to give tragach interesting work, not to be a benchmark — and there's plenty of room to improve. It also runs standalone as a Firebird stress harness.
Where it's headed.
The current CLI binaries are the foundation. Reasonable next steps:
* Prometheus / OpenTelemetry / Grafana exporters so tragach plugs into
whatever observability stack your shop already uses, instead of being
one more thing to learn.
* Daemon mode hosting multiple probes simultaneously, with a control plane
over Unix socket or HTTP.
* The USDT upstream collaboration above. If that lands, the per-version
fragility goes away and tragach becomes meaningfully better for everyone.
* Classic / SuperClassic and Firebird v4 / v6 support, once v5
foundations are exercised in the wild.
Status. 1.0.0-beta — first public pre-release. Both probes implemented, validated against a live Firebird, overhead measured (~10% wall-clock at 3,300 stmts/sec for slowquery, ~0.6% for iowait on a Firebird workload). CI matrix builds and releases on Ubuntu 22.04 / 24.04 and Debian 12 / 13.
Apache-2.0.
Repo: https://github.com/ZlatanOmerovic/tragach
Forge: https://github.com/ZlatanOmerovic/tragach-forge
Feedback, issues, and especially Firebird-devs interested in the USDT path — very welcome.
Best,
Zlatan
Repo: https://github.com/ZlatanOmerovic/tragach
Forge: https://github.com/ZlatanOmerovic/tragach-forge
Feedback, issues, and especially Firebird-devs interested in the USDT path — very welcome.
Best,
Zlatan
Zlatan Omerović
May 13, 2026, 8:01:39 AM (13 days ago) May 13
to firebird-general
One side I forgot to mention:
eBPF is available on Windows too. I did the Linux-only version as a proof-of-concept just to showcase what's possible and what are the future prospects of this.
I will develop a Windows version of tragach (tragach-nt) during this week.
I will develop a Windows version of tragach (tragach-nt) during this week.
Nikolai Davydik
May 13, 2026, 9:21:24 AM (12 days ago) May 13
to firebird-general
Looking forward to test Windows version!
Reply all
Reply to author
Forward
0 new messages