ANN: TurboBean – A Fast Beancount Implementation

[Moritz Drexl]

Hi all,

I'd like to share a project I'm working on: A from-scratch implementation of

Beancount that is fast and independent of the Python ecosystem.

https://github.com/themoritz/turbobean

Why?

I love Beancount, but as my journal has grown, processing takes a considerable
amount of time. I came across the Beancount Vnext: Goals & Design document
which talks about how to address this with re-implementing the core in C++. And
since Martin has so little time working on Beancount these days, I thought why
not work on this evolved version of Beancount myself? I added some of my own
design ideas to the mix and the result is TurboBean.

Philosophy

My vision is to have this very minimalist and fast core Beancount
implementation that is not tied to any language or ecosystem. It has some
essentials included (in particular the LSP) to get started quickly but then
mainly just interfaces with other languages via Protobuf.

- Single binary, zero dependencies. You download one small binary and it just
  works — no Python environment, no pip, no virtualenvs.
- Instant processing. Even large journals are parsed and processed in
  milliseconds.
- Plugins in Lua. Extend and customize processing without the overhead of
  a full Python runtime.
- Interop with other languages. Just use templating to generate .bean files.
  Pipe the output of processed transactions into any other language via
  Protobufs.
- LSP and formatter built in. Every editor gets first-class support out of the
  box — jump to account definitions, hover for balances, auto-completion and
  renaming for accounts/tags/links.
- Practical Web UI included. A built-in web server gives you basic journal,
  balance sheet, and income statement views to get started quickly. Similar to
  Fava.

Compatibility

TurboBean aims to be as compatible with existing .bean files as possible. Most
files should work without changes. Where breaking changes exist, they stem from
the redesigned booking system inspired by Vnext. The key changes are described
in this document.

Current Status

The project is at a point where I use the LSP for editing all my .bean files
and the Web UI for basic reports.

I plan to work next on Lua plugins, the formatter as well as Protobuf output.

The project is definitely still exploratory so it would be great to have people
play with it, and get feedback on design decisions since that's always the
toughest part.

Best,
Moritz

[Martin Blais]

Great work Moritz! Thank you for sharing it broadly :-)

--
You received this message because you are subscribed to the Google Groups "Beancount" group.
To unsubscribe from this group and stop receiving emails from it, send an email to beancount+...@googlegroups.com.
To view this discussion visit https://groups.google.com/d/msgid/beancount/a2b794df-0e57-4bbb-abe3-1df34db824e3n%40googlegroups.com.

[Timothy Jesionowski]

Hi Moritz, two thoughts. First of all, can I query the ledger using TurboBean? And second of all, how large are your ledgers that processing takes a lot of time? I have kept a fairly detailed ledger for three years and any processing takes less than a tenth of a second. 

Sincerely,

Timothy Jesionowski

[David Avraamides]

Hi Timothy,

As a data point, I have a ledger with 59k directives, 113k postings and 56k transactions covering a little over 13 years of data (I imported my history when I moved to Beancount a few years ago). If I delete my .picklecache file and start up bean-query (which processes my full ledger), it takes 3.1s on my M4 Pro Mac mini (48GB of memory). While 3 seconds doesn't sound like much, when you hit it over and over (say when doing EOM recs), it does add notable friction. If I'm doing read-only work where it can use the cache, it's better, but still 1.1s per invocation.

-David

[Moritz Drexl]

Hey Timothy,

It's mostly like David describes. While I don't have as many transactions as he has (only ~6 years worth of data), running a query still takes on the order of 1s on my M1 MacBook Air. It's not preventing me from maintaining my ledger but adds friction when iterating a lot (recs). I really enjoyed that instant feedback feeling when starting out with Beancount and no longer have it. And it's only going to get worse every year that my ledger grows. All the existing editor extensions and LSPs also call out to Python Beancount behind the scenes (afaik) so I also have the 1s delay for editor diagnostics. In Fava, the delay is a lot higher (~6s) for any change that I make to my .bean files to appear in the UI. I think it's because it's refreshing a lot of caches so that navigation between accounts doesn't take the full 1s every time.

Re querying the ledger, I'm still torn on whether that is something that belongs into the core. For now my thinking is that querying is a different component that just works off of the Protobuf output of the core. But on the other hand maybe that adds too much overhead, and the implementation requires too much knowledge of the Beancount internals (like booking) anyway? I haven't looked into it too closely yet and am open both ways. I'm also not sure what percentage of users rely on it out of the box. I for example use Fava most of the time and bean-query only once in a while for advanced stuff like when I do my taxes.

Happy to hear your thoughts!

Moritz

[Justus Pendleton]

Sounds like a fun project!

I tried it but nothing happened, unfortunately:

> ~/Downloads/turbobean-x86-linux-0.0.2/turbobean serve my.bean                                                                                                                                            ─╯
[error]: FileNotFound

> ~/Downloads/turbobean-x86-linux-0.0.2/turbobean -h                                                                                                                                                       ─╯
Turbobean is a fast Beancount implementation. (version 0.0.2)

[..... the rest cut....]

[redst...@gmail.com]

As another data point: I load only ~8 years of my ledger (~20k entries); anything more is unusably slow. Even this subset takes 3–4 seconds on an Intel Core Ultra 7 / 16 GB, making editing with Fava open quite frustrating.

A faster implementation without Python overhead would be a major improvement, and would also finally make running Beancount as an LSP during editing feasible.

[redst...@gmail.com]

Wow, very cool project. It’s impressively fast! It’s currently not usable for me due to the surrounding ecosystem and plugin support, but nevertheless, this is very promising!

I’m curious:

• Roughly how many hours did it take to go from zero to this state?
• I assume this was largely AI-assisted? If so, what inputs did you provide (current Beancount repo, v3 docs, design notes, etc.), and what was the AI technology used?
• How helpful was AI for the frontend work? Did you reference Fava in the process?

A few architectural questions:

• Why not expose a Python interface so existing plugins can be reused? In my experience, plugins don’t materially impact performance.
• Fava has accumulated a large feature set over many years. Would integrating with Fava be difficult? Given that it now makes fewer direct Beancount library calls, could this be handled by constructing the expected Python data structures instead of protobufs?
• Similarly for Beanquery: how complex would integration be, and what would the expected performance tradeoff look like?

​