A declarative idempotent rule-based transaction import engine - beanhub-import

[Fang-Pen Lin]

Hi beancount users,

I love beancount and use it daily, but if there's one thing I wish there was a better way to do, it is dealing with importing transactions from CSV files. There are issues to deal with, such as duplicate transactions. Once transactions are added, applying changes to all of them is hard. If I want to move transactions around to different files, it's also not easy to do. There's also a lack of features like getting data from multiple sources and merging it into a single transaction. Of course, as a software engineer myself, I can write code and modify existing importers to meet my own needs. But I always wonder if there's a better way to do it.

With that in mind, I spent the past few days building a whole new beancount importer from the ground up. While it's not 100% done yet, it's already at the point I am happy with. I can now easily import transactions from CSV files with the new tool, which I used to need a complex custom Python script to do the same job. I open-sourced the project from the very beginning. Now you can find it at

https://github.com/LaunchPlatform/beanhub-import

Here's an example of its superpower: say you don't want the imported transactions live in the book grouped by year, and you want it to be grouped by quarter instead. You can easily change the import rule like this:



And you rerun the import command, then voila!



It knows how to remove the imported transactions from the year file and move them into new quarter files instead.

Yet another example: say you want to add a new tag to all the imported transactions. You can simply add the tags in the transaction template and rerun import:



And here you go. It automatically adds tags to all the corresponding transactions.



This project is still in its early stages and subject to major changes. If you want to find out how it works, you can read the how-it-works. Currently, the extraction of transactions relies on another library, beanhub-extract:

https://github.com/LaunchPlatform/beanhub-extract

For the purpose of my use case and also as a proof of concept, it only supports Mercury Bank CSV files for now. However, I will add support for more banks' CSV files in the future. I will also make beancount-import able to support third-party extractors.

Besides supporting more banks, I am also making generating transactions from multiple sources possible. Here's an example of the merge rule I envisioned:

merges:
- match:
  - name: mercury
    extractor:
      equals: "mercury"
    desc: "Credit card payment"
    merge_key: "{{ date }}:{{ amount }}"
  - name: chase
    extractor:
      equals: "chase"
    desc: "Payment late fee"
    merge_key: "{{ post_date }}:{{ amount }}"
  actions:
    - txn:
        narration: "Paid credit card"
        postings:
          - account: Expenses:CreditCardPayment
            amount:
              number: "{{ -mercury.amount }}"
              currency: "{{ mercury.currency | default('USD', true) }}"
          - account: Expenses:LateFee
            amount:
              number: "{{ -chase.amount }}"
              currency: "{{ chase.currency | default('USD', true) }}"

Let me know what you think or any suggestions are welcome 😄

Best,
Fang-Pen Lin.