Frustration

[Dustin Farris]

I've been using beancount for a little over a year to manage my personal finances as well as some side self-employment accounting.  I have twice gotten frustrated to the point of trying other products (specifically Personal Capital, Mint, Quicken, and QuickBooks) but give up and come back to command-line accounting for reasons probably familiar to everyone in this group.

So here are my frustrations and what I'm going to attempt to do for myself to address them.  I'm mainly venting here, but if anyone has any suggestions I'd really appreciate them.

• I update my journal every month.  Getting updated transactions from 25+ different accounts every month is very time consuming.
  • Todo: Research programmatic downloads of transactions from all banks (has this been done already?)
• The beancount file is getting overwhelmingly large after just 1 year.  This is making it hard for me to jump around and find/fix things.  I often have personal transactions that cross equity accounts into our rental business, or my software engineering self employment.
  • Todo: split personal.beancount into smaller journals (by month?)
• omni-complete in vim is an awkward keyboard chord and account completions in vim are sometimes broken depending on what you last typed
  • Todo: try VSCode which can now embed neovim and has it's own beancount extension that should make the editing experience nicer
    • except VSCode folding is weird so this might actually make things worse
• I still don’t understand how reporting works, and part of that is because I don't use it enough.  I feel like I'm relearning Beancount Query Language every time I do need something.
  • Todo: keep a list of commonly-used queries
• reconciling receipts / splitting transactions is time consuming, although vim macros help here
  • Todo: Improve importer recognition of payees and likely expense accounts
• no way to attach receipt pictures to transactions?
  • Todo: Research beancount tooling or other apps to capture receipts
• fava is nice, but could be nicer
  • Todo: Research writing a new frontend reporting/visualization tool and/or contribute to fava
  
  

[Martin Blais]

On Sun, Feb 7, 2021 at 10:24 AM Dustin Farris <dustin...@gmail.com> wrote:

I've been using beancount for a little over a year to manage my personal finances as well as some side self-employment accounting.  I have twice gotten frustrated to the point of trying other products (specifically Personal Capital, Mint, Quicken, and QuickBooks) but give up and come back to command-line accounting for reasons probably familiar to everyone in this group.

So here are my frustrations and what I'm going to attempt to do for myself to address them.  I'm mainly venting here, but if anyone has any suggestions I'd really appreciate them.

• I update my journal every month.  Getting updated transactions from 25+ different accounts every month is very time consuming.
  • Todo: Research programmatic downloads of transactions from all banks (has this been done already?)

Look for Yodlee, or Plaid.

If you're willing to give up your passwords, those could save you time by lumping together a bunch of imports.

(I tend to only update the most important ones - checking, credit card, investing - frequently, and the others lag behind a bit.)

Also, there are simplifications coming up to the importing framework which is going to make it substantially simpler to use, though I'm not sure if those changes will address your specific needs.

 

• The beancount file is getting overwhelmingly large after just 1 year.  This is making it hard for me to jump around and find/fix things.  I often have personal transactions that cross equity accounts into our rental business, or my software engineering self employment.
  • Todo: split personal.beancount into smaller journals (by month?)

Sure, you can split. It won't make it much faster though. (The ongoing rewrite in C++ is going to address that.)

I use outline-minor-mode. Another clever trick is to realize that like in life, you only ever go to a few places in that file - depending how you organize it - and insert unique comment tags and just i-search for them to go there directly (*). Huge time saver. The new import framework might even auto-insert in front of those tags.

I think if you have the patience, you could refine your previous transaction detection so that it's nearly flawless, automatically categorize your transactions, and implement auto-insert in your file and trust it. Would save some time, but it'll be time-consuming to get it right IMO (you'll spend the time coding instead of updating; it's unclear to me in that situation which will be better). beangulp will also provide more hooks for you to refine previous-import-detection, even per importer. (For more on upcoming changes on that look here https://docs.google.com/document/d/1O42HgYQBQEna6YpobTqszSgTGnbRX7RdjmzR2xumfjs/edit)

(*) In fact, I would say if you're using emacs this is the fourth stage of evolution of Emacs users about cursor movement: first you use the cursor keys, then you learn to use the relative word and paragraph movement, then you graduate to sexp movements (e.g. beginning of function), and when you nearly achieve enlightenment you are nearly always moving via interactive-search (https://www.masteringemacs.org/article/effective-editing-movement). There is a final stage of awakening called "ace-jump-mode", kept as a closely guarded secret by those of use experiencing a permanent state of non-duality ;-).

 

• omni-complete in vim is an awkward keyboard chord and account completions in vim are sometimes broken depending on what you last typed
  • Todo: try VSCode which can now embed neovim and has it's own beancount extension that should make the editing experience nicer
    • except VSCode folding is weird so this might actually make things worse
• I still don’t understand how reporting works, and part of that is because I don't use it enough.  I feel like I'm relearning Beancount Query Language every time I do need something.
  • Todo: keep a list of commonly-used queries

Write your queries as code, and run your own scripts. There's a neat API.

You can also use the Query directive to save pre-made queries in your Beancount file and execute them by name.

Easier reporting is something I'd like to refine in the next version. With a more generic SQL or providing a library that works with petl.

• reconciling receipts / splitting transactions is time consuming, although vim macros help here
  • Todo: Improve importer recognition of payees and likely expense accounts

Yep.

(v3's new parser will bring another dimension to this: instead of having to write code to fixup the payees in memory only, the new rewriting capabilities will make it possible to rewrite the actual file by changing the AST and writing it back out.)

 

• no way to attach receipt pictures to transactions?
  • Todo: Research beancount tooling or other apps to capture receipts

See the ingest framework's filling ability. It renames files that have been identified prepending a date and can stash them in a repository under the same account hierarchy. bean-web used to serve those directives right next to the transactions and you could insert a common link to link them together (or write a plugin that attempts to match them up automatically)

 

• fava is nice, but could be nicer
  • Todo: Research writing a new frontend reporting/visualization tool and/or contribute to fava

Go for it!

Custom one-off renderings are also useful (e.g. like in beangrow).

 

-- 

You received this message because you are subscribed to the Google Groups "Beancount" group.
To unsubscribe from this group and stop receiving emails from it, send an email to beancount+...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/beancount/4bd97aa7-ef37-4f08-9b65-d9555acbe7fbn%40googlegroups.com.

[Dustin Farris]

Wow, thanks for the fast and thorough response.  It is very encouraging, and really what I needed for my mental health haha.

A couple reactions at random:

Yodlee, or Plaid

👍👍👍 will check these out

Sure, you can split. It won't make it much faster though.

I am excited about performance coming with the C++ rewrite, but this was not so much about addressing performance as my ability to find things.  (but to be clear).  The headers I’ve created in my journal are per-institution-account (e.g. fidelity 401k) with sub-headers by year and then by month.  These headers with their folding in vim (similar to emacs outline mode) are helpful, but I think I can do better.  It is also a less-than-nice experience for on the fly entries such as invoices, tracking kids’ allowance, and so on.  I need to just set aside a few hours to ponder this and come up with the right organization and workflow.

There is a final stage of awakening called "ace-jump-mode", kept as a closely guarded secret

Is there a vim equivalent of nirvana? :P  Because of beancount, I gave emacs+orgmode a real try — a month with spacemacs, and a month with doom, but ultimately reverted back to what I know (vim).  I’m hoping to get my beancount vim+VSCode experience leveled up.

new rewriting capabilities will make it possible to rewrite the actual file by changing the AST and writing it back out.

This sounds really exciting.

in the next version. With a more generic SQL or providing a library that works with petl

Being able to reason about my financials using dataframes sounds empowering.  Looking forward to it.

-- 

Dustin Farris

(646) 671-2007

To view this discussion on the web visit https://groups.google.com/d/msgid/beancount/CAK21%2BhNsEmRrdxnDCbrZXqxam2pphcGysM4WpEmb0CS-wd3R6w%40mail.gmail.com.

[Ben Blount]

> Importing is time consuming

I use github.com/jbms/beancount-import and am quite happy with it for the import UI, transaction dedupe logic, transaction clearing, and ML-based categorization. Admittedly I've spent tens of hours programming importers and scrapers and possibly have not made any net time savings. But it was a fun project!

I signed up for a plaid developer account and wrote a small script to download transactions. Then I wrote a beancount-import source for it.

I'm excited for Beancount v3's importer changes, it will hopefully mean that we can all merge our efforts on importers into one compatible format. I'd be happy to collaborate with you on design for that Martin.

> Journal file being too large to navigate

I share this frustration, though mine is more of a frustration that it's difficult to keep it organized in any kind of sane way. Beancount v3's ability to rewrite the ledger file in code will hopefully make it trivial to make a beancount formatter, which is super exciting.

The beancount emacs plugin is /by far/ the best transaction entry system I've used so far. I too am a longtime vim user and have been learning doom emacs for a few months. So far I am thrilled with it and am increasingly using it full-time. I don't think it'll replace Jetbrains IDEs for heavy coding but it's replacing all my adhoc editing / light coding needs.

I am using this forked version of the beancount emacs plugin since I am currently using jbms/beancount-import's journal format which splits accounts and transactions into separate files. I haven't learned elisp yet or I'd consider trying to get this patch upstreamed. I should probably adopt the 'one journal file' approach instead.

> petl, visidata, data frames and beancount queries

I love Martin's principles around promoting interop with powerful recomposable tools like this. I look forward to seeing this workflow realized and collaborating to make a new cookbook of useful queries.

To view this discussion on the web visit https://groups.google.com/d/msgid/beancount/AC64FB84-AF87-4AA7-AB57-CF0667DE1728%40gmail.com.

[Tuno Tunante]

Hi Dustin,

I'm using VSCode for a while and I use vim regularly for my job, so I know both systems. You must try VS-Code with the extension VSCode-Beancount from Lencerf  you'll never go back ;-)

For example, if you start a new line with a "2" that write automatically the date. It runs a bean-check each time you save the file (I do automatically each 30 seconds) and tell you about the errors, etc.

When you pass the mouse over an account, e.g. Expenses:Kids -50 USD, etc. gives you the total amount for this account and more things like highlight, etc.

Try ;-)

To view this discussion on the web visit https://groups.google.com/d/msgid/beancount/AC64FB84-AF87-4AA7-AB57-CF0667DE1728%40gmail.com.

[Justus Pendleton]

On Monday, February 8, 2021 at 12:28:16 AM UTC+7 dustin...@gmail.com wrote:

The headers I’ve created in my journal are per-institution-account (e.g. fidelity 401k) with sub-headers by year and then by month.

You might want to reconsider whether this level of organization (or any organization, really) within the beancount file actually provides any value or if you're just creating busy work for yourself. Don't get me wrong, I totally know where you are coming from and started out trying to organize my beancount file the same. But I eventually gave up because....why do I actually need it organized by account or by month? Anytime I want to actually see transactions that way I can easily get bean-report or bean-query or fava to show it to me.

At this point I consider the transaction ledger an append-only (and largely write-only) document and I don't even care if things aren't in any order at all.

[Martin Blais]

If you work like that you can also use the "location" column in bean-query in combination with Emacs to jump around the selected transactions, e.g. with "next-error".

--

You received this message because you are subscribed to the Google Groups "Beancount" group.
To unsubscribe from this group and stop receiving emails from it, send an email to beancount+...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/beancount/856c269a-93f2-48f3-bdbf-3799c5184bdfn%40googlegroups.com.

[TRS-80]

The only problem with that is that Beancount's time reolution is only
down to the day, after which it falls back to file order.

Not trying to take away from your point, as it's a good one and should
at least be considered.

It is also entirely possible that I am still just firmly in the former
camp and have not yet been able to let go of my rigidly heirarchical
file structure in order to fully embrace your Zen-like realization.
Probably because I am descended from Germans... :D

Cheers,
TRS-80

[TRS-80]

On 2021-02-07 10:23, Dustin Farris wrote:
> I've been using beancount for a little over a year [...] I have twice

> gotten frustrated to the point of trying other products (specifically
> Personal Capital, Mint, Quicken, and QuickBooks) but give up and come
> back to command-line accounting for reasons probably familiar to
> everyone in this group.

Don't feel bad. I think it took me like 3 or more tries over some years
to finally get off the ground.

I think the most important takeaway is that we each have to find some
workflow that suits us. It's highly personal, IMO.

Not only that, but there are a few related proficiencies / tools that I
think I finally got into place /around/ Beancount to facilitate the
process in a way that worked for me (and I will touch more on these
below).

> * I update my journal every month. Getting updated transactions from

> 25+ different accounts every month is very time consuming.
>

> * Todo: Research programmatic downloads of transactions from all banks

> (has this been done already?)

When first starting out, I remember reading Martin's advice to handle
things more manually, and more frequently. Like you, I thought the way
to go was to automate as much as possible. So I spent a lot of time
(and a couple complete failures) going down that path.

I think I can attribute a big part of my recent success doing a more
"manual" and frequent approach. Which really means, I created some
tools to very quickly and easily facilitate this "manual" entry process
(more on this further down).

Personally I also found the automatically imported data quite ugly (all
caps, etc.) and often not very descriptive, either, which bothered me a
lot.

> * The beancount file is getting overwhelmingly large after just 1

> year. This is making it hard for me to jump around and find/fix
> things. I often have personal transactions that cross equity accounts
> into our rental business, or my software engineering self employment.
>

> * Todo: split personal.beancount into smaller journals (by month?)

To split files or not is a subject you can find a lot of discussion
about on this list, but I decided on "one big file" because I seem to
recall reading about / experiencing some other small/subtle issues that
can happen when splitting files (account name completion, etc.).

OTOH, I keep a lot of my (non-Beancount) personal notes in a more
Zettelkasten-like style (many smaller notes) so perhaps that helps you
get your head around things, particularly separate entities like rental
properties. I think if/when I start another business entity again, I
might probably do a separate file for that. So, this one I think, is
personal...

Anyway, it sounds to me like your real problem is navigation. Nothing
to add to what Martin said, get that sorted and I think it will solve
that problem for you.

Like Martin I am also an Emacs user. Which is something else that took
me many tries to get off the ground with, in it's own right... So
either give Emacs /yet another/ try (and keep doing so until you
succeed), or figure out a way to do similar things in other editor of
your choice. But this is why we love Emacs so much, it is a completely
customizeable environment and can do so much, once you get the hang of
it.

> * I still don’t understand how reporting works, and part of that is

> because I don't use it enough. I feel like I'm relearning Beancount
> Query Language every time I do need something.
>

> * Todo: keep a list of commonly-used queries

I do this for all manner of code snippets and other seldom used
information. A personal knowledge base, if you will. Of course, again
implemented in Orgmode / Emacs. ;)

> * reconciling receipts / splitting transactions is time consuming,

> although vim macros help here
>

> * Todo: Improve importer recognition of payees and likely expense
> accounts

I devised a little "worksheet" based on a couple Orgmode tables to help
facilitate the splitting process (on nontrivial, i.e., "long" receipts,
with many different Expense accounts) more quickly and easily. I have
been meaning to make a video to demonstrate, but if there is interest
perhaps I could start a new thread on the list to discuss that
separately.

> * no way to attach receipt pictures to transactions?
>
> * Todo: Research beancount tooling or other apps to capture receipts

In fact, the tool I wrote handles all of the above in one go. I simply
take photos (with smartphone, usually immediately) of all of my
receipts. Which are then synced automatically across devices with
SyncThing. Later at desktop/laptop I invoke my function from Dired
in Emacs which prompts me for things like Payee, amount, etc. from the
receipt. It then re-names the receipt photo filename accordingly, and
also generates the Beancount transaction, because I realized it's
largely the same information. But most importantly, you only need to
"touch" each receipt once. And the whole process really is facilitated
by nice completing-read style interfaces, based on Controlled Vocabulary
of Payees and things like that, making it very fast and easy and
eliminating things like typos from truly manual entry.

I posted to the list about my workflow starting with taking a photo of a
receipt once before, but there was no interest it seemed. If that has
changed, I would be happy to being it up again (perhaps in a new
thread).

I think I finally realized, that a lot of the guys on this list are
probably making so much money that they are more interested in tracking
their investments / net worth than their expenses any more. :D

Cheers,
TRS-80

[TRS-80]

On 2021-02-07 10:23, Dustin Farris wrote:

> * I update my journal every month. Getting updated transactions from

> 25+ different accounts every month is very time consuming.

I just noticed this, while reading back over the last email I sent.
That is a tremendous number of accounts (I am assuming by "account" here
you mean bank/brokerage/card or whatever). I can only imagine how many
/transactions/ that actually works out to.

Therefore maybe some automation /is/ in order, in your case. Or hiring
a bookkeeper / assistant! At minimum, I would suggest putting aside a
significant amount of your own time to be able to handle the workload
properly. But I am sure you are aware of that already.

Anyway, the sheer volume of work here is an aspect of your "problem"
that I feel should be identified in it's own right, and addressed.
Although the "right" solution will depend on what suits you and your
situation.

In a way, text based accounting may be the only tool which makes such a
complicated and voluminous workload possible to keep track of
(especially when considering long term archival, and eschewing lock-in).
Perhaps that's why you keep coming back, maybe you are (consciously or
not) realizing the limitations of off-the-shelf tools. As mentioned in
my other email already, maybe it's then juat a matter of fashioning some
tools /around/ Beancount to help you keep up with the volume.

Cheers,
TRS-80

[Dustin Farris]

the sheer volume of work here is an aspect of your "problem"
that I feel should be identified in it's own right, and addressed.

☝️I’m glad to hear you say that — I couldn’t tell if I’m just doing it wrong or if 25+ really is a lot.

I feel like I should probably consolidate this somehow.  My wife and I have separate checking/savings accounts.  I’m tracking our insurance account HRA transactions.  5 credit cards.  We have multiple investment accounts each with Fidelity, Vanguard, SallieMae, Barclays.  And then miscellaneous “crowd funded” investments e.g. lending, private equity, real estate, digital currency — most of which do not provide any kind of useful export and have to be entered manually.  And we have several foreign accounts.  And in case that comes off sounding like we have a lot of money, we don’t, we’ve just dabbled around with various instruments/institutions over the years; a couple thousand here, couple thousand there, and now I'm in accounting hell.

-- 

Dustin Farris

(646) 671-2007

--
You received this message because you are subscribed to the Google Groups "Beancount" group.
To unsubscribe from this group and stop receiving emails from it, send an email to beancount+...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/beancount/3c70ec07dcec427af8f8cdaa56ff5156%40isnotmyreal.name.

[TRS-80]

On 2021-02-14 15:12, Dustin Farris wrote:
>> the sheer volume of work here is an aspect of your "problem"
>> that I feel should be identified in it's own right, and addressed.
>
> ☝️I’m glad to hear you say that — I couldn’t tell if I’m
> just doing it wrong or if 25+ really is a lot.
>
> I feel like I should probably consolidate this somehow. My wife and I
> have separate checking/savings accounts. I’m tracking our insurance
> account HRA transactions. 5 credit cards. We have multiple
> investment accounts each with Fidelity, Vanguard, SallieMae, Barclays.
> And then miscellaneous “crowd funded” investments e.g. lending,
> private equity, real estate, digital currency — most of which do not
> provide any kind of useful export and have to be entered manually.
> And we have several foreign accounts. And in case that comes off
> sounding like we have a lot of money, we don’t, we’ve just dabbled
> around with various instruments/institutions over the years; a couple
> thousand here, couple thousand there, and now I'm in accounting hell.

Really glad to be able to help.

I think this would be a big step in the right direction. I think I like
to dabble / experiment in various things, too, but to everything (turn,
turn, turn) there is a season (turn, turn, turn) and maybe the great
wheel is swinging around now to a big consolidation / simplification
phase for you.

Get the biggest monkey off your back, first, then come back and let us
know how it's going and we can continue to refine the workflow
incrementally. Additional refinements may or may not even be needed,
depending...

And still, no one seems to be interested in my receipt -> photo ->
Beancount workflow. I am starting to wonder if I am mad. I really
thought I might have had my first taker there for a moment. :D

Cheers,
TRS-80

[Martin Blais]

On Sun, Feb 14, 2021 at 3:12 PM Dustin Farris <dustin...@gmail.com> wrote:

the sheer volume of work here is an aspect of your "problem"
that I feel should be identified in it's own right, and addressed.

☝️I’m glad to hear you say that — I couldn’t tell if I’m just doing it wrong or if 25+ really is a lot.

I feel like I should probably consolidate this somehow.  My wife and I have separate checking/savings accounts.  I’m tracking our insurance account HRA transactions.  5 credit cards.  We have multiple investment accounts each with Fidelity, Vanguard, SallieMae, Barclays.  And then miscellaneous “crowd funded” investments e.g. lending, private equity, real estate, digital currency — most of which do not provide any kind of useful export and have to be entered manually.  And we have several foreign accounts.  And in case that comes off sounding like we have a lot of money, we don’t, we’ve just dabbled around with various instruments/institutions over the years; a couple thousand here, couple thousand there, and now I'm in accounting hell.

You'll get the most bang for your time investment in Beancount then. 

It's super nice to have a fully consolidated view of your entire portfolio, across accounts, across multiple currencies.

I've enjoyed that for many years now, I can't imagine the opacity around my assets if I hadn't built this... everyone needs this.

(I'll revive the "baskets" project at some point, which goes one step further into disaggregating all your holdings back to their stock components and then allows you to reaggregate all the bits and pieces arbitrarily.)

 

To view this discussion on the web visit https://groups.google.com/d/msgid/beancount/ED91E53B-E55A-414B-9F02-3B8472D384AF%40gmail.com.

[Daniele Nicolodi]

Hello Dustin,

On 07/02/2021 16:23, Dustin Farris wrote:
> I've been using beancount for a little over a year to manage my personal
> finances as well as some side self-employment accounting.  I have twice
> gotten frustrated to the point of trying other products (specifically
> Personal Capital, Mint, Quicken, and QuickBooks) but give up and come
> back to command-line accounting for reasons probably familiar to
> everyone in this group.

It seems that most of your frustrations are not rooted in how Beancount
works, but are

> So here are my frustrations and what I'm going to attempt to do for
> myself to address them.  I'm mainly venting here, but if anyone has any
> suggestions I'd really appreciate them.
>

> * I update my journal every month.  Getting updated transactions from

> 25+ different accounts every month is very time consuming.

This is not specific to Beancount, is it? As it has been mentioned by
others there are services that do the data aggregation for you, if
trusting an online app with your accounts credentials is something for you.

One thing I can suggest is to take an incremental approach to the number
of accounts you import, to the frequency at which you do it, and to the
amount of detail you record. Having less to manage while you build your
workflow will make things easier. It is easy to go back and add
information when you need it.

> o Todo: Research programmatic downloads of transactions from all

> banks (has this been done already?)

Back when I started working with Ledger (before switching to Beancount)
I wrote some code to download statements for some of my accounts. Then I
realized that writing them and keeping them working takes much more time
than I would ever save compared to downloading manually once a month or
so. Even more so nowt that I have two-factor authentication for all my
accounts.

> * The beancount file is getting overwhelmingly large after just 1

> year.  This is making it hard for me to jump around and find/fix
> things.  I often have personal transactions that cross equity
> accounts into our rental business, or my software engineering self
> employment.

> o Todo: split personal.beancount into smaller journals (by month?)

Also this does not seems to be an issue specific to Beancount. Once you
have a lot of data (for some definition of "a lot") you need to have a
way to sort through it. I don't know of any system that would solve this
magically. You need to identify what is the key you most often would
like to use to look-up a transaction and organize your data accordingly
and maybe write tools to help (bean-doctor could serve as an example, if
you are thinking about writing some tools to find things in your ledger).

I use a single gigantic Beancount file where transactions are sorted by
date. I have outline-mode headers that split it up by year and month,
but those mostly help when I cut-and-paste in the imported transactions.
Most often, when I am looking up a transactions, I know the date or the
time range in which it occurred. I have an Emacs function that (assuming
the ledger is sorted by date) looks up a specific date. I use this both
for inserting transactions manually and for looking up transactions. It
is rare that I have to manually amend transactions once they are imported

> * omni-complete in vim is an awkward keyboard chord and account

> completions in vim are sometimes broken depending on what you last typed

You should try Emacs and beancount-mode :-)

> * I still don’t understand how reporting works, and part of that is

> because I don't use it enough.  I feel like I'm relearning Beancount
> Query Language every time I do need something.

> o Todo: keep a list of commonly-used queries

As it has been suggested already, you can have queries recorded in the
ledger itself with the "query" directive.

> * reconciling receipts / splitting transactions is time consuming,

> although vim macros help here

> o Todo: Improve importer recognition of payees and likely expense
> accounts

This is also not Beancount specific. I actually thing that the text
ledger format helps immensely here (assuming you use a text editor that
cooperated and not one that stands in the way).

Probably some other application embed some machine learning that
partially automates this, but you can have the same for Beancount
imports. There is smart_importer, or you can roll your own. I have my
own solution leveraging some ML in scikit-learn and it is less than 50
lines of code (it is so short that I never really felt it worth to be
published). The rule-based system I was using before is more lines of code.

> * no way to attach receipt pictures to transactions?
> o Todo: Research beancount tooling or other apps to capture receipts

You can associate metadata to transactions. Beancount does not interpret
it, but you can put the path to a receipt picture in there. An Emacs
function to display the receipt would probably be around 10 lines of
emacs-lisp.

> * fava is nice, but could be nicer
> o Todo: Research writing a new frontend reporting/visualization

> tool and/or contribute to fava

Every nice thing could be nicer :-)

Cheers,
Dan

[Daniele Nicolodi]

On 07/02/2021 18:55, Ben Blount wrote:
> I am using this forked version
> <https://github.com/cnsunyour/beancount.el> of the beancount emacs

> plugin since I am currently using jbms/beancount-import's journal format
> which splits accounts and transactions into separate files. I haven't
> learned elisp yet or I'd consider trying to get this patch

Uh? I wasn't aware of the fork. Completing account names form multiple
files is supported in the "official" beancount-mode (although I don't
use it and it may be broken).

I haven't checked carefully, but the only change applied n the fork is
substituting a defvar with a defcustom, which is more an user interface
change than a functional change. If it being a defcustom makes life
easier to anyone, I'll be happy to make the change.

Cheers,
Dan

[Ben Blount]

Good to know, thanks Daniele! I have subsequently merged my ledger into one file anyway, many of the tools are a little easier with one file.

--
You received this message because you are subscribed to the Google Groups "Beancount" group.
To unsubscribe from this group and stop receiving emails from it, send an email to beancount+...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/beancount/8c5e80ef-5d4e-7d25-9c22-9b2d75b1d5a5%40grinta.net.

[redst...@gmail.com]

• I update my journal every month.  Getting updated transactions from 25+ different accounts every month is very time consuming.
  • Todo: Research programmatic downloads of transactions from all banks (has this been done already?)

This was the biggest frustration for me. I spent a lot of time getting to a "one-button update". Today, I'm 90% there. I have ~45 accounts.

- 14 of these don't have a scriptable ofx API (aka ofx/qfx via quicken directconnect). For these, I have to login to the website and download the ofx

- with the remaining, I use ofxget and some homecooked scripts to automatically get them. My login info is stored in pass, which truly makes this a matter of typing a single command to download dozens of ofx

- I spent tons of time writing importers with reusable code to process these

- Investment accounts are predictable to build transactions for. For credit card accounts, I use the awesome smart_importer

My total time to update is now less than 5 minutes. This doesn't include three or four accounts which I have to submit GDPR requests to get data for, and such, but I don't care about updating these more frequently than 3-4 times a year.

Bringing my total data update time to under 5mins was critical to me in making personal finance productive and fun, by being able to focus on the actual financial parts.

(TODO to myself: write a blog/article about my setup)

 

• The beancount file is getting overwhelmingly large after just 1 year.  This is making it hard for me to jump around and find/fix things.  I often have personal transactions that cross equity accounts into our rental business, or my software engineering self employment.
  • Todo: split personal.beancount into smaller journals (by month?)

As others said, why bother when it can be automated? I used to have a file tree that mirrored my account hierarchy. I still have different source files for different real-world accounts, which makes it easy to debug, and keeps the file sizes somewhat small. I patch beancount's extract.py to directly append to these files. Happy to share the patch.

• omni-complete in vim is an awkward keyboard chord and account completions in vim are sometimes broken depending on what you last typed
  • Todo: try VSCode which can now embed neovim and has it's own beancount extension that should make the editing experience nicer
    • except VSCode folding is weird so this might actually make things worse

I use something similar to supertab, which works 100% of the time by just pressing Tab on an account.

• I still don’t understand how reporting works, and part of that is because I don't use it enough.  I feel like I'm relearning Beancount Query Language every time I do need something.
  • Todo: keep a list of commonly-used queries

fava:

2017-01-01 query "Expense breakdown" "SELECT * FROM ...."

 

• reconciling receipts / splitting transactions is time consuming, although vim macros help here
  • Todo: Improve importer recognition of payees and likely expense accounts

smart_importer gets it right for me 90% of the time

 

• no way to attach receipt pictures to transactions?
  • Todo: Research beancount tooling or other apps to capture receip

What I do:

1) drop all my receipt photos into a single directory (sync's from my phone automatically)

2) I use tesseract + simple scripts to OCR the receipts and find a date for them

3) Use beancount and fava's document feature to make them appear next to Expense transactions. A bit more work could associate these with specific transactions via links or some such, but I personally don't have the need to do so.

(TODO to myself: write a blog/article about my setup)

 

• fava is nice, but could be nicer
  • Todo: Research writing a new frontend reporting/visualization tool and/or contribute to fava

Any specifics on what would make it nicer?

[Dustin Farris]

My total time to update is now less than 5 minutes.

Wow!  That is truly inspiring.  I am looking forward to your forthcoming blog post(s) on all of this.  Do you have anything already published I could flip through?

14 of these don't have a scriptable ofx API (aka ofx/qfx via quicken directconnect)

Are you using quicken to access the APIs that area available?

I use something similar to supertab, 

I tried getting supertab to work and did not succeed.  I’m currently trying to get something going with neovim embedded in VSCode; but the folding in VSCode leaves a lot to be desired.

1) drop all my receipt photos into a single directory (sync's from my phone automatically)

2) I use tesseract + simple scripts to OCR the receipts and find a date for them

3) Use beancount and fava's document feature to make them appear next to Expense transactions. A bit more work could associate these with specific transactions via links or some such, but I personally don't have the need to do so.

Very cool.  I would love to steal some or all of this.  I’ve thought about using Apple’s document scanning into Files which also does OCR — but no idea how accessible the OCR data is (like is it embedded in the file, or kept in some other untouchable index).

Any specifics on what would make [fava] nicer?

A few things off the cuff:

* The visualizations seem constrained by the size of the screen

* I’m not a big fan of the styling overall

* Would like to see better print styles/formatting, e.g. to export an income statement pdf for my accountant

This is not a great list, but my kid is running circles around me right now.

-- 

Dustin Farris

(646) 671-2007

--
You received this message because you are subscribed to the Google Groups "Beancount" group.
To unsubscribe from this group and stop receiving emails from it, send an email to beancount+...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/beancount/fe4d3301-7ee4-4b01-a0c3-ec8bef2266a2n%40googlegroups.com.

[redst...@gmail.com]

My total time to update is now less than 5 minutes.

Wow!  That is truly inspiring.  I am looking forward to your forthcoming blog post(s) on all of this.  Do you have anything already published I could flip through?

I'll try to share at least an outline in the next couple weeks.

14 of these don't have a scriptable ofx API (aka ofx/qfx via quicken directconnect)

Are you using quicken to access the APIs that area available?

Not Quicken, but the API that Quicken uses, via https://ofxtools.readthedocs.io/en/latest/

 

I use something similar to supertab, 

I tried getting supertab to work and did not succeed.  I’m currently trying to get something going with neovim embedded in VSCode; but the folding in VSCode leaves a lot to be desired.

My bad: turns out I use this single, simple line in my .virmc (in addition to the vim-beancount plugin):

autocmd FileType beancount inoremap <buffer> <Tab> <C-x><C-o>

More later.

[redst...@gmail.com]

I've started writing up my import setup in a series of articles titled "The Five-Minute Ledger Update". It is work in progress. Feel free to comment on the articles there to tell me what you would like to hear about.

Link to articles.

(graph of articles is here).

cc @Dustin Farris

[Dustin Farris]

This is awesome.  Going to read through it this week.  Thanks for publishing.

-- 

Dustin Farris

(646) 671-2007

--
You received this message because you are subscribed to the Google Groups "Beancount" group.
To unsubscribe from this group and stop receiving emails from it, send an email to beancount+...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/beancount/b33ade50-2053-4545-995b-91d2cb5e31a0n%40googlegroups.com.

[Luciano Fiandesio]

You may want to take a look at my project, Beanborg: https://github.com/luciano-fiandesio/beanborg

I have used it for over a year to automatically import several types of external CSV files from different financial institutions. It doesn't take long to setup and once is configured, assuming you have the CVS ready, it's a matter of launching a script. I do my weekly "imports" in less than 5 mins, and between my wife and myself, we have 8 different accounts.

Feel free to reach out if you need more info.

L

[Ben Blount]

Sorry it's been a rather crazy time and I don't remember.

There are several nice importers in github.com/jbms/beancount-import in addition to a nice webUI for merging / reviewing postings. beancount-import supports using beancount native importers, so in a sense the merging I described is already there in one direction.

On Tue, Nov 16, 2021 at 2:14 PM Bman Q <mpl...@gmail.com> wrote:

"I'm excited for Beancount v3's importer changes, it will hopefully mean that we can all merge our efforts on importers into one compatible format. I'd be happy to collaborate with you on design for that Martin."

-could you please explain what changes are you talking about? (i am using v2 and didn't look into v3 yet)

To view this discussion on the web visit https://groups.google.com/d/msgid/beancount/da9b922d-fa61-4eec-8ab6-e0cb7b55fc5dn%40googlegroups.com.

[Samuel Lotz]

I agree with Blais that there really isn't a better solution to all of this. I think there is a misconception that because it is conceivable that all of your transactions can be automated (and they are for individual banks & institutions) that it should be easy to automate or that there is already a solution. It sucks to have to do it all manually, but the pain of entering hundreds of "Expenses:Coffee 3.00 USD" transactions now makes me think twice about it! (or to just use cash and book them all in a single pad entry :P) It also has really made me appreciate the complexity of some financial instruments, which in my case has made me less afraid of them since I know that I can keep track of them all as I want to start doing crazy things to make money :)

I really can't imagine having to do all of the importers myself and maintain them since the banks are always changing and no two are alike. I think its just easier to do by hand as of now since I know after working a few hours I will have a product, rather than spending a few hours automating and it just sucks at the end anyways. I would definitely pay someone to provide up to date importers and a UI for quickly categorizing them (since the banks never get this right for my purposes).

I have heard of people outsourcing their bookkeeping to traditional book keepers (I'm guessing they only do mainstream software like Quick Books) and also to overseas virtual assistants which will do anything you pay them for. I am betting that if you find one you can work with long enough that they would be able to do most of the drudge work if you just give them your statements.

[redst...@gmail.com]

I agree with Blais that there really isn't a better solution to all of this. I think there is a misconception that because it is conceivable that all of your transactions can be automated (and they are for individual banks & institutions) that it should be easy to automate or that there is already a solution. It sucks to have to do it all manually, but the pain of entering hundreds of "Expenses:Coffee 3.00 USD" transactions now makes me think twice about it! (or to just use cash and book them all in a single pad entry :P) It also has really made me appreciate the complexity of some financial instruments, which in my case has made me less afraid of them since I know that I can keep track of them all as I want to start doing crazy things to make money :)

Indeed, close to all of your transactions can be automated close to all of the time. ofx rarely breaks and is indeed highly automatable. csv may take longer to develop and might occasionally break. Getting it from 95% to 100% is usually not worth it in my experience. But getting it to to the 95% point is extremely valuable, and the point of the 5-minute ledger update.

PS: I haven't entered "hundreds of 'Expenses:Coffee 3.00 USD' transactions" or similar in at least half a decade. Nobody needs to do this.

 

I really can't imagine having to do all of the importers myself and maintain them since the banks are always changing and no two are alike. I think its just easier to do by hand as of now since I know after working a few hours I will have a product, rather than spending a few hours automating and it just sucks at the end anyways. I would definitely pay someone to provide up to date importers and a UI for quickly categorizing them (since the banks never get this right for my purposes).

With the ingest frameworks available (including mine), the average user should be able to recoup their cost in say 2-5 import cycles. If you're like me and like to import several times a month, there's a tremendous ROI in automating. And no, it's extremely far from "just sucking at the end anyways".

A whole bunch of people have shared importers. A pypi-like system to list, install, and manage importers would be awesome. Without that, most users need to put in some time up front to find the importers they need.

Of course, all this depends on where you live and whom you bank with. My personal experience is as a user in the US.

I'm curious as to where your impressions above come from, given mine is far from it. Perhaps if you could share whom you bank with and what specific problems you've had writing importers, that would help.

[Samuel Lotz]

I guess I've always meant to do the automation but have never gotten to it as I haven't seen enough actual success stories with it for basically the reasons you just posted (I don't typically read the mailing list). I've been through too many "works for me" recomendations (for other technologies) that don't work to trust it at face value, so I would have to spend significant effort to test and validate all these different experiences. I am also in the US with some credit unions etc. and I couldn't even get them to connect with Mint even though I had friends that could do the same. Lesson learned there was just to take ownership of it and no one will do it for you.

These days I just don't generate enough "1 USD coffee" transactions to make me motivated to automate them, part of which I credit to painfully hand entering all these transactions :) It also forces me to look at each transaction to know if any are fraudulent. For a beginner to beancount or just someone thats financially unsophisticated I think its valuable to do it by hand. But all that said, yes I would like to automate the tedious parts. I'll try to do a little bit more investigation, but I would like to see a "Do it this way" kind of guide that isn't just "works for me" with a reasoned explanation of all the problems, gotchas, etc. in detail so I know what I'm getting into and where something may or may not work.

[redst...@gmail.com]

I guess I've always meant to do the automation but have never gotten to it as I haven't seen enough actual success stories with it for basically the reasons you just posted (I don't typically read the mailing list). I've been through too many "works for me" recomendations (for other technologies) that don't work to trust it at face value, so I would have to spend significant effort to test and validate all these different experiences. I am also in the US with some credit unions etc. and I couldn't even get them to connect with Mint even though I had friends that could do the same. Lesson learned there was just to take ownership of it and no one will do it for you.

All excellent points and they mostly line up with my experiences too.

BTW, one exception: getting your ingest to directly download from your bank is is actually only a small part of the picture. This is why I recommend having that part be one of the last things you automate. See here.

 

These days I just don't generate enough "1 USD coffee" transactions to make me motivated to automate them, part of which I credit to painfully hand entering all these transactions :) It also forces me to look at each transaction to know if any are fraudulent. For a beginner to beancount or just someone thats financially unsophisticated I think its valuable to do it by hand.

Agree again: doing it by hand first is very educational.

 

But all that said, yes I would like to automate the tedious parts. I'll try to do a little bit more investigation, but I would like to see a "Do it this way" kind of guide that isn't just "works for me" with a reasoned explanation of all the problems, gotchas, etc. in detail so I know what I'm getting into and where something may or may not work.

You captured exactly why I wrote the Five-Minute Ledger Update. Here is the mailing list thread if you haven't seen it.

For this particular problem, sharing code = "works for me". The guide above is the "do it this way".

I'm happy to hear about what helped you and what didn't from the guide above if you read/use it.