Beanborg

[Luciano Fiandesio]

Hello, cheers! I would like to share Beanborg, a small project I have been working on to semi-automatically import CSV transactions into a Beancount ledger.

From the README:

Goals and key features

Beanborg has two main design goals:

• automatic matching of transaction data with the correct Expense accounts
• speed, the tool is designed to process several financial CSV file in few seconds

Other features:

• a sophisticated and extendible rule-based system
• avoid duplicates during the import
• highly configurable
  
• smart archiving function: when archiving a CSV file, the file is renamed using the start and end date of the CSV file

I have been using this tool for the last 18 months. Happy to share in case someone wants to try it out: https://github.com/luciano-fiandesio/beanborg

Feedback welcome!

Luciano

[redst...@gmail.com]

Nice to see others writing importers. And great, comprehensive documentation too! Thanks for sharing this.

I'm wondering if your import scripts can be used for building investment transactions? How do you handle balance assertions? How do you handle institutions that include multiple tables in a single csv? And I'm curious: why not use bean-file (I don't know what it's called in beangulp) for archiving?

Finally, would you be willing to share institution specific configs with the community?

[Martin Blais]

Great stuff Luciano, thank you for sharing, I'll definitely have a good look at it when I find a bit of time.

Red: Related to investments, I'm in the process of cleaning up and building common data structures specifically for trading accounts, in a new repo called "beanbuff". Find related codes here:

https://github.com/beancount/beanbuff/tree/master/beanbuff

At the moment it's just the importers I'm sharing, but I'm working on making the conversion to an intermediate trade log consistent across them and to convert that trade log to Beancount. I'm finding that having a flat table trade log is more useful for analysis and am still on the fence about representing thousands of 1000's of transactions for temporary positions in Beancount inputs. I tend to separate the long term stuff (e.g., a long position on an ETF held for years, with associated covered calls) from active trading (e.g., a strangle on BIDU held for 20 days, a /ZCN21-/ZCZ21 calendar spread, etc.), those things don't matter much in the long run and I think it might make more sense to summarize the impact of trading to just a single transaction per week and to keep all the detail in separate files analyzed by custom scripts doing various breakdowns on P/L.

--
You received this message because you are subscribed to the Google Groups "Beancount" group.
To unsubscribe from this group and stop receiving emails from it, send an email to beancount+...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/beancount/cf06f1e0-a09b-4ae6-950b-a341dfb8a89en%40googlegroups.com.

[redst...@gmail.com]

Awesome, thanks for sharing beanbuff. March seems to be the #ingest month :).

I'm curious though: what is the fundamental concern around representing thousands of temporary positions in Beancount? Is it performance, and if so, won't v3 solve this? Is it to not pollute your ledger with distracting short-term transactions, and if so, wouldn't a better approach be to put it in appropriate accounts in the hierarchy that don't distract? I'd imagine the pros of putting it in Beancount would far outweigh the cons (performance not withstanding), given you can focus your analysis tools on one system (Beancount) rather than building a parallel system for it.

Also: why "buff"? :)

[Luciano Fiandesio]

Hi, try to answer some of the questions:

> I'm wondering if your import scripts can be used for building investment transactions?

I'm not sure, I don't track my investments using beancount. I wish I could give a better answer. Perhaps we could discuss the use case and try to figure out how to incorporate investment imports into beanborg.

> How do you handle balance assertions?

I don't. Beanborg main goal is to import transactions and assign expenses based on some rules.

> How do you handle institutions that include multiple tables in a single csv?

I have never encountered such case, do you have a "sanitized" CSV I can look at?

> And I'm curious: why not use bean-file (I don't know what it's called in beangulp) for archiving?

Sorry, I don't understand this question...what is a bean-file? Archiving in beanborg, is just about moving the imported CVS file into an archive folder.

> would you be willing to share institution specific configs with the community?

100%, this was actually a goal of the project.

[Martin Blais]

On Thu, Apr 1, 2021 at 6:09 AM Luciano Fiandesio <luc...@fiandesio.com> wrote:

Hi, try to answer some of the questions:

> I'm wondering if your import scripts can be used for building investment transactions?

I'm not sure, I don't track my investments using beancount. I wish I could give a better answer. Perhaps we could discuss the use case and try to figure out how to incorporate investment imports into beanborg.

> How do you handle balance assertions?

I don't. Beanborg main goal is to import transactions and assign expenses based on some rules.

> How do you handle institutions that include multiple tables in a single csv?

I have never encountered such case, do you have a "sanitized" CSV I can look at?

> And I'm curious: why not use bean-file (I don't know what it's called in beangulp) for archiving?

Sorry, I don't understand this question...what is a bean-file? Archiving in beanborg, is just about moving the imported CVS file into an archive folder.

"bean-file" is just that: a tool to file away an imported download to a cleanly renamed directory following the hierarchy of accounts.

This is why the importers have an method to return the associated account.

It's now called "archive" in the updated beangulp, and is just one of the subcommands.

 

> would you be willing to share institution specific configs with the community?

100%, this was actually a goal of the project.

On Thursday, 1 April 2021 at 04:33:34 UTC+2 redst...@gmail.com wrote:

Nice to see others writing importers. And great, comprehensive documentation too! Thanks for sharing this.

I'm wondering if your import scripts can be used for building investment transactions? How do you handle balance assertions? How do you handle institutions that include multiple tables in a single csv? And I'm curious: why not use bean-file (I don't know what it's called in beangulp) for archiving?

Finally, would you be willing to share institution specific configs with the community?

On Tuesday, March 30, 2021 at 1:46:53 PM UTC-7 Luciano Fiandesio wrote:

Hello, cheers! I would like to share Beanborg, a small project I have been working on to semi-automatically import CSV transactions into a Beancount ledger.

From the README:

Goals and key features

Beanborg has two main design goals:

• automatic matching of transaction data with the correct Expense accounts
• speed, the tool is designed to process several financial CSV file in few seconds

Other features:

• a sophisticated and extendible rule-based system
• avoid duplicates during the import
• highly configurable
  
• smart archiving function: when archiving a CSV file, the file is renamed using the start and end date of the CSV file

I have been using this tool for the last 18 months. Happy to share in case someone wants to try it out: https://github.com/luciano-fiandesio/beanborg

Feedback welcome!

Luciano

--
You received this message because you are subscribed to the Google Groups "Beancount" group.
To unsubscribe from this group and stop receiving emails from it, send an email to beancount+...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/beancount/212c4c75-bcc4-40bf-8b4d-23f4c0276f88n%40googlegroups.com.

[redst...@gmail.com]

On Thursday, April 1, 2021 at 3:09:36 AM UTC-7 Luciano Fiandesio wrote:

Hi, try to answer some of the questions:

> I'm wondering if your import scripts can be used for building investment transactions?

I'm not sure, I don't track my investments using beancount. I wish I could give a better answer. Perhaps we could discuss the use case and try to figure out how to incorporate investment imports into beanborg.

This article covers the basics required for investment accounts (buys, sells, commissions/fees, cash/money market balance, commodity-leaf accounts).

 

> How do you handle balance assertions?

I don't. Beanborg main goal is to import transactions and assign expenses based on some rules.

I've written about why automated balance assertions are critical to ensuring import integrity here. There are some exceptions, but CSV files don't generally include balances; OFXs do.

 

> How do you handle institutions that include multiple tables in a single csv?

I have never encountered such case, do you have a "sanitized" CSV I can look at?

This is the unfortunate part of sharing ingest code: it's hard to share CSVs. Schwab uses multitable CSVs as does Vanguard AFAIK. See this article for more.

 

Unrelated to the above: have you considered releasing your rule-based categorizer as a hook for beancount's import process? This would make it an alternative approach to smart_importer.

[Martin Michlmayr]

* redst...@gmail.com <redst...@gmail.com> [2021-04-02 01:27]:

> Unrelated to the above: have you considered releasing your rule-based
> categorizer as a hook for beancount's import process? This would make it an
> alternative approach to smart_importer

> <https://github.com/beancount/smart_importer>.

There are now at least four different solutions for a rules-based
importer. I'm wondering if we could jointly work on a design and
implementation for one solution that would work for everyone.

I've wanted to bring up this topic here for almost a year (when I
wrote one, which unfortunately I never published). I'll try to put
together an email soon with some thoughts and links to the different
solutions.

--
Martin Michlmayr
https://www.cyrius.com/

[Martin Blais]

Consider doing it integrated well with beangulp. I won't have time to work on it, but I think it's worth making this easy and now common

--
You received this message because you are subscribed to the Google Groups "Beancount" group.
To unsubscribe from this group and stop receiving emails from it, send an email to beancount+...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/beancount/20210402110115.GA9213%40jirafa.cyrius.com.