Best path for some simple importers

144 views
Skip to first unread message

Eric Altendorf

unread,
Jul 16, 2023, 11:34:59 PM7/16/23
to bean...@googlegroups.com
I have a few CSV files with basic transactions (basically sent and received assets & amounts, plus commission cost).  These are all I need to process.  It would be pretty easy for me to just translate them line by line into Beancount format inputs, but I like to do things "the right way".  Problem is, I can't figure out what "the right way" is :)

I spent a long time exploring the importer framework before realizing that it was the old one, tossed out for v3.  I have now read the design doc for the v3 importer system, the readme, skimmed the code and some examples, and then read some threads on this list which also led me to RedS's CSV framework, which I also looked into.

Am I trying too hard?  Should I just ignore the frameworks and spit out text files of the right format?  If I should use a framework...which would folks recommend?

thanks,
eric

Red S

unread,
Jul 16, 2023, 11:58:18 PM7/16/23
to Beancount
I'd recommend exactly that as the first step---spitting out text files of the write format, and using that with your workflow for a while. You'll either find that sufficient, or truly understand the specific problems you want to solve, and be able to pick the right framework with that knowledge.

Beancount has a built in generic CSV importer, I'd suggest trying that first. That'll get you bean-{identify, extract, file} for free. If you need the heavy lifting of investments, find yourself solving similar problems across multiple importers, or find your workflow could be smoother, my importer framework can help.

Eric Altendorf

unread,
Jul 17, 2023, 12:52:49 PM7/17/23
to bean...@googlegroups.com
Thanks for the quick reply!

On Sun, Jul 16, 2023 at 8:58 PM Red S <redst...@gmail.com> wrote:
I'd recommend exactly that as the first step---spitting out text files of the write format, and using that with your workflow for a while. You'll either find that sufficient, or truly understand the specific problems you want to solve, and be able to pick the right framework with that knowledge.

Cool, makes sense.
 
Beancount has a built in generic CSV importer, I'd suggest trying that first. That'll get you bean-{identify, extract, file} for free. If you need the heavy lifting of investments, find yourself solving similar problems across multiple importers, or find your workflow could be smoother, my importer framework can help.

I'm not even sure if identify/extract/file is useful to me.  I literally have 4 CSV files that have my all time historical transactions, that are already filed in the place I want them, and I just want to calculate cap gains and transaction fees for tax purposes.  I'll take a look at the generic importer and see if I can figure it out; it may be easier to start from scratch.

thanks!

 

On Sunday, July 16, 2023 at 8:34:59 PM UTC-7 erical...@gmail.com wrote:
I have a few CSV files with basic transactions (basically sent and received assets & amounts, plus commission cost).  These are all I need to process.  It would be pretty easy for me to just translate them line by line into Beancount format inputs, but I like to do things "the right way".  Problem is, I can't figure out what "the right way" is :)

I spent a long time exploring the importer framework before realizing that it was the old one, tossed out for v3.  I have now read the design doc for the v3 importer system, the readme, skimmed the code and some examples, and then read some threads on this list which also led me to RedS's CSV framework, which I also looked into.

Am I trying too hard?  Should I just ignore the frameworks and spit out text files of the right format?  If I should use a framework...which would folks recommend?

thanks,
eric

--
You received this message because you are subscribed to the Google Groups "Beancount" group.
To unsubscribe from this group and stop receiving emails from it, send an email to beancount+...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/beancount/c091350e-c0f5-45af-9a64-35bd17ae76f7n%40googlegroups.com.

Martin Blais

unread,
Jul 17, 2023, 7:34:20 PM7/17/23
to bean...@googlegroups.com
On Tue, Jul 18, 2023 at 12:52 AM Eric Altendorf <erical...@gmail.com> wrote:
Thanks for the quick reply!

On Sun, Jul 16, 2023 at 8:58 PM Red S <redst...@gmail.com> wrote:
I'd recommend exactly that as the first step---spitting out text files of the write format, and using that with your workflow for a while. You'll either find that sufficient, or truly understand the specific problems you want to solve, and be able to pick the right framework with that knowledge.

Cool, makes sense.
 
Beancount has a built in generic CSV importer, I'd suggest trying that first. That'll get you bean-{identify, extract, file} for free. If you need the heavy lifting of investments, find yourself solving similar problems across multiple importers, or find your workflow could be smoother, my importer framework can help.

I'm not even sure if identify/extract/file is useful to me.  I literally have 4 CSV files that have my all time historical transactions, that are already filed in the place I want them, and I just want to calculate cap gains and transaction fees for tax purposes.  I'll take a look at the generic importer and see if I can figure it out; it may be easier to start from scratch.

I agree; if you do a one-off conversion from just a few sources of data the whole importer/ingest framework doesn't buy you much.
I'd just write a script.


 

thanks!

 

On Sunday, July 16, 2023 at 8:34:59 PM UTC-7 erical...@gmail.com wrote:
I have a few CSV files with basic transactions (basically sent and received assets & amounts, plus commission cost).  These are all I need to process.  It would be pretty easy for me to just translate them line by line into Beancount format inputs, but I like to do things "the right way".  Problem is, I can't figure out what "the right way" is :)

I spent a long time exploring the importer framework before realizing that it was the old one, tossed out for v3.  I have now read the design doc for the v3 importer system, the readme, skimmed the code and some examples, and then read some threads on this list which also led me to RedS's CSV framework, which I also looked into.

Am I trying too hard?  Should I just ignore the frameworks and spit out text files of the right format?  If I should use a framework...which would folks recommend?

thanks,
eric

--
You received this message because you are subscribed to the Google Groups "Beancount" group.
To unsubscribe from this group and stop receiving emails from it, send an email to beancount+...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/beancount/c091350e-c0f5-45af-9a64-35bd17ae76f7n%40googlegroups.com.

--
You received this message because you are subscribed to the Google Groups "Beancount" group.
To unsubscribe from this group and stop receiving emails from it, send an email to beancount+...@googlegroups.com.

Eric Altendorf

unread,
Jul 18, 2023, 9:10:09 PM7/18/23
to bean...@googlegroups.com
FWIW I went ahead and wrote my importers to the Beangulp API.  I'm developing them here if anyone wants to follow along, contribute, or comment:

Currently covers Coinbase, Coinbase Pro, and Gate.io.

For now I just have a dumb shell script that runs each importer and merges their output textually.  I'll be investigating some of the deduping frameworks.

To view this discussion on the web visit https://groups.google.com/d/msgid/beancount/CAK21%2BhPF1xj9Xw15Vkj911e5MOg7PaUEgz8LghywLcpw69tuwQ%40mail.gmail.com.

Martin Blais

unread,
Jul 30, 2023, 12:12:26 PM7/30/23
to bean...@googlegroups.com

On Wed, Jul 19, 2023 at 9:10 AM Eric Altendorf <erical...@gmail.com> wrote:
FWIW I went ahead and wrote my importers to the Beangulp API.  I'm developing them here if anyone wants to follow along, contribute, or comment:

Currently covers Coinbase, Coinbase Pro, and Gate.io.

Eric Altendorf

unread,
Aug 1, 2023, 3:20:00 PM8/1/23
to bean...@googlegroups.com
On Sun, Jul 30, 2023 at 9:12 AM Martin Blais <bl...@furius.ca> wrote:

On Wed, Jul 19, 2023 at 9:10 AM Eric Altendorf <erical...@gmail.com> wrote:
FWIW I went ahead and wrote my importers to the Beangulp API.  I'm developing them here if anyone wants to follow along, contribute, or comment:

Currently covers Coinbase, Coinbase Pro, and Gate.io.


Thanks.  FWIW, the current state isn't really usable by others yet.  I haven't fully disentangled the custom logic specific to my accounts & setup from the generic importers and support libraries.  If anyone is interested in these reach out to me.
 

Eric Altendorf

unread,
Aug 9, 2023, 12:33:57 AM8/9/23
to bean...@googlegroups.com
On Tue, Aug 1, 2023 at 12:19 PM Eric Altendorf <erical...@gmail.com> wrote:
On Sun, Jul 30, 2023 at 9:12 AM Martin Blais <bl...@furius.ca> wrote:

On Wed, Jul 19, 2023 at 9:10 AM Eric Altendorf <erical...@gmail.com> wrote:
FWIW I went ahead and wrote my importers to the Beangulp API.  I'm developing them here if anyone wants to follow along, contribute, or comment:

Currently covers Coinbase, Coinbase Pro, and Gate.io.


Thanks.  FWIW, the current state isn't really usable by others yet.  I haven't fully disentangled the custom logic specific to my accounts & setup from the generic importers and support libraries.  If anyone is interested in these reach out to me.

Just following up on this; these are now disentangled and the code should now be runnable by anyone.  Not easily; it's still very rough and very much a power-user's tool, but it's at the point where someone else with similar needs, some Python skills, and motivation, could contribute and also get value.

I see in the archives over the years a multitude of crypto folks who have shown up with very similar issues as me -- if any of you are still around and lurking, and want to get this problem solved, reach out :)

package has been renamed to https://github.com/ericaltendorf/magicbeans ; i couldn't resist.....

eric
Reply all
Reply to author
Forward
0 new messages