Newbie Setting Up CSV Import / Ingest

1,335 views
Skip to first unread message

trs...@tutanota.com

unread,
Jun 22, 2018, 12:08:09 PM6/22/18
to Beancount Mailing List
OK, so this is quite challenging for someone who doesn't really know Python. However I think it's a good exercise not only for myself but also to help other newbies who would like to try and get this awesome feature working.

I have read everything I can in source and mailing list about CSV Import / Ingest and I've made some progress, but now I'm stuck.

Apologies in advance for ugly formatting, Google Groups apparently do not support inline text formatting, and I am communicating with the group via email.

I've tried to (mostly) follow the naming conventions in the examples but it seems they have changed over time. Anyway, file structure looks like so:
~/fin
    |---documents
    |---Downloads
    |---importers
    |    |---suncoast_g
    |         |---__init__.py   (this file shared below)
    |    |---__init__.py        (this file is empty)
    |---ledger.beancount
    |---ledger.config         (I have seen this also referenced as .import in docs)

Here is my ledger.config file:
--------------------(begin ledger.config file)--------------------
#!/usr/bin/env python3
"""Example import configuration."""

# Insert our custom importers path here.
# (In practice you might just change your PYTHONPATH environment.)
import sys
from os import path
sys.path.insert(0, path.join(path.dirname(__file__)))

from importers import suncoast_g
#from importers import acme_pdf

from beancount.ingest import extract
#from beancount.ingest.importers import ofx


# Setting this variable provides a list of importer instances.
#
# Removed the following from below to replace with my own, saved for reference
#
#    utrade.Importer("USD",
#                    "Assets:US:UTrade",
#                    "Assets:US:UTrade:Cash",
#                    "Income:US:UTrade:{}:Dividend",
#                    "Income:US:UTrade:{}:Gains",
#                    "Expenses:Financial:Fees",
#                    "Assets:US:BofA:Checking"),
#
#    ofx.Importer("379700001111222",
#                 "Liabilities:US:CreditCard",
#                 "bofa"),
#
#    acme_pdf.Importer("Assets:US:AcmeBank"),
#
CONFIG = [
    suncoast_g.Importer("Assets:Suncoast:Checking-G"),
]


# Override the header on extracted text (if desired).
extract.HEADER = ';; -*- mode: org; mode: beancount; coding: utf-8; -*-\n'
--------------------(end ledger.config file)--------------------

OK now the __init__.py that is in suncoast_g contains following:
--------------------(begin __init__.py file)--------------------
#!/usr/bin/env python3

#
# Configuration file for extracting Suncoast-G data
#

from beancount.ingest import regression
from beancount.ingest.importers import csv

from beancount.plugins import auto_accounts


class Importer(csv.Importer):

    config = {csv.Col.DATE: 'Posted Date',
              csv.Col.TXN_DATE: 'Transaction Date',
              csv.Col.NARRATION: 'Description',
              csv.Col.AMOUNT_CREDIT: 'Deposit',
              csv.Col.AMOUNT_DEBIT: 'Withdrawal',
              csv.Col.BALANCE: 'Balance'}

    def __init__(self, account):
        csv.Importer.__init__(
            self, self.config,
            account, 'Currency',
            ('Posted Date,Transaction Date,Description,'
             'Deposit,Withdrawal,Balance'),
            1)

    def get_description(self, row):
        payee, narration = super().get_description()
        narration = '{} ({})'.format(narration, row.category)
        return payee, narration
--------------------(end __init__.py file)--------------------

I have just copied this stuff and tried to figure it out. I'm sure I've got something wrong in here but I don't really know what I'm doing. FYI here is what the data looks like which is in G.csv in Downloads:

Posted Date,Transaction Date,Description,Deposit,Withdrawal,Balance
6/4/2018,6/4/2018,Withdrawal Debit Card SOME BAR & GRILL CITY ST Card XXXX,,($59.83),$229.15

OK I think that's all the relevant info. So now when I do:

~/fin$ bean-identify ledger.config Downloads

I get:

**** /home/myname/fin/Downloads/A Sunnet History 6186156 23032018_21062018.csv
**** /home/myname/fin/Downloads/G.csv

Which I think means it is identifying those 2 files (the only ones in there) as CSV, correct? I will point out that G.csv is an Asset account and is my first target here. The other one is a Liability account (credit card) and therefore has different fields (only one amount, and no balance). But I figure once I get this one working, that other one (and subsequent others) should be pretty easy.

OK so now when I do:

~/fin$ bean-extract ledger.config Downloads

I get:

**** /home/myname/fin/Downloads/A Sunnet History 6186156 23032018_21062018.csv                                                          
**** /home/myname/fin/Downloads/G.csv                                                                                                   
ERROR:root:Importer importers.suncoast_g.Importer: "Assets:Suncoast:Checking-G".extract() raised an unexpected error: CSV config without header has non-index fields: {<Col.DATE: '[DATE]'>: 'Posted Date', <Col.TXN_DATE: '[TXN_DATE]'>: 'Transaction Date', <Col.NARRATION: '[NARRATION1]'>: 'Description', <Col.AMOUNT_CREDIT: '[CREDIT]'>: 'Deposit', <Col.AMOUNT_DEBIT: '[DEBIT]'>: 'Withdrawal', <Col.BALANCE: '[BALANCE]'>: 'Balance'}                                                                                                                              
ERROR:root:Traceback: Traceback (most recent call last):                                                                                  
  File "/usr/local/lib/python3.6/dist-packages/beancount/ingest/extract.py", line 187, in extract                                         
    allow_none_for_tags_and_links=allow_none_for_tags_and_links)                                                                          
  File "/usr/local/lib/python3.6/dist-packages/beancount/ingest/extract.py", line 69, in extract_from_file                                
    new_entries = importer.extract(file, **kwargs)
  File "/usr/local/lib/python3.6/dist-packages/beancount/ingest/importers/csv.py", line 189, in extract
    iconfig, has_header = normalize_config(self.config, file.head())
  File "/usr/local/lib/python3.6/dist-packages/beancount/ingest/importers/csv.py", line 340, in normalize_config
    "{}".format(config))
ValueError: CSV config without header has non-index fields: {<Col.DATE: '[DATE]'>: 'Posted Date', <Col.TXN_DATE: '[TXN_DATE]'>: 'Transaction Date', <Col.NARRATION: '[NARRATION1]'>: 'Description', <Col.AMOUNT_CREDIT: '[CREDIT]'>: 'Deposit', <Col.AMOUNT_DEBIT: '[DEBIT]'>: 'Withdrawal', <Col.BALANCE: '[BALANCE]'>: 'Balance'}

;; -*- mode: org; mode: beancount; coding: utf-8; -*-

And this is where I'm currently stuck. I feel like it's something dumb, something not pointing at something else correctly but I don't know enough Python (yet) to figure it out myself. Any halp would be greatly appreciated. :)

TRS-80
--
Securely sent with Tutanota. Claim your encrypted mailbox today!
https://tutanota.com

trs...@tutanota.com

unread,
Jun 22, 2018, 2:20:01 PM6/22/18
to bean...@googlegroups.com
OK I sought and received some help in @python. I think I am on a much better track now. I don't know where I got my original __init__.py from, some similar thread here I think.

But now I have downloaded from source the utrade one from: https://bitbucket.org/blais/beancount/src/65212d1176bb427a7883d2593edbd0e0545a145a/examples/ingest/office/importers/utrade/__init__.py?at=default&fileviewer=file-view-default and am modifying that to my needs. I now see that I missed a whole bunch of the methods listed in "Writing an Importer" section of "Importing External Data" Docs. It will take me a while to work through it but I will post something back later, including results. I just didn't want anyone to spend time posting a long reply in the meantime.

Fun fun! :)


TRS-80

--
Securely sent with Tutanota. Claim your encrypted mailbox today!
https://tutanota.com

22. Jun 2018 12:08 by trs...@tutanota.com:

--
You received this message because you are subscribed to the Google Groups "Beancount" group.
To unsubscribe from this group and stop receiving emails from it, send an email to beancount+...@googlegroups.com.
To post to this group, send email to bean...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/beancount/LFcF9ZJ--3-0%40tutanota.com.
For more options, visit https://groups.google.com/d/optout.

trs...@tutanota.com

unread,
Jun 22, 2018, 7:21:32 PM6/22/18
to bean...@googlegroups.com
Yeah I was completely on the wrong track before (I think). But I am on the right one now (I think)?

So what I have done is just copy the csv.py file and save it as __init__.py in my importers/suncoast_g directory. Then I put the following into ledger.config: https://paste.pound-python.org/show/popHoa0wvVE2OiPCqIAL

But now when doing bean-extract I get "ValueError: CSV config without header has non-index fields: {'[DATE]': 'Posted Date', '[TXN_DATE]': 'Transaction Date', '[NARRATION1]': 'Description', '[CREDIT]': 'Deposit', '[DEBIT]': 'Withdrawal', '[BALANCE]': 'Balance'}"

Yes my CSV have headers. I been searching the internet for that error, but still scratching my head. Also tried to change '[DATE]' to 'DATE' etc. but that didn't seem to make a difference either.

Of course, I could be completely off track (this is my fourth different approach). I been flailing around at this all day and a good part of yesterday too. Early in the morning until late at night. At this point I would be willing to send someone a few dollars to help me get this set up. I am sure I could get other accounts working and maintain it once I can just get the first one working.

When I first saw my credit union's CSV file I thought "this should be easy" because it's very straightforward. I don't need all this complicated parsing like I have seen in some of the other Importers I have been studying. Just a straight CSV import. Or so I thought... :/

Anyway, any help at all would be greatly appreciated at this point. Any clue might help!


TRS-80
--
Securely sent with Tutanota. Claim your encrypted mailbox today!
https://tutanota.com

22. Jun 2018 14:19 by trs...@tutanota.com:

trs...@tutanota.com

unread,
Jun 24, 2018, 11:58:37 AM6/24/18
to Beancount Mailing List
OK, I am overjoyed to report that after days of frustration, I finally got the first recognizable output from bean-import (I feel like: "it actually works for you. Angels sing, and a light suddenly fills the room" (see: https://github.com/git/git/blob/e83c5163316f89bfbde7d9ab23ca2e25604af290/README)! XD

It still needs some further tweaking (adding the other leg of the transaction for instance) but I wanted to pause and take a moment to share where I am up to by now.

In my case, I had to actually identify the columns by number instead of name to get past the error in my last email ("ValueError: CSV config without header has non-index fields"...). Not sure if that is normal or what, but it worked for me. So here is my working config file now (note that they are 0 indexed):

----------------------------------------------begin------------------------------------------

#!/usr/bin/env python3
"""Example import configuration."""

# Insert our custom importers path here.
# (In practice you might just change your PYTHONPATH environment.)
import sys
from os import path
sys.path.insert(0, path.join(path.dirname(__file__)))

#from importers import suncoast_g

#from importers import acme_pdf

from beancount.ingest import extract
from beancount.ingest.importers import csv


# Setting this variable provides a list of importer instances.
#
# Removed the following from below to replace with my own, saved for reference
#
#    utrade.Importer("USD",
#                    "Assets:US:UTrade",
#                    "Assets:US:UTrade:Cash",
#                    "Income:US:UTrade:{}:Dividend",
#                    "Income:US:UTrade:{}:Gains",
#                    "Expenses:Financial:Fees",
#                    "Assets:US:BofA:Checking"),
#
#    ofx.Importer("379700001111222",
#                 "Liabilities:US:CreditCard",
#                 "bofa"),
#
#    acme_pdf.Importer("Assets:US:AcmeBank"),
#
CONFIG = [
    csv.Importer({csv.Col.DATE: 0,
                  csv.Col.TXN_DATE: 1,
                  csv.Col.NARRATION: 2,
                  csv.Col.AMOUNT_DEBIT: 4,
                  csv.Col.AMOUNT_CREDIT: 3,
                  csv.Col.BALANCE: 5},
                  'Assets:US:Suncoast:Checking',
                  'USD',
                  'Posted Date,Transaction Date,Description,Deposit,Withdrawal,Balance')

]

# Override the header on extracted text (if desired).
#xtract.HEADER = ';; -*- mode: org; mode: beancount; coding: utf-8; -*-\n'
------------------------------------------EOF-----------------------------------------------

So then I kept getting errors like " ValueError: ('Unknown string format:', 'Transaction Date') " At this point I honestly don't remember exactly how I fixed that.

Disclaimer: I don't really know Python and so I just change one thing at a time in my config file and run bean-extract and then see what output I get. ¯\_(ツ)_/¯ Sorry! lol

Anyway, I think maybe this is when I changed column numbering from starting with 1 to starting with 0.

Then the next error I got was " decimal.InvalidOperation: [<class 'decimal.ConversionSyntax'>] " which I got past by actually modifying my CSV file (specifically, removing all the $ from the file).

Again, could be getting the details not exactly correct, but my hope here is to convey the overall idea.

Then I think I got the same error and realized I would also need to remove all the parens which were around numbers which were already in the debit column (my bank uses separate debit and credit columns, so having parens around a number in debit is like a double negative, making it show up as positive).

This was another "Aha!" moment. I guess I thought maybe the CSV parser was smart enough to figure some of this stuff out? But I guess not. I suppose this is what is meant whenever the discussion of importers comes up (I had read several of those threads before jumping into this) and it's said just how customized each implementation is.

But by all means, please correct me if I am wrong, or have missed something.

So now that I have attained some success, and see the light at the end of the tunnel, it looks like I will have to do ~ the following:
1.Manually download CSV file from bank.
2.Do some pre-processing, either manually or with macros in Emacs, or (more likely) programatically, using scripts and sed, etc. to remove parens and $s.
3.Run the actual bean-import.
4.Run some post processing (I would like to change date: metadata name to transaction_date: because I think it's more descriptive).
5.And then finally hand copy these transactions into my main .beancount file, double checking and tweaking (aka "clearing") them in the process, categorizing remaining ones into Expense accounts and perhaps updating my scripts in the process.

I suppose 2, 4, and 5 could be done all in Emacs, but I'll just have to figure out some workflow now that works for me.

Also not mentioned is somehow programatically inserting the other leg of the transaction (which Expense account). I agree with Martin's basic philosophy on this, and still plan on manually reviewing everything, however I am already seeing that the bulk of transactions are the same places in my case and could easily be categorized with some simple matching (either in a post matching script or within bean-extract using categorizer). I need to look into this more, and also experiment or read up on how the de-duplication works, as I think it's probably related.

Anyway, I will continue to report on what I find as I go along, and even though I'm not getting any replies hopefully this will either encourage others to try and set this up or perhaps help other noobs who come along later looking for more in depth info (or perhaps stumble across similar error messages searching the internet) and it eventually helps someone.

Helpful tips, encouraging words, or even just letting me know if anyone is actually reading my idiotic ramblings are always welcomed. :D


TRS-80
--
Securely sent with Tutanota. Claim your encrypted mailbox today!
https://tutanota.com

22. Jun 2018 19:21 by trs...@tutanota.com:

Martin Blais

unread,
Jun 24, 2018, 3:21:46 PM6/24/18
to Beancount
On Sun, Jun 24, 2018 at 11:58 AM <trs...@tutanota.com> wrote:
[...]But by all means, please correct me if I am wrong, or have missed something.


So now that I have attained some success, and see the light at the end of the tunnel, it looks like I will have to do ~ the following:
1.Manually download CSV file from bank.
Yes
 
2.Do some pre-processing, either manually or with macros in Emacs, or (more likely) programatically, using scripts and sed, etc. to remove parens and $s.
You can write code in your importer to do that.
 
3.Run the actual bean-import.
You mean bean-extract.

4.Run some post processing (I would like to change date: metadata name to transaction_date: because I think it's more descriptive).
Do that in your importer code as well.
 

5.And then finally hand copy these transactions into my main .beancount file, double checking and tweaking (aka "clearing") them in the process, categorizing remaining ones into Expense accounts and perhaps updating my scripts in the process.
Yes.

I suppose 2, 4, and 5 could be done all in Emacs, but I'll just have to figure out some workflow now that works for me.
Yes.
 

Also not mentioned is somehow programatically inserting the other leg of the transaction (which Expense account). I agree with Martin's basic philosophy on this, and still plan on manually reviewing everything, however I am already seeing that the bulk of transactions are the same places in my case and could easily be categorized with some simple matching (either in a post matching script or within bean-extract using categorizer). I need to look into this more, and also experiment or read up on how the de-duplication works, as I think it's probably related.

You can write some function for your importer to do that with your particular rules if it saves you time.


Anyway, I will continue to report on what I find as I go along, and even though I'm not getting any replies
Short emails with direct questions -> more replies more quickly

 
hopefully this will either encourage others to try and set this up or perhaps help other noobs who come along later looking for more in depth info (or perhaps stumble across similar error messages searching the internet) and it eventually helps someone.

Helpful tips, encouraging words, or even just letting me know if anyone is actually reading my idiotic ramblings are always welcomed. :D

Sounds like you're making great progress!
Unfortunately automating the importing still requires writing Python code and I see no way around that, I wish it was easier.

 

trs...@tutanota.com

unread,
Jun 25, 2018, 4:34:00 PM6/25/18
to bean...@googlegroups.com
OK, stayed up late last night and actually got all my character stripping accomplished in Python within the provided tools. Yay me (first Python code I ever wrote)! :)

OK so basic CSV importers are working, now trying to figure out where to stick the categorizer code I found here: https://bitbucket.org/blais/beancount/pull-requests/24/improve-ingestimporterscsv/diff

I been trying here and there without success as of yet. Any hints/pointers would be greatly appreciated!


TRS-80
--
Securely sent with Tutanota. Claim your encrypted mailbox today!
https://tutanota.com

24. Jun 2018 15:21 by bl...@furius.ca:

trs...@tutanota.com

unread,
Jul 19, 2018, 8:32:39 AM7/19/18
to bean...@googlegroups.com
It is still unclear to me where to put this categorizer code? I have tried putting it here, there, and everywhere. I am using the provided generic CSV importer, which calls it, but I cannot figure out where to put it or how to instantiate it or whatever it is you need to do in Python.

Since I don't really know Python, I am happy to pay someone few bucks to help me get this working.

(from https://bitbucket.org/blais/beancount/pull-requests/24/improve-ingestimporterscsv/diff):

def dumb_categorizer(txn):
    # At this time the txn has only one posting
    try:
        posting1 = txn.postings[0]
    except IndexError:
        return txn

    # Guess the account(s) of the other posting(s)
    if 'nutella' in txn.narration.lower():
        account = 'Expenses:Food'
    else:
        return txn

    # Make the other posting(s)
    posting2 = posting1._replace(
        account=account,
        units=-posting1.units
    )

    # Insert / Append the posting into the transaction
    if posting1.units < posting2.units:
        txn.postings.append(posting2)
    else:
        txn.postings.insert(0, posting2)

    return txn


--
Securely sent with Tutanota. Claim your encrypted mailbox today!
https://tutanota.com

25. Jun 2018 16:33 by trs...@tutanota.com:

OK, stayed up late last night and actually got all my character stripping accomplished in Python within the provided tools. Yay me (first Python code I ever wrote)! :)

OK so basic CSV importers are working, now trying to figure out where to stick the categorizer code I found here: https://bitbucket.org/blais/beancount/pull-requests/24/improve-ingestimporterscsv/diff

I been trying here and there without success as of yet. Any hints/pointers would be greatly appreciated!

TRS-80
--
Securely sent with Tutanota. Claim your encrypted mailbox today!
https://tutanota.com

24. Jun 2018 15:21 by bl...@furius.ca:


For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "Beancount" group.
To unsubscribe from this group and stop receiving emails from it, send an email to beancount+...@googlegroups.com.
To post to this group, send email to bean...@googlegroups.com.

trs...@tutanota.com

unread,
Jul 19, 2018, 8:52:06 AM7/19/18
to bean...@googlegroups.com
I suppose I should have included a link to the CSV importer source: https://bitbucket.org/blais/beancount/src/80d30d6896cf5fdcff8c1156cab77107ee8e0f96/beancount/ingest/importers/csv.py?at=default&fileviewer=file-view-default

Down toward the bottom (line 283) is where the categorizer gets called.

Last night at my local LUG, I volunteered to do a talk next month on plain text accounting, and got the green light. So it would be nice to get this working by then. :)


TRS-80
--
Securely sent with Tutanota. Claim your encrypted mailbox today!
https://tutanota.com

19. Jul 2018 08:32 by trs...@tutanota.com:

--
Securely sent with Tutanota. Claim your encrypted mailbox today!
https://tutanota.com

25. Jun 2018 16:33 by trs...@tutanota.com:

OK, stayed up late last night and actually got all my character stripping accomplished in Python within the provided tools. Yay me (first Python code I ever wrote)! :)

OK so basic CSV importers are working, now trying to figure out where to stick the categorizer code I found here: https://bitbucket.org/blais/beancount/pull-requests/24/improve-ingestimporterscsv/diff

I been trying here and there without success as of yet. Any hints/pointers would be greatly appreciated!

TRS-80
--
Securely sent with Tutanota. Claim your encrypted mailbox today!
https://tutanota.com

24. Jun 2018 15:21 by bl...@furius.ca:


For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "Beancount" group.
To unsubscribe from this group and stop receiving emails from it, send an email to beancount+...@googlegroups.com.
To post to this group, send email to bean...@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "Beancount" group.
To unsubscribe from this group and stop receiving emails from it, send an email to beancount+...@googlegroups.com.
To post to this group, send email to bean...@googlegroups.com.

trs...@tutanota.com

unread,
Jul 19, 2018, 10:44:30 AM7/19/18
to bean...@googlegroups.com
OK, I am successfully calling dumb_categorizer from CSV Importer by defining it at beginning of .config file, and then passing categorizer = dumb_categorizer to CSV Importer. I know this because I replaced it with a simple print("something") and I got a bunch of "something" on stdout. So the categorizer is getting called, it's just either not matching or not attaching the other leg... ?

Any help would be greatly appreciated.


TRS-80
--
Securely sent with Tutanota. Claim your encrypted mailbox today!
https://tutanota.com

19. Jul 2018 08:52 by trs...@tutanota.com:

I suppose I should have included a link to the CSV importer source: https://bitbucket.org/blais/beancount/src/80d30d6896cf5fdcff8c1156cab77107ee8e0f96/beancount/ingest/importers/csv.py?at=default&fileviewer=file-view-default

Down toward the bottom (line 283) is where the categorizer gets called.

Last night at my local LUG, I volunteered to do a talk next month on plain text accounting, and got the green light. So it would be nice to get this working by then. :)


TRS-80
--
Securely sent with Tutanota. Claim your encrypted mailbox today!
https://tutanota.com

--
Securely sent with Tutanota. Claim your encrypted mailbox today!
https://tutanota.com

25. Jun 2018 16:33 by trs...@tutanota.com:

OK, stayed up late last night and actually got all my character stripping accomplished in Python within the provided tools. Yay me (first Python code I ever wrote)! :)

OK so basic CSV importers are working, now trying to figure out where to stick the categorizer code I found here: https://bitbucket.org/blais/beancount/pull-requests/24/improve-ingestimporterscsv/diff

I been trying here and there without success as of yet. Any hints/pointers would be greatly appreciated!

TRS-80
--
Securely sent with Tutanota. Claim your encrypted mailbox today!
https://tutanota.com

24. Jun 2018 15:21 by bl...@furius.ca:


For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "Beancount" group.
To unsubscribe from this group and stop receiving emails from it, send an email to beancount+...@googlegroups.com.
To post to this group, send email to bean...@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "Beancount" group.
To unsubscribe from this group and stop receiving emails from it, send an email to beancount+...@googlegroups.com.
To post to this group, send email to bean...@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "Beancount" group.
To unsubscribe from this group and stop receiving emails from it, send an email to beancount+...@googlegroups.com.
To post to this group, send email to bean...@googlegroups.com.

trs...@tutanota.com

unread,
Jul 19, 2018, 12:22:48 PM7/19/18
to bean...@googlegroups.com
I figured it out. The dumb_categorizer does .lower(): and I was passing it a search term with a capital letter in it. Now I'm off to the races.. :)

I think maybe I might publish my working setup once I get it all cleaned up, as yet another example for others to follow.

TRS-80

--
Securely sent with Tutanota. Claim your encrypted mailbox today!
https://tutanota.com

19. Jul 2018 10:44 by trs...@tutanota.com:

OK, I am successfully calling dumb_categorizer from CSV Importer by defining it at beginning of .config file, and then passing categorizer = dumb_categorizer to CSV Importer. I know this because I replaced it with a simple print("something") and I got a bunch of "something" on stdout. So the categorizer is getting called, it's just either not matching or not attaching the other leg... ?

Any help would be greatly appreciated.

TRS-80
--
Securely sent with Tutanota. Claim your encrypted mailbox today!
https://tutanota.com

19. Jul 2018 08:52 by trs...@tutanota.com:

I suppose I should have included a link to the CSV importer source: https://bitbucket.org/blais/beancount/src/80d30d6896cf5fdcff8c1156cab77107ee8e0f96/beancount/ingest/importers/csv.py?at=default&fileviewer=file-view-default

Down toward the bottom (line 283) is where the categorizer gets called.

Last night at my local LUG, I volunteered to do a talk next month on plain text accounting, and got the green light. So it would be nice to get this working by then. :)


TRS-80
--
Securely sent with Tutanota. Claim your encrypted mailbox today!
https://tutanota.com

--
Securely sent with Tutanota. Claim your encrypted mailbox today!
https://tutanota.com

25. Jun 2018 16:33 by trs...@tutanota.com:

OK, stayed up late last night and actually got all my character stripping accomplished in Python within the provided tools. Yay me (first Python code I ever wrote)! :)

OK so basic CSV importers are working, now trying to figure out where to stick the categorizer code I found here: https://bitbucket.org/blais/beancount/pull-requests/24/improve-ingestimporterscsv/diff

I been trying here and there without success as of yet. Any hints/pointers would be greatly appreciated!

TRS-80
--
Securely sent with Tutanota. Claim your encrypted mailbox today!
https://tutanota.com

24. Jun 2018 15:21 by bl...@furius.ca:


For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "Beancount" group.
To unsubscribe from this group and stop receiving emails from it, send an email to beancount+...@googlegroups.com.
To post to this group, send email to bean...@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "Beancount" group.
To unsubscribe from this group and stop receiving emails from it, send an email to beancount+...@googlegroups.com.
To post to this group, send email to bean...@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "Beancount" group.
To unsubscribe from this group and stop receiving emails from it, send an email to beancount+...@googlegroups.com.
To post to this group, send email to bean...@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "Beancount" group.
To unsubscribe from this group and stop receiving emails from it, send an email to beancount+...@googlegroups.com.
To post to this group, send email to bean...@googlegroups.com.

Martin Blais

unread,
Jul 19, 2018, 11:01:48 PM7/19/18
to Beancount
Great to hear it's working.
Setting up importers is still, I'm afraid, a programming task, but once it'll be working for you that's all going to be incremental work and you should be good for a while.


To view this discussion on the web visit https://groups.google.com/d/msgid/beancount/LHnLRo---3-0%40tutanota.com.

dosh...@gmail.com

unread,
Sep 16, 2018, 12:15:32 AM9/16/18
to Beancount
Hey, 

I'm in a very similar boat, were you able to post your importer files publicly? I think seeing the conversation of you working through this, along with your finished files would make your files a lot more easier to understand than the current examples I've seen.

Cheers,

Martin Blais

unread,
Sep 16, 2018, 9:39:12 AM9/16/18
to Beancount
There are also some examples in the source code, here:




To view this discussion on the web visit https://groups.google.com/d/msgid/beancount/660e92ff-2ba4-4c47-9fbd-eb76b8ec6571%40googlegroups.com.

trs...@tutanota.com

unread,
Apr 16, 2019, 2:52:23 PM4/16/19
to Beancount Mailing List
Aha! Found it! :)

I'm bumping this so I can reply from my new email.

TRS-80

--
Securely sent with Tutanota. Get your own encrypted, ad-free mailbox:


Sep 16, 2018, 8:38 AM by bl...@furius.ca:

Alen Šiljak

unread,
Apr 22, 2019, 3:05:37 AM4/22/19
to Beancount
Have you perhaps tried Ledger's CSV import? By reading the docs I had an impression that it was fairly easy. All it requires is using Ledger's headers in the CSV file and it would use the appropriate columns. I haven't tried it yet, though.
If that works, you should be able to convert the resulting output to beancount syntax. That requires installing and running a Perl script, though.
Just an idea.
Reply all
Reply to author
Forward
0 new messages