Debugging Importers

[Mattijs Hoitink]

Hi,

I've started using beancount just a short while ago, reading the documentation first and now trying my hand at some custom importers. This is going pretty well and I'm getting my transactions extracted.

I'd like to get some insights in what is going on under the hood and how data flows through the beancount code. I was hoping to get some tips on debugging importers and beancount in general.

So far I've used vim + pdb and also trying to get the PyCharm debugger to work.

cheers,

mattijs

[Martin Blais]

On Sun, Jul 24, 2016 at 7:45 PM, Mattijs Hoitink <mattijs...@gmail.com> wrote:

Hi,

I've started using beancount just a short while ago, reading the documentation first and now trying my hand at some custom importers. This is going pretty well and I'm getting my transactions extracted.

I'd like to get some insights in what is going on under the hood and how data flows through the beancount code. I was hoping to get some tips on debugging importers and beancount in general.

Check out the section on regression testing from this doc:

https://docs.google.com/document/d/11EwQdujzEo2cxqaF5PgxCEZXWfKKQCYSMfdJowp_1S8/edit#heading=h.re9y2huiep1o

This is what I use to test my own. 

In one sentence, the easiest (laziest) way to write these tests is to use real downloaded files and to compare the output of the importer against an expected file, and ensure that as you change your importer those don't change. Of course, a good coverage with unit tests would be better, but in practice these importers have many unspecified quirks and no specification and can be unreasonably time-consuming or even impossible to test comprehensively. So I encourage you to be scrappy and just write the minimum you need and incrementally fix your importers when unexpected inputs appear.

So far I've used vim + pdb and also trying to get the PyCharm debugger to work.

The usual Python tools should all work.

 

cheers,

mattijs

--
You received this message because you are subscribed to the Google Groups "Beancount" group.
To unsubscribe from this group and stop receiving emails from it, send an email to beancount+...@googlegroups.com.
To post to this group, send email to bean...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/beancount/ff4b9186-5ceb-4c68-9ffd-2431b5a8e980%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Mattijs Hoitink]

Thanks for the pointers, I read the ingestion test guide but have not set that up yet.

I was trying to get some breakpoints set up in PyCarm so I can inspect the running code. For some reason my breakpoint fires inside the import config file but not inside the importer file.

I was wondering if this might be because the importer is run through runpy.run_path? Maybe this makes the code special? I also understand that you might not be a PyCharm expert but any help in narrowing this down would be very helpful :)

cheers,

mattijs

[Martin Blais]

Not sure, but this might have something to do with bean-extract itself importing your configuration file.

You can try adding a explicit call to the main function for extrawct from your config file itself and running it itself, that may solve your problem.

To view this discussion on the web visit https://groups.google.com/d/msgid/beancount/324f82e4-39a4-4b8e-a26b-6b4d2789c1da%40googlegroups.com.