plugin proposal - file_ordering

[Stefano Zacchiroli]

Dear Beancounters,

attached you can find a plugin proposal which I'm using for my personal
Beancount needs but might be useful for others. If there is interest,
I'll be happy to submit it via the BTS, possibly adapting it to be more
generic before doing so.

The plugin validates that the in-file ordering of directives is
consistent with date ordering. That is not what you always want; and the
fact that Beancount does not care by default about file ordering is
indeed a great design principle.

But *if* you adopt the style of linear, date-ordered Beancount file(s),
then this plugin might help you in detecting errors. The typical error
that I'm successful avoiding with this plugin is copy-pasting an old
transaction as a more recent one, but forgetting to bump its date.

I haven't done so because I don't personally need it, but it should be
trivial to make the plugin configurable to, e.g., allow reverse ordering
and/or ensure that transactions are strictly ordered according to
metadata other than date. I'll be happy to support this if there is
interest from others.

As this is my first Beancount plugin, I also welcome comments on the
code and approach.

What do you think?
Cheers.
--
Stefano Zacchiroli . za...@upsilon.cc . upsilon.cc/zack . . o . . . o . o
Computer Science Professor . CTO Software Heritage . . . . . o . . . o o
Former Debian Project Leader . OSI Board Director . . . o o o . . . o .
« the first rule of tautology club is the first rule of tautology club »

[Simon Michael]

On 11/21/16 1:50 AM, Stefano Zacchiroli wrote:
> But *if* you adopt the style of linear, date-ordered Beancount file(s),
> then this plugin might help you in detecting errors. The typical error
> that I'm successful avoiding with this plugin is copy-pasting an old
> transaction as a more recent one, but forgetting to bump its date.
>
> I haven't done so because I don't personally need it, but it should be
> trivial to make the plugin configurable to, e.g., allow reverse ordering
> and/or ensure that transactions are strictly ordered according to
> metadata other than date. I'll be happy to support this if there is
> interest from others.
>
> As this is my first Beancount plugin, I also welcome comments on the
> code and approach.

Hi Stefano,

just on the general idea: I think it's a good one! At least in my
workflow, it's surprisingly easy to make this mistake and to lose time
tracking it down during reconciliation. I'm going to start doing a
similar check.

[Stefano Zacchiroli]

On Mon, Nov 21, 2016 at 06:31:00AM -0800, Simon Michael wrote:
> just on the general idea: I think it's a good one! At least in my workflow,
> it's surprisingly easy to make this mistake and to lose time tracking it
> down during reconciliation. I'm going to start doing a similar check.

Thanks Simon, I'm glad it's useful for your workflow too!

Martin: can you please advise on how you want to go about contributed
plugins? I see various options, e.g.:

- people just publish them independently (or keep them private)

- you declare a fairly liberal policy into accepting contributed plugins
(similar with what the docs say about additional SQL functions), and
all plugins get collected in the main Beancount repo. In this case,
I'll be happy to submit file_ordering via an issue on bitbucket

- some sort of "beancount forge" emerges, possibly blessed by you, as
central place where non-mainline plugins get contributed, with some
sanity checking on namespace to avoid naming clashes

Do you have any preference?

Cheers

[Dominik Aumayr]

> - some sort of "beancount forge" emerges, possibly blessed by you, as
> central place where non-mainline plugins get contributed, with some
> sanity checking on namespace to avoid naming clashes

We could create a separate repository under https://github.com/beancount for plugins from the community.

- Dominik

> --
> You received this message because you are subscribed to the Google Groups "Beancount" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to beancount+...@googlegroups.com.
> To post to this group, send email to bean...@googlegroups.com.
> To view this discussion on the web visit https://groups.google.com/d/msgid/beancount/20161202080355.5rfm4c32wlenwiqa%40upsilon.cc.
> For more options, visit https://groups.google.com/d/optout.

[Stefano Zacchiroli]

On Fri, Dec 02, 2016 at 09:42:29AM +0100, Dominik Aumayr wrote:
> We could create a separate repository under
> https://github.com/beancount for plugins from the community.

Speaking of which, how about setting up
https://github.com/beancount/beancount as an always up-to-date git
mirror of the hg repo on bitbucket?

I'm using git-remote-hg locally and it works perfectly on the beancount
repo, so automating with a cron a "git pull" + "git push --mirror" seems
pretty straightforward. Martin: would you object to that? Of course the
github-based repo should come with a description stating that it's just
a mirror, and that the main development happens on bitbucket.

Cheers.

[Martin Blais]

On Thu, Dec 1, 2016 at 10:42 PM, Dominik Aumayr <dom...@aumayr.name> wrote:

> - some sort of "beancount forge" emerges, possibly blessed by you, as
>  central place where non-mainline plugins get contributed, with some
>  sanity checking on namespace to avoid naming clashes

We could create a separate repository under https://github.com/beancount for plugins from the community.

+1

 

- Dominik

> Am 02.12.2016 um 09:03 schrieb Stefano Zacchiroli <za...@upsilon.cc>:
>
> On Mon, Nov 21, 2016 at 06:31:00AM -0800, Simon Michael wrote:
>> just on the general idea: I think it's a good one! At least in my workflow,
>> it's surprisingly easy to make this mistake and to lose time tracking it
>> down during reconciliation. I'm going to start doing a similar check.
>
> Thanks Simon, I'm glad it's useful for your workflow too!
>
> Martin: can you please advise on how you want to go about contributed
> plugins? I see various options, e.g.:
>
> - people just publish them independently (or keep them private)

Best to start this way IMHO. 

See http://furius.ca/beancount/doc/contrib

Publish it and I'll add a link there.

This does seem like something that could go into the main source code eventually.

> - you declare a fairly liberal policy into accepting contributed plugins
>  (similar with what the docs say about additional SQL functions), and
>  all plugins get collected in the main Beancount repo. In this case,
>  I'll be happy to submit file_ordering via an issue on bitbucket

The main difference there is testing. My experience with various OSS projects is that many people who share things often provide an insufficient amount of testing, and invariably, adding good test coverage leads to much more work afterwards - you always discover flaws with tests. If you build good test coverage it makes it much, much easier to integrate a submission. Beancount has great support for building tests using Beancount syntax in docstrings; see the source code itself for examples.

> - some sort of "beancount forge" emerges, possibly blessed by you, as
>  central place where non-mainline plugins get contributed, with some
>  sanity checking on namespace to avoid naming clashes

I kind-of tried that with Ledgerhub - for sharing importers, I thought this would be popular, alas, so now it's gone - so I think the way it's built now it's very flexible (anything in your pythonpath) and I prefer to keep a hands-off approach. I would say, start by building plugins which are useful to you in a separate repo. Share them if you like. I like the idea of putting things under github.com/beancount/*, that's why it's there. Maybe in the future banding together with others to make a consolidated repo of plugins might be a good idea, if there's a theme. If some of those provide broadly useful functionality and have good test coverage, I'd be happy to fold them into the main source code. In general, I like the idea of letting the dust settle a bit and get a fair amount of real usage before doing that.

 

>
> Do you have any preference?

Thanks for asking.

 

>
> Cheers
> --
> Stefano Zacchiroli . za...@upsilon.cc . upsilon.cc/zack . . o . . . o . o
> Computer Science Professor . CTO Software Heritage . . . . . o . . . o o
> Former Debian Project Leader . OSI Board Director  . . . o o o . . . o .
> « the first rule of tautology club is the first rule of tautology club »
>
> --
> You received this message because you are subscribed to the Google Groups "Beancount" group.

> To unsubscribe from this group and stop receiving emails from it, send an email to beancount+unsubscribe@googlegroups.com.

> To post to this group, send email to bean...@googlegroups.com.
> To view this discussion on the web visit https://groups.google.com/d/msgid/beancount/20161202080355.5rfm4c32wlenwiqa%40upsilon.cc.
> For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "Beancount" group.

To unsubscribe from this group and stop receiving emails from it, send an email to beancount+unsubscribe@googlegroups.com.

To post to this group, send email to bean...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/beancount/39D793D4-11BF-4FDB-9C92-BF4047A91FF3%40aumayr.name.

[Martin Blais]

On Fri, Dec 2, 2016 at 12:06 AM, Stefano Zacchiroli <za...@upsilon.cc> wrote:

On Fri, Dec 02, 2016 at 09:42:29AM +0100, Dominik Aumayr wrote:
> We could create a separate repository under
> https://github.com/beancount for plugins from the community.

Speaking of which, how about setting up
https://github.com/beancount/beancount as an always up-to-date git
mirror of the hg repo on bitbucket?

I'm using git-remote-hg locally and it works perfectly on the beancount
repo, so automating with a cron a "git pull" + "git push --mirror" seems
pretty straightforward. Martin: would you object to that? Of course the
github-based repo should come with a description stating that it's just
a mirror, and that the main development happens on bitbucket.

Thanks for LMK it works. It's on my list of things to do. I have to take care of this and when I do, make sure the mirror is always up-to-date thereafter, I'll need to setup something on my server to that whenever I push it automatically pushes to github too, that's likely the best way to make sure all copies are sync'ed and up-to-date.

Cheers.

--
Stefano Zacchiroli . za...@upsilon.cc . upsilon.cc/zack . . o . . . o . o
Computer Science Professor . CTO Software Heritage . . . . . o . . . o o
Former Debian Project Leader . OSI Board Director  . . . o o o . . . o .
« the first rule of tautology club is the first rule of tautology club »

--
You received this message because you are subscribed to the Google Groups "Beancount" group.

To unsubscribe from this group and stop receiving emails from it, send an email to beancount+unsubscribe@googlegroups.com.

To post to this group, send email to bean...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/beancount/20161202100608.naj4serghkpjeeld%40upsilon.cc.

[Martin Blais]

I've created a mirror at https://github.com/beancount/beancount

I'm using hg-git, which works well enough.

I'll try to keep this in sync with the Mercurial repo.

(It seems like I can basically just push from my Mercurial client to the github git repo and it converts on-the-fly.)

Cheers,

[Stefano Zacchiroli]

On Fri, Dec 02, 2016 at 04:55:25PM -1000, Martin Blais wrote:
> I've created a mirror at https://github.com/beancount/beancount
> I'm using hg-git, which works well enough.
> I'll try to keep this in sync with the Mercurial repo.
> (It seems like I can basically just push from my Mercurial client to the
> github git repo and it converts on-the-fly.)

Awesome, thanks a lot for this!

[Stefano Zacchiroli]

On Fri, Dec 02, 2016 at 11:28:18AM -1000, Martin Blais wrote:
> > > - people just publish them independently (or keep them private)
>
> Best to start this way IMHO.
> See http://furius.ca/beancount/doc/contrib
> Publish it and I'll add a link there.

*nod*

This is now at https://github.com/zacchiro/beancount-plugins-zack .

Taking inspiration from the other plugins listed on that page, I've put
everything in the root dir, so that people can import it however they
like in their PYTHONPATH.

It would be useful at some point if you could document best practices or
other expectations on how 3rd party plugins should be "bundled up". That
would make it easy and consistent to use them for Beancount users.

> The main difference there is testing. My experience with various OSS
> projects is that many people who share things often provide an insufficient
> amount of testing, and invariably, adding good test coverage leads to much
> more work afterwards - you always discover flaws with tests. If you build
> good test coverage it makes it much, much easier to integrate a submission.
> Beancount has great support for building tests using Beancount syntax in
> docstrings; see the source code itself for examples.

It would be great if you could document what the expectations are on
that front (e.g., 100% coverage or not?), and how to best integrate with
the main Beancount test suite. I'll then be happy to play guinea pig
for the documented expectations, and submit file_ordering for
integration in the main code base, if you want me to.

> I kind-of tried that with Ledgerhub - for sharing importers, I thought this
> would be popular, alas, so now it's gone - so I think the way it's built
> now it's very flexible (anything in your pythonpath) and I prefer to keep a
> hands-off approach. I would say, start by building plugins which are useful
> to you in a separate repo. Share them if you like. I like the idea of
> putting things under github.com/beancount/*, that's why it's there. Maybe
> in the future banding together with others to make a consolidated repo of
> plugins might be a good idea, if there's a theme. If some of those provide
> broadly useful functionality and have good test coverage, I'd be happy to
> fold them into the main source code. In general, I like the idea of letting
> the dust settle a bit and get a fair amount of real usage before doing that.

Sounds good to me.

[Martin Blais]

On Sat, Dec 3, 2016 at 1:48 AM, Stefano Zacchiroli <za...@upsilon.cc> wrote:

On Fri, Dec 02, 2016 at 11:28:18AM -1000, Martin Blais wrote:
> > > - people just publish them independently (or keep them private)
>
> Best to start this way IMHO.
> See http://furius.ca/beancount/doc/contrib
> Publish it and I'll add a link there.

*nod*

This is now at https://github.com/zacchiro/beancount-plugins-zack .

Thanks!

I added a link from the contrib doc so people can find it.

Taking inspiration from the other plugins listed on that page, I've put
everything in the root dir, so that people can import it however they
like in their PYTHONPATH.

It would be useful at some point if you could document best practices or
other expectations on how 3rd party plugins should be "bundled up". That
would make it easy and consistent to use them for Beancount users.

I keep it simple: if it's in your PYTHONPATH, and it has a __plugins__ attribute, it's all good. There's no convention beyond that. Documented here:

https://docs.google.com/document/d/1QftxNvQPdH-MikMBHupftU6F4IsNZP5FlFh1LCbVgk8/edit#heading=h.s0kq0i1h7gdl

Note: I think most Python projects organize their files a bit differently than Beancount's source code. If Beancount was organized like most projects, its Python source code would be user beancount/beancount/* instead of beancount/src/python/beancount/*. (The reason it's different is that when I rewrote it I wasn't sure which language I'd commit to, so src/<language>/* was how I started.) You may want to follow what other people generally do. I might even change it eventually (it's pretty solidly anchored in Python at this point).

> The main difference there is testing. My experience with various OSS
> projects is that many people who share things often provide an insufficient
> amount of testing, and invariably, adding good test coverage leads to much
> more work afterwards - you always discover flaws with tests. If you build
> good test coverage it makes it much, much easier to integrate a submission.
> Beancount has great support for building tests using Beancount syntax in
> docstrings; see the source code itself for examples.

It would be great if you could document what the expectations are on
that front (e.g., 100% coverage or not?), and how to best integrate with
the main Beancount test suite.  I'll then be happy to play guinea pig
for the documented expectations, and submit file_ordering for
integration in the main code base, if you want me to.

If you want examples of testing plugins, see here: https://bitbucket.org/blais/beancount/src/tip/src/python/beancount/plugins/

I would say at the very minimum, every function/method should be called at least once; that's a good baseline. If you have complicated logic, you should have tests that exercise each of the main different use cases.

Beyond that, it doesn't have to be exhaustive.

For example, if you look at my plugins tests, I think they're a bit on the lighter side of testing IMO.

I use the tests to iterate on the source code, I've found it saves me a lot of time. There's really nice helpers to build test input from docstrings, it makes it fun.

Otherwise I try to follow the Google style for Python development, except with 4 space indent and underscore_case instead of camelCase. I run pylint (see config under etc/) regularly on everything, and some custom scripts to avoid unwanted dependency order.

> I kind-of tried that with Ledgerhub - for sharing importers, I thought this
> would be popular, alas, so now it's gone - so I think the way it's built
> now it's very flexible (anything in your pythonpath) and I prefer to keep a
> hands-off approach. I would say, start by building plugins which are useful
> to you in a separate repo. Share them if you like. I like the idea of
> putting things under github.com/beancount/*, that's why it's there. Maybe
> in the future banding together with others to make a consolidated repo of
> plugins might be a good idea, if there's a theme. If some of those provide
> broadly useful functionality and have good test coverage, I'd be happy to
> fold them into the main source code. In general, I like the idea of letting
> the dust settle a bit and get a fair amount of real usage before doing that.

Sounds good to me.

Cheers
--
Stefano Zacchiroli . za...@upsilon.cc . upsilon.cc/zack . . o . . . o . o
Computer Science Professor . CTO Software Heritage . . . . . o . . . o o
Former Debian Project Leader . OSI Board Director  . . . o o o . . . o .
« the first rule of tautology club is the first rule of tautology club »

--
You received this message because you are subscribed to the Google Groups "Beancount" group.
To unsubscribe from this group and stop receiving emails from it, send an email to beancount+unsubscribe@googlegroups.com.
To post to this group, send email to bean...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/beancount/20161203114840.lhqi4cfeztkgst2g%40upsilon.cc.

[Dominik Aumayr]

> I would say at the very minimum, every function/method should be called at least once; that's a good baseline. If you have complicated logic, you should have tests that exercise each of the main different use cases.
> Beyond that, it doesn't have to be exhaustive.
> For example, if you look at my plugins tests, I think they're a bit on the lighter side of testing IMO.
> I use the tests to iterate on the source code, I've found it saves me a lot of time. There's really nice helpers to build test input from docstrings, it makes it fun.
>
> Otherwise I try to follow the Google style for Python development, except with 4 space indent and underscore_case instead of camelCase. I run pylint (see config under etc/) regularly on everything, and some custom scripts to avoid unwanted dependency order.

We could create a repository under github.com/beancount/plugins-contrib, which has an integration for Travis CI (like Fava does) which defines and automatically runs the tests upon each commit/PR.

[Stefano Zacchiroli]

On Sat, Dec 03, 2016 at 09:21:35AM -1000, Martin Blais wrote:
> On Sat, Dec 3, 2016 at 1:48 AM, Stefano Zacchiroli <za...@upsilon.cc> wrote:
> >
> > It would be useful at some point if you could document best
> > practices or other expectations on how 3rd party plugins should be
> > "bundled up". That would make it easy and consistent to use them for
> > Beancount users.
>
> I keep it simple: if it's in your PYTHONPATH, and it has a __plugins__
> attribute, it's all good. There's no convention beyond that.

Sorry, I wasn't clear about what I meant with "bundle" here. I'm good on
the Python API between Beancount and plugins; it's great, clear, and
couldn't be simpler.

But 3rd party plugins distributed as repos or tarballs can implement
that API in many different ways. You can have a top-level Python module
that you just git clone, or a deep module namespace that is grafted
(beancount.plugins.AUTHOR_NAME.pluginN), or not
(AUTHOR_NAME.beancount.pluginN), into the beancount one.

From the point of view of plugin users, dealing with these difference is
gonna become tiresome at some point. So having some "Beancount 3rd party
plugin *distribution* guidelines", with emphasis on distribution, would
be nice.

> If you want examples of testing plugins, see here:
> https://bitbucket.org/blais/beancount/src/tip/src/python/beancount/plugins/

Here too, I did see that, which is helpeful if your plugin is eventually
going to be integrated into the Beancount main codebase. But if your
plugin is going to remain 3rd party, you need quite a bit of scaffolding
before being able to test your plugin. Having a set of guidelines about
how to do that for *external* plugins would be nice. Same thing for
linting. You've some pylint settings sprinkled around the Beancount
codebase, either in pylintrc or in individual files; for external
plugins is not clear which rules should apply when using pylint.

Maybe, joining this discussion with Dominik's proposal of CI plugin
testing, what we need is a repo with 3rd party plugin *template*, that
plugin author should start from. It can contain the desired module
structure, testing scaffolding, and Travis integration.

Hope this clarifies,
Cheers.

[Martin Blais]

On Sun, Dec 4, 2016 at 12:29 AM, Stefano Zacchiroli <za...@upsilon.cc> wrote:

On Sat, Dec 03, 2016 at 09:21:35AM -1000, Martin Blais wrote:
> On Sat, Dec 3, 2016 at 1:48 AM, Stefano Zacchiroli <za...@upsilon.cc> wrote:
> >
> > It would be useful at some point if you could document best
> > practices or other expectations on how 3rd party plugins should be
> > "bundled up". That would make it easy and consistent to use them for
> > Beancount users.
>
> I keep it simple: if it's in your PYTHONPATH, and it has a __plugins__
> attribute, it's all good. There's no convention beyond that.

Sorry, I wasn't clear about what I meant with "bundle" here. I'm good on
the Python API between Beancount and plugins; it's great, clear, and
couldn't be simpler.

But 3rd party plugins distributed as repos or tarballs can implement
that API in many different ways. You can have a top-level Python module
that you just git clone, or a deep module namespace that is grafted
(beancount.plugins.AUTHOR_NAME.pluginN), or not
(AUTHOR_NAME.beancount.pluginN), into the beancount one.

From the point of view of plugin users, dealing with these difference is
gonna become tiresome at some point. So having some "Beancount 3rd party
plugin *distribution* guidelines", with emphasis on distribution, would
be nice.

*Giggles* I wish my project was _that_ popular we have to establish a convention.

> If you want examples of testing plugins, see here:
> https://bitbucket.org/blais/beancount/src/tip/src/python/beancount/plugins/

Here too, I did see that, which is helpeful if your plugin is eventually
going to be integrated into the Beancount main codebase. But if your
plugin is going to remain 3rd party, you need quite a bit of scaffolding
before being able to test your plugin. 

You can import the helper code from Beancount directly. No scaffolding required: 

1. create a <myplugin>_test.py,

2. import stuff from beancount.utils.test_utils and

3. use nosetests to run it. 

That's it.

If you end up needing more test helpers, send me a patch, those should most likely want to live in the Beancount source itself, where they can be reused across plugins.

 

Having a set of guidelines about
how to do that for *external* plugins would be nice.  Same thing for
linting. You've some pylint settings sprinkled around the Beancount
codebase, either in pylintrc or in individual files; for external
plugins is not clear which rules should apply when using pylint.

What you're seeing in the source code might be selectively disabling some of the checks on specific pieces of code. Those are sprinkled here and there where I need to quiet down pylint for some exceptions. There are no pylint settings other than in the pylintrc file. 

Maybe, joining this discussion with Dominik's proposal of CI plugin
testing, what we need is a repo with 3rd party plugin *template*, that
plugin author should start from. It can contain the desired module
structure, testing scaffolding, and Travis integration.

In the libertarian spirit, the template I'd recommend myself is "no template." Contribute us a template if you'd like to see a particular format get popular. I'm happy to put a note in the contrib file to guide people to an example.

Hope this clarifies,
Cheers.

--
Stefano Zacchiroli . za...@upsilon.cc . upsilon.cc/zack . . o . . . o . o
Computer Science Professor . CTO Software Heritage . . . . . o . . . o o
Former Debian Project Leader . OSI Board Director  . . . o o o . . . o .
« the first rule of tautology club is the first rule of tautology club »

--
You received this message because you are subscribed to the Google Groups "Beancount" group.
To unsubscribe from this group and stop receiving emails from it, send an email to beancount+unsubscribe@googlegroups.com.
To post to this group, send email to bean...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/beancount/20161204102910.wiicnec56yud4xzi%40upsilon.cc.