LWN reviews beancount

[Martin Michlmayr]

LWN has reviewed beancount to see if it would fit their needs:

https://lwn.net/SubscriberLink/785553/39167e23980e7e4c/

--
Martin Michlmayr
https://www.cyrius.com/

[Zhuoyun Wei]

> For example, there is no way in the base format to note whether a transaction has been reconciled or not.

By "reconciled", I am assuming the LWN editor is refer to the difference between "*" and "!" transactions?

--
Zhuoyun Wei

> --
> You received this message because you are subscribed to the Google
> Groups "Beancount" group.
> To unsubscribe from this group and stop receiving emails from it, send
> an email to beancount+...@googlegroups.com.
> To post to this group, send email to bean...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/beancount/20190412054207.GF1719%40jirafa.cyrius.com.
> For more options, visit https://groups.google.com/d/optout.
>

[Martin Michlmayr]

* Zhuoyun Wei <wzy...@wzyboy.org> [2019-04-12 23:27]:

> > For example, there is no way in the base format to note whether a transaction has been reconciled or not.
> By "reconciled", I am assuming the LWN editor is refer to the difference between "*" and "!" transactions?

I'm not sure what his process is exactly. Using flags might be part
of it.

But it might also be things like: when you write a check, you create a
transaction for the check (normally you record the money as leaving
your bank account when you write the check and not when the person
deposits the check, but in reality the money is on your bank account
until it gets deposited by the other person). So you may have to
reconcile why your bank account balance is different to what your
books say by looking at a list of uncashed checks.

That kind of things. Basically, your books won't always show bank
account balances all the time.

[Zhuoyun Wei]

In the case of writing checks, my approach is to use two transactions:


2019-01-01 * "John Doe" "Settle up Splitwise" #check ^check123
Assets:Chase:Checking -100.00 USD
Assets:Chase:Checking:Payable

2019-02-01 * "CHECK 123" ^check123
document: "2019-02-01.check-123.pdf"
Assets:Chase:Checking:Payable -100.00 USD
Equity:ARAP:JohnDoe


The first trasnaction is entered manually by me, on the day of writing the check, in a separate file ("checks.bean"). The second transaction is imported by "bean-extract" from the CSV my bank provides, at the end of each month (along with the image of the check).

Sometimes it takes quite a while for your checks to clear, so using two transactions is essential. This way, you can easily determine how many checks and money are not cleared by examing the transactions in the ":Payable" account, and also avoid blank checks / NSF by examing the balance of the parant account (excluding the leaf account). The bonus touch is, the two transactions are linked ("^check123"), so in Fava you could click the link to show corresponding trasactions, no matter how many days they are apart.

I remember someone in the mailing list wrote a plugin called ZeroSumAccounts, which did similar things.


--
Zhuoyun Wei

[Zhuoyun Wei]

One thing that surprises me (I noticed this just now, when writing the last reply) is that Beancount "balance" directive computes the balance of the parant account along with all child accounts.

Say you have 1000.00 USD in the checking account, and you write a check of 100.00 USD to your friend. Now you have 900.00 USD in the parent account alone and 100.00 USD in the child / leaf ":Payable" account. And the assertion:

20XX-XX-XX balance Assets:Bank:Checking 1000.00 USD

passes. This very behavior makes it easy for you to compare the balance in Beancount with the balance in your bank. At the same time, you could know how many money you could spend (without overdraft) by manually subtract the amount in ":Payable" from the parent tree.

--
Zhuoyun Wei

[Aamer Abbas]

I agree, this behavior is somewhat surprising. I might be misisng something, but this seems to conflict with Martin's advice to view account names as "tags" (https://groups.google.com/d/msg/beancount/MqIk8nfgtx0/8YSzZpQTEAAJ).

In this case, the hierarchy does matter since the balance directive will sum up to the parent account. Similarly, Fava's UI seems to also indicate that hierarchy is important.

--
You received this message because you are subscribed to the Google Groups "Beancount" group.
To unsubscribe from this group and stop receiving emails from it, send an email to beancount+...@googlegroups.com.
To post to this group, send email to bean...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/beancount/360165f8-609c-4c84-a66b-77fe80458421%40www.fastmail.com.

[rse...@gmail.com]

Rather than booking outstanding checks as an asset under "Assets:Chase:Checking:Payable", you could track them under Liabilities. Here's how I handle checks:

2019-03-01 * "My Building" "March 2019 Rent" #rent

  Liabilities:AccountsPayable 1000.00 USD

  Assets:Checking:Chase

2019-03-10 * "My Building" "Check #12345" #rent

  Liabilities:AccountsPayable -1000.00 USD

  Expenses:Rent:MyAddress

This way, the money "leaves" your checking account as soon as you write the check, but it is still tracked as an outstanding liability. When the check is cached, it gets booked under the appropriate expense account.

To unsubscribe from this group and stop receiving emails from it, send an email to bean...@googlegroups.com.

[Zhuoyun Wei]

On Sun, Apr 14, 2019, at 02:11, Aamer Abbas wrote:

> In this case, the hierarchy does matter since the balance directive
> will sum up to the parent account. Similarly, Fava's UI seems to also
> indicate that hierarchy is important.

Fava does quite a nice job handling this:

- if the parent account has no balance, the UI shows a grey number (the sum of all child accounts);
- if the parent account has balance, the UI shows a black number (the balance of the parent account);
- if you fold the parent account tree, the UI always shows a grey number (the sum of all child accounts).

--
Zhuoyun Wei

[Zhuoyun Wei]

On Sun, Apr 14, 2019, at 06:09, rse...@gmail.com wrote:
> Rather than booking outstanding checks as an asset under
> "Assets:Chase:Checking:Payable", you could track them under
> Liabilities. Here's how I handle checks:
>
> 2019-03-01 * "My Building" "March 2019 Rent" #rent
> Liabilities:AccountsPayable 1000.00 USD
> Assets:Checking:Chase
>
> 2019-03-10 * "My Building" "Check #12345" #rent
> Liabilities:AccountsPayable -1000.00 USD
> Expenses:Rent:MyAddress
>
> This way, the money "leaves" your checking account as soon as you write
> the check, but it is still tracked as an outstanding liability. When
> the check is cached, it gets booked under the appropriate expense
> account.

I also considered booking the outstanding checkings in another account, like Liabilities or Equities, but the problem with this is that the balance of your checking account in Beancount is different than the balance in your bank, so that you cannot use "balance" directive to assert the balance.

--
Zhuoyun Wei

[Martin Blais]

Thank you for sharing, Martin.

This is a reasonably well-written and fair enough review, though with a bit of a snarky tone and perhaps some misconceptions (though I can't blame the author for not seeing through all the nooks and crannies in just a few days exploration). Here's a response that attempts to address the points made in the article.

First, I'm impressed that he managed to import his QuickBooks data in just a few hours. I didn't realize it had become that easy.

RECONCILIATION. 

The reviewer mentioned the lack of reconciliation as a "nearly fatal flaw." What he's referring to is not specifically the absence of a flagging mechanism (because we have one) but rather the lack of support for different dates in a single transaction. A proposal for solving this was sketched here:

https://docs.google.com/document/d/1x0qqWGRHi02ef-FtUW172SHkdJ8quOZD-Xli7r4Nl_k/

While in practice fudging one of the dates has worked fine for personal finances (and I've always advocated for doing that until that's resolved), I can see that for more serious bookkeeping needs - where one may desire to produce a precise journal of postings for an account, with all the dates matching exactly with those of the institution - it would be important to support it (and I'd also like to start using it like that too for myself). It is also a recurring topic on the list from people who've used commercial accounting software. The proposal mentioned above would work fine with transfer accounts (I actually don't know how "real" accounting software deals with this behind the scenes, transfer accounts too?) and has little in the way of blockers -- it wouldn't add complexity to the rest of the system (though it would split up a lot of transactions into two transactions). Basically adding support for inputting per-posting alternate dates and having a plugin split up those transactions to post to/from one (or more) transfer account(s). Also supporting the /merging/ of transactions with a single posting each would be useful too (just another way to achieve the same thing). 

The reviewer does mention "the reconciliation pass." This obviously comes from a history of working without assertions. To a large extent, assertions obviate the need for this, but he missed that you can use flags for this anyway (that's their main purpose, e.g. using '*' and '!' to make transactions are correct or incomplete) and suggests the use of metadata instead, which is a bit overkill in terms of syntax. To go through the set of unreconciliated transactions that's nothing to build, one can just use a SQL command, something like SELECT location WHERE date >= 2019-02-01 and flag ='!' and iterate using "next-error" in Emacs. Perhaps that should be documented better.  We could also build some features to ensure that all transactions before some specific date are flagged done and correct (trivially easy).

CHECKS & TRANSACTION NUMBERS.  

For those you can definitely use links or tags, or metadata and the reviewer mentions this possibility, but alludes that it would be work to "actually write the accounting system." I'd like to know what kind of processes he'd like to see supported. I'm not too familiar with what type of support is provided for checks, for example, all I can imagine is (a) checking that no two check numbers are repeated, and/or that all consecutive numbers from the smallest one are present, or (b) special reporting by order of check number. Both of these ideas would be very simple to implement as plugins, but I'm not sure whether they are canonical enough to provide as default? (I'd imagine if that were the case over the last 10 years someone would have made noises about it.) In any case, those would be easy to support.  Generally speaking we can't possibly cover all the cases - but if someone with more knowledge of e.g. QuickBooks would like to prepare a list of essential functionality, we can consider building plugins for those features in the repo.

IGNORED LINES.

I think that comment stems from outdated documentation.

Org-mode / Markdown start-line characters are now ignored explicitly, see here:

https://bitbucket.org/blais/beancount/src/fa1edde3bcd02a277fac193f460a39c9a1461161/beancount/parser/lexer.l#lines-173

So you can probably sleep at night. But if you're concerned with this I'd like to challenge you to come up with examples of silently ignored input and file a ticket, I would certainly fix this ASAP.

ACCOUNTING vs. BOOKKEEPING.

The reviewer mentions that many features one would expect from an accounting system are absent. That is correct -- the word "accounting" is used rather loosely here (and in all PTA systems) and it would be more fair to call it a "book-keeping" system. It's a parser and a good, simple schema that applies a simple constraint and generates a stream of transactions, + a mish-mash of various loose functionalities built on top. Generating invoices would require someone to build an item (inventory?) tracking subsystem and client information database. These are considered out of scope for Beancount, but I can easily imagine something that would integrate well with it. Besides, we have users all over the world, and building such a system, with the inevitable myriad features and variants it would involve, would be worthy of a full project on its own. Good design (e.g. UNIX) is that we build simple tools that do one thing and just one thing well yet allow composition with other tools and this is precisely the point here. Someone ought to build a free, OSS invoicing system and the integration with our book-keeping system would be easy. A similar comment applies for the reviewer's notice of the absence of customers or vendors. I'd have expected an LWN reviewer to see through that.

I think the reviewer's comments about "tracking invoices" and the absence of a dedicated concept for accounts payable / receivable refers to ensuring that invoices have been paid. Since invoices aren't present as concepts in the ledger, AP / AR aren't treated in any other way than other accounts. However, I've shown in the past that you can build your own custom tracking by providing a 90-line plugin that does that:

https://bitbucket.org/blais/beancount/src/default/beancount/plugins/tag_pending.py

Again, given it's so simple to add that in via a small piece of code I have to wonder whether it's worthwhile to add a first-class concept for that in Beancount itself. Maybe writing a doc about one would go about doing that is the best way to proceed?  Maybe invoicing / closing an invoice ought to be first-class concepts in the schema?

The author mentions the difficulty of handling duplicate transactions; that's probably from a thread on the mailing-list where we discussed *automatically detecting* such duplicates, which is more demanding than explicitly handling that via unique numbers. By implementing very simple constraints on links on transactions - treating a subset of link names as "unique transaction numbers" if you will - one would be able to achieve this with a 20 LOC plugin. Again, I can imagine many variants for this, so it's best left to a bit of custom code IMO for your own interpretation of metadata. Maybe we should have a built-in plugin provided that does that (e.g. recognize some metadata or "regexp on link" pattern and ensure each value occurs no more than once on the stream - easy).

Maybe what we need is a new, middle layer, that builds in a bunch of these concepts at once on top of the existing schema. Maybe plugins aren't formal enough.

REPORTING.

The author mentions that it is impossible to restrict a report to a given period of time; that is incorrect. He must have missed that part from my long intro: https://docs.google.com/document/d/100tGcA4blh6KSXPRGCZpUlyxaRUwFHEvnz_k9DyZFn4/  where I describe how I translate the process of truncation of the past and future as "open" and "close" operations on the stream of directives, and the associated features on the SQL syntax where you can trigger these from a query. One of the design aspects of Beancount is that we can always defer period restrictions to reports and keep reading the full stream of transactions over all time every time. What may have happened here is that the now more-or-less obsolete bean-web tool and bean-report tools do lack that capability and the reviewer must have stumbled upon that. In the next version, these will be removed and the SQL capability will be expanded (and forked to a separate project). Being able to report over any period of time is the norm.

About the slowness: this is true, it's a bit slow (the cache helps, but it's still slightly frustrating), and this will be handled in the next major version. (I think of the next version as essentially a rewrite in 3 parts: (1) rewrite just the core in C++ keeping the very same schema and same Python API but with protobufs instead of namedtuples as storage (cross language), (2) fork the SQL query tool to its own project, adding missing features and making it able to work on any data, not just Beancount, and (3) delete much of the custom reporting code, deferring users to either Fava or using the SQL tool (with extra whizbang) instead. Moreover, I'd like to move that next version to a development model more open to contributions, where I reduce the ruthlessly-unchanging and fiercely-defended-against-changes part to a small core written in C++ and the rest opens up more liberally than I have in the past, maybe even move to github, etc.)

The author mentions that no free accounting systems generate government-mandated forms; you will never see any free software doing that. There are simply too many variants of forms, too much data that isn't part of the transaction flow, too many countries, etc. and most importantly, it's just not a sexy job. That's why companies like Intuit have thousands of developers being paid to process the minute details some obscure tax rule in a particular country for a particular year and figure out how to map some specific constrainted subset of accounts in their ledger to produce some number in some form, etc.  Not Beancount nor any of the other PTA systems will ever do that. Nobody in their right mind would work on that on their free time on the weekend. I'd never expect that from any free system in the first place and I have to wonder why the reviewer did. What I think we might see instead, is a generic PDF overlay project, where one could combine a PDF file - from a government form - with an associated template with some input file to insert text where the template specifies it should (i..e template says "field 'name' is at <coordinates>", input file says "name = 'Joe'"). This could be easily integrated with something like Beancount by writing a script to pull the right numbers to generate the input file and then produce the PDF. That's a conceivable way such form generation could proceed in the OSS world (I started something like that a little while ago, but time is scarce).

I enjoyed the review, thanks for sharing it.

Cheers,

--
You received this message because you are subscribed to the Google Groups "Beancount" group.

To unsubscribe from this group and stop receiving emails from it, send an email to beancount+...@googlegroups.com.

To post to this group, send email to bean...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/beancount/20190412054207.GF1719%40jirafa.cyrius.com.

[Martin Blais]

Seems to me you're doing it backwards, I'd have recognized the expense first, then booked the asset change when the money actually leaves the checking account.

To unsubscribe from this group and stop receiving emails from it, send an email to beancount+...@googlegroups.com.

To post to this group, send email to bean...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/beancount/a06a8f13-3caf-442a-a2e3-8b52a6fbc999%40googlegroups.com.

[Chary Chary]

Martin,

great that you share your future plan.

 

(I think of the next version as essentially a rewrite in 3 parts: (1) rewrite just the core in C++ keeping the very same schema and same Python API but with protobufs instead of namedtuples as storage (cross language),

Will it still be possible to write plugins?

 

(2) fork the SQL query tool to its own project, adding missing features and making it able to work on any data, not just Beancount,

what kind other "any data" do you have in mind? I understood, that bean query was made specific to double entry accounting-like data.

 

[Martin Blais]

On Sun, Apr 14, 2019 at 3:56 PM Chary Chary <char...@gmail.com> wrote:

Martin,

great that you share your future plan.

 

(I think of the next version as essentially a rewrite in 3 parts: (1) rewrite just the core in C++ keeping the very same schema and same Python API but with protobufs instead of namedtuples as storage (cross language),

Will it still be possible to write plugins?

Yes, same API, but potentially also C++ support.

 

 

(2) fork the SQL query tool to its own project, adding missing features and making it able to work on any data, not just Beancount,

what kind other "any data" do you have in mind? I understood, that bean query was made specific to double entry accounting-like data.

Any other data. At the very least, anything that can be converted a flat list of rows and columns of basic types.

The only thing specific to double-entry accounting-like data in bean-query is the aggregation functions for Position and Inventory types, and their associated rendering routines.

The vision I have for this is to generalize it to any data (e.g. csv files, any protocol buffer type, other sources of data, e.g., metadata on images, html tables from web pages) and to allow the addition of custom types to this tool, which is what Beancount would use, + the addition of convenient features for reporting (built-in pivot clause, and aggregation of columns as a special "bottom" line).

This scope would be that of a data science tool, beyond the scope of Beancount. As long as it's customizable enough that wouldn't be a problem. One of the central aspects of this tool which has been underserved by other data science frameworks is the automatic inference of schemas; this should, ideally, just work. Ideally you should be able to run joins and filters on a good old CSV file.

 

[Stefano Zacchiroli]

On Sun, Apr 14, 2019 at 11:34:13AM -0400, Martin Blais wrote:
> RECONCILIATION.
[...]

> one can just use a SQL command, something like SELECT location WHERE
> date >= 2019-02-01 and flag ='!' and iterate using "next-error" in
> Emacs. Perhaps that should be documented better.

I totally missed this was a possibility and it is indeed really cool.

I randomly checked the BQL documentation (which is what I used to learn
about the query syntax at the time) and there is no mention of "SELECT
location" which, OTOH, is well documented in "help select" in the BQL
shell. Quoting from it for others who might have missed it to:

'location': The filename:lineno where the posting was parsed from or
created. If you select this column as the first column, because it
renders like errors, Emacs is able to pick those up and you can
navigate between an arbitrary list of transactions with next-error
and previous-error. Type: str.

It might indeed deserve some mention in the main BQL documentation, as a
relevant use case.

> CHECKS & TRANSACTION NUMBERS.
[...]

> of support is provided for checks, for example, all I can imagine is
> (a) checking that no two check numbers are repeated, and/or that all
> consecutive numbers from the smallest one are present, or (b) special

Just as a random feedback: some banks here in France give out
chequebooks where check numbers are randomized (within the chequebook).
So any strict check on sequentially should allow some sort of override.

Cheers
--
Stefano Zacchiroli . za...@upsilon.cc . upsilon.cc/zack . . o . . . o . o
Computer Science Professor . CTO Software Heritage . . . . . o . . . o o
Former Debian Project Leader & OSI Board Director . . . o o o . . . o .
« the first rule of tautology club is the first rule of tautology club »

[Jonathan Corbet]

On Sun, 14 Apr 2019 11:34:13 -0400
Martin Blais <bl...@furius.ca> wrote:

> This is a reasonably well-written and fair enough review, though with a bit
> of a snarky tone and perhaps some misconceptions (though I can't blame the
> author for not seeing through all the nooks and crannies in just a few days
> exploration). Here's a response that attempts to address the points made in
> the article.

And here's a quick response to just a couple of things...

> First, I'm impressed that he managed to import his QuickBooks data in just
> a few hours. I didn't realize it had become that easy.

The tricky part is extracting it from QB. Obviously, once you have the
data, formatting it properly for beancount is easy.

> The reviewer mentioned the lack of reconciliation as a "nearly fatal flaw."
> What he's referring to is not specifically the absence of a flagging
> mechanism (because we have one) but rather the lack of support for
> different dates in a single transaction.

Not really. What matters is not the two dates; what matters is knowing
what is outstanding so you can (1) calculate the "real" account balances
and (2) reconcile against a bank statement that doesn't show that balance.

> The reviewer does mention "the reconciliation pass." This obviously comes
> from a history of working without assertions. To a large extent, assertions
> obviate the need for this, but he missed that you can use flags for this
> anyway (that's their main purpose, e.g. using '*' and '!' to make
> transactions are correct or incomplete) and suggests the use of metadata
> instead, which is a bit overkill in terms of syntax.

I very much did *not* miss the use of flags. I also didn't miss that the
documentation describes them as how one deals with deduplication or for
marking transactions that "look incorrect", which are different problems.
I can certainly see how they could be used for this purpose, but that also
did not seem to be the intent of the feature.

Assertions would only be helpful here if they ignore transactions with
specific flag values. If that is the case, the documentation doesn't
reflect it (as far as I can see).

> CHECKS & TRANSACTION NUMBERS.
>
> For those you can definitely use links or tags, or metadata and the
> reviewer mentions this possibility, but alludes that it would be work to
> "actually write the accounting system." I'd like to know what kind of
> processes he'd like to see supported.

The biggest thing, of course, is tracking which checks have cleared. It's
important information to put on related documents, such as the summary we
provide to authors about the payments we have sent them. Otherwise they
are really just debits like any other. Automatic number generation for
entry and printing is nice, but a secondary feature.

> IGNORED LINES.
>
> I think that comment stems from outdated documentation.

Certainly I went from the documentation (which says explicitly that this
happens); didn't have the time to experiment with it.

> A similar comment
> applies for the reviewer's notice of the absence of customers or vendors.
> I'd have expected an LWN reviewer to see through that.

I'm not quite sure what I should "see through". I made it clear in the
review that I was looking for features that were beyond the scope of what
beancount was meant to provide. It was all about suitability for a
specific task, not a criticism of the system as a whole.

> The author mentions that no free accounting systems generate
> government-mandated forms; you will never see any free software doing that.

Almost, but not entirely true.

> There are simply too many variants of forms, too much data that isn't part
> of the transaction flow, too many countries, etc. and most importantly,
> it's just not a sexy job. That's why companies like Intuit have thousands
> of developers being paid to process the minute details some obscure tax
> rule in a particular country for a particular year and figure out how to
> map some specific constrainted subset of accounts in their ledger to
> produce some number in some form, etc. Not Beancount nor any of the other
> PTA systems will ever do that. Nobody in their right mind would work on
> that on their free time on the weekend. I'd never expect that from any free
> system in the first place and I have to wonder why the reviewer did.

Did I say I expected that? I simply said that it is not present.
Remember, we're talking about escape from proprietary systems, and that is
one of the challenges inherent in that task.

That said, one does occasionally see support for specific forms pop up in
some free systems. But it's rare, incomplete, and usually poorly
maintained.

> What I
> think we might see instead, is a generic PDF overlay project, where one
> could combine a PDF file - from a government form - with an associated
> template with some input file to insert text where the template specifies
> it should (i..e template says "field 'name' is at <coordinates>", input
> file says "name = 'Joe'"). This could be easily integrated with something
> like Beancount by writing a script to pull the right numbers to generate
> the input file and then produce the PDF. That's a conceivable way such form
> generation could proceed in the OSS world (I started something like that a
> little while ago, but time is scarce).

That does sound like a promising way forward, depending on just how badly
specific governments obfuscate their PDF forms...

Overall I get the sense that you saw the review as an attack on your work.
That was not the intent, and I tried to be clear about that in the
conclusion. Beancount is good stuff with clear applicability in a number
of places, and the fact that it doesn't solve the specific problem I had,
when you never set out to solve that problem, does not detract from that.

Thanks,

jon

[rse...@gmail.com]

I agree it is a bit backwards from an accounting perspective. But some of my payees wait up to several weeks to cash checks, and I prefer to have an accurate picture of my spendable balances. My mental model has the money leaving the account when I issue a check, so that whenever the payee ends up cashing it is immaterial to me.

[Martin Blais]

On Mon, Apr 15, 2019 at 12:39 PM Jonathan Corbet <cor...@lwn.net> wrote:

On Sun, 14 Apr 2019 11:34:13 -0400
Martin Blais <bl...@furius.ca> wrote:
> The reviewer mentioned the lack of reconciliation as a "nearly fatal flaw."
> What he's referring to is not specifically the absence of a flagging
> mechanism (because we have one) but rather the lack of support for
> different dates in a single transaction.

Not really.  What matters is not the two dates; what matters is knowing
what is outstanding so you can (1) calculate the "real" account balances
and (2) reconcile against a bank statement that doesn't show that balance.

I don't understand what you're saying. What's the "real" account balance?

Be specific.

The whole reconciliation thing... I haven't seen anybody discuss this cogently.

I think this stems from a historical method of working--please prove me wrong.

I can see possible example interpretations of what "reconciling" could mean to you:

1. "Leave some state on this transaction indicating that I've verified it manually and it's correct"

That's a flag.

2. "Indicate that until now all of the past for some account is frozen (and correct) after some point in time."

We don't support that. We allow changing the past, especially where it's incorrect. 

Your numerous assertions are the backbone of your correctness.

3. "I need to integrate pending information that's not materialized as a transaction yet."

You use queries for that, and you can sum values from different accounts to derive numbers from them, e.g. checking + checks-made-not-cleared-yet.

Maybe that should be formalized more (well there's the Query directive that allows you to define common queries you may want to run).

4. Something else?

Please illuminate me.

> The reviewer does mention "the reconciliation pass." This obviously comes
> from a history of working without assertions. To a large extent, assertions
> obviate the need for this, but he missed that you can use flags for this
> anyway (that's their main purpose, e.g. using '*' and '!' to make
> transactions are correct or incomplete) and suggests the use of metadata
> instead, which is a bit overkill in terms of syntax.

I very much did *not* miss the use of flags.  I also didn't miss that the
documentation describes them as how one deals with deduplication or for
marking transactions that "look incorrect", which are different problems.

How would you use flags to deal wiith deduplication?

They don't do that.

 

I can certainly see how they could be used for this purpose, but that also
did not seem to be the intent of the feature.

Assertions would only be helpful here if they ignore transactions with
specific flag values.  If that is the case, the documentation doesn't
reflect it (as far as I can see).

Assertions that ignore a subset of transactions are an interesting concept.

How would they be useful to you?

I think generally I'd like to have the ability to embed any conditional - as SQL expressions - that are always checked like assertions.

Basically, assertions could become programmable, that's a really powerful idea.

You could do temporal checks like that, which would lead to budgeting-like features ("assert that no more than X is spent between time A and B").

It's not there yet.

> CHECKS & TRANSACTION NUMBERS.
>
> For those you can definitely use links or tags, or metadata and the
> reviewer mentions this possibility, but alludes that it would be work to
> "actually write the accounting system." I'd like to know what kind of
> processes he'd like to see supported.

The biggest thing, of course, is tracking which checks have cleared. 

Once the check hits the account, it's cleared. 

You recognize the payment as a transaction, post it to some other pending account, and when the check is cleared you'll see a transaction from your checking account.

I don't understand what you need.

Tell us how you do it in QuickBooks.

Be specific. I'm ignorant.

 

It's
important information to put on related documents, such as the summary we
provide to authors about the payments we have sent them.  Otherwise they
are really just debits like any other.  Automatic number generation for
entry and printing is nice, but a secondary feature.

> IGNORED LINES.
>
> I think that comment stems from outdated documentation.

Certainly I went from the documentation (which says explicitly that this
happens); didn't have the time to experiment with it.

> A similar comment
> applies for the reviewer's notice of the absence of customers or vendors.
> I'd have expected an LWN reviewer to see through that.

I'm not quite sure what I should "see through".  I made it clear in the
review that I was looking for features that were beyond the scope of what
beancount was meant to provide.  It was all about suitability for a
specific task, not a criticism of the system as a whole.

What I mean is that coming from a Linux background, you should be a position to appreciate that most of the tools surviving the test of time are simple and designed to do just one thing well. All those big integrated "applications"  (e.g. stuff usually built with desktop UI toolkits on Gnome, KDE, etc.) come and go, most often fall into disrepair after a few years, and almost always fall way short of their commercial equivalents. You still run "ls" but that windowed file manager is probably the 12th iteration you've seen (and ignore every time it pops up).

What the PTA tools are attempting to achieve is different: design the simplest systems that will get the most amount of mileage to run bookkeeping out of text files and get creative about how to combine them with scripts to solve custom problems. We don't even invert the signs on Income accounts!  It's not a surprise they don't have invoicing, printing cheques, tax / forms support, vendor lists, and nor built-in integration with your bank, etc. This is more like "vi" less like MSDEV. Discussing those things as if they were flaws or even shortcomings is missing the point a bit.

You could do it on the basis of coordinates on top of the PDF.

Xournal could be modified to include the concept of "boxes" in its output data format.

One could then very rapidly produce templates matching each form, manually giving field names to the boxes.

Those could be shared as text files between users in a repository.

The most common forms would probably all be there already (e.g. W4).

This is a simple project idea with a lot of potential impact.

 

Overall I get the sense that you saw the review as an attack on your work.

I don't know why you say that, I just responded to your review, but in any case I'm completely unfazed.

The main thing I care about is whether I might have missed some important idea in the design or schema (or proposals).

It provided yet another reminder that the settlement dates proposal really matters and needs to be done at some point, and that having more built-in plugins that do things by default might be more appreciated by users that I usually think it is.

[Martin Michlmayr]

* rse...@gmail.com <rse...@gmail.com> [2019-04-15 18:11]:

> I agree it is a bit backwards from an accounting perspective. But
> some of my payees wait up to several weeks to cash checks, and I
> prefer to have an accurate picture of my spendable balances. My
> mental model has the money leaving the account when I issue a check,
> so that whenever the payee ends up cashing it is immaterial to me.

This is exactly how companies handle it in the real world (at least in
the UK).

That's why they do regular bank reconciliations where one thing you do
is compare the actual bank balance with your books and take uncashed
checks into account to reconcile.

[Jonathan Corbet]

On Tue, 16 Apr 2019 00:22:25 -0400
Martin Blais <bl...@furius.ca> wrote:

> > Not really. What matters is not the two dates; what matters is knowing
> > what is outstanding so you can (1) calculate the "real" account balances
> > and (2) reconcile against a bank statement that doesn't show that balance.
> >

> I don't understand what you're saying. What's the "real" account balance?
> Be specific. The whole reconciliation thing... I haven't seen anybody
> discuss this cogently. I think this stems from a historical method of
> working--please prove me wrong.

OK, for purposes of illustration, imagine that we send a check for, say,
$1000 to an author on December 20; they cash it on January 4. The
statement from the bank at the end of December will not include that check,
but we know that we have $1000 less than the bank thinks; that money is
spent. The lower balance is what I was referring to as the "real" balance -
the amount of money we actually have.

This situation arises in other settings as well. We've been using
TransferWise for international payments (a great resource, BTW), and there
is often a delay between the initiation of a transfer and when it hits the
bank account.

If we want to verify that we are in agreement with the bank on how much
money we have at the end of the year, we need to calculate a balance
without the outstanding items. But we need those items recorded; we will,
for example, include that December 20 check as an expense in our tax filing
for that year. It's also important for cash-flow management.

"Reconciliation" is just the process of (1) being sure that you recognize
and agree with every item in the statement from the bank, and (2) checking
that the balances match after taking the outstanding items into account.
Once the checks pass, the items reported by the bank are marked as being
reconciled (or "cleared").

> How would you use flags to deal wiith deduplication?
> They don't do that.

From the docs:

> Finally, when I know I import just one side of these, I select the other
> account manually and I mark the posting I know will be imported later
> with a flag, which tells me I haven’t de-duped this transaction yet

Perhaps I was confused by the inconsistent use of "flag" here, apologies.
In general, the discussion of flags as a way of marking transactions that
"might be incorrect" suggests this kind of use.

> > I can certainly see how they could be used for this purpose, but that also
> > did not seem to be the intent of the feature.
> >
> > Assertions would only be helpful here if they ignore transactions with
> > specific flag values. If that is the case, the documentation doesn't
> > reflect it (as far as I can see).
>
> Assertions that ignore a subset of transactions are an interesting concept.
> How would they be useful to you?

Hopefully the above discussion makes that clear?

In truth, though, it wouldn't be as useful as I had thought. You could
certainly check the reconciliation of a bank statement that way, but when
the items clear later on, the assertion will break.

> > CHECKS & TRANSACTION NUMBERS.
> > >
> > > For those you can definitely use links or tags, or metadata and the
> > > reviewer mentions this possibility, but alludes that it would be work to
> > > "actually write the accounting system." I'd like to know what kind of
> > > processes he'd like to see supported.
> >
> > The biggest thing, of course, is tracking which checks have cleared.
>
> Once the check hits the account, it's cleared.
> You recognize the payment as a transaction, post it to some other pending
> account, and when the check is cleared you'll see a transaction from your
> checking account.
> I don't understand what you need.
> Tell us how you do it in QuickBooks.
> Be specific. I'm ignorant.

The two-transaction approach could work, certainly, but it complicates
things. A system like QB just keeps a "reconciled" flag on each
transaction to track state. You don't track the date that the item actually
cleared, that information is not normally useful for anything.

> What I mean is that coming from a Linux background, you should be a
> position to appreciate that most of the tools surviving the test of time
> are simple and designed to do just one thing well.

Remember that I contribute to a project that's 25 million lines of code and
counting...:)

> All those big integrated
> "applications" (e.g. stuff usually built with desktop UI toolkits on
> Gnome, KDE, etc.) come and go, most often fall into disrepair after a few
> years, and almost always fall way short of their commercial equivalents.
> You still run "ls" but that windowed file manager is probably the 12th
> iteration you've seen (and ignore every time it pops up).
> What the PTA tools are attempting to achieve is different: design the
> simplest systems that will get the most amount of mileage to run
> bookkeeping out of text files and get creative about how to combine them
> with scripts to solve custom problems. We don't even invert the signs on
> Income accounts! It's not a surprise they don't have invoicing, printing
> cheques, tax / forms support, vendor lists, and nor built-in integration
> with your bank, etc. This is more like "vi" less like MSDEV. Discussing
> those things as if they were flaws or even shortcomings is missing the
> point a bit.

No, it's really not. I am evaluating a tool for suitability to a specific
task. As I said in the article, the fact that it's not suitable for the
task I have in mind is neither a flaw nor a shortcoming; you didn't set out
to solve my problem! But somebody who is going through a similar
evaluation is going to want to know those things.

> > Overall I get the sense that you saw the review as an attack on your work.
>
> I don't know why you say that

See above "flaws or even shortcomings" language; that's how I came to that
conclusion.

Anyway, I'll repeat that beancount is good stuff, even if it doesn't solve
every problem that somebody might throw at it.

Thanks,

jon

[Stefano Zacchiroli]

Hi Jonathan, nice to read you here!

On Tue, Apr 16, 2019 at 12:03:39PM -0600, Jonathan Corbet wrote:
> OK, for purposes of illustration, imagine that we send a check for,
> say, $1000 to an author on December 20; they cash it on January 4.
> The statement from the bank at the end of December will not include
> that check, but we know that we have $1000 less than the bank thinks;
> that money is spent. The lower balance is what I was referring to as
> the "real" balance - the amount of money we actually have.

The traditional way of tracking this in Beancount/Ledger (and, in fact,
in double-entry accounting with accrual basis) would be:

2018-12-20 * "author payment" "Mr Foo Bar"
; cheque: "12345678"
Expenses:Writing 1000.00 USD
Liabilities:Payables -1000.00 USD

2019-01-04 * "cashing in author cheque" "Mr Foo Bar"
; cheque: "12345678"
Liabilities:Payables 1000.00 USD
Assets:Checking -1000.00 USD

Unless I'm misunderstanding your needs, this allows to verify at any
point in time both (1) the amount of money you are supposed to have in
the bank (by querying Assets:Checking) and (2) what you called the
"real" balance, by querying Income/Expenses over the year of
interest. ... or any variations of it you might need: a common one is
your Net worth, i.e., Assets - Liabilities.

I've never used a GUI accounting tool, nor I'm an actual accounting
expert. But what I'm guessing from the conversation here is that the
accounting systems you're used to actually do the above, but "hide" the
two transactions behind a single transaction with a "cleared" flag,
whose state changes are timestamped and monitored by the system.


This is not to say that Beancount is or isn't suitable for specific use
cases (yours or others). I see value in both "low level" accounting
tools like Beancount (and I've enjoyed learning more about accounting
theory thanks to that) and in higher-level ones that focus more on other
aspects of accounting. Take your pick :)

[Aamer Abbas]

I don't want to speak on his behalf, but I think his concern is a bit different. In your example, I think he wants his checking balance to reflect that he has $1000 less than the actual amount. Because at any moment, that check could be cashed and he needs to ensure he is setting it aside so that money is not spent (and also I guess he wants to recognize it that way for tax purposes).

Your example does certainly make that possible, but it requires you to do two queries and then subtract Liabilities:Payable from Assets:Checking. It's a bit of a hoop to jump through. I'm not an accountant, but I think from an accrual accounting point of view, he also would want to recognize that money as being gone from his bank account the moment the check is sent out.

Again, don't want to speak on his behalf, but that's how I interpret it.

--
You received this message because you are subscribed to the Google Groups "Beancount" group.
To unsubscribe from this group and stop receiving emails from it, send an email to beancount+...@googlegroups.com.
To post to this group, send email to bean...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/beancount/20190417125602.pdcdofpgkdqfsdhg%40upsilon.cc.

[Stefano Zacchiroli]

On Wed, Apr 17, 2019 at 04:29:03PM +0300, Aamer Abbas wrote:
> I don't want to speak on his behalf, but I think his concern is a bit
> different. In your example, I think he wants his checking balance to
> reflect that he has $1000 less than the actual amount. Because at any
> moment, that check could be cashed and he needs to ensure he is setting it
> aside so that money is not spent (and also I guess he wants to recognize it
> that way for tax purposes).

Right. I didn't mention it explicitly in my previous post, but I do face
that very same problem for my personal finances: I've a bank account and
I want to make sure that the amount of known outstanding payments that
might hit it cannot make it go red. (In my cases those payments include
not a lot of cheques, but a potentially substantial monthly credit card
bill.)

> Your example does certainly make that possible, but it requires you to do
> two queries and then subtract Liabilities:Payable from Assets:Checking.

... and that is precisely what I do. I have Beancount canned queries
that takes into account the relevant accounts (unintended pun :-)) and
show me the total. If it's below 0, I should better move money from
savings accounts back to that checking account.

(Arguably, those queries could be simplified if we had tags/metadata on
accounts and the ability to filter accounts based on those, rather than
only string matching. But that is a separate topic, which IIRC we have
discussed in the past on this list.)

Cheers
--
Stefano Zacchiroli . za...@upsilon.cc . upsilon.cc/zack . . o . . . o . o
Computer Science Professor . CTO Software Heritage . . . . . o . . . o o

Former Debian Project Leader . OSI Board Director . . . o o o . . . o .

[Martin Michlmayr]

* Stefano Zacchiroli <za...@upsilon.cc> [2019-04-17 14:56]:

> The traditional way of tracking this in Beancount/Ledger (and, in fact,
> in double-entry accounting with accrual basis) would be:
>
> 2018-12-20 * "author payment" "Mr Foo Bar"

Note that you mixed up the order of payee and narration in the example.

> ; cheque: "12345678"
> Expenses:Writing 1000.00 USD
> Liabilities:Payables -1000.00 USD

BTW, technically this may not even be the whole storyfor the accrual
basis because the liability probably arises earlier. There would in
fact be 3 entries:

1) author submits article/invoice -> liability arises
2) LWN posts checks a week later
3) Author deposits check a few weeks later

> 2019-01-04 * "cashing in author cheque" "Mr Foo Bar"
> ; cheque: "12345678"
> Liabilities:Payables 1000.00 USD
> Assets:Checking -1000.00 USD
>
> Unless I'm misunderstanding your needs

I think what you're missing is that checks are handled in a special
way. Checks are treated as cash and for accounting purposes the
money leaves your bank account the moment you write your check (not
when the check is actually deposited).

This is why a bank reconciliation is needed: your books will say one
bank balance but your actual bank balance is different. That's why
you add up all outstanding checks during the bank reconciliation.

And this is also why the "deduplication" is needed: because at some
point you import the bank transactions, and voila, the check has been
deposited and now your bank accounts goes down by $1000 twice - once
for when you recorded the check manually, and once for the bank
transaction.

See e.g. https://www.bizfilings.com/toolkit/research-topics/marketing/advertising-pr/accounting-for-cash-transactions
for some background on this.

I tried to find out if checks are handled in different ways in cash vs
accruals accounting. While I couldn't find as much information as I
had hoped, I think it's the same. (But of course the time of the
expense can be different: in accruals, you accrue the expense when the
author submits the invoice, as mentioned above; on the cash basis,
it's at the time you send the check.)

So the question is how to best model this in beancount. I see two
options:

1) when you send the check, you decrease the bank account. You then
tell your bank importer script to ignore all entries where checks are
cashed (it might be something like "CHECK POSTED"). This solves the
deduplication problem, but imho doesn't give a satisfactory answer for
the reconciliation step since it's difficult to get a list of all
outstanding checks.

2) You create an account to track checks. This isn't how you'd report
it to your accountant, but to use internally, so e.g. (assuming
accruals basis):

2018-12-10 * "Foo Bar" "submits article"

Expenses:Writing 1000.00 USD
Liabilities:Payables -1000.00 USD

2018-12-20 * "Foo Bar" "Wrote check for Foo Bar"

; cheque: "12345678"
Liabilities:Payables 1000.00 USD

Liabilities:Checks -1000.00 USD

(If you're on the cash basis, you'd ignore the first transaction and
use 'Expenses:Writing' in the second transaction instead of payables.)

2019-01-04 * "Foo Bar" "cashing in author cheque"
; cheque: "12345678"
Liabilities:Checks 1000.00 USD
Assets:Checking -1000.00 USD

(When you important the bank transactions, you'll have to go through
all outstanding checks and assign the metadata for cashsed checks
manually, or you might be able to write a progream to match on author
name or something clever.)

So now you can easily track which checks are still outstanding.
Just look at Liabilities:Checks and do a GROUP BY on the 'cheque'
metadata. Those that have been cashed will be 0, the rest will
show up. It also solves the deduplication problem.

But it doesn't give the view that accountants would expect and
it doesn't show what the account balance is if you take the uncashed
checks into account (although, as Zack pointed out, you could simply
do a balance on Assets:Checking *and* Liabilities:Checks). For that
I'd write a beancount plugin which would make 2 changes to my
transactions:

1) For transactions involving Liabilities:Payables and
Liabilities:Checks, it would change Liabilities:Checks to
Assets:Checking.
2) For transactions involving Liabilities:Checks and Assets:Checking
it would drop the whole transaction (deduplication).

Then if I want to do a bank reconciliation and see what checks are
outstanding, I would turn the plugin off. Otherwise it's on.

I'm not really satisfied by this answer, but that's the best approach
I can think of. Maybe someone else has a more elegant solution.

[Martin Michlmayr]

* Martin Michlmayr <t...@cyrius.com> [2019-04-17 23:01]:

> 1) when you send the check, you decrease the bank account. You then
> tell your bank importer script to ignore all entries where checks are
> cashed (it might be something like "CHECK POSTED"). This solves the
> deduplication problem, but imho doesn't give a satisfactory answer for
> the reconciliation step since it's difficult to get a list of all
> outstanding checks.

Ok, you could use flags for that.

When you write a check, you add a manual entry like this:

2018-12-20 * "Foo Bar" "Wrote check for Foo Bar"

check: 12345678
Liabilities:Payables 1000.00 USD
! Assets:Checking -1000.00 USD

And next month you import the bank data. You can now use flags to
find all outstanding checks:
select ANY_META('check'), position WHERE account = 'Assets:Checking' AND flag = '!'
Note that this doesn't work at the moment due to this bug:
https://bitbucket.org/blais/beancount/issues/353/posting-flag-ignored-how-to-query-posting

And when you import the bank data, you change the flag from ! to *
when you see it on your bank statement. (Remember the deduplication
is already done by the importer ignoring the entry).

And that's it. That seems like the easiest workflow to me.

BTW, I didn't notice but Zack's and mine example used ledger-syntax
for metadata (; foo: bar); you have to drop the semicolon for
beancount.

[Martin Blais]

Hi Martin,

This is as good an explanation as I've seen. Thanks for chipping in.

To me this is yet another one of those odd things people familiar with accounting might do and don't question the basis for. It's also very strange to me someone would call their checking account balance "real" when it doesn't actually match what the bank would report. I still see this just as a payable account. I do buy your explanation that this does hint at some sort of special handling for checks from something like QB, that would make sense indeed.

Here's a third way to handle this in Beancount: create two subaccounts, e.g.

  Assets:Bank:Checking:Actual

  Assets:Bank:Checking:Pending

Post the check to pending as you would a payable, when it clears, book that in Actual against Pending.

In other words, treat the Pending account as a payable account dedicated to the checking account.

Actual just tracks the ingested transactions and should reflect the bank statement.

Now, if you need the reconciled account (what personally I'd call the "real" one), report on Actual.

If  OTOH you need your combined balance (including pending checks, what others might - perhaps foolishly - call "real"), report on the Checking parent account, which will include transactions from the two.

--
You received this message because you are subscribed to the Google Groups "Beancount" group.
To unsubscribe from this group and stop receiving emails from it, send an email to beancount+...@googlegroups.com.
To post to this group, send email to bean...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/beancount/20190417160131.GK1709%40jirafa.cyrius.com.