Beancount Updates - 2016-10-30

[Martin Blais]

It has been a good 6 months since the last update.

Lots of important changes recently - in particular, moving to the newer and better booking algorithm, requiring Python 3.5 and I've deprecated a bunch of old deprecated stuff. It's possible you will experience user-visible changes.

Full details below.

2016-10-30

  - Merge the current version of the 'booking' branch and made the "FULL" booking

    algorithm the default one.

    * The "experiment_booking_algorithm" option has been renamed to

      "booking_algorithm". Its default value is now "FULL"; everyone will be

      using the new booking code. If for some reason you have a transitional

      need to revert to the older booking method use this option.

        option "booking_algorithm" "SIMPLE"

    * The new booking algorithm is _MUCH_ more powerful than the one old. I'll

      describe it soon in a new document, but briefly:

      - It supports many more interpolation capabilities; try omitting amounts

        or currencies which you think it ought be able to figure out

        automatically and you might find it does.

      - All positions carry a date. This means that if you have transactions

        that would merge because they weren't specified with a date, like this

        you may face a problem:

          2016-01-15 *

            Assets:Investments          70 UUG {10.00 USD}

            ...

          2016-02-16 *

            Assets:Investments          25 UUG {10.00 USD}

            ...

          2016-03-30 *

            Assets:Investments         -80 UUG {10.00 USD}

            ...

        Those used to resolve automatically because both augmenting lots would

        merge into a single one without a date. Now that all positions have an

        acquisition date, the -80 reducing posting is ambiguous, because it

        matches against these:

            Assets:Investments          70 UUG {10.00 USD, 2016-01-15}

            Assets:Investments          25 UUG {10.00 USD, 2016-02-16}

        Either you'll have to disambiguate, or let Beancount do so. One way to

        do this is to add more specific matching data, and split the posting.

        An even easier way to do this is to make the default booking method of

        this account to "FIFO" or "LIFO", which will automatically select the

        oldest (newest) postings without issuing a warning.

        If you find yourself in a bind, please email the mailing-list with a

        specific and as small as possible data input that reproduces the

        problem.

  - Removed parsing of the pipe (|) symbol as a separator for strings on

    transaction description lines. This was vestigial from the days long long

    ago when Beancount was attempting to be compatible with Ledger syntax; this

    is wholly unnecessary how.

    You will have to convert transactions like this:

      2015-08-25 * "Tunnel" | "Coffees"

        Expenses:Food:Coffee           15 CAD

        Assets:Cash

    Into transactions like this:

      2015-08-25 * "Tunnel" "Coffees"

        Expenses:Food:Coffee           15 CAD

        Assets:Cash

    This can be done using a sed script, or as a macro in your favorite text editor.

    If you need a temporary respite, you can use the following option, which

    will replace all the error messages by a single warning:

      option "allow_pipe_separator" "TRUE"

    Support for this option will be removed eventually.

  - Moved the {{...}}} total cost syntax from a legacy support with only "/"

    separator and a date, no label, to full support. Previously, only these syntaxes

    were supported for total cost:

      Assets:Account     100 MSFT {{2000.00 USD}}

      Assets:Account     100 MSFT {{2000.00 USD / 2016-10-30}}

    Changes:

    * The "/" separator support has been removed, you must now use a comma, like

      for the {...} full booking syntax.

    * The same syntax as for the regular {...} compound cost is supported;

      however, specifying a compound amount will raise an error, and the

      per-unit cost is interpreted as a total cost.

    For example, this:

      Assets:Account     100 MSFT {{2000.00 USD}}

    is interpreted as:

      Assets:Account     100 MSFT {0 # 2000.00 USD}

    which is equivalent to:

      Assets:Account     100 MSFT {20.00 USD}

    And the following will raise an error, it is not legal syntax:

      Assets:Account     100 MSFT {{0 # 2000.00 USD}}

    Finally, specifying a date and a label can be done, just like for {...}:

      Assets:Account     100 MSFT {{2000.00 USD, 2011-10-30, "ref32"}}

    The intention here is to deprecate the older limited syntax while keeping

    support for total cost input using {{...}}. Based on previous questions, I

    suspect that many users are still using {{...}} instead of {{# ...}}, so I

    won't remove it just yet.

  - Removed very old compatibility code in beancount.core.amount that lived

    since I cleaned up the definition of the D() constructor and moved it to

    beancount.core.number.

  - Removed built-in aliases for auto_accounts and implicit_prices modules, from

    beancount.ops to beancount.plugins.

  - Removed deprecated option "default_tolerance" (which had been renamed to

    "inferred_tolerance_default").

  - Removed deprecated option "tolerance".

  - Remove deprecated use of 'plugin' as an option. The directive ought to be

    the only way.

  - Removed deprecated option "experiment_explicit_tolerances".

  - Renamed "experiment_booking_algorithm" to "booking_algorithm", and changed

    the default value to "FULL". The old "SIMPLE" algorithm will be deprecated

    eventually.

  - Removed old code that would warn if the user set account_rounding to a full

    account name; it just needs be a subaccount.

  - Commit to Python 3.5. 3.5.0 released more than a year ago. You need to

    update or stick with the 2.0b13 release.

  - Remove old code to dispatch between an external Enum class and the built-in

    one that's included in 3.5 (commit to 3.5).

  - Added type annotations to beancount.core.data. This will grow more over

    time. (I also need to add rules to use those.)

2016-10-23

  - Modified parsing of transaction strings and links and tags. I did this to

    factor out the parsing of tags and links in order to add it to other

    transactions and minimize grammar code. There is user-visible impact to this

    change: all of a transaction's strings must appear before tags or links. You

    cannot place tags or links before or in-between strings anymore. This is a

    reasonable restriction and I doubt anybody did anyway.

  - Added support for tags and links to the Document directive. This works just

    as for Transactions, e.g.

      2016-10-23 document Assets:Checking "/statements/oct.pdf" #banking ^9fed846348c2

    The current tags on the tag stack are also included, like for transactions.

2016-10-10

  - Removed "experiment_explicit_tolerances" flag; made this supported by

    default permanently.

2016-10-08

  - Documented a subtle problem reported about balances against inventory of

    mixed at-cost and no-cost, added more context to the error and a unit test

    for balance checks with mixed at-cost and no-cost positions.

  - Fixed a bug in the full booking matching code whereby an existing position

    not held at cost would trip up the matching code. (Credits to Son Nguyen for

    finding and reporting.)

  - During ingest, when multiple extractors match a particular file, select the

    first matching account instead of raising an error.

  - Fixed accidentally colliding key-binding for invoking bean-price from Emacs,

    to C-c p.

2016-09-25

  - Fixed typo in grammar that was parsing a transaction flag as '#' into '*'.

  - Added "cost_date" and "cost_label" columns in the SQL shell.

2016-09-04

  - Implemented new "run" shell command that allows you to run a named query.

    For example, if an input file has a Query directive like this:

      2016-09-01 query "home" "select account, sum(position) where account ~ 'Home' group b$

    And you input this command:

      run home

    The entries will be closed at tue query date (2016-09-01) and the query

    automatically run. If you want to always include all the transactions, set a

    date far into the future.

    This works from the command-line and from the shell, equally. You can also

    run all the queries in an input file, like this:

      run *

    To view the list of available queries, just type the "run" command without

    an argument.

2016-08-21

  - Fixed a SQL shell bug with extracting UNITS() from a Position column.

  - Finally properly fixed the timezone of the Google Finance price fetcher.

    (It was done incorrectly and I found my unit tests failing whenever I would

    work on Beancount from another timezone.)

2016-08-17

  - Added beancount.core.amount.abs() function for absolute values of Amount.

  - Added rendering of links to beancount.plugins.split_expenses plugin, and

    made it quiet by default. Added --output-stdout option to make it do that

    again.

  - Created a new experimental script that attempts to match postings between

    two ledgers. A pair of (filename, account-regexp) is provided for each, and

    then the amounts and links are compared and paired up, and remaining

    unmatched postings are printed out. This is very useful to reconcile trip

    expenses between a personal ledger and the ledger for a trip or external

    project where some of the expenses are paid from the personal ledger as

    contributions to the trip or project.

  - Made the upload-csv-to-google-sheet script use the basename (without

    extension) of the given filenames as sheet names instead of "Sheet X"

    numbers. (Note: This entire thing be rewritten using the new Google Sheets

    API which won't require gdata and which will be more stable.)

  - Added a quick implementation of b.q.query_render.render_csv()

    and refactored b.q.query_render.render_text() to use common code. Both

    functions now have a new 'expand' option which will create multiple rows for

    cells which return a list of strings. The new default behavior is to

    concatenate the strings with commas.

    The shell (bean-query) has a new 'expand' shell variable to match. Also, the

    -f/--format option of it is now supported, and in interactive mode it sets

    the default value for 'expand'.

  - Since output to CSV is now implemented, I enabled the --output-csv option of

    b.p.split_expenses as well.

2016-08-14

  - Fixed bug in beancount.utils.file_type for Python3, whereby magic now returns

    bytes, not a string

2016-08-04

  - Made account name components allowed to have two characters (beyond the

    first component). For instance, Assets:Investments:F is now a valid account

    name.

2016-07-11

  - Added parsing a transaction date column to metadata in the CSV importer.

2016-07-01

  - bean-format now accepts input from stdin.

2016-06-30

  - Make it possible for a BALANCES statement to have a WHERE clause. BALANCES

    is now equivalent to

      SELECT account, sum(position) FROM ... WHERE ... GROUP BY account

    Inching one step closer to removing that WHERE clause eventually.

2016-06-18

  - Added support for columns AMOUNT_DEBIT and AMOUNT_CREDIT in the csv

    importer.

  - Merged ongoing progress from 'booking' branch; if you don't use any of the

    in-progress new booking methods, this should not have any effect.

    However, the 'booking_method has changed names and you may have to adjust

    its values if you've been tinkering with it:

    * 'booking_method' used to take on value "SIMPLE" or "FULL" and dispatch

      betwee the current (old) booking implementation and the new one which will

      support many other booking methods. This option is now named

      'experiment_booking_algorithm' (a new option) and takes on the same

      values. The default value is "SIMPLE", which keeps the behavior unchanged.

      Use "FULL" if you want to test out the new booking algorithm: the new

      algorithms should work except for the AVERAGE cost method. WARNING: This

      is still under flux and considered unreleased.

    " 'booking_method': This is a different option now, and takes the value of

      any one of the booking methods visible here:

      https://bitbucket.org/blais/beancount/src/7b9e00e6e19136f680befc1d4322f47cdc294cf6/sr$

2016-06-13

  - Ported beancount.docs.download_docs to the v3 drive API and the use of a

    service account instead of a full access to a user account. Service accounts

    are easier to configure and work with.

  - Ported beancount.docs.upload_options to the v3 drive API.

2016-06-05

  - Added a new column to the posting context: "other_accounts" is a set of

    account strings for the other postings of the transaction.

  - Added a new JOINSTR() generic function to the shell, to reduce the repeated

    column above to a single comma-separated string.

2016-05-28

  - Made various changes to the shell in order to make it possible to convert to

    the market value:

    * CONVERT() has been converted to convert to the market value, when applied

      to a position held at cost. However, it still requires a target currency

      and it may fail if you apply it to a position for which the cost currency

      differs from that of the target currency, i.e., it will not automatically

      chase down the transitive conversions toward the desired target currency.

    * A new function VALUE() has been added, which is similar to CONVERT() but

      which when applied to a position held at cost converts to the cost

      currency at the current price. It's similar to CONVERT() but you don't

      have to provide the target currency. And similarly to CONVERT(), you may

      provide a particular date, and if at that date there is no available

      conversion rate in the prices database, the conversion will fail

      analogously and spit out an error message.

    * PRICE() has been renamed to GETPRICE() to avoid some confusion. PRICE()

      does not convert, it merely gets the price from the price database.

      GETPRICE() is thus a more appropriate name.

2016-05-22

  - Created the beancount.plugins.mark_unverified plugin as a response to this

    thread: https://groups.google.com/d/msg/beancount/hQoOSJE7qNo/-E__v-XhEgAJ

    This plugin makes it possible to compute the balances as of the last Balance

    directive of each account by adding metadata to postings which appear after

    that account's last balance. You'd filter out unverified postings with an SQL

    query on meta-data like this:

      SELECT account, sum(position)

      WHERE NOT meta('unverified')

      GROUP BY 1

      ORDER BY 1

    Note that the resulting balanaces almost certainly don't sum up to zero,

    because of differing dates of the Balance directives. Also, note that

    accounts without a single Balance directive will not have their postings

    marked at all.

2016-05-08

  - In the output of bean-price, sort prices exclusively by currency, regardless

    of date. This is because on weekends you'll tend to get slightly varying

    dates, and manual/occasional price updates are usually put in "chunks" in a

    file section. It's easier to visually find the price by scanning by currency

    order.

2016-05-05

  - Added a shell function to filter a set of strings (for tags or links) with a

    regular expression and return the first one. This way you can extract some

    of the tags or links with a pattern as columns.

2016-05-01

  - Fixed issue #124, a leak in the grammar parser which would leak the entire

    set of entries on every instance of parse.

  - Fixed some warnings about unclosed files which occur only on default python

    branch.

2016-04-29

  - Don't just log an extraction error to logging.error() in the regression

    tests; let the exception through. This makes building and debugging

    importers much easier.

  - Created an experimental script to extract date/location information required

    for a naturalization application.

2016-04-24

  - Rewrote the section on "Core Data Structures" of the Design Doc, to reflect

    the newer internals of Beancount, those which have been merged from the

    ongoing 'booking' branch.

    See http://furius.ca/beancount/doc/design-doc.

2016-04-23

  - Added beancount.ingest.importers.fileonly, a simplistic importer which can

    be instantiated to just file away some downloads without extracting anything

    from them.

2016-04-18

  - In bean-doctor context, all the accounts should appear in the

    post-transaction balances, even if their balance is empty. This had been

    fixed for the pre-transaction balances but I stumbled upon a case of

    post-transaction balances not showing an empty account.

2016-04-17

  - Added support for the BEANCOUNT_LOAD_CACHE_FILENAME environment variable,

    which can be set in order to overridde the location of the pickle cache.

    This variable may contain the string "{filename}", and if it is present the

    basename of the beancount file will be spliced in. This can be useful if,

    for example, you are reading your Beancount input file from a read-only

    filesystem and still want to use the cache by specifying an alternative

    filename.

  - Fixed a bug when reading from a read-only filesystem, the loader cache would

    fail to be removed and raise an exception.

  - Fixed an important bug in beancount.ingest whereby instances of

    b.i.cache.FileMemo weren't being shared as widely as they could have been.

    FileMemo instances now have to be created via b.i.cache.get_file() and

    FileMemo is now a hidden class _FileMemo. See this discussion for more details:

    https://groups.google.com/d/msg/beancount/b-ZqYqag7AU/N-AaGSZOAwAJ

2016-04-10

  - Create a new 'beancount.plugins.fix_payees' plugin that allows the user to

    provide a set of rules to rewrite and clean up the names of the payees from

    the Beancount file.

    While it would be ideal to do this at import time, in practice there are a

    lot of downloaded files' "memo" fields which get set as the payee

    automatically. These "memos" aren't very clean and readable strings: they

    sometimes contain numbers, partial city names, and other junk. In order to

    produce clean reports where one might want to aggregate by payee, it's

    useful to rewrite the payees to make them nice and clean. You can use this

    plugin to do that.

  - Made the Custom directive accept ACCOUNT tokens. You can now do something

    like this:

      2014-06-20 custom "budget" Assets:Account2 "balance < 200.00 USD"

    Prior to this, the account name had to be represented as a string. (Users of

    the burgeoning Custom directive not like this very much.)

    Note that account names and regular strings are both output from the parser

    as 'str' objects, so in order to make this possible, I had to change the

    representation of the values to a pair of (value, dtype). Account names have

    a 'dtype' of beancount.core.account.TYPE. (Eventually I will probably want

    to make account strings represented with their own custom datatypes, but

    that will require a fair amount of work on the codebase; this substitute is

    fine for now, as there are no other places where such a disambiguation is

    necessary.)

2016-04-04

  - Added a test to make sure that building a source dist includes the header

    files and C implementation files.

  - Sort Document directives after Transactions when they occur on the same day.

    The reason for this is that statements typically include transactions on

    that final day, and we'd like for them to occur at the right place in the

    journals. See originating thread:

    https://groups.google.com/d/msg/beancount/3ItOYS9DJx4/VyTJB3ssBQAJ

2016-04-03

  - Fixed a bug in "bean-doctor context" which wouldn't report the full set of

    accounts when an account's before-balance would be empty.

  - Fixed a bug in "bean-doctor linked" whereby the balances wouldn't get

    printed out if there was no link. Print the balances before and after; in

    fact, maybe this command should be renamed to "balances" and have an option

    to support links.

2016-03-31

  - Added TODAY() function to extract today's date. Not sure how useful it'll

    be.

2016-03-27

  - Added a PRICE() function to extract entries from the price database. You can

    now query for "SELECT PRICE('USD', 'CAD', date)", for example.

  - Added a boolean renderer for the SQL shell.

2016-03-26

  - Added an optional date argument to convert, so you can provide the requested

    date in a SQL query, as in "CONVERT(sum(position), "CAD", 2015-06-01)".

  - Improved the rendering of the context, removed the weird comments it used to

    render with, render an account name if the inventory is empty.

  - Ignore .DS_Store files from the file finder in ingest.

2016-03-22

  - Support multiple accounts in same ofx file. See

    https://bitbucket.org/blais/beancount/pull-requests/10/. I applied this

    one-liner: issue a balance for each statement, use the maximum date for the

    file_date() method, and added unit tests for this.

2016-03-21

  - Fixed bug in beancount.ingest.file whereby two files being filed to the same

    destination filename would clobber each other; with this fix, instead, an

    error will be generated.

  - Create an example of an importer just filing a PDF file using the external

    tool PDFminer2. If the tool isn't installed, nothing should "break" (but the

    importer won't really do its job). The tests should be skipped

    automatically, and the importing will just ignore the PDF file to be filed.

[cgr...@gmail.com]

Thanks Martin. I look forward to trying the new booking algorithm.

Thanks for all the work you and the larger team does on Beancount. 

Long live text based accounting.

-Chris

On Sunday, October 30, 2016 at 9:31:57 PM UTC-4, Martin Blais wrote:

It has been a good 6 months since the last update.

Lots of important changes recently - in particular, moving to the newer and better booking algorithm, requiring Python 3.5 and I've deprecated a bunch of old

[snip] 

[yegle]

Hi Martin,

The # char in cost basis is a new thing to me and I don't think the syntax is mentioned in the language syntax doc https://docs.google.com/document/d/1wAMVrKIA2qtRGmoVDSUBJGmYZSygUaR0uOMW1GV3YE0/edit# Could you document it there? You used `Assets:Account     100 MSFT {0 # 2000.00 USD}` in the example. I don't think the {{...}} syntax is mentioned in the doc as well.

Also, I'm sad to know AVERAGE booking is not available yet :-(

--
You received this message because you are subscribed to the Google Groups "Beancount" group.
To unsubscribe from this group and stop receiving emails from it, send an email to beancount+unsubscribe@googlegroups.com.
To post to this group, send email to bean...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/beancount/CAK21%2BhNTCgSibc5xVYzCkPyknz%3DFmN3cTM4VEawEEmwxH2zE6g%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

--

Yuchen Ying
http://about.me/yegle

[Martin Blais]

On Tue, Nov 1, 2016 at 2:38 AM, yegle <cny...@gmail.com> wrote:

Hi Martin,

The # char in cost basis is a new thing to me and I don't think the syntax is mentioned in the language syntax doc https://docs.google.com/document/d/1wAMVrKIA2qtRGmoVDSUBJGmYZSygUaR0uOMW1GV3YE0/edit# Could you document it there? You used `Assets:Account     100 MSFT {0 # 2000.00 USD}` in the example. I don't think the {{...}} syntax is mentioned in the doc as well.

I'll have to update the docs; taking note, thanks. This requires time I don't have for a couple of weeks.

Essentially, there are two numbers, e.g.

   Assets:Investments     100 MSFT {42.00 # 9.95 USD}

The first (42.00) is the number of USD per share of MSFT. 

The second is the total number of USD for the posting.

Thus, this posting's per-unit cost is (100 * 42.00 + 9.95) / 100 = 42.0995 USD / MSFT

Also, I'm sad to know AVERAGE booking is not available yet :-(

Progress doesn't always move in the order one wants. This is progress in the same direction, however.

If you'd like to have a go at it, here's where that would happen:

https://bitbucket.org/blais/beancount/src/fdd1b562fd53db239da1b0338a2634d05e1fe48e/src/python/beancount/parser/booking_full.py?at=default&fileviewer=file-view-default#booking_full.py-691

It turns out it needs a lot of unit testing to complete this, but at this point all the scaffolding around it is there and merged.

 

To view this discussion on the web visit https://groups.google.com/d/msgid/beancount/CAFL5w3UCJTuT0PqzYQM-STwAfOPk5_LHFOOdDhtHDwzKMiEM4g%40mail.gmail.com.