Beancount Updates - 2015-06-29

[Martin Blais]

Here's a buffet of updates from the last month of development on Beancount.

(From the CHANGES file; only includes changes on default branch.)

2015-06-27

  - Complete the design doc and made it public.

2015-06-26

  - Created a generic script that can be used to automate the upload of

    arbitrary CSV files to a Google Spreadsheet. This can be used along with the

    holdings reports in order to create a spreadsheet tracking your assets.

    Unfortunately, this script has to be written in Python 2.x, because of

    the unavailability of the gdata-python-client library which is not available

    at this time for Python 3.x. The script is self-contained. If you want

    to use this, you must install the following libraries in your Python 2.x

    install:

    * gdata-python-client

    * oauth2client

    * httplib2

2015-06-25

  - Posting objects now have (filename, lineno) in their metadata. This could

    potentially be helpful for people who want to insert text programmatically

    in their input file without having to write their own parser.

    See: https://groups.google.com/d/msg/beancount/zNCFk-YGdr8/8P2_vGGH9xoJ

2015-06-15

  - All the code to bake a static HTML archive of a bean-web interface has been

    rewritten.

    * There used to be a number of bugs caused by the wget tool. These bugs have

      been fixed.

    * The new code does not depend on wget -- Beancount does not require wget

      anymore. Explicit code to do web scraping has been implemented. On the

      other hand, a new dependency is introduced: lxml. This dependency was

      already required for development and install in the same way as all the

      other dependencies: using pip3. Should be a breeze to install on all

      platforms.

    * A verification layer has been tacked on after the web scraping step to

      check that there are no dangling links.

    * Intentionally skipped links are now converted to plain text, such that

      there are no dangling links for pages we aren't saving (these just used

      to be ignored).

    I repeat: Beancount does not depend on wget anymore.

    Please install lxml with: "pip3 install lxml"

  - The 'exportpf' report which is used to produce OFX that can be imported in

    Google Finance now issues a warning if you did not provide either 'ticker'

    or 'export' metadata for holdings that get exported. The reason for the new

    adminition is that it trips up the OFX import and when Google Finance sees

    a ticker it does not understand, the import fails. At least with the warning

    we're telling the user they should be providing tickers or export directives

    for all their commodities if they export the portfolio.

2015-06-10

  - Removed dependency on curses. If you don't have curses, everything should

    work fine now, errors are being caught. Curses was only used to provide good

    default values for rendering tables in terminals on e.g. bean-report

    journal and to decide whether to use a pager in multi-page bean-query.

    The default values are fixed if curses isn't available, e.g. 80 chars wide,

    but as previously, you can override them manually. I also removed the

    dependency from checkdeps and from the install document.

2015-06-09

  - Rewrote the Installation Instructions document completely to make it

    simpler. Some users had found it daunting.

    http://furius.ca/beancount/doc/install

  - Tested Windows installation using Cygwin and added it to the installation

    instructions.

    http://furius.ca/beancount/doc/install

2015-06-07

  - Moved the following representation related symbols to their own module:

      beancount.core.amount.D          ->  beancount.core.number.D

      beancount.core.amount.Decimal    ->  beancount.core.number.Decimal

      beancount.core.amount.ZERO       ->  beancount.core.number.ZERO

      beancount.core.amount.ONE        ->  beancount.core.number.ONE

      beancount.core.amount.HALF       ->  beancount.core.number.HALF

      beancount.core.amount.round_to   ->  beancount.core.number.round_to

    I also moved the tests. All the code specifically related to numbers is

    now in beancount.core.number. I also removed the relative imports, as

    Ledgerhub is importing from the Beancount source code now.

    This is not a user-visible change. However, if you wrote scripts against

    the codebase, you will need to rename your imports. In the meantime I added

    stubs which issue a warning if you import from the obsoleted names.

  - Moved documentation scripts from experiments to the main Beancount codebase.

    There is a new package for it: beancount.docs. These codes are used to maintain

    the Beancount documentation and there is common code for authentication in there,

    code that I plan to use for automating the exporting ot portfolios.

    This introduces some optional dependencies:

    * google-api-python-client

    * oauth2client

    * httplib2

    These are optional, and at the moment none of the actual Beancount tools

    make any use of them. I moved this into the main Beancount codes because

    the common authentication code will be used to export portfolios directly

    to a Google spreadsheet in Google Drive, which will be super convenient

    to build custom portfolio tracking views.

2015-06-06

  - Added an 'inferred_tolerance_multiplier' option to allow the user to adjust

    the magnitude of the inferred tolerance.

    When the tolerance values aren't specified explicitly via the

    'default_tolerance' option, the tolerance is inferred from the numbers in

    the input file. For example, if a transaction has posting with a value

    like '32.424 CAD', the tolerance for CAD will be inferred to be 0.001

    times some multiplier. This is the muliplier value.

    We normally assume that the institution we're reproducing this posting

    from applies rounding, and so the default value for the multiplier is

    0.5, that is, half of the smallest digit encountered.

    You can now customize this multiplier by changing this option, typically

    expanding it to account for amounts slightly beyond the usual tolerance,

    for example, if you deal with institutions with bad of unexpected rounding

    behaviour.

  - Promoted the 'experiment_infer_tolerance_from_cost' option to

    'infer_tolerance_from_cost': not an experiment anymore. See thread on

    mailing-list w/ mharris. This feature works well and actually solves real

    problems for users.

    A corresponding section has been added in the documentation at

    http://furius.ca/beancount/doc/tolerances

2015-06-04

  - When using the 'experiment_infer_tolerance_from_cost' option, the inferred

    tolerance is now expanded to include the SUM of all maximum rounding errors

    from postings held at cost, not just the maximum. Users with multiple

    rounding events in a single transactions could occur rounding that many

    times. We assume no more rounding events than the number of postings held at

    cost.

  - I wrote a new document to explain how to deal with restricted stock units

    vesting events:

    http://furius.ca/beancount/doc/vesting

2015-06-03

  - bean-doctor context now outputs weight and residual numbers at their full

    precision instead of their rounded one.

2015-06-02

  - Created a new document to keep track of user contributions:

    http://furius.ca/beancount/doc/contrib

2015-05-30

  - Converted the Syntax Cheat Sheet to a Google Doc and removed it here.

  - Added a check to bake the source code's checksums into the C extension

    module and upon loading beancount.parser.parser check that the checksums

    match. This automatically issues a warning on all programs that use the

    parser if the compiled extension module is out-of-date and the user needs to

    rebuild. This should explain the occasional unexpected situation where the

    extension module doesn't match the Python code, for those running from

    source or during development and in switching branches.

  - Added an experimental boolean option: 'experiment_infer_tolerance_from_cost'

    that expands the inferred tolerance to include values inferred from the cost

    and price amounts on those cost and price currencies.

    Here is the doc for it:

      Enable an EXPERIMENTAL feature that expands the maximum tolerance inferred

      on transactions to include values on cost currencies inferred by postings

      held at-cost or converted at price. Those postings can imply a tolerance

      value by multiplying the smallest digit of the unit by the cost or price value

      and taking half of that value.

      For example, if a posting has an amount of "2.345 RGAGX {45.00 USD}"

      attached to it, it implies a tolerance of 0.001 x 45.00 / 2 = 0.045 USD

      and this is added to the mix to enlarge the tolerance allowed for units of

      USD on that transaction. All the normally inferred tolerances (see

      http://furius.ca/beancount/doc/tolerances) are still taken into account.

      Enabling this flag only makes the tolerances potentially wider.

      WARNING: This feature may go away in the future. It is an exploration to see

      if it is truly useful.

  - Added a dependency check for 'curses' library and updated documentation.

    This is for OpenSuse which packages curses separately.

2015-05-22

  - Fixed a simple bug (#47) where the default value used for legacy tolerances

    wasn't being initialized properly and added a test for it.

2015-05-20

  - Fixed a potential important bug (#48) just introduced a few days ago whereby

    enabling the 'account_rounding' option would inadvertently disable balance

    checks. Thanks to Matthew Harris for finding and reporting it.

  - Added a 'dedent' option to both beancount.parser.parser.parse_string() and

    beancount.loader.load_string(). This is a convenience for writing quick

    tests.

[Simon Michael]

Nice! Fewer clearer more cross-platform install steps is always good.

Installation worked for me on osx just now. I may have had some
prerequisites installed from before, such as python3 from brew.

I thought I was done when bean-check ran, but just out of interest I ran
bean-doctor from the next section and got a traceback showing that I
still needed lxml. I confirmed that by re-reading your release notes and
installed it with pip3 install lxml. The next traceback mentioned
datetime, so pip3 install python-datetime. Finally bean-doctor showed a
nice list of dependencies and their status. I had bottle from before,
and upgraded it with pip3 install bottle --upgrade, and ply sounds
useful so pip3 install ply.

bean-doctor is nice, it helps reassure me that I won't be getting
dependency-related tracebacks later while using the tool. Here are some
ideas make it even better:
- always show the dependency list instead of a traceback
- include the new lxml dependency in the list
- indicate which libs are required
- for not-yet-installed libs, suggest the pip command to install them
- maybe have each bean tool check for required dependencies at startup ?

On 6/28/15 11:09 PM, Martin Blais wrote:
> 2015-06-27
>
> - Complete the design doc and made it public.

[Martin Blais]

On Wed, Jul 1, 2015 at 10:54 AM, Simon Michael <si...@joyful.com> wrote:

Nice! Fewer clearer more cross-platform install steps is always good. 

Installation worked for me on osx just now. I may have had some
prerequisites installed from before, such as python3 from brew.

I thought I was done when bean-check ran, but just out of interest I ran
bean-doctor from the next section and got a traceback showing that I
still needed lxml. I confirmed that by re-reading your release notes and
installed it with pip3 install lxml.

Ah! How could I forget. Thanks for pointing this out. 

I added lxml to checkdeps.

 

The next traceback mentioned
datetime, so pip3 install python-datetime.

You mean python-dateutil, not python-datetime.

(datetime is part of the stdlib and should require no install.)

 

Finally bean-doctor showed a
nice list of dependencies and their status. I had bottle from before,
and upgraded it with pip3 install bottle --upgrade, and ply sounds
useful so pip3 install ply.

You need PLY for bean-query.

bean-doctor is nice, it helps reassure me that I won't be getting
dependency-related tracebacks later while using the tool.

"bean-doctor checkdeps" is only required to run Beancount directly from the clone.

I'd been having problems with setup.py in that it's supposed to be able to install dependencies by default, so in theory you shouldn't have to install anything but Beancount now. It hadn't been working before. 

Here's some background, in case anyone's interested: Python has two active supporting projects for packaging: 

(1) distutils, which comes with it in the stdlib, and 

(2) setuptools, which is newer and better, but which needs to be installed on top of the stdlib. 

There's a deep history of why that is and many projects have come and gone trying to make this better (e.g. distribute). The main issue with disutils is that it does not support automatic installation of dependencies (it's really old but a ton of packages still depend on it, that's the price of popularity). Setuptools does support it, the packager can declare dependencies and pip3 should be able to pull in and install all requirements automatically.  I could not get setuptools to install my dependencies due to a namespace collision with the 'parser' module (was my fault).

Now that I've made changes, Beancount supports setuptools and installs dependencies automatically. If you don't have setuptools installed, it falls back on distutils (and you have to install the dependencies manually, e.g. it's a one-liner:  "pip3 install python-dateutil lxml bottle ply").

 

Here are some
ideas make it even better:
- always show the dependency list instead of a traceback

Hmm I've done such things in the past, and my experience is that it's often better to just let an unexpected exception trickle through: if I put too much magic in the runner and then _that_ magic fails, then you have an even more cryptic error message: the original exception, and maybe another exception occurring in the error detection code on top of it. Installing dependencies is boiled down to something pretty simple at this point:

  pip3 install python-dateutil lxml bottle ply

That's it. Beancount isn't much different than other Python softwares and the other packages don't wrap all their imports with try-except all over the place. The only difference is that many users will be running from source.  I prefer to keep things simple.  Nonetheless, I'll experiment with the idea.

 

- include the new lxml dependency in the list

Done.

 

- indicate which libs are required

In docs and checkdeps output.

 

- for not-yet-installed libs, suggest the pip command to install them

That's great idea. Done. 

I've added a message to the checkdeps output.

 

- maybe have each bean tool check for required dependencies at startup ?

Thanks a lot Simon.

Can I count you as one of my users now? ;-)

Thanks,

On 6/28/15 11:09 PM, Martin Blais wrote:
> 2015-06-27
>
>   - Complete the design doc and made it public.

> 2015-06-09
>
>   - Rewrote the Installation Instructions document completely to make it
>     simpler. Some users had found it daunting.
>     http://furius.ca/beancount/doc/install
>
>   - Tested Windows installation using Cygwin and added it to the
> installation
>     instructions.
>     http://furius.ca/beancount/doc/install

--

---
You received this message because you are subscribed to the Google Groups "Ledger" group.
To unsubscribe from this group and stop receiving emails from it, send an email to ledger-cli+...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Martin Blais]

Okay, an update: the state of setuptools / distutils Python distribution tools is in a really sorry state indeed. I've spend much of my afternoon dealing with some more setuptools brokenness and running around its sad, sad source code. When I use

  python3 setup.py build_ext -i

with setuptools installed, it puts the _parser.so library under src/python and not under src/python/beancount/parser where it should be, so that's a no go.

This is using setuptools v15. Manually installing the latest v18 puts the .so in the right place but there is an annoying warning about not running it with pip3.

#$&*# this. No seriously, I've had enough. I want to spend my time building features instead of packaging, and I've wasted the better part of a beautiful vacation day just dealing with setuptools and reading conflicting information on the web. I'm ditching support for setuptools. 

Beancount will use distutils, which is basic, comes with all versions of Python, and does the right thing by default. It's only drawback is that you need to install the dependencies manually. It won't do the dependencies automatically. It's a one-liner to install dependencies. No big deal in my book.

Python seriously needs a brand new, rewritten from scratch packaging system.

[Wm...]

Mon, 29 Jun 2015 02:09:01
<CAK21+hMxAOeFow5GNrO=RqcOYevrY146Zqm...@mail.gmail.com>
gmane.comp.finance.beancount Martin Blais
<bl...@furius.ca>

>2015-06-09
>
>  - Rewrote the Installation Instructions document completely to make
>it
>    simpler. Some users had found it daunting.
>    http://furius.ca/beancount/doc/install
>
>  - Tested Windows installation using Cygwin and added it to the
>installation
>    instructions.
>    http://furius.ca/beancount/doc/install

Suggest use cygwin rather than pip where packages are available as it
mitigates the setuptools / distutils contre temps a bit

i.e. where your list in whatever you've called the doc today says
===
Install the latest Cygwin. This may take a while (it downloads a lot of
stuff), but it is well worth it in any case. But before you kick off the
install, make sure the following packages are all manually enabled in
the interface provided by setup.exe (they’re not selected by default):
python3
python3-setuptools
mercurial
make
gcc-core
flex
bison
===

add

===
lxml
ply
===

and adjust the pip list accordingly in

===
Install dependencies using pip:
pip3 install python-dateutil bottle ply lxml
===

Further, I needed to use easy_install-3.4 to get the right pip

=================

I may be wrong but I think if you get the beancount+cygwin process right
you'll be ahead of the other ledger+friends implementations on Win.

Doing it without cygwin is a different barrel of fish and you may want
to measure need before wandering down that route.

If you redo your cygwin and therefore general install instructions and
re-publish them in your quaint way I'll try following them to see if
they work and let you know of any further changes.

P.S. nice to see SQL wasn't as bad an idea as you first thought, eh?

--
Wm...

[Gary Peck]

I fully understand your pain and frustration with setuptools, but I'm hoping you might reconsider. With a few small changes to setup.py, I have not been able to find an installation method that fails. If you let me know environments where the installation fails, I'm happy to try to fix those environments too. I'm trying to get others in my company to use beancount for our bookkeeping and it's really nice to be able to point people at a single installation command, as well as being able to use tools like pipsi to install beancount.

I tested the following commands under Ubuntu 15.04 and (a) a virtualenv with python 3.4.3, pip 7.1.0, and setuptools 18.0.1, as well as (b) a pyvenv with python 3.4.3, pip 1.5.6, setuptools 12.2:

- pip3 install -e . (this is recommended by the python packaging guide for development installs rather than using "python3 setup.py install" directly)

- pip3 install .

- python3 setup.py install (yes, you get an annoying warning about not running it under pip, but everything works)

- python3 setup.py build_ext -i (copies _parser.so to src/python/beancount/parser)

Here's a diff of the changes I had to make. I think the reason you were having trouble with _parser.so getting copied to the right place is because the first argument to Extension needs to be a python package path, not a filesystem path.

diff -r b5d8ae469380 setup.py

--- a/setup.py Mon Oct 05 11:31:52 2015 +0800

+++ b/setup.py Wed Oct 07 15:36:42 2015 -0700

@@ -18,7 +18,7 @@

 

 # Import setup().

 setup_extra_kwargs = {}

-from distutils.core import setup, Extension

+from setuptools import setup, Extension

 

 # Unused support for setuptools. Setuptools is seriously broken:

 #

@@ -132,7 +132,7 @@

     scripts=install_scripts,

 

     ext_modules=[

-        Extension("beancount/parser/_parser",

+        Extension("beancount.parser._parser",

                   sources=[

                       "src/python/beancount/parser/lexer.c",

                       "src/python/beancount/parser/grammar.c",

@@ -142,6 +142,13 @@

                                   '"{}"'.format(hash_parser_source_files()))]),

     ],

 

+    install_requires=[

+        'python-dateutil',

+        'bottle',

+        'ply',

+        'lxml',

+    ],

+

     # Add optional arguments that only work with some variants of setup().

     **setup_extra_kwargs

 )

Thanks,

Gary

[Martin Blais]

I created a ticket:

https://bitbucket.org/blais/beancount/issues/72/consider-installing-with-setuptools-again

It would need a lot more testing, e.g., on Windows, Mac, etc., in order to make sure this works and if I recall all the crazy time I sunk into this, I'm pretty sure I don't want to burn all that time all over again. I'll reconsider it but it's at the very bottom of the list of things I need to do. The price to pay to install the other dependencies explicitly is really small IMO (and you don't have to install setuptools, which is a plus).

--
You received this message because you are subscribed to the Google Groups "Beancount" group.
To unsubscribe from this group and stop receiving emails from it, send an email to beancount+...@googlegroups.com.
To post to this group, send email to bean...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/beancount/957bf596-26da-4f6b-b72d-512e9d20ce4b%40googlegroups.com.

[Gary Peck]

Thanks. I'm not even sure how you can end up without setuptools installed with a modern python3 installation. I think as long as you require that the user has pip3 installed, through whatever means is appropriate to their OS/environment, it'll pull in setuptools.

Gary

[Martin Blais]

Fair enough.

The main problem is that I'm just not really into going back down the rabbit hole and burn another few precious weekend days to setup new Windows VMs and futz around with docker to test on various distros just to avoid users having to type a single command to install dependencies, I already burn that time, and I'm really busy. I deal with enough piping and plumbing at work that on my own time I want to write brand new code fast and build features or learn new things instead of spending all that time worrying about packaging.

If you really care about this, and want to help, test your change on Windows, Linux (Ubuntu, Debian, RedHat) and recent Mac OS X, and if all pans out, I'll change it over. Otherwise my feeling is what's there - distutils - is good enough.

--
You received this message because you are subscribed to the Google Groups "Beancount" group.
To unsubscribe from this group and stop receiving emails from it, send an email to beancount+...@googlegroups.com.
To post to this group, send email to bean...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/beancount/692f65af-a1ef-453b-94c8-20885965f223%40googlegroups.com.