Getting start with Ledger and Python

[John Wiegley]

Hello all,

I think it's time for a primer on using Python to extend your Ledger
experience. But first, a word must be said about Ledger's data model, so that
other things make sense later.

------------------------------------------------------------------------------
# Basic data traversal

Every interaction with Ledger happens in the context of a Session. Even if
you don't create a session manually, one is created for you by the top-level
interface functions. The Session is where objects live like the Commodity's
that Amount's refer to.

The make a Session useful, you must read a Journal into it, using the function
`read_journal`. This reads Ledger data from the given file, populates a
Journal object within the current Session, and returns a reference to that
Journal object.

Within the Journal live all the Transaction's, Posting's, and other objects
related to your data. There are also AutomatedTransaction's and
PeriodicTransaction's, etc.

Here is how you would traverse all the postings in your data file:

import ledger

for xact in ledger.read_journal("sample.dat").xacts:
for post in xact.posts:
print "Transferring %s to/from %s" % (post.amount, post.account)

------------------------------------------------------------------------------
# Raw vs. Cooked

Ledger data exists in one of two forms: raw and cooked. Raw objects are what
you get from a traversal like the above, and represent exactly what was seen
in the data file. Consider this journal:

= true
(Assets:Cash) $100

2012-03-01 KFC
Expenses:Food $100
Assets:Credit

In this case, the *raw* regular transaction in this file is:

2012-03-01 KFC
Expenses:Food $100
Assets:Credit

While the cooked form is:

2012-03-01 KFC
Expenses:Food $100
Assets:Credit $-100
(Assets:Cash) $100

So the easy way to think about raw vs. cooked is that raw is the unprocessed
data, and cooked has had all considerations applied.

When you traverse a Journal by iterating its transactions, you are generally
looking at raw data. In order to look at cooked data, you must generate a
report of some kind by querying the journal:

for post in ledger.read_journal("sample.dat").query("food"):
print "Transferring %s to/from %s" % (post.amount, post.account)

The reason why queries iterate over postings instead of transactions is that
queries often return only a "slice" of the transactions they apply to. You
can always get at a matching posting's transaction by looking at its "xact"
member:

last_xact = None
for post in ledger.read_journal("sample.dat").query(""):
if post.xact != last_xact:
for post in post.xact.posts:
print "Transferring %s to/from %s" % (post.amount,
post.account)
last_xact = post.xact

This query ends up reporting every cooked posting in the Journal, but does it
transaction-wise. It relies on the fact that an unsorted report returns
postings in the exact order they were parsed from the journal file.

------------------------------------------------------------------------------
# Queries

The Journal.query() method accepts every argument you can specify on the
command-line, including --options.

Since a query "cooks" the journal it applies to, only one query may be active
for that journal at a given time. Once the query object is gone (after the
for loop), then the data reverts back to its raw state.

------------------------------------------------------------------------------
# Embedded Python

Can you embed Python into your data files using the 'python' directive:

python
import so
def check_path(path_value):
print "%s => %s" % (str(path_value), os.path.isfile(str(path_value)))
return os.path.isfile(str(path_value))

tag PATH
assert check_path(value)

2012-02-29 KFC
; PATH: somebogusfile.dat
Expenses:Food $20
Assets:Cash

Any Python functions you define this way become immediately available as
valexpr functions.

------------------------------------------------------------------------------
# Amounts

When numbers come from Ledger, like post.amount, the type of the value is
Amount. It can be used just like an ordinary number, except that addition
and subtraction are restricted to amounts with the same commodity. If you
need to create sums of multiple commodities, use a Balance. For example:

total = Balance()
for post in ledger.read_journal("sample.dat").query(""):
total += post.amount
print total

John

[Simon Michael]

Wow! Nice functionality (python everywhere) and nice docs, John.

[Raphael Lorenzeto]

Hello to everyone!

This is my first post here. I have to confess that I was very impressed by the features of ledger.

As an avid GNUCash user I was a little disappointed with the absence of GNUCash reading capabilities.

I managed to write a perl script to parse and output a ledger-formatted file and than I started looking for ways to present my data.

Long story short, I haven't found a way to access this Python interface, my intent is to produce a series of scripts to output HMTL5 files with my reports on a regular basis (probably a cron-job). How can I do this? Is there any way to communicate with Ledger without having to put it in REPL mode and creating the interface myself?

Thanks!

[John Wiegley]

>>>>> Raphael Lorenzeto <raphael.lorenzeto-Re...@public.gmane.org> writes:

> Long story short, I haven't found a way to access this Python interface, my
> intent is to produce a series of scripts to output HMTL5 files with my
> reports on a regular basis (probably a cron-job). How can I do this? Is
> there any way to communicate with Ledger without having to put it in REPL
> mode and creating the interface myself?

Make sure to pass --python to acprep when you build (you'll have to make
distclean if you're just adding that flag now). Then you use 'ledger python
foo.py' to run a Python script with the ability to 'import ledger' in that
script. See earlier postings in this mailing list for some examples of what
you can do then...

John

[Raphael Lorenzeto]

Thanks! I managed to compile with the --python passed to acprep but I get the following error while trying to execute the examples shown on this post:

./ledger/ledger python ledger-script.py 

Traceback (most recent call last):

  File "ledger-script.py", line 6, in <module>

    for post in post.xact.posts:

TypeError: 'instancemethod' object is not iterable

Abort trap: 6

On Sunday, March 11, 2012 7:37:32 AM UTC-3, John Wiegley wrote:

>>>>> Raphael Lorenzeto  writes:

[John Wiegley]

>>>>> Raphael Lorenzeto <raphael.lorenzeto-Re...@public.gmane.org> writes:

> Thanks! I managed to compile with the --python passed to acprep but I get
> the following error while trying to execute the examples shown on this post:

> ./ledger/ledger python ledger-script.py Traceback (most recent call last):
> File "ledger-script.py", line 6, in <module>
> for post in post.xact.posts:
> TypeError: 'instancemethod' object is not iterable Abort trap: 6

It should be post.xacts.posts, I believe.

John

[John Wiegley]

>>>>> John Wiegley <jwiegley-Re5JQE...@public.gmane.org> writes:

>> ./ledger/ledger python ledger-script.py Traceback (most recent call last):
>> File "ledger-script.py", line 6, in <module> for post in post.xact.posts:
>> TypeError: 'instancemethod' object is not iterable Abort trap: 6

> It should be post.xacts.posts, I believe.

No, you're right, post.xact.posts. I'll enter a bug for this into Bugzilla.

John

[Craig Earls]

Added to docs.

--
Craig, Corona De Tucson, AZ
enderw88.wordpress.com

[Chris Rossi]

On Sunday, March 11, 2012 6:37:32 AM UTC-4, JohnW wrote:

>>>>> Raphael Lorenzeto writes:

I've tried the following under Ubuntu 12.04:

$ git clone https://github.com/jwiegley/ledger.git 

$ cd ledger

$ git checkout v2.6.3

$ ./acprep --python

$ make

$ ./ledger python -f path/to/ledgerfile

The result is:

Error: Unrecognized command 'python'

I don't notice any errors during the build process.  Any idea what I'm doing wrong?

Chris

[John Wiegley]

>>>>> Chris Rossi <chris-Nf+wZpSdgwelM...@public.gmane.org> writes:

> The result is:

> Error: Unrecognized command 'python'

> I don't notice any errors during the build process. Any idea what I'm doing
> wrong?

v2.6.3 never had a Python command. Could you try building the 'master'
branch?

Thanks,
John

[henry atting]

Just out of curiousity...
I never managed to get any small python script running, ledger built
with/or without python or whatsoever.
My current version is a build from today: Ledger 3.0.0-20110325; with
--python
dir(ledger) displays what it should.
No script provided in this thread does work though.
E.g. this one:

import ledger

for xact in ledger.read_journal("sample.dat").xacts:
for post in xact.posts:
print "Transferring %s to/from %s" % (post.amount,
post.account)

Leads to this error message:

Traceback (most recent call last):

File "/home/me/script.py", line 3, in <module>

for xact in ledger.read_journal("sample.dat").xacts:

AttributeError: 'module' object has no attribute 'read_journal'

Any hint appreciated.
henry

[John Wiegley]

How are you invoking the script? Please try "ledger python foo.py".

John

[henry atting]

Yes, this is exactly what I do.

On May 12, 12:17 am, John Wiegley <jwieg...@gmail.com> wrote:

[henry atting]

Mmh, that's it? I cannot bring ledger (great) together with python
(great, too)?

What I did so far:

1. Compile ledger from git with --python
2. dir(ledger) shows everything form ACCOUNT_EXT_AUTO_VIRTUALIZE', to
'value_context'
3. Invoking this script:

#!/usr/bin/env python
import ledger

for xact in ledger.read_journal("ledger.dat").xacts:

for post in xact.posts:
print "Transferring %s to/from %s" % (post.amount, post.account)

in this way:

ledger python script.py

leads to:

Traceback (most recent call last):

File "scr.py", line 4, in <module>
for xact in ledger.read_journal("ledger.dat").xacts:

AttributeError: 'module' object has no attribute 'read_journal'

henry

[John Wiegley]

>>>>> henry atting <me.henryatting-Re5...@public.gmane.org> writes:

> Traceback (most recent call last):
> File "scr.py", line 4, in <module>
> for xact in ledger.read_journal("ledger.dat").xacts:
> AttributeError: 'module' object has no attribute 'read_journal'

Ah, try this instead:

for xact in read_journal("ledger.dat").xacts:
...

The Python bridge is still evolving.

John

[Tim Crews]

I have spent most of three days trying to figure out why this doesn't work, because I am very motivated to get ledger working with Python.

The problem is that the boost::python declaration for the "xacts" member is using "def" instead of "property" (py_journal.cc line 301).  The example python code works if you change it to:

for xact in ledger.read_journal("ledger.dat").xacts():   # Note the () after xacts
   for post in xact.posts():  # Note the () after posts

      print "Transferring %s to/from %s" % (post.amount, post.account)

I'm still waiting for a confirmation e-mail so I can add this information to the bugzilla bug that was recently opened.

[Tim Crews]

On the subject of getting the example Python API code to run:

On Sunday, May 27, 2012 4:39:38 PM UTC-7, Tim Crews wrote:

for xact in ledger.read_journal("ledger.dat").xacts():   # Note the () after xacts
   for post in xact.posts():  # Note the () after posts
      print "Transferring %s to/from %s" % (post.amount, post.account)

Reading my original post a day later, I see that it probably came across as abrupt.  I also posted without providing any context.  I'm sorry about that.

Anyway, I read the Boost::Python documents to figure out how to make an iterator wrapper that is also a property.  The example given at http://www.boost.org/doc/libs/1_49_0/libs/python/doc/tutorial/doc/html/python/iterators.html uses a ".property" function for this purpose, but that doesn't compile.

So I tried ".add_property" instead, and this works.  If you change py_journal.cc to use .add_property for xacts, and change py_xact.cc to use .add_property for posts, then the original Python sample code will run as written.

I've been reviewing the entire Python API with an eye towards putting together an equivalent RESTful web API (see other post), and in the process I think I see a few other functions that might be more "Pythonic" if they were exposed as read-only properties.  I was planning to publish a summary soon of the entire Python API, and I will note these.  

Does anyone have an opinion about what form a Python API summary document ought to take?  I would be willing to add it to the docs or the wiki, or just post it on the list.  Or am I duplicating someone else's effort?


Tim Crews

[John Wiegley]

>>>>> Tim Crews <tim-QJ3Fn9itRil4wfmgmkLWVgC/G2K4...@public.gmane.org> writes:

> Reading my original post a day later, I see that it probably came across as
> abrupt. I also posted without providing any context. I'm sorry about that.

You weren't, Tim, I just haven't been very available to respond to you this
weekend. If you could find me on irc.freenode.net/#ledger sometime, I'd be
happy to work this out with you in real-time. I'll be on there later tonight.

John

[John Wiegley]

>>>>> Tim Crews <tim-QJ3Fn9itRil4wfmgmkLWVgC/G2K4...@public.gmane.org> writes:

> I've been reviewing the entire Python API with an eye towards putting
> together an equivalent RESTful web API (see other post), and in the process
> I think I see a few other functions that might be more "Pythonic" if they
> were exposed as read-only properties. I was planning to publish a summary
> soon of the entire Python API, and I will note these.

If you think we can get to where we want with Python + Javascript, then I can
be talked away from trying to develop a full RESTful API.

John

[Stefan Tunsch]

Hi!

I am a Python developer, also interested in developing a good interface
from Python to ledger.
I have also taken a look at projects like beancount, who are
reimplementations of ledger in Python, but I would prefer to stick to
the original project.

That said, I am willing to invest time developing this.
So Tim and John, if you need help, let's team up and see if we can get
this going.

Regards, Stefan Tunsch

[Tim Crews]

On Wednesday, May 30, 2012 1:34:07 AM UTC-7, JohnW wrote:

If you think we can get to where we want with Python + Javascript, then I can
be talked away from trying to develop a full RESTful API.

John

I hope I'm not displaying my ignorance, but I think I might be missing something.  I'm in favor of a RESTful API.

Are you referring to client-side JavaScript or server-side?  I can imagine a client-side HTML application that interfaced to a ledger server, and this would involve plenty of client-side JavaScript, but that code would still need to communicate with the server to access Ledger functions.  I think a REST API would be a great way to model that.

I don't have a strong opinion that this is the best possible system architecture; this is just the model that I assumed.  Did you have a different model in mind?  I noticed in another thread that you mentioned a "pure JavaScript" implementation, and I didn't know what that meant.

Tim Crews

[Tim Crews]

On Wednesday, May 30, 2012 1:46:49 AM UTC-7, stunsch wrote:

I am willing to invest time developing this.
So Tim and John, if you need help, let's team up and see if we can get
this going.

Regards, Stefan Tunsch

I have some down-time, I love to write Python, I am good at C++, and I have received a lot of value from Ledger, so I see this as an opportunity to contribute.  I do not want to give the impression that I'm trying to take charge of anything!

With that said, here are the tasks that I was planning to start chipping away at.  None of this has been coordinated with John.  I would welcome any feedback if my work is misguided.

1. I created a github fork at https://github.com/code-affinity/ledger
2. I want to produce a Python API document.  I don't think the majority of Python developers would be comfortable crawling through the C++ code to discover the capabilities.  Since the exposed Python functions don't have any docstrings (yet), the usual Pythonic approach to exploration from the Python REPL isn't very productive either.
3. I created a python_api_reference topic branch for this purpose.
4. I've started by editing docs/ledger3.texi to add documentation for each Python class.  This is quickly getting large, and the section nesting levels are getting out of hand.  I think it might need to be a separate document.  Opinions?
   
5. So far, I'm only listing classes, methods, and properties by name, and listing parameters and return types.  There is no descriptive documentation yet.
6. For each method/property, if I see any problems, I'm adding a @comment right in the texinfo document to describe the problem.  I was planning to update Bugzilla bug 306 with my findings once I'm done.  Summarizing what I've found so far:
   
1. Lots of methods that would be more Pythonic if they were represented as read-only properties (for example the xacts() and posts() methods that have been discussed so much in this thread).
   
2. Several native C++ types need converting to native Python types to be useful to a Python developer.  For example, datetime and date, boost::filesystem::path, and perhaps FileInfo.  
3. A few C++ containers are returned directly without giving them iterator wrappers first.
   
4. The C++ code uses boost::optional, and I don't think Boost::Python knows how to wrap that without help, so I think a conversion might be needed to produce the Python None type as appropriate.  I could be wrong about that.
   
7. This process takes a lot of time!  I've probably put 20 hours into it so far, and even with this incomplete documentation, I've only finished Account, AccountXData, AccountXDataDetails, Amount, FileInfo, Journal, PostCollectorWapper, and PostHandler.  I guess this is about 25% of the overall work required.  Ouch!
8. Depending on any feedback from John Wiegley or other interested parties, update the Python bridge implementation, and modify the documentation to match.
9. Fill out the documentation with user-friendly descriptions, perhaps stealing comments from the C++ code where they exist.
10. Modify the Python bridge code to include Python docstrings for each class, method, and property.  (This is not hard with boost::python.)  Python developers really like their docstrings!
    
11. Time permitting, I really want to figure out why ledger python segfaults so often.  Even if it were feature-complete, the frequent segfaults make it very difficult for me to use and explore.  Also, after ledger python exits, my shell is usually in some weird state that doesn't show subsequent typed input.  That is also bad for productivity.

And none of this involves writing any Python!  I don't enjoy this kind of work, but I see it as preparatory for the more-fun stuff.

Any feedback?

Tim Crews

[Christopher Allan Webber]

There's no reason a RESTful API couldn't be done also with python on the
backend.

[Simon Michael]

On 5/30/12 9:17 AM, Tim Crews wrote:
> Any feedback?

This is excellent work Tim. Until more help shows up maybe you could pick just a small subset of the API to polish, so
you can get to the segfault/terminal issues and then some fun and motivating demo ?

[John Wiegley]

>>>>> Tim Crews <t...@code-affinity.com> writes:

> With that said, here are the tasks that I was planning to start chipping away
> at. None of this has been coordinated with John. I would welcome any feedback
> if my work is misguided.
>

> 1. I created a github fork at https://github.com/code-affinity/ledger
> 2. I want to produce a Python API document. I don't think the majority of

> Python developers would be comfortable crawling through the C++ code to
> discover the capabilities. Since the exposed Python functions don't have
> any docstrings (yet), the usual Pythonic approach to exploration from the
> Python REPL isn't very productive either.

> 3. I created a python_api_reference topic branch for this purpose.
> 4. I've started by editing docs/ledger3.texi to add documentation for each

> Python class. This is quickly getting large, and the section nesting
> levels are getting out of hand. I think it might need to be a separate
> document. Opinions?

> 5. So far, I'm only listing classes, methods, and properties by name, and

> listing parameters and return types. There is no descriptive documentation
> yet.

> 6. For each method/property, if I see any problems, I'm adding a @comment

> right in the texinfo document to describe the problem. I was planning to
> update Bugzilla bug 306 with my findings once I'm done. Summarizing what
> I've found so far:

> 1. Lots of methods that would be more Pythonic if they were represented as

> read-only properties (for example the xacts() and posts() methods that
> have been discussed so much in this thread).

> 2. Several native C++ types need converting to native Python types to be

> useful to a Python developer. For example, datetime and date,
> boost::filesystem::path, and perhaps FileInfo.

> 3. A few C++ containers are returned directly without giving them iterator
> wrappers first.
> 4. The C++ code uses boost::optional, and I don't think Boost::Python

> knows how to wrap that without help, so I think a conversion might be
> needed to produce the Python None type as appropriate. I could be
> wrong about that.

> 7. This process takes a lot of time! I've probably put 20 hours into it so

> far, and even with this incomplete documentation, I've only finished
> Account, AccountXData, AccountXDataDetails, Amount, FileInfo, Journal,
> PostCollectorWapper, and PostHandler. I guess this is about 25% of the
> overall work required. Ouch!

> 8. Depending on any feedback from John Wiegley or other interested parties,

> update the Python bridge implementation, and modify the documentation to
> match.

> 9. Fill out the documentation with user-friendly descriptions, perhaps

> stealing comments from the C++ code where they exist.

> 10. Modify the Python bridge code to include Python docstrings for each class,

> method, and property. (This is not hard with boost::python.) Python
> developers really like their docstrings!

> 11. Time permitting, I really want to figure out why ledger python segfaults so

> often. Even if it were feature-complete, the frequent segfaults make it
> very difficult for me to use and explore. Also, after ledger python exits,
> my shell is usually in some weird state that doesn't show subsequent typed
> input. That is also bad for productivity.
>
> And none of this involves writing any Python! I don't enjoy this kind of work,
> but I see it as preparatory for the more-fun stuff.

This is fantastic, Tim. I hope you can get to the Python part sooner rather
than later!

I'll help you with the segfaults as you go, though my time may be a little
limited this summer (several work contracts ongoing).

John

[John Wiegley]

>>>>> Stefan Tunsch <stu...@gmail.com> writes:

> Hi! I am a Python developer, also interested in developing a good interface
> from Python to ledger. I have also taken a look at projects like beancount,
> who are reimplementations of ledger in Python, but I would prefer to stick
> to the original project.
>
> That said, I am willing to invest time developing this. So Tim and John, if
> you need help, let's team up and see if we can get this going.

Stefan, I would love to have your help on that. Maybe you can coordinate with
Tim and Simon here on the mailing list as the Python work progresses?

John