Ledger 3.0 performance compared with older versions

[Vinod Kurup]

I tried ledger 3.0 after using a combination of ledger 2.5 and 2.6 happily for many years. I've found 3.0 to be too slow for my needs. My ledger file is huge and I use the emacs reconcile mode extensively, which calls ledger frequently, so performance issues are very noticeable.

Please note that this really isn't a complaint. I know 3.0 brings a lot of new functionality and bug fixes, and performance may be perfectly fine for normal size ledger files. I just wanted to report this in case I'm doing something wrong or there exist some easy ways to improve 3.0 performance. For the moment, I'm sticking with 2.5 since it works perfectly for me. 

All of the following done on a quad core dell desktop, ubuntu 10.10:

TL;DR=> 

2.5 (register = 5 sec, balance = 0.06 sec)

2.6 (register = 3 min, balance = 0.9 sec)

3.0 (register = 65 min, balance = 4.6 sec)

vinod@cartman:~ $ uname -a

Linux cartman 2.6.35-30-generic-pae #59-Ubuntu SMP Tue Aug 30 16:16:14 UTC 2011 i686 GNU/Linux

# FILE IS BIG (dating back to 1991)

vinod@cartman:~ $ wc -l $LEDGER_FILE

85406 /home/vinod/Dropbox/ledger.dat

# ledger 2.5

vinod@cartman:~ $ ./src/ledger-2.5/ledger -v

Ledger 2.5, the command-line accounting tool

# Register command

vinod@cartman:~ $ time ./src/ledger-2.5/ledger -V reg | wc -l

54363

real 0m5.516s

user 0m5.416s

sys 0m0.104s

# Balance command

vinod@cartman:~ $ time ./src/ledger-2.5/ledger -V bal 

real 0m0.062s

user 0m0.036s

sys 0m0.024s

# ledger 2.6

vinod@cartman:~ $ ledger -v

Ledger 2.6.2, the command-line accounting tool

# Register command

vinod@cartman:~ $ time ledger -V reg | wc -l

46072

real 3m4.285s

user 3m3.159s

sys 0m1.048s

# Balance command

vinod@cartman:~ $ time ledger -V bal 

real 0m0.945s

user 0m0.696s

sys 0m0.248s

# ledger 3.0

vinod@cartman:~ $ ./src/ledger/ledger -v

Ledger 3.0.0-20110325, the command-line accounting tool

# Register command

vinod@cartman:~ $ time ./src/ledger/ledger -V reg | wc -l

47378

real 65m59.653s

user 65m52.407s

sys 0m2.424s

# Balance command

vinod@cartman:~ $ time ./src/ledger/ledger -V bal 

real 0m4.662s

user 0m4.084s

sys 0m0.052s

[John Wiegley]

>>>>> Vinod Kurup <vvkurup-Re5JQEe...@public.gmane.org> writes:

> I tried ledger 3.0 after using a combination of ledger 2.5 and 2.6 happily
> for many years. I've found 3.0 to be too slow for my needs. My ledger file
> is huge and I use the emacs reconcile mode extensively, which calls ledger
> frequently, so performance issues are very noticeable.

> Please note that this really isn't a complaint. I know 3.0 brings a lot of
> new functionality and bug fixes, and performance may be perfectly fine for
> normal size ledger files. I just wanted to report this in case I'm doing
> something wrong or there exist some easy ways to improve 3.0
> performance. For the moment, I'm sticking with 2.5 since it works perfectly
> for me.

Hi Vinod!

As always, I am extremely concerned about performance, and will go to great
lengths to achieve it, since this is a CLI tool meant to be executed many
times. Now, there are some correctness issues which will prevent 3.0 from
being faster than 2.x at times, but it shouldn't be pathologically slower --
as your data definitely shows.

So, my first questions are: What OS? What compiler? How did you build Boost
and which compiler flags? How did you build Ledger and which compiler flags?

Thanks,
John

[Vinod Kurup]

On Sun, Oct 9, 2011 at 4:02 AM, John Wiegley <jwie...@gmail.com> wrote:

So, my first questions are: What OS?  What compiler?  How did you build Boost

and which compiler flags?  How did you build Ledger and which compiler flags?

Hi John! Thanks for the quick response!

OS: Ubuntu 10.10

uname: Linux cartman 2.6.35-30-generic-pae #59-Ubuntu SMP Tue Aug 30 16:16:14 UTC 2011 i686 GNU/Linux

vinod@cartman:~ $ gcc -v

Using built-in specs.

Target: i686-linux-gnu

Configured with: ../src/configure -v --with-pkgversion='Ubuntu/Linaro 4.4.4-14ubuntu5' --with-bugurl=file:///usr/share/doc/gcc-4.4/README.Bugs --enable-languages=c,c++,fortran,objc,obj-c++ --prefix=/usr --program-suffix=-4.4 --enable-shared --enable-multiarch --enable-linker-build-id --with-system-zlib --libexecdir=/usr/lib --without-included-gettext --enable-threads=posix --with-gxx-include-dir=/usr/include/c++/4.4 --libdir=/usr/lib --enable-nls --with-sysroot=/ --enable-clocale=gnu --enable-libstdcxx-debug --enable-objc-gc --enable-targets=all --disable-werror --with-arch-32=i686 --with-tune=generic --enable-checking=release --build=i686-linux-gnu --host=i686-linux-gnu --target=i686-linux-gnu

Thread model: posix

gcc version 4.4.5 (Ubuntu/Linaro 4.4.4-14ubuntu5) 

Boost: I just installed the libraries and headers from Ubuntu. Version: 1.42.0-3ubuntu1

Ledger: I just did a plain './configure', but here is the line from my config.log:

  $ configure CPPFLAGS=-isystem /usr/local/include -isystem /usr/include/python2.6 CXXFLAGS=-Wconversion -pthread -g -pipe LDFLAGS=-fpch-deps -g -L/usr/local/lib -L/usr/lib -L/usr/lib/python2.6/config --with-included-gettext --enable-python --enable-pch --enable-debug

I'm not sure which config information is useful so I've attached my config.log here: https://gist.github.com/1273610

I wonder if those '--enable-debug' lines are what's hurting me. Let me know if I can provide any other info.

Thanks again,

Vinod

[thierry]

Hi Vinod,

Please refer to this thread and particularly last message of the
thread.
http://groups.google.com/group/ledger-cli/browse_thread/thread/da86281229a3fabd/

I believe that performance issue about ledger v3 are related to "price
relationships". In your example, you used "ledger -V", and here is the
point. With this command, for every date, the ledger program has to
find the price that is in your price.db file. In my case, I use "-X",
but like "-V", it computes the price relationships.


I solved this issue by (I tested this, and this really helps):
- using ledger -M or -Y. By computing "-V" only once per month or per
year, you reduce a lot the price relationship to compute. Yes, I
abandoned to use "-V" for computing daily values.
- reducing the price.db file, with only one value per month. This also
reduce the price relations to compute by ledger


There is also one point that can help. I did not tested it with real
performance tests. First let's build the pricemap with commands:

$ ledger pricemap > /tmp/pricemap.dot
$ circo -Tpdf -o/tmp/pricemap.pdf /tmp/pricemap.dot

By looking to /tmp/pricemap.pdf, you should only have hub (or star)
with your reference currency in the middle. You should not have
triangle, or square, or any polygons, that would mean you have
circular price relationship. At this point, I am sure that those
polygons can not speed up ledger, only add confusion to ledger.

Thierry

On 9 oct, 00:38, Vinod Kurup <vvku...@gmail.com> wrote:
> I tried ledger 3.0 after using a combination of ledger 2.5 and 2.6 happily
> for many years. I've found 3.0 to be too slow for my needs. My ledger file
> is huge and I use the emacs reconcile mode extensively, which calls ledger
> frequently, so performance issues are very noticeable.
>
> Please note that this really isn't a complaint. I know 3.0 brings a lot of
> new functionality and bug fixes, and performance may be perfectly fine for
> normal size ledger files. I just wanted to report this in case I'm doing
> something wrong or there exist some easy ways to improve 3.0 performance.
> For the moment, I'm sticking with 2.5 since it works perfectly for me.
>
> All of the following done on a quad core dell desktop, ubuntu 10.10:
>

> *TL;DR=> *
> *2.5 (register = 5 sec, balance = 0.06 sec)*
> *2.6 (register = 3 min, balance = 0.9 sec)*
> *3.0 (register = 65 min, balance = 4.6 sec)*

[Vinod Kurup]

On Sun, Oct 9, 2011 at 8:35 AM, thierry <thierry....@free.fr> wrote:

Please refer to this thread and particularly last message of the
thread.
http://groups.google.com/group/ledger-cli/browse_thread/thread/da86281229a3fabd/

Wow, I don't know how I missed this thread previously. Let me try some of the the recommendations from there and I'll report back if I'm still experiencing issues.

Thanks!

Vinod 

[John Wiegley]

>>>>> Vinod Kurup <vinod-46o65aAd...@public.gmane.org> writes:

> I wonder if those '--enable-debug' lines are what's hurting me. Let me know
> if I can provide any other info.

Oh, yes indeed. The debug build of 3.0 is significantly slower. Please try
this:

./acprep opt

That should cure some of your problems, with the exception that -V can become
quite slow in certain circumstances.

John

[Vinod Kurup]

On Sun, Oct 9, 2011 at 10:14 PM, John Wiegley <jwie...@gmail.com> wrote:

Oh, yes indeed.  The debug build of 3.0 is significantly slower.  Please try

this:

 ./acprep opt

That should cure some of your problems, with the exception that -V can become
quite slow in certain circumstances.

Thanks, that was it. After building the optimized version, here are the new results:

2.5 (register = 5 sec, balance = 0.06 sec)

2.6 (register = 3 min, balance = 0.9 sec)

3.0 (register = 11 min, balance = 0.9 sec)

And, if I use -Y instead of -V as you and Thierry suggested, then the register time comes down to 7 seconds, which is awesome. The 'ledger -V reg' is actually a command I never use... I always limit it by account or some other criteria.

I'm going to start using 3.0 regularly. Thanks again!

Vinod

[Simon Michael]

Just curious, could you share roughly how many transactions and accounts are in that file ?

Related, I note that ledger stats counts postings but not transactions, while hledger stats does the reverse.

[Vinod Kurup]

On Mon, Oct 10, 2011 at 3:11 PM, Simon Michael <si...@joyful.com> wrote:

Just curious, could you share roughly how many transactions and accounts are in that file ?

Related, I note that ledger stats counts postings but not transactions, while hledger stats does the reverse.

Sure,

ledger 3.0 reports:

Unique payees: 3633

Unique accounts: 820

Number of postings: 47493

hledger reports:

Accounts: 820 (depth 5)

Payees/descriptions: 3633

Transactions: 19303

Vinod

[Simon Michael]

Thanks! Real-world numbers are useful. Ignore me if you have better things to do, but now I wonder how long your
optimized ledger and hopefully optimized hledger (the default with cabal install) took to complete the stats command.

[Vinod Kurup]

On Mon, Oct 10, 2011 at 6:48 PM, Simon Michael <si...@joyful.com> wrote:

Thanks! Real-world numbers are useful. Ignore me if you have better things to do, but now I wonder how long your optimized ledger and hopefully optimized hledger (the default with cabal install) took to complete the stats command.

Happy to provide any info that I can!

hledger stats: 0m10.977s

ledger 3.0 (optimized) stats: 0m1.084s

I installed hledger by installing haskell-platform in Ubuntu 10.10 and then doing 'cabal update cabal-install' and then using the new cabal to install hledger.

Vinod

--
Vinod Kurup, MD
http://kurup.org