Generating general ledger is very long
B.V.
Working on a project of accounting migration to Tryton, we found some
performance issues with the general ledger.
The problems we met should be split in two parts:
- report data generation
- ODT rendering, then PDF conversion
With data from a customer, we built a database with more than 21000
account move lines (21735 to be accurate). The database contains 913
accounts.
Generating data for the general ledger is done in 12 minutes and 58
seconds. This time does not include ODT rendering/PDF conversion.
Very worried, we have built a test database with generated data. This
database contains 21819 account move lines. On this database, data for
the general ledger are generated in 35 seconds. Searching what could
impact so much performances, we realized that the test database is
built with default account chart: 7 accounts.
As the accounting migration is a work in progress, on the customer
database, we have easily decreased the number of accounts, while
keeping the same number of account move lines. Now the database has
only 45 accounts, and generating data for the general ledger takes 58
seconds, which is a pretty long time, but quite better than our first
result.
So the conclusion is that the data generation time for the general
ledger dramatically increase with the number of accounts, even if
those accounts have no lines.
We found that ODT rendering and PDF conversion only depend on the
number of pages. Rendering about 800 pages takes between 6 and 7
minutes. Converting the ODT into PDF takes generally 10 minutes.
All tests were done on the same server (same source code and same
machine), a Core2Duo / 3GbRAM server.
The source code is a regular trytond 1.4 server, with standard
modules, excepted account module, in which we have backported some
patches from development account module [1] [2] [3]. Before we did
that, the data generation on the database with 913 accounts may last
more than 1H.
This post is related to [4].
B.V.
[1] http://hg.tryton.org/hgwebdir.cgi/modules/account/rev/43675304b84c
[2] http://hg.tryton.org/hgwebdir.cgi/modules/account/rev/be90e2906108
[3] http://hg.tryton.org/hgwebdir.cgi/modules/account/rev/b76372879e72
[4] https://bugs.tryton.org/roundup/issue1399
ced
> ledger dramatically increase with the number of accounts, even if
> those accounts have no lines.
To solve this kind of performance issue, it requires some times
because
it needs to setup a benchmark environment for the specific report to
optimize.
I don't think that any one without this specific needs, will spend his
free-time on
this. You should ask to some companies on the tryton's services pages
[1].
We (B2CK) are convinced we could optimize it within one work day.
>
> We found that ODT rendering and PDF conversion only depend on the
> number of pages. Rendering about 800 pages takes between 6 and 7
> minutes. Converting the ODT into PDF takes generally 10 minutes.
For me the ODT format (like doc) is not designed to handle so much
pages.
I think for this kind of long listing should be generated directly in
PDF file
using ReportLab [2].
We already made some tests with ReportLab to generate the same kind of
report (a list of tables with many rows and columns):
N° pages Time
266 12s
1319 1m27s
6593 10m30s
[1] http://www.tryton.org/services.html
[2] http://www.reportlab.com/software/opensource/