Memory footprint of character collections

[Christoph Burgmer]

I am currently experimenting with providing several character instances to the
training step [1]. This increases the number of Character objects by factor 5
to 10, in my latest set 55103 instances. I seem to hit a memory limit, with
the systems starting to trash once python uses 70% of memory. While I
currently run scripts on a 2 gig machine, my local system is pretty old with
less ram, so I can't even view the resulting XML files with tegaki-train.

I wonder if there's any possibility to decrease the memory footprint of
scripts working with character collections. It's only an issue for
development, end users only work with resulting model files. Still it might be
necessary once working with larger collections. So feel free to suggest
optimizations ;)

-Christoph

[1] http://cburgmer.nfshost.com/content/improving-recognition-handwriting-
component-samples

[Mathieu Blondel]

On Fri, Nov 20, 2009 at 9:54 PM, Christoph Burgmer <cbur...@ira.uka.de> wrote:
> I am currently experimenting with providing several character instances to the
> training step [1].

Nice work. Seems like we've got the new default models for next release then :)

I also liked your idea of using the KanjiVG data as evaluation data.
It's pretty dangerous to do the evaluation with the few samples that
we have in tegaki-lab/. Since the amount of data is so small, the
improvements that we may get by tweaking our algorithms or models
against this data may well be pure luck and not reproduce on larger
data.

KanjiVG has only one sample per character but it has over 6000
characters so evaluation results are much more reliable. Even though
KanjiVG is for Japanese, we can reasonably assume that the
improvements we get also apply to Chinese. Using KanjiVG for
evaluation data means that we should refrain from using it for
training data as well otherwise our evaluation results will be biased.

> This increases the number of Character objects by factor 5
> to 10, in my latest set 55103 instances. I seem to hit a memory limit, with
> the systems starting to trash once python uses 70% of memory. While I
> currently run scripts on a 2 gig machine, my local system is pretty old with
> less ram, so I can't even view the resulting XML files with tegaki-train.
>
> I wonder if there's any possibility to decrease the memory footprint of
> scripts working with character collections. It's only an issue for
> development, end users only work with resulting model files. Still it might be
> necessary once working with larger collections. So feel free to suggest
> optimizations ;)

I definitely agree that this needs to be reworked. Currently, the
entire XML file is read and a Character instance is kept in memory for
each character in the collection... This approach makes certain things
easier: combining collections together (the + operator), modifying
character instances in the collection (tegaki-train), searching for
characters... We need to make the code more lazy, i.e. access
characters only when necessary. Python generators (the yield operator)
may be helpful here. However, I think this involves a major rewrite of
the class :-/

Mathieu

[Ian Johnson]

What about using something like sqlite so we can store and retrieve characters as needed? This wouldn't solve the issue of having to rewrite/restructure the class but it might scale better than reading bigger and bigger xml files. It could also later be ported to a heavier-duty DB if needed.

Ian

--

You received this message because you are subscribed to the Google Groups "Tegaki Handwriting Recognition Project" group.
To post to this group, send email to tegak...@googlegroups.com.
To unsubscribe from this group, send email to tegaki-hwr+...@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/tegaki-hwr?hl=.

--
Ian Johnson

[Mathieu Blondel]

Yeah that seems like a good idea. And sqlite is part of Python since
2.5. I wonder if we should throw away the existing CharacterCollection
class or make a new class which is API compatible.

Mathieu

[Mathieu Blondel]

I merged your latest changes in my repo, Christoph. Sorry for taking
some time. We also need to discuss what to do about your other
branches. For release 0.4, I want to focus my attention on recognizers
that use sub-units like components or strokes to recognize entire
characters, and the necessary infrastructure for that. I'll start a
thread about that later.

So far, I though your use of master and branches was very appropriate.
We should avoid to use master for too experimental stuff or things
that require prior discussion.

Also I'm planning to tackle the sqlite backend issue in the next few
days so don't waste your time working on it.

Mathieu

[Christoph Burgmer]

Am Freitag, 27. November 2009 schrieb Mathieu Blondel:
> I merged your latest changes in my repo, Christoph. Sorry for taking
> some time. We also need to discuss what to do about your other
> branches. For release 0.4, I want to focus my attention on recognizers
> that use sub-units like components or strokes to recognize entire
> characters, and the necessary infrastructure for that. I'll start a
> thread about that later.

Cool. I'm unsure how to integrate some of this stuff properly so we need to
discuss this. If we want to boost instance numbers, we need to find an
appropriate way how to build such models.

> So far, I though your use of master and branches was very appropriate.
> We should avoid to use master for too experimental stuff or things
> that require prior discussion.

I agree.

> Also I'm planning to tackle the sqlite backend issue in the next few
> days so don't waste your time working on it.

Sounds cool. I'm pretty busy with other stuff right now, so I'll just watch for
now.

-Christoph

[Mathieu Blondel]

OK, I committed the new CharacterCollection implementation to the repo
(about one and a half day worth of work...).

So character collections can now be stored into 2 formats: XML
(.charcol) or Sqlite (.chardb).

I updated tegaki-train to be able to open/save both .charcol and
.chardb formats. Loading large databases is now instantaneous with the
latter format.

tegaki-convert can now convert collections to database files if the
output file ends with .chardb.

Although you can still give several character collections to
tegaki-eval, tegaki-build and others, this will be slow because
character collections need be merged into one. So if you're dealing
with a lot of data, it's better to generate your .chardb file once for
all with tegaki-convert and then give it as a parameter to the tool
you're using (-b or --db option).

Now for the API part (you can skip this if you're not developing with Tegaki).

#####

I took the opportunity to move CharacterCollection to a separate file,
so now you need to import it like that:
from tegaki.charcol import CharacterCollection

Other than that, it should be backward-compatible with the previous
implementation (my unit-tests say so at least).

You can now give a database file parameter to the constructor:
charcol = CharacterCollection("myfile.chardb")

Otherwise it's kept in memory like before:
charcol = CharacterCollection()

In the former case, changes are not saved back to the file until you
do charcol.commit().

When you get a character object from the database, normally the
changes you make to that object aren't reflected back to the database.

char = charcol.get_all_characters(limit=1)[0]
char.set_utf8(newvalue)

In the above example, newvalue would normally not be reflected back to
the database as the new utf8 value for this character. So I created a
simple object proxy system that forwards the modification requests to
the database. If you don't want the overhead of using proxy objects
(e.g. if you only want to read data), this feature can be disabled by
doing

charcol.PROXY = False

I'm still hesitating what should be the default value for this option.

The code still needs to be polished (e.g. error checking on the
inputs) and some tools can be rewritten to use more optimized methods
that have been introduced.

Any comment is welcome, in particular on the table structure. Thanks
to Ian for suggesting Sqlite. It was perfect for the job! (in memory
databases and transactions made my life much easier)

Mathieu

[Christoph Burgmer]

Looks good. Especially sticking to well known design patterns, and keeping API
compatibility.

Are you working on an auto_commit feature? I found clear_pool_threshold() that
would only write out the cached data if the pool surpasses a certain size, but
this is not used in the CharacterCollection.

My problem so far was memory consumption, and while the new code does solve
this, I need to manually keep track of reducing the in-memory size.

How do we now write out generated CharacterCollections? The tool itself needs
to take care of either calling to_xml() or commit(), right? Could we find a
unified way of handling this? Does that make sense?

Oh, and SQLite is a very good option, as this is the only db you can certainly
use everywhere, even on mobile phones.

-Chris

> --
>
> You received this message because you are subscribed to the Google Groups
> "Tegaki Handwriting Recognition Project" group. To post to this group,
> send email to tegak...@googlegroups.com.
> To unsubscribe from this group, send email to
> tegaki-hwr+...@googlegroups.com. For more options, visit this

> group at http://groups.google.com/group/tegaki-hwr?hl=en.
>

[Mathieu Blondel]

On Tue, Dec 1, 2009 at 10:52 PM, Christoph Burgmer <cbur...@ira.uka.de> wrote:
> Looks good. Especially sticking to well known design patterns, and keeping API
> compatibility.
>
> Are you working on an auto_commit feature?

No I'm not but that could be an option indeed. Python's sqlite3 API
has an option isolation_level that does just that. But not committing
by default is really great. That makes it possible to edit your
collection in tegaki-train and pushing the "save" button when you're
done.

> My problem so far was memory consumption, and while the new code does solve
> this, I need to manually keep track of reducing the in-memory size.

Sure, the new implementation should make things easier but it doesn't
mean that tools can be written without care, especially if you
retrieve a lot of character objects at once. Your tools may benefit
from generator-based equivalents to get_characters(set_name) and
get_all_characters(), I think.

> How do we now write out generated CharacterCollections? The tool itself needs
> to take care of either calling to_xml() or commit(), right? Could we find a
> unified way of handling this? Does that make sense?

I'll try to do something about it.

Mathieu

[Mathieu Blondel]

On Tue, Dec 1, 2009 at 11:32 PM, Mathieu Blondel <mblo...@gmail.com> wrote:
>> How do we now write out generated CharacterCollections? The tool itself needs
>> to take care of either calling to_xml() or commit(), right? Could we find a
>> unified way of handling this? Does that make sense?
>
> I'll try to do something about it.

OK I added a new optional parameter to the constructor and a new save
method, for both Character and CharacterCollection. So now saving
should be transparent for the user. For more details, you can generate
the API documentation with "epydoc tegaki".

Mathieu