New feature of Tagaini - kanji selector build on KanjiVG component information

[Alexandre Courbot]

Hey everybody,

I wanted to do that one since a loooong time ago (since I learned
about KanjiVG, actually), so I christmas-treated myself and coded it
during the holidays. :p

Basically, this is the "new" (as far as I know - that is not very far)
kanji input method that is based on selecting components that make the
kanji. Think of it as a radical selector on steroids. Or rather,
forget about radicals - thanks to the very rich component information
of KanjiVG, any part of the kanji that you are able to recognize can
be the end of the thread that will lead you to it. Once you have found
that end, other components that can complement this one to make a
kanji are displayed, and you can incrementally build your way to the
kanji you are looking for. Jeroen had explained a similar and more
complete concept in his great thesis
(http://handle.jeroenhoek.nl/hoek_2009 ), I think the present work can
constitute the first step to its implementation.

To demonstrate it I have put a short video on Youtube. I think it
explains better than any description I could make.

http://www.youtube.com/watch?v=RhuhMpKv7BU

The feature should make it into 0.2.5 - 0.3.0 (or by the meantime into
a development build, if people ask for it). It is still rough on the
edges, and I realized some data is missing, but even in the current
state it is the fastest way for me to find a totally unknown kanji. 4
clicks max, provided I can identify a component. I believe any student
with a very basic knownledge about kanjis can use it to its full
power.

I'm curious to know what you guys think about it, whether it is
actually something new (at least, I haven't seen anything similar in
any software or web site), and how it could be improved.

Alex.

[Benoit Cerrina]

nice,
it is similar to the improvements I did to the radical search in the latest version of shinkanji:

http://www.youtube.com/watch?v=yeciVor0Ggg

but in this case I only use the radkfile and I disable radicals which won't lead to anything.
The idea to draw on the vector description (kanjiVG for you but for shinkanji it would be taka) is nice,
I would have to test the resulting speed on the iphone.

I am wondering a couple things:
1) how did you select the root components
2) how do you select the new components to display? Do you display every one of those which can be combined into a target kanji (including some which are already a combination of simpler components)

Best regards
Benoit

[Alexandre Courbot]

Hi Ben,

> it is similar to the improvements I did to the radical search in the latest version of shinkanji:

Indeed - I think jisho.org also features a similar radical search
tool: http://www.jisho.org/kanji/radicals/

> but in this case I only use the radkfile and I disable radicals which won't lead to anything.

Being limited to radicals is exactly what I wanted to avoid - radicals
are good for classification, or for making an index for a book - but
as far as searching is concerned, they are very limited. For instance,
皮 is not the radical for 彼, but using a computer I see no reason why I
should not be able to access it from it. All the same, I should be
able to find 蘯 through 艹, 氵 and 日. I don't really learn Japanese
through textbooks, so I don't know much about radicals - and very
often fail at finding a kanji using them.

> 1) how did you select the root components

Expressing SQL in English is not easy, but basically, they are the
components which are the parent of no other component in at least one
kanji.

> 2) how do you select the new components to display?  Do you display every one of those which can be combined into a target kanji (including some which are already a combination of simpler components)

Yes, although the plan is to simplify the complements list by removing
those that are already covered by the current selection. For that I
need to add component coverage information into the database.

Selecting the new components is another rough line of SQL. It
basically says "select all components which are part of kanji that
contains all the components already selected", but it is more
complicated than it looks. I had to write a custom aggregation
function in SQLite to get it to work properly, and since I want to
treat radicals and their original kanji (e.g. 人 and 亻) the same, this
makes things even more complex. But thanks to SQLite, things go very
fast and the update is real-time.

Alex.

[Benoit Cerrina]

On Jan 2, 2010, at 6:28 PM, Alexandre Courbot wrote:

Hi Ben,

it is similar to the improvements I did to the radical search in the latest version of shinkanji:

Indeed - I think jisho.org also features a similar radical search
tool: http://www.jisho.org/kanji/radicals/

but in this case I only use the radkfile and I disable radicals which won't lead to anything.

Being limited to radicals is exactly what I wanted to avoid - radicals
are good for classification, or for making an index for a book - but
as far as searching is concerned, they are very limited. For instance,
皮 is not the radical for 彼, but using a computer I see no reason why I
should not be able to access it from it.

That works with my radical search

All the same, I should be
able to find 蘯 through 艹, 氵 and 日.

that didn't, I agree it should.

I don't really learn Japanese
through textbooks, so I don't know much about radicals - and very
often fail at finding a kanji using them.

1) how did you select the root components

Expressing SQL in English is not easy, but basically, they are the
components which are the parent of no other component in at least one
kanji.

It is clear, for me it would probably mean the components which are "simple" meaning not composed of other subcomponents. 

2) how do you select the new components to display?  Do you display every one of those which can be combined into a target kanji (including some which are already a combination of simpler components)

Yes, although the plan is to simplify the complements list by removing
those that are already covered by the current selection. For that I
need to add component coverage information into the database.

Selecting the new components is another rough line of SQL. It
basically says "select all components which are part of kanji that
contains all the components already selected", but it is more
complicated than it looks. I had to write a custom aggregation
function in SQLite to get it to work properly, and since I want to
treat radicals and their original kanji (e.g. 人 and 亻) the same, this
makes things even more complex. But thanks to SQLite, things go very
fast and the update is real-time.

That is because you are on a desktop.

My radical filtering which is probably simpler was running real time from the start on the simulator but was very slow on the device.

Now after several important optimizations (including dedicated additional tables) it is still not real time in the device although it is usable (1.5 to 2s) I had to make it run in a separate thread though to keep the UI responsive and add to had a little spinner...

Alex.

Benoit

[Neil Caldwell]

Have you thought about actually separating the radicals with tabs or
something instead of just sorting them?

I'm just wondering if that would make it easier to find radicals. Or
perhaps having the numbers that sort them hide and unhidden all the
radicals with that number of strokes.

Neil

2010/1/3 Benoit Cerrina <benoit...@gmail.com>:

[Alexandre Courbot]

> Have you thought about actually separating the radicals with tabs or
> something instead of just sorting them?
>
> I'm just wondering if that would make it easier to find radicals. Or
> perhaps having the numbers that sort them hide and unhidden all the
> radicals with that number of strokes.

Not sure, as for some combinations the number of complements is so
small that the right ones are easily spotted as-is. The display of the
complements is probably the thing that could be best improved though.

Alex.

[Alexandre Courbot]

> It is clear, for me it would probably mean the components which are "simple"
> meaning not composed of other subcomponents.

In an idea world, yes - unfortunately in KanjiVG some of the
components are not correctly split and doing so would render some
radicals inaccessible. But I agree this is how things should work.
Maybe if I simplify the KanjiVG model I import in Tagaini I could do
as you say.

> That is because you are on a desktop.
> My radical filtering which is probably simpler was running real time from
> the start on the simulator but was very slow on the device.
> Now after several important optimizations (including dedicated additional
> tables) it is still not real time in the device although it is usable (1.5
> to 2s) I had to make it run in a separate thread though to keep the UI
> responsive and add to had a little spinner...

Which made me wonder: what is the technology involved in the radical
search of ShinKanji? Are you using SQLite or a custom indexing
solution?

Alex.

[Benoit Cerrina]

SQLite,
I'll send you the structures of the tables. The issue is really that sqlite on an iphone is really slow compared to the same on the desktop + I have space restrictions, the db should not be too big.
Best regards
Benoit

[Alexandre Courbot]

> SQLite,
> I'll send you the structures of the tables.  The issue is really that sqlite on an iphone is really slow compared to the same on the desktop + I have space restrictions, the db should not be too big.

Yes please. Maybe there are things that can be optimized in your
indexes. Sometimes having a foreign eye looking at things can help.
But I guess performance on the iPhone will be limited by its amount of
memory.

Alex.

[Benoit Cerrina]

The main tables used are this 2

CREATE TABLE multirads (multirad_ucs_code INTEGER NOT NULL, stroke_count INTEGER NOT NULL, element_id INTEGER, ja_frequency INTEGER,  PRIMARY KEY (multirad_ucs_code));

CREATE TABLE character_multirads (character_id INTEGER NOT NULL, multirad_ucs_code INTEGER NOT NULL, PRIMARY KEY ( multirad_ucs_code, character_id));

CREATE INDEX character_multirads1 ON character_multirads(multirad_ucs_code);

Mostly I use the second which gives the list of radicals which are contained in any character.   The first one is used to define each radical to display the table listing them.

To perform a search I create a temp table:

CREATE TEMP TABLE IF NOT EXISTS t_rad_results (character_id INTEGER PRIMARY KEY,      ja_ucs_code INTEGER default NULL,   stroke_count INTEGER default NULL

I retrieve the list of results using a query built depending on the selection criteria:

the base statement is:

SELECT c.ja_ucs_code,  c.character_id FROM character_multirads m, characters c, element_glyphs g WHERE    m.multirad_ucs_code = ? AND g.glyph_variant = 1  AND c.ja_element_id  = g.element_id AND c.character_id = m.character_id  AND g.stroke_count = ? 

or 

SELECT c.ja_ucs_code,  c.character_id FROM character_multirads m, characters c, element_glyphs g WHERE    m.multirad_ucs_code = ? AND g.glyph_variant = 1  AND c.ja_element_id  = g.element_id AND c.character_id = m.character_id  AND g.stroke_count >= ? AND g.stroke_count <= ?

or 

with optionally 

AND ja_frequency > 0

the complete statement is this base statement repeated once per chosen radical and intersected with all the others + ORDER BY 2

that gives me a result which I then insert in the temp table using 

INSERT OR IGNORE INTO t_rad_results (ja_ucs_code,  character_id)

the temp table gives me the result of the radical query and I find all the radicals which should be enabled by using the query

SELECT m.multirad_ucs_code FROM  t_rad_results r, character_multirads m WHERE r.character_id = m.character_id

The process is  a bit complex as I have already tried to optimize it.  I didn 't initially use a temp table.

Moreover sqlite is configured such that the temp table is build in memory

Best regards

Benoit

[qborki]

Hello.

I find it hard to use the radical selector. Could you make it look more like http://jisho.org/kanji/radicals/? Which is IMHO almost perfect.
Main problems/requests:

1. Radicals are too small. Couldn't find a way to change size.
2. The window is also too small with no way to resize it. I absolutely don't want a scrollbar.
3. Please, do NOT hide radicals which won't lead to anything. Make them gray or something. It is a lot harder to find a radical if it doesn't keep it's position.
4. Couldn't find kozatohen (left village radical, 2 strokes). Some radicals seem to be missing.

And a minor request about UI layout. Could you move radical/component selector to a separate pane? The popup is quite annoying. Besides I don't think you really need both selectors. A radio or a checkbox should suffice.

Thank you.