Using an algorithm to identify all kanji components and phonetics
karl rosvold
component information in KanjiVG.
Matthew's IDSGrep which he mentioned several days ago can find kanji
by specifying that a particular component exists in a particular
position within a kanji.
For example the following kanji all appear to have 菐(83D0) as their
component on the right:
撲(50D5)僕(64B2)樸(6A38)璞(749E)蹼(8E7C)
And 濮(6FEE) is made up of the water radical on the left 氵(6C35)and
僕(64B2) on the right, which can further be broken down as above.
Of course KanjiVG draws all of these characters just fine, but When
actually looking at the KanjiVG files for these characters, however,
only these two characters: 撲(50D5), 僕(64B2) have the component:
菐(83D0) marked. 濮(6FEE) has 僕(64B2) marked, but within that component
the further division into 亻(4EBB) and 菐(83D0) are not marked.
While all of these characters have a phonetic component, it is only
marked in 撲(50D5)僕(64B2). I would actually argue that the phonetic of
濮(6FEE) is 僕(64B2) , not 菐(83D0) because the structure of kanji is 1
semantic component + 1 phonetic component, where the phonetic
components can in principal be taken from the entire set of kanji.
These characters are just one example. I shudder at the idea of
looking through 6000+ SVG files to look for the presence or absence of
component and phonetic notation. Would it be possible to make an
algorithm that looks for collections of strokes that fit the pattern
of particular components but which aren't marked? I have my own
phonetic information for approximately the same 6000+ kanji covered by
KanjiVG. Perhaps this can be incorporated into the KanjiVG data, or at
least a separate complementary file made which gives lists of kanji
and their phonetics.
Does anybody have any ideas or feedback?
Karl
msk...@ansuz.sooke.bc.ca
> ǎ(83D0) marked. �(6FEE) has �W(64B2) marked, but within that component
> the further division into ��(4EBB) and ǎ(83D0) are not marked.
To clarify: � (U+6FEE) is marked up as a left and right combination of
��(U+6C35) and �W (U+64B2), but then �W (U+64B2) inside � (U+6FEE) is
marked up as a left-right combination of ��(U+4EBB) and something
anonymous; whereas �W (U+64B2) in its own entry is a left-right
combination of ��(U+4EBB) and ǎ (U+83D0). The inconsistency is that �W
isn't described the same way in the different contexts; ǎ isn't always
marked when it occurs. As a consequence, a search for "everything
containing ǎ" doesn't return �.
> marked in ��(50D5)�W(64B2). I would actually argue that the phonetic of
> �(6FEE) is �W(64B2) , not ǎ(83D0) because the structure of kanji is 1
That may actually be why ǎ isn't marked inside �, if someone thought "we
will mark one phonetic and one semantic component"; but if that's the
policy, it wouldn't explain why ��(U+4EBB) *is* marked inside �. I would
advocate marking things wherever it's possible to do so, even when (as
here) it means marking some things that are neither semantic nor phonetic
components of the entire top-level kanji.
I think some even more drastic inconsistencies may occur in the database -
for instance, the same three-component stack split up as (A+B)+C in one
place and A+(B+C) in another - but those are by nature very hard to search
for.
A similar issue, which I mentioned recently to Alexandre off-list, has to
do with consistent use of the kvg:position attribute. It makes sense to
me that "left" and "right" are a pair that go together - when you use one,
you use the other - and "top" and "bottom" are a pair that go together in
the same way; you don't mix the left/right with the top/bottom pair; and
when you use any of those four things it's always one child of a group
that has exactly two children, the other being the matching position type.
All those rules are violated in the database.
Consider a kanji like �H (U+50C5). That appears to me to be a left-right
combination of ��(U+4EBB) and something which has no code point; the thing
on the right is in turn a top-bottom combination of ܳ (U+8279) and
something else resembling but not identical to �� (U+91CC). (I think Karl
told me that the top of the right may derive from إ (U+5EFF) instead, but
KanjiVG marks it as U+8279 and that looks right in the font my terminal
window uses.)
Instead, the 20111029 version of the database describes �H as having three
components all at the same level: one "left," one "right," and one
"bottom." The "right" component is further subdivided into just one
piece, which is tagged as "top." This all seems to be a change from the
previous version, in which there were a "left," a "top," and a "bottom"
all at the same level, with no "right" at all. Neither is easy to
interpret. The kanji's actual structure is left and right at the highest
level, and then the right further subdivided into top and bottom; for
IDSgrep, which needs that structure, I can sort of patch around this kind
of thing by adding ad-hoc rules to recognize it and translate it into a
consistent tree language, but since the rules for when right, left, top,
and bottom can be used together don't seem to be documented (if there are
any rules consistently followed at all) any such solution is going to be
fragile. For �H it may be a simple bug - the top-level "bottom" component
should be moved inside the "right", at which point it will match with the
existing "top" component there - but there are other kanji with similar
issues where it looks like it may be more deliberate.
It'd be easy to generate a list of all the kanji containing this kind of
conflict. We could get close just by skimming the output of my
KanjiVG-to-EIDS converter, which generates a complaint on every record it
doesn't understand. Unmarked components (like the missing ǎ in �) seem
a lot harder to catch automatically because it amounts to handwriting
recognition.
--
Matthew Skala
msk...@ansuz.sooke.bc.ca People before principles.
http://ansuz.sooke.bc.ca/
karl rosvold
I think what you're saying is that kanji which have more than one
component should be described as a spatial relationship with a
determined number of arguments, with kanji components filling in those
arguments. However, the element fitting in the place for an argument
can itself have a breakdown. My apologizes if my terminology isn't
very precise. My point (I think your point) is that any given spatial
relationship takes a fixed number of arguments (no more, no less) and
they can be nested.
If that's what you're saying, I completely agree, and if it is
possible to make all KanjiVG data conform to this paradigm, I agree.
If someone can think of an example of a kanji that doesn't fit this
description, please mention it, and I can look at what I've done in my
own kanji database.
It would be too time consuming to look through all of the SVG files,
but if Matthew can generate some kind of visual list of components and
their spatial relationships as they are listed in the database, I
volunteer to go through the kanji in the database and verify that
their structures are correct. It would be even better if you sorted
the list into "seems correct" and "seems incorrect."
Similarly, if a list of kanji and their phonetics can be organized by
phonetic, I can go through and verify that the phonetic data is
correct as well.
2012/2/27 <msk...@ansuz.sooke.bc.ca>:
> On Mon, 27 Feb 2012, karl rosvold wrote:
>> 菐(83D0) marked. 濮(6FEE) has 僕(64B2) marked, but within that component
>> the further division into 亻(4EBB) and 菐(83D0) are not marked.
>
> To clarify: 濮 (U+6FEE) is marked up as a left and right combination of
> 氵(U+6C35) and 僕 (U+64B2), but then 僕 (U+64B2) inside 濮 (U+6FEE) is
> marked up as a left-right combination of 亻(U+4EBB) and something
> anonymous; whereas 僕 (U+64B2) in its own entry is a left-right
> combination of 亻(U+4EBB) and 菐 (U+83D0). The inconsistency is that 僕
> isn't described the same way in the different contexts; 菐 isn't always
> marked when it occurs. As a consequence, a search for "everything
> containing 菐" doesn't return 濮.
>
>> marked in 撲(50D5)僕(64B2). I would actually argue that the phonetic of
>> 濮(6FEE) is 僕(64B2) , not 菐(83D0) because the structure of kanji is 1
>
> That may actually be why 菐 isn't marked inside 濮, if someone thought "we
> will mark one phonetic and one semantic component"; but if that's the
> policy, it wouldn't explain why 亻(U+4EBB) *is* marked inside 濮. I would
> advocate marking things wherever it's possible to do so, even when (as
> here) it means marking some things that are neither semantic nor phonetic
> components of the entire top-level kanji.
>
> I think some even more drastic inconsistencies may occur in the database -
> for instance, the same three-component stack split up as (A+B)+C in one
> place and A+(B+C) in another - but those are by nature very hard to search
> for.
>
> A similar issue, which I mentioned recently to Alexandre off-list, has to
> do with consistent use of the kvg:position attribute. It makes sense to
> me that "left" and "right" are a pair that go together - when you use one,
> you use the other - and "top" and "bottom" are a pair that go together in
> the same way; you don't mix the left/right with the top/bottom pair; and
> when you use any of those four things it's always one child of a group
> that has exactly two children, the other being the matching position type.
> All those rules are violated in the database.
>
> Consider a kanji like 僅 (U+50C5). That appears to me to be a left-right
> combination of 亻(U+4EBB) and something which has no code point; the thing
> on the right is in turn a top-bottom combination of 艹 (U+8279) and
> something else resembling but not identical to 里 (U+91CC). (I think Karl
> told me that the top of the right may derive from 廿 (U+5EFF) instead, but
> KanjiVG marks it as U+8279 and that looks right in the font my terminal
> window uses.)
>
> Instead, the 20111029 version of the database describes 僅 as having three
> components all at the same level: one "left," one "right," and one
> "bottom." The "right" component is further subdivided into just one
> piece, which is tagged as "top." This all seems to be a change from the
> previous version, in which there were a "left," a "top," and a "bottom"
> all at the same level, with no "right" at all. Neither is easy to
> interpret. The kanji's actual structure is left and right at the highest
> level, and then the right further subdivided into top and bottom; for
> IDSgrep, which needs that structure, I can sort of patch around this kind
> of thing by adding ad-hoc rules to recognize it and translate it into a
> consistent tree language, but since the rules for when right, left, top,
> and bottom can be used together don't seem to be documented (if there are
> any rules consistently followed at all) any such solution is going to be
> fragile. For 僅 it may be a simple bug - the top-level "bottom" component
> should be moved inside the "right", at which point it will match with the
> existing "top" component there - but there are other kanji with similar
> issues where it looks like it may be more deliberate.
>
> It'd be easy to generate a list of all the kanji containing this kind of
> conflict. We could get close just by skimming the output of my
> KanjiVG-to-EIDS converter, which generates a complaint on every record it
> doesn't understand. Unmarked components (like the missing 菐 in 濮) seem
> a lot harder to catch automatically because it amounts to handwriting
> recognition.
> --
> Matthew Skala
> msk...@ansuz.sooke.bc.ca People before principles.
> http://ansuz.sooke.bc.ca/
>
> --
> You received this message because you are subscribed to the "KanjiVG" group.
> For options and unsubscribing, visit this group at
> http://groups.google.com/group/kanjivg
msk...@ansuz.sooke.bc.ca
> I think what you're saying is that kanji which have more than one
> component should be described as a spatial relationship with a
> determined number of arguments, with kanji components filling in those
> arguments. However, the element fitting in the place for an argument
> can itself have a breakdown. My apologizes if my terminology isn't
> very precise. My point (I think your point) is that any given spatial
> relationship takes a fixed number of arguments (no more, no less) and
> they can be nested.
Yes - that kind of consistency is necessary for it to be represented in
Unicode IDS format (which defines a strict list of the "spatial
relationships" it will allow and how many arguments each one takes) or
any similar system, such as my own EIDS, or HanGlyph from Wong et al.
> If that's what you're saying, I completely agree, and if it is
> possible to make all KanjiVG data conform to this paradigm, I agree.
> If someone can think of an example of a kanji that doesn't fit this
> description, please mention it, and I can look at what I've done in my
> own kanji database.
KanjiVG's desire to represent stroke order by the order of tags in the XML
file creates issues because it's not possible to represent BOTH the stroke
order AND the containment structure by means of the XML tag sequence.
For example, 锟紷 (U+5712). That is represented in KanjiVG as three parts:
First part: 锟斤拷, general radical, no position
Second part: 袁, phonetic (internally subdivided into a top and a bottom)
Third part: 锟斤拷, general radical, position="kamae"
Unicode and IDSgrep would represent this as a "surround" relation with the
outside equal to 锟斤拷 and the inside equal to 袁. (But, by the way, note
that the inside does not actually look exactly like 袁 and it's not marked
as a variant.) So it's necessary to change the three-argument
relationship into a two-argument relationship when we want to process it
spatially; but KanjiVG's mandate of representing stroke order requires the
database to keep the two parts of 锟斤拷 separate and in the sequence shown.
KanjiVG usually marks this kind of case using the "kvg:part" attribute -
the first occurrence of 锟斤拷 is "part 1" and the second is "part 2." It
doesn't seem to be hard to look for that attribute and combine any such
split elements. I'm not clear on what happens if that syntax is used
twice inside the same character, but that seems to be rare.
Similar issues arise any time the stroke order does not match the
containment hierarchy - for instance, in 锟斤拷, which KanjiVG represents as
four parts: the middle stroke, the 锟斤拷 on the left, the 锟斤拷 on the right,
and then the remainder of 山. None of them has a specified "position"
attribute. Even if we combine the two parts of 山 we're still left with a
three-argument spatial relationship that has no name.
> It would be too time consuming to look through all of the SVG files,
> but if Matthew can generate some kind of visual list of components and
> their spatial relationships as they are listed in the database, I
I can probably do something like that. We should talk, maybe off-list,
about just what you'd like to see in such a "visual list"; if I try to
represent all the data that's in the XML file, then it'll basically just
be a printout *of* the XML file, whereas if I filter out some fields then it
won't be much help in debugging the information in those fields. At some
point, once issues are identified, someone is going to have to read the
raw XML, but that doesn't necessarily have to be you if some more "visual"
format would be helpful in identifying the issues.
Alexandre Courbot
Lot of interesting discussion taking place lately!
On Feb 27, 2012 9:39 PM, <msk...@ansuz.sooke.bc.ca> wrote:
> To clarify: 濮 (U+6FEE) is marked up as a left and right combination of
> 氵(U+6C35) and 僕 (U+64B2), but then 僕 (U+64B2) inside 濮 (U+6FEE) is
> marked up as a left-right combination of 亻(U+4EBB) and something
> anonymous; whereas 僕 (U+64B2) in its own entry is a left-right
> combination of 亻(U+4EBB) and 菐 (U+83D0). The inconsistency is that 僕
> isn't described the same way in the different contexts; 菐 isn't always
> marked when it occurs. As a consequence, a search for "everything
> containing 菐" doesn't return 濮.
You pointed out one of the biggest issue at the moment in KanjiVG: data redundancy. Every time a component is included, its structure information is repeated again and sometimes may not match, either on purpose or by mistake.
When this is on purpose, this should be because we have a special case of a variant. In this case, the group tag should specify the variant keyword, and the structure information be specified.
But for the vast majority of the cases, where a component is structurally identical to its standalone kanji, why repeat the grouping again? The structure should just be obtained by referencing the kanji file describing the component (or just specifying the kanji itself) and only the stroke paths should be present.
Actually in Tagaini this is exactly the approach that I have: every component is assumed to have a file of its own (which is mostly true - only a few components are missing), and the structure information is always taken from it. It takes much less space to store, and is exact most of the time.
What we are clearly in need for here is the following:
* A clear DTD to specify consistent XML rules (KanjiVG should be a subset of SVG).
* Rules to define when a component's information should be repeated and when it should not (presence of the "variant" attribute?)
* In general, *written rules*. Unfortunately my understanding of kanji is too limited for me to produce them. We need an expert like Ulrich for that.
> A similar issue, which I mentioned recently to Alexandre off-list, has to
> do with consistent use of the kvg:position attribute. It makes sense to
> me that "left" and "right" are a pair that go together - when you use one,
> you use the other - and "top" and "bottom" are a pair that go together in
> the same way; you don't mix the left/right with the top/bottom pair; and
> when you use any of those four things it's always one child of a group
> that has exactly two children, the other being the matching position type.
> All those rules are violated in the database.
I think in most (all) of the cases this can be workarounded by assuming the missing component when meeting this pattern. After all, kanji do not have that many different structures so there is not much room for mistake.
But I agree that ultimately a format update is in order. We could have a group with an IDS attribute containing the appropriate number of sub-groups. I don't know if that could be enforced by the DTD - if it could, that would be best.
> It'd be easy to generate a list of all the kanji containing this kind of
> conflict. We could get close just by skimming the output of my
> KanjiVG-to-EIDS converter, which generates a complaint on every record it
> doesn't understand.
Feel free to have such a list generated - I also wonder if this could be accurately fixed by a script
> Unmarked components (like the missing 菐 in 濮) seem
> a lot harder to catch automatically because it amounts to handwriting
> recognition.
Right. On the other hand, having a list of kanji with missing components would also be a good start. The amount cannot be that insane.
Don't hesitate to clone the git repo and to give a shot to any of your ideas! ;) I would *love* to see one of these issues addressed in a satisfactory way.
Alex.
msk...@ansuz.sooke.bc.ca
> But for the vast majority of the cases, where a component is structurally
> identical to its standalone kanji, why repeat the grouping again? The
> structure should just be obtained by referencing the kanji file describing
> the component (or just specifying the kanji itself) and only the stroke
> paths should be present.
As I understand it, SVG doesn't support any feature that would allow a tag
to say "now include this file, scaled and transformed in so-and-so way."
Is that right? If so, then it seems like the commitment to SVG format
forces you to include the entire kanji in each record instead of referring
to parts of it elsewhere.
However, I suppose that *even with the current data* a smart reader that
wasn't forced to read it as SVG, would be free to chase the kvg:element
fields; as soon as it sees one, it would go look in the relevant record
and ignore whatever structural information appears inside the current
tag. I guess that's what you mean in the below:
> Actually in Tagaini this is exactly the approach that I have: every
> component is assumed to have a file of its own (which is mostly true - only
> a few components are missing), and the structure information is always
> taken from it. It takes much less space to store, and is exact most of the
> time.
> * In general, *written rules*. Unfortunately my understanding of kanji is
> too limited for me to produce them. We need an expert like Ulrich for that.
Yes, I'd say that is a priority. I fear that the rules that end up being
written down won't be the ones most convenient for my own purposes, but
any clear documentation that would allow me to say "this structure is
WRONG" instead of just "this structure is WEIRD" would be a big win.
> I think in most (all) of the cases this can be workarounded by assuming the
> missing component when meeting this pattern. After all, kanji do not have
> that many different structures so there is not much room for mistake.
Statements like this worry me, especially in the absence of written rules,
because they mean that some information the file format could provide will
instead bey supplied by the reader's guesses, and there's no way of
knowing that two different readers will guess the same. If the file says
"A - left; B - top; C - no position specified", does that mean (excuse my
EIDS) [lr]A[tb]BC, or [tb]B[lr]AC? Are we supposed to guess from stroke
order? We've already established that stroke order is sometimes
unpredictable, does not necessarily follow layout, and even experts may
disagree on it if they just follow their intuitive feelings of what works
without having a defined standard. If there's any possibility of writing
the XML in an unambiguous way, I think that's preferable.
> But I agree that ultimately a format update is in order. We could have a
> group with an IDS attribute containing the appropriate number of
> sub-groups. I don't know if that could be enforced by the DTD - if it
> could, that would be best.
I don't think any format changes are necessary if the use of the current
format could be better disciplined. A DTD could certainly enforce this
stuff if we could define our own tag names for it, but I don't know if
it could do it while retaining SVG compatibility - and attempts to also
retain stroke order might force further compromise.
> > It'd be easy to generate a list of all the kanji containing this kind of
> > conflict. We could get close just by skimming the output of my
> Feel free to have such a list generated - I also wonder if this could be
> accurately fixed by a script
I'll put it on my to-do list..
> > Unmarked components (like the missing ǎ in �) seem
> > a lot harder to catch automatically because it amounts to handwriting
> > recognition.
>
> Right. On the other hand, having a list of kanji with missing components
> would also be a good start. The amount cannot be that insane.
There are a *lot* of what I'd be inclined to call "anonymous nodes" in
this database, and my first thought was that I'd be impressed if the
average was less than one per kanji. A quick scan of the XML shows 5996
"g" tags without "kvg:element" attributes, not counting the group at the
top level of each kanji. That is less than one per kanji, so I am
impressed. But it's not *much* less than one per kanji, and it's still a
lot of components without element attributes.
Many of these will be components that don't correspond to any defined code
point, for which it's not possible to give them an element attribute short
of defining special notation; but I don't know of a quick way to exclude
such cases without a human looking at them. Karl has gone ahead and
defined special notation for handling this kind of thing in his software;
my own project allows the nodes to remain anonymous, but imposes other
constraints that KanjiVG doesn't. Maybe the highest-priority ones to
consider should be the ones that have a phonetic value marked, of which I
count 159; those certainly seem like they should correspond to "elements."
> Don't hesitate to clone the git repo and to give a shot to any of your
> ideas! ;) I would *love* to see one of these issues addressed in a
> satisfactory way.
Although it's not exactly a solution to KanjiVG's own problems, I've got
code to work around some of them in my KanjiVG-to-EIDS converter, which
is included in IDSgrep.