parsing broke down
Sebastian Drude
in particular those who have helped me in the past and have access to the Aweti-Parsing project on the LanguageDepot (awe-flex) (Kevin, Andy, Ann, Andreas, Jean Paul):
Jean Paul Gotopo Maldonado
--
"FLEx list" messages are public. Only members can post.
flex_d...@sil.org
http://groups.google.com/group/flex-list.
---
You received this message because you are subscribed to the Google Groups "FLEx list" group.
To unsubscribe from this group and stop receiving emails from it, send an email to flex-list+...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/flex-list/fa78c5fa-112e-4e71-8f1a-42de6f95770an%40googlegroups.com.
Sebastian Drude
I am not sure, though, that I understand your proposal. When you write "in my FLEX"; do you mean in a project of yours in the LanguageDepot? What is "change control in FLEx"?
Andy Black
The zãᵀ entry has several affix process rules which have an empty position. For example:
Try clicking in the empty spot above 2 and pressing the delete key if it is not supposed to be there. If it is, then set it to a natural class or phoneme or whatever it should be.
The other error messages in Try a Word point to similar entries (two of them are in variants, so you'll have to find those and fix them).
The ∅ entries also do not need to have an affix process rule. Just use a plain affix form and constrain them with an appropriate environment. I'm also a bit puzzled as to why some of the null entries have one allomorph before/after a vowel and the other one is before/after a consonant...
If you make these changes, do things come out better?
--Andy
sebadr...@gmail.com
Dear Andy, all,
A quick follow-up and one further question.
Thanks, Andy, for replying and pointing this out.
I was aware of these empty positions. They crept in when, perhaps with some update of FLEX?, some natural classes I used in my phonological and affix rules where substituted by “automatically generated class for… [some rule]”, which I deleted, which made the slots go empty.
Still, these empty slots and the errors they generated were not responsible for the errors that I reported here a few weeks ago, because some parses worked despite them.
Just a follow-up: With a lot of work, I was able to make the parsing of my FLEX project functional again. In some cases, I had to delete a lexical entry and re-create it with apparently exactly the same content. I also discovered that, apparently, the ordering of affix rules is also significant, not only the ordering of regular allomorph forms restricted to environment.
The only remnant that is still there from the “crisis” (that I am aware of) are a number of spurious entries which show up when I ask for seeing all detailed steps in parsing, entries which are described as “Automatically generated null affix for the […] irregularly inflected form”. I do not know why these entries were created, nor how to access them and less so how to delete them. Can you give me a hint here? I suspect they slow the parsing down.
Thanks again for your support,
Sebastian
Andy Black
I'm glad you got things working!
As for those automatically generated null affixes, these come from your irregularly inflected forms which have the Slots field filled out:
- 3rd person
- 2 singular imperative
- 2 plural imperative
The issue here is that these irregularly inflected forms already include the inflection from the slots mentioned. FLEx generates a null for these so that any slots mentioned will have something to fill them when processing inflectional templates containing those slots. Perhaps there is another way to handle this situation, but that is what FLEx currently does.
--Andy
sebadr...@gmail.com
Thanks a lot, Andy!
IAs to the null affixes – I see. They were not there a few months back, so it seems in one of the updates the parsing mechanism was altered in some way.
I hope it works from now on…
Thanks again for your support!
Sebastian
Sebastian Drude
- when creating something new from the "analysis" tab in the Texts and Words area, I often get the message "Can only merge two newly-created units of work", for no evident reason and no apparent consequences.
- the option "parse current word" does not seem to work at all (or takes to long for me to wait for the result)
- even when I manually provide the morpheme breaks, the glosses offered for the individual affixes are weird, including entries/senses which are not compatible with the stem (nominal prefixes fith verbal stems or similar)
Andy Black
I played with the form ekozokotu to see what might be done to improve parser performance. This form did parse but it took about 9 minutes on my machine.
Making the following changes brought it down to about 100 seconds:
First, there is duplication among the three null entries. For example, ∅ t.v has two affix process rules for the null: one before a consonant and one before a vowel. Since this boils down to just a plain null, I changed the Lexeme Form from being an affix process rule to just a plain form and deleted the second affix process rule. It now just has
Then both of the other two null entries (0.pers and n.rel) have a similar situation along with two other affix process rules. One of these other affix process rules specifies the glottal stop overtly. Since glottal stop is already in [C], there should not be a reason to have a separate rule for it. The second other affix process rule appears to find a glottal stop and delete it:
So I changed all of these to two: the lexeme form is now
and the other rules (except for the one that deletes a glottal stop) are deleted.
So that removed some duplicated forms from these entries, making the parser have less to look for.
The other thing I tried was for the ll red entry. Since the Active verb category does not have any inflectional templates (but its two subcategories do) and since no lexical roots/stem are marked as active verb, there is no need for have any Active verb > Active verb mappings. You can just use the Stative verb > Stative verb, Intransitive verb > Intransitive verb, and Transitive verb > Transitive verb mappings.
Doing these things made ekozokotu parse in about one minute and 40 seconds. Not great, but much better than 9 minutes...
--Andy
--
"FLEx list" messages are public. Only members can post.
flex_d...@sil.org
http://groups.google.com/group/flex-list.
---
You received this message because you are subscribed to the Google Groups "FLEx list" group.
To unsubscribe from this group and stop receiving emails from it, send an email to flex-list+...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/flex-list/6c76ba94-b3ab-40ed-8871-5530c3ffb76bn%40googlegroups.com.
Michael Maxwell
Are you certain there is a truncation affix allomorph (the glottal stop
deletion rule)?
I ask because morphological truncation affixes are vanishingly rare, and
deleting a word-initial glottal stop seems especially odd:
pre-glottalized consonants in word-initial position are rare (in
Tzeltal, for example, the only pre-glottalized consonant is /'t/, and it
occurs only in medial position), and the distinction between
vowel-initial and glottal+vowel initial words is seldom phonemic, I think.
Optional phonological deletion (or epenthesis) rules for word-initial
glottal stops are probably more common, although in that case it's
debatable whether the glottal stop is really phonemic, at least in that
position. IIRC, Tagalog vowel-initial words are really glottal+vowel
initial, but the glottal is not written in the orthography.
So I'm just checking that there is a genuine meaning difference (i.e. an
affix) between words with and without an initial glottal stop. If not,
then this might be a phonological rule rather than a morphological rule.
BTW, the effect of the glottal deletion rule is that the parser attempts
for every word to find both a glottal-initial stem and a non-glottal
initial stem, which more or less doubles the search space and time. If
that's the correct analysis, then you'll have to put up with it, but if
it's not, then Andy's 100 second parse time might be cut to 50 seconds.
Another possibility is that the glottal-deletion only happens before a
vowel (if there are no stems with pre-glottal word-initial consonants).
If so, then the morphological rule could be
' V X ==> 2 3
1 2 3
which would narrow the search space. The two rules--one with and one
without an explicit V--would be the same in practice, although the V
would in some sense be superfluous.
Mike Maxwell
On 7/5/2024 4:54 PM, Andy Black wrote:
> Hi, Sebastian.
>
> I played with the form ekozokotu to see what might be done to improve
> parser performance. This form did parse but it took about 9 minutes on
> my machine.
>
> Making the following changes brought it down to about 100 seconds:
>
> First, there is duplication among the three null entries. For example,
> ∅ t.v has two affix process rules for the null: one before a consonant
> and one before a vowel. Since this boils down to just a plain null, I
> changed the Lexeme Form from being an affix process rule to just a plain
> form and deleted the second affix process rule. It now just has
>
>
> similar situation along with two other affix process rules. One of
> these other affix process rules specifies the glottal stop overtly.
> Since glottal stop is already in [C], there should not be a reason to
> have a separate rule for it. The second other affix process rule
> appears to find a glottal stop and delete it:
>
>
>
>
>> newly-created units of work", for no evident reason and no
>> apparent consequences.
>> entries/senses which are not compatible with the stem (nominal
>> prefixes fith verbal stems or similar)
>>
>> And besides sometimes freezing while working on a parsing, now the
>> project has crashed FLEX unexpectedly a number of times.
>>
>> I have uploaded the latest version of the database to the language
>> depot. If you wizards could have a look, or tell me how to determine
>> what may be causing the different, very slow behavior of the parser, I
>> would be very grateful.
>>
>> Many thanks in advance, as so often...
>> Sebastian
>>
>> --
>> "FLEx list" messages are public. Only members can post.
>> flex_d...@sil.org
>> http://groups.google.com/group/flex-list.
>> ---
>> You received this message because you are subscribed to the Google
>> Groups "FLEx list" group.
>> To unsubscribe from this group and stop receiving emails from it, send
>> an email to flex-list+...@googlegroups.com.
>> To view this discussion on the web visit
> --
> "FLEx list" messages are public. Only members can post.
> flex_d...@sil.org
> http://groups.google.com/group/flex-list
> You received this message because you are subscribed to the Google
> Groups "FLEx list" group.
> To unsubscribe from this group and stop receiving emails from it, send
> an email to flex-list+...@googlegroups.com
Sebastian Drude
Dear Andy, Mike, all,
thanks so much for your very helpful replies, again.
My answers, and open problems, below here:
The question about the (optional) glottal stop deletion rule has the following background:
Glottal stops are phonemic in Awetí and only occur before vowels, mostly between vowels, as they often get elided when in a (rare) consonantal encounter.
There are many minimal pairs of stems one of which begins with the glottal stop, the other one without, such as /ok/ 'house' vs. /?ok/ <'ok> 'tuber'. They take different prefix-allomorphs (cf. tok 'his house' vs. i'ok 'its (the plant's) tuber').
Now, word initially, the glottal stop is often dropped, at least in writing. So some may write 'ok 'tuber (in general)', but others write ok, which becomes ambiguous, also meaning 'house (in general)'.
The two rules were my attempt at formulating this optional rule.
I tested that many months ago, and it seemed to work. My problem
with using a phonological rule is that the rule cannot be optional
(I asked this group about optional "phonological" (in fact, rather
orthographic) rules), which means that the variant spelling <'ok>
would not be recognized. The implementation as an affix also has
the advantage that I can mark the resulting form as
"not-relative", which means that it can not be combined with
prefixes. I need that category independently.
It is certainly a very good idea to reduce the search space by
using ' [V] X as the input.
I followed your, Andy, other hints with reducing the in some
cases spurious (leftover from earlier formulations) empty-affix
rules. Thanks for checking this so thoroughly.
To my surprise, indeed, leaving only the glottal stop deletion
rule explicit still does make it optional, as the general empty
affix rule apparently also holds (which I find weird).
With your suggested changes, parsing has indeed improved somewhat, but other aspects do still not work. For instance, the form tupejat 'the one who is staying' composed of t- up -eju -at does not parse at all; I tried letting it run for hours and hours. Not even toat (to -at)
'the one who goes/went' seems to be parsing, at least not in a
reasonable time (I gave up after 5 minutes). I believe that an initial
or and/or final t make the parsing extremely slow, but I do not know how to fix that.
What I also see and do not understand is how empty affixes are dealt
with by the 'try a word' procedure, as the outcome is quite
heterogeneous, in some cases it appears explicitly, in others not.
Take the two forms ok and 'ok, which I mentioned earlier.
Here are the results for ok:
The output is correct, in principle: besides the absolute form of ok 'house' (the third parse), it can be the unmarked-for-person form of both 'ok and ok,
and the parsing details show that
the empty 0.pers-prefix has been applied also in the case of the first
parse, but there it is not shown in the results (which is what I prefer,
actually), but is made explicit in the second parse. Why?
(There could also be a parse recognizing that the form could also be the
spelling variant without the glottal stop of the absolute (prefix-less)
form of 'ok, but I have not formulated any rule for that
(because that would imply formulating either a phonological rule, which
can not be optional, or an empty affix rule, but that form is
affix-less, which is exactly the point).
Here is the output for 'ok:
This is also correct, the form can be analyzed as the absolute form, or as the unmarked-for-person relative form, and that is what is happening underneath, as the parsing details show. They do not show the empty prefix in the results themselves (which is what I prefer), but why are these then shown in the case of the second parse of ok?
So these are the open questions. The latter ones are more for my understanding of how the parsing works internally, but the not parsing of tupejat or toat is indeed a problem.
Thanks again in advance for your help.
Best, Sebastian
Sebastian Drude
Thanks again to all who will have a look at this project.
Andy Black
Looking through the data, I notice that there are several affixes which have both allomorphs and variant forms where the variant forms are the same as the allomorphs. Since both the main entry (and all its allomorphs) and the variant forms are tried, this produces double work any time any of the these forms is found. Is there a reason why you need affixes to have variant forms?
Also, try using environments to try and restrict forms as much as possible. This is especially important for any null forms. If these happen to only occur word initially, then add that to the environment. For example, something that is word initial and before a vowel would have an environment of
/# _ [V]
Whenever possible use allomorph forms instead of affix process rules. This is especially the case when something can only occur word initially or finally because then you can set the environment to say so. This can greatly reduce the work of the parser.
When you have variation (like some of the glottal initial forms), you can add a form with the same environments as the form that does not have the variation. One place to do this is with the 'tuber' entry. The conceptual intro document has this section that can help here:
3.1.4.1 Free Fluctuation
Sometimes a language has free fluctuation between two allomorphs of a morpheme. In such cases, one should create both allomorphs and condition them exactly the same way (in terms of environments and inflection classes). One should also order them one after the other. The FieldWorks Language Explorer default parser will try both forms in such cases.
Consider removing the null t.v. entry altogether. You can make the slot it's in optional I see that it does occur in an intransitive template that requires more derivation but couldn't you make the now optional nom slot be required for this case and leave the TematicV/Syl slot be optional?
Hopefully these will help improve processing time some. I got the toat parsing to be down to about 20 seconds by doing some of these...
--Andy
To view this discussion on the web visit https://groups.google.com/d/msgid/flex-list/ade4101f-18d8-45f6-9593-5a1ab65253e6n%40googlegroups.com.
Sebastian Drude
Thanks a lot indeed for your time and effort, Andy.
Really much appreciated.
A general comment: I am trying to model the linguistic analysis
that I believe to be accurate for Awetí in FLEX, with some
necessary adaptions. For one thing, I actually do not believe in
0-morphemes (just in affix-less forms), but in order to implement
situations where affixes only occur with a certain class of stems
or only in forms that belong to a certain category (inflection
feature), while other forms do not have any affix, I understand
that 0-morphemes are necessary. If I can do without, I am all in
for that.
To your individual observations and suggestions:
Looking through the data, I notice that there are several affixes which have both allomorphs and variant forms where the variant forms are the same as the allomorphs. Since both the main entry (and all its allomorphs) and the variant forms are tried, this produces double work any time any of the these forms is found. Is there a reason why you need affixes to have variant forms?
Usually, the variants are marked as "abstract forms" and are, if I understood that correctly, not considered by the parser. They are there in order to appear in the dictionary, so that somebody who does not know the language and comes across such a different allomorph has a chance to find the relevant main entry with the main headword.
Am I wrong assuming that "abstract forms" are ignored by the parser?
Also, try using environments to try and restrict forms as much as possible. This is especially important for any null forms. If these happen to only occur word initially, then add that to the environment. For example, something that is word initial and before a vowel would have an environment of
/# _ [V]
Fine, that's a good hint. I will
implement that for all affixes which occur only at the very
beginning or end of words then.
Whenever possible use allomorph forms instead of affix process rules. This is especially the case when something can only occur word initially or finally because then you can set the environment to say so. This can greatly reduce the work of the parser.
Yes, fine. On the other hand, Mike
Maxwell said that in his experience, mixing in the same entry
allomorphs with environments and allomorph processes is usually
a bad idea, which is why I switched to use only processes in
entries where at least one process was needed. Would you advise
me to not do so?
One case I tested just now is one of
the alphabetically first affixes, -ap. When I use the
affix allomorph ap in the environment /[V]_ , then I
cannot have both toap and twap to be parsed,
which is what I need (besides inconsistencies of the different
transcribers, the rules which determine whether /o/ turns to /w/
are too complex to implement in FLEX, and they refer to the
position of the word accent, which is not written in Awetí
texts). The latter has to be done by a process X [ou] =>
1 w + ap. If I put the regular affix after/below that
process, only twap parses, not toap, and if I
put it above, the reverse holds. So have returned to indicating
two processes side by side, and indeed, then both forms parse.
When you have variation (like some of the glottal initial forms), you can add a form with the same environments as the form that does not have the variation. One place to do this is with the 'tuber' entry. The conceptual intro document has this section that can help here:
3.1.4.1 Free Fluctuation
Yes, thanks, I saw that. Problem with the glottal stop is that
it can be dropped only word-initially, not between prefix and
stem. If I indicate this to be free variation, all word forms of
'house' ok would incorrectly also parse as possible
variant forms of 'ok 'tuber'.
Consider removing the null t.v. entry altogether. You can make the slot it's in optional I see that it does occur in an intransitive template that requires more derivation but couldn't you make the now optional nom slot be required for this case and leave the TematicV/Syl slot be optional?
I will try that. I have checked, there are only very few
instances where the same root can be both, of the inflectional
class e-Verb (where the -e thematic vowel is required) and a
regular verb, and in most cases they are synonymous.
Perhaps I can also do without the 0-prefix for person, making the person slot for the modal forms optional (which comes closer to my analysis anyways). It would not capture that the absolute forms like mo 'hand' only occur without a possessor while the relative unmarked for person forms like po 'hand (of)' only occur after a possessor, but this would not be such a hindrance in parsing the texts.
I will do these tests and come back to you. Thanks again.
SebastianMichael Maxwell
> On the other hand, Mike Maxwell said that in his experience,
> mixing in the same entry allomorphs with environments and allomorph
> processes is usually a bad idea
probably slower than affix allomorphs. But afaik, there's no reason
they can't be mixed if need be, that is there's no reason a single affix
(morpheme) can't have both allomorphs and affix processes, although I
would guess that the need for that is rare.
BTW, my *experience* using this parser is slim. I wrote the original
Hermit Crab parser back in the 1990s, and it's been re-implemented by
someone else since then.
Mike Maxwell
Sebastian Drude
Perhaps I remember it wrong and you said something different, or
somebody different said that.
I have a few instances where such a mix would make sense.
Thanks again!
S
Andy Black
...
To your individual observations and suggestions:
Looking through the data, I notice that there are several affixes which have both allomorphs and variant forms where the variant forms are the same as the allomorphs. Since both the main entry (and all its allomorphs) and the variant forms are tried, this produces double work any time any of the these forms is found. Is there a reason why you need affixes to have variant forms?
Usually, the variants are marked as "abstract forms" and are, if I understood that correctly, not considered by the parser. They are there in order to appear in the dictionary, so that somebody who does not know the language and comes across such a different allomorph has a chance to find the relevant main entry with the main headword.
Am I wrong assuming that "abstract forms" are ignored by the parser?
From what I can tell, yes, abstract forms are ignored by the parser.
--Andy