Moving from Toolbox to FLEx?

[Friederike Kronauer]

I've been using Toolbox for a long time and now, finally, would like to move to FLEx (my dictionary and interlinear texts).  Unfortunately, I can't find any information about how to do it! Is there a tutorial or something similar? Can somebody help?

Also: is there anything specifically about the differences between Toolbox and FLEx and about how to start coming from Toolbox?

I would be really grateful!!!

[Françoise Rose]

Hello,

to answer you best, it would be useful to know why you want to change: what you don’t like in toolbox, and what you expect to find in Flex.

I’ve gone through that process and would like to underline two basic facts:

1)      in this process of shifting from Toolbox to Flex, you will lose all your morphological analysis and will have to redo it all (these took me several years, one hour per day).

2)      as I’ve come to understand (only once my data was in Flex), most Flex users do not use an automatic parse. The automatic parse through the Hermit Crab Parser needs to enter a lot of complex parameters, that you can hopefully succeed to do (I haven’t), only when knowing well a language. The very positive aspect of Flex that makes the manual parse much faster is that Flex stores the analysis of words previously parsed, so you basically only have to accept or correct previous analyses for new occurrence of words already analyzed. Actually, there is a way to somehow do something similar in Toolbox, by creating lexicon of “words”.

If you find it necessary, we can discuss these options offline.

Best,

Françoise

 

De : 'Friederike Kronauer' via FLEx list <flex...@googlegroups.com>
Envoyé : mercredi 24 février 2021 08:31
À : FLEx list <flex...@googlegroups.com>
Objet : [FLEx] Moving from Toolbox to FLEx?

 

I've been using Toolbox for a long time and now, finally, would like to move to FLEx (my dictionary and interlinear texts).  Unfortunately, I can't find any information about how to do it! Is there a tutorial or something similar? Can somebody help?

Also: is there anything specifically about the differences between Toolbox and FLEx and about how to start coming from Toolbox?

I would be really grateful!!!

--
"FLEx list" messages are public. Only members can post.
flex_d...@sil.org
http://groups.google.com/group/flex-list.
---
You received this message because you are subscribed to the Google Groups "FLEx list" group.
To unsubscribe from this group and stop receiving emails from it, send an email to flex-list+...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/flex-list/1fef389c-7d96-4502-b52d-9f9596033ab5n%40googlegroups.com.

[Friederike Kronauer]

Thanks, Françoise!

Ok, to answer your question: the main reason why I'd like to use FLEx is, that I've heard it's possible to publish an online dictionary (even if it's still a work-in-progress). Apart from that, I'm quite happy with Toolbox! 

(I would also like to find out a way to "publish" the interlinear text, but so far, I don't have any idea how to do that.)

I'm not sure I can imagine what manual parsing in FLEx would look like. Do you have any example?

[Kevin Warfel]

Unless your data is fairly simple, importing from Toolbox into FLEx is not a trivial task. SIL's Dictionary & Lexicography Services has a team of specialists who do this as a paid service. You can get a no-obligation estimate by submitting the application found here: https://www.webonary.org/request-help-with-dictionary-conversion-into-flex/.

It is true that, once it's in FLEx, you can publish your dictionary data on Webonary.org as early and as often as you wish by clicking the File menu and choosing Upload to Webonary. (It is really that simple!) What Françoise has said is also true with regard to the disadvantages, especially concerning the interlinearization already done in Toolbox. (For this reason, if you are going to move your data from Toolbox to FLEx at some point, it is better to do it sooner rather than later, as the amount of work to potentially be redone will only increase.) Françoise's data includes a lot of variants, many of which are homographs; I believe that is one of the main reasons that it was so difficult for her to get the automatic parser to function effectively. She can correct me if I'm wrong.

fwiw,
Kevin Warfel
Associate Dictionary & Lexicography Services Coordinator
SIL International

[Claire Bowern]

If the main reason to convert to flex is to publish a dictionary, there are other (probably) ways to get data into publishable formats. For example, you can use Lexique Pro with toolbox files directly.

Claire

To view this discussion on the web visit https://groups.google.com/d/msgid/flex-list/ac74ccb8-df21-d97d-0ef4-6f9a198e656b%40sil.org.

--

Claire Bowern

Professor

Editor: Diachronica

Department of Linguistics, Yale University

she/her or they/them

[Friederike Kronauer]

Thanks a lot!

But now, I have more questions:

- How can I find out whether my dictionary is simple or not? What would make it complicated?

- What are the differences between Lexique Pro and Webonary?

- Is it possible to publish a work-in-progress with Lexique Pro?
- If I decide to try to move to FLEx, is there any information about how to do it?

- Would I have to interlinearize all the text again?

- I've never managed to use the formulas in Toolbox... Would that actually make it easier to move to FLEx? Because I wouldn't loose too much work?

[Kevin Warfel]

There are others who could answer your questions more competently than I, but I'll give them my best shot. See my responses in context below:

Kevin Warfel

On 2/25/2021 9:16 AM, 'Friederike Kronauer' via FLEx list wrote:

Thanks a lot!

But now, I have more questions:

- How can I find out whether my dictionary is simple or not? What would make it complicated? If your data has subentries (\se) that are related to a single sense (\sn) of the headword (\lx) rather than to the entry as a whole, that is potentially complicated. If there are subentries (\se) that have multiple senses (\sn), that is very complicated to import into FLEx and requires the application of a Perl script to restructure your Toolbox data prior to import. There are likely other things that would make the import complicated, but those are two that I am aware of. In general, the more "links" you have between entries in your data, the more complicated it is to convert it to another format without breaking those links. If your data consists only of independent entries (i.e., no cross-references, no lexical relations, no subentries), importing it into FLEx is more straightforward.

- What are the differences between Lexique Pro and Webonary? Lexique Pro is a program/app, so far as I know, while Webonary is a website: https://www.webonary.org/ that serves as an aggregator of lexical data from around the world.

- Is it possible to publish a work-in-progress with Lexique Pro? I know too little about Lexique Pro to answer this question.

- If I decide to try to move to FLEx, is there any information about how to do it? I'm not sure. FLEx Help files certainly have something, but it may be insufficient, depending on your data.

- Would I have to interlinearize all the text again? Yes. This is probably the primary deterrent to moving data from Toolbox to FLEx. However, as I mentioned in my previous post, if you will be making that move sometime, it's better to do it sooner rather than later, as the amount of work that will have to be redone will only increase over time.

- I've never managed to use the formulas in Toolbox... Would that actually make it easier to move to FLEx? Because I wouldn't loose too much work? I have no idea what formulas are in Toolbox, so I can't answer this question.

To view this discussion on the web visit https://groups.google.com/d/msgid/flex-list/2499f5ca-4180-42f5-b06f-bf536ec9644fn%40googlegroups.com.

[Aaron Broadwell]

Just a comment on one aspect of this thread -- Lexique Pro was a terrific step forward when it was produced.  But it has not been updated since 2012, and I think it is currently unsupported.  So if you run into a problem, I am not sure who can help you.

I had some early projects in Lexique Pro, but I have moved them all to Webonary.  In contrast to Lexique Pro, Webonary has an active development process and there are people (especially the very helpful Verna Stutzman) who can assist you.

[Jeff Shrum]

Here, Here! For Verna Stutzman and her team.

 

 

Jeff Shrum

 

SIL International

Language Technology Consultant

Dallas, TX

USA

Phone: +1-(972) 708-7579

To view this discussion on the web visit https://groups.google.com/d/msgid/flex-list/abd88d51-f5ba-4f89-a4ab-1d0258233ea0n%40googlegroups.com.

[Friederike Kronauer]

Thanks a lot to all of you!!! It's a lot clearer now..., even if there still are a lot of questions. 

Maybe I'm lucky and my dictionary isn't too complicated after all. And it's definitely a good point to try to switch sooner rather than later.

I'll see how far I can get!

[Murray Honsberger]

Further to Kevin's comment "importing from Toolbox into FLEx is not a trivial task", I would highly recommend investigating the paid service by the specialists. I made the move a number of years ago, and a "specialist" ran my Toolbox data through software called SOLID. I think this would still be one of the first steps. That showed many, many dead end references in my Toolbox data, such as references (\cf, \lv, \va etc.) to an old spelling of an entry, or references to entries that never actually got added. That is a major shortcoming of a Toolbox database (and I am considered to be a detailed person). FLex automatically steps you through creating an entry if you add any kind of reference to something that is not yet in your database. Or, it automatically corrects those references if you change the spelling of the original entry.

You might very well find it was worth the cost of at least doing the SOLID process, because it could very well result in a much improved Toolbox database.

However, I have no idea of the pricing of the service. (My "specialist" was in our branch, and happened to be (one of?) the SOLID programmer(s))

Happy lexicography!

Murray 

To view this discussion on the web visit https://groups.google.com/d/msgid/flex-list/ffc362a8-9cfa-4b51-ac42-19a13a2cbcdan%40googlegroups.com.

[david...@alum.mit.edu]

That said, @Aaron Broadwell, I still use Lexique Pro at times, including making a standalone lexicon (which has some advantages) (from a LIFT export from FLEx). I find it works fine, even if it hasn't been upgraded for 8+ years. And couldn't some kinds of problems, if encountered,  be discussed on this forum?

[Friederike Kronauer]

Hi again,

I finally found the courage to try out FLEx and even import a small part of a dictionary as a test.

Here are some questions I have so far:

- I've used the \a and \u fields to help Toolbox parse correctly. Importing the \a field is no problem, but what about the \u? Both: how do I deal with it during or after import and how does it work in FLEx for parsing a word without it?

- if a word can be both a noun and a verb: do I have to always enter a new sense? And what about suffixes?

- is there a way to tentatively assign a part of speech? (You can say "not sure", but that's not the same.)

- is it possible to keep a different database for suffixes and vocabulary? (I find it a bit confusing if everything's mixed.)

- in this language, they use the apostrophe to separate suffixes from proper names. So far, FLEx puts a space before and after the apostrophe, (and also before a comma and a period).

- is there a way to break/number text lines? Can you use that as a reference when entering an example into the lexicon?

- is it possible to tell FLEx that a certain postposition requires a certain case?

That's it for the moment, though I'm sure there will be more... :)

[Michael Maxwell]

On 3/9/2021 10:20 AM, 'Friederike Kronauer' via FLEx list wrote:
> is it possible to tell FLEx that a certain postposition requires a
> certain case?

Just a comment on this from a linguistic standpoint: this is about
syntax, which I'm pretty sure means FLEx can't do it, unless the
postposition is written solid with the case-marked word (which would be
treating it as if it were a clitic, or even as a suffix).

One issue that arises in syntax which essentially does not arise in
morphology is discontinuous dependencies. By that I mean that in many
languages, an arbitrary number of words could intervene between a
postposition and the noun whose case it governs. Hard to illustrate
this in English, since probably the only postposition we have is 'ago',
but using prepositions: a determiner and any number of adjectives can
intervene between a preposition and the noun it governs:
...from the big, bad, mean, and awful wolf
(Of course there's no significant case marking on English nouns, but you
get the idea, and can probably make a similar example in German.)

Of course if nouns are final in their NP, and the postposition
immediately follows the NP, then they will be adjacent, which would make
the syntactic dependency easier to deal with. Although I still suspect
FLEx does not have a mechanism for that.
--
Mike Maxwell
"Digital objects last forever--or five years,
whichever comes first." --Jeff Rothenberg

[Andreas_Joswig]

Dear Friederike,

I'm trying to give a few responses between your questions, in addition to what Mike already said.

Best wishes,

Andreas

On 3/9/21 4:20 PM, 'Friederike Kronauer' via FLEx list wrote:

Hi again,

I finally found the courage to try out FLEx and even import a small part of a dictionary as a test.

Here are some questions I have so far:

- I've used the \a and \u fields to help Toolbox parse correctly. Importing the \a field is no problem, but what about the \u? Both: how do I deal with it during or after import and how does it work in FLEx for parsing a word without it?

There is no equivalent to the \u field in FLEx. The best place to put the underlying form is to use the 'Lexeme form' field. You can then get at the surface forms by using allomorphs, or, if you feel brave, by using the rule-based parser Hermit Crab. This requires a lot of set-up work, but will eventually lead to very satisfying results (if you don't give up in frustration long before you get there, which can happen).

- if a word can be both a noun and a verb: do I have to always enter a new sense? And what about suffixes?

yes. If they use different affixes, that will be sorted out by using the affix templates for each different category, one of the very strong features of FLEx.

- is there a way to tentatively assign a part of speech? (You can say "not sure", but that's not the same.)

no. You can place a note in the entry that your PoS-assignment is tentative, but this is all there is.

- is it possible to keep a different database for suffixes and vocabulary? (I find it a bit confusing if everything's mixed.)

no. If it is one project, all information is in the same database. But you can use filters to only show stems and phrases.

- in this language, they use the apostrophe to separate suffixes from proper names. So far, FLEx puts a space before and after the apostrophe, (and also before a comma and a period).

apostrophes are a big pain in the neck for various reasons, as the character normally entered via your keyboard is interpreted by FLEx as a punctuation character, which is its purpose according to the Unicode. You can use a different character, but that normally requires a keyboard solution. I think you can also define the apostrophe as a wordforming character in the writing system setup, but I'm not exactly sure how this works.

- is there a way to break/number text lines? Can you use that as a reference when entering an example into the lexicon?

text lines are numbered in the gloss and analysis view, and in the text-chart view

- is it possible to tell FLEx that a certain postposition requires a certain case?

You can leave a grammar note with the lexical entry, but it will have no effect on parsing

That's it for the moment, though I'm sure there will be more... :)

On Wednesday, February 24, 2021 at 8:30:54 AM UTC+1 Friederike Kronauer wrote:

I've been using Toolbox for a long time and now, finally, would like to move to FLEx (my dictionary and interlinear texts).  Unfortunately, I can't find any information about how to do it! Is there a tutorial or something similar? Can somebody help?

Also: is there anything specifically about the differences between Toolbox and FLEx and about how to start coming from Toolbox?

I would be really grateful!!!

--

"FLEx list" messages are public. Only members can post.
flex_d...@sil.org
http://groups.google.com/group/flex-list.
---
You received this message because you are subscribed to the Google Groups "FLEx list" group.
To unsubscribe from this group and stop receiving emails from it, send an email to flex-list+...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/flex-list/9a229aa3-699e-4ec8-a0f5-d34f91a423d1n%40googlegroups.com.

[Kevin Warfel]

A big thank you to Andreas for responding to all of your questions, Friederike. I'm writing to supplement the one detail where he expressed some uncertainty.

You wrote: in this language, they use the apostrophe to separate suffixes from proper names. So far, FLEx puts a space before and after the apostrophe, (and also before a comma and a period)

Andreas commented: apostrophes are a big pain in the neck for various reasons, as the character normally entered via your keyboard is interpreted by FLEx as a punctuation character, which is its purpose according to the Unicode. You can use a different character, but that normally requires a keyboard solution. I think you can also define the apostrophe as a wordforming character in the writing system setup, but I'm not exactly sure how this works.

I will now add this:
SIL's Dictionary & Lexicography Services is developing an online lexicography course (https://sites.google.com/sil.org/dls-course). The course is not yet complete, but one of the lessons that is done is Writing Systems (https://sites.google.com/sil.org/dls-course/home-lessons/b-database-preparation/writing-systems). In that lesson, we provide detailed instructions about how to configure your project so that characters which are defined in Unicode as "punctuation" are treated in FLEx as "word-forming". I suggest that you read through that lesson.

Since the apostrophe is a word-forming character in the language you are working with, you will need to tell FLEx how you want it treated in the sort order. Should an apostrophe sort before all alphabetic characters or after all alphabetic characters or somewhere else, or should it simply be ignored? To specify this in FLEx, you'll need to use a rule similar to the one shown on slide 53 of the Writing Systems lesson. (Note: In the lesson, we talk about the apostrophe as representing a phonemic glottal stop, because we are under the impression that the majority of orthographies that use an apostrophe as a word-forming character use it to represent this phoneme. You'll just need to ignore the reference to glottal stop and use the principle explained to apply to your language's use of the apostrophe.)

If you need more help with this particular aspect of FLEx, I will be happy to provide what assistance I can. Just contact me off-list.

Hope that helps.

Kevin Warfel
Associate Dictionary & Lexicography Services Coordinator
SIL International

To view this discussion on the web visit https://groups.google.com/d/msgid/flex-list/f1f3da0d-0aa4-0998-e2f4-30e9a660f42c%40sil.org.

[Friederike Kronauer]

Thank you all so much!!! I'd be completely lost without your help.

Well, that's a lot of food for thought and experimentation... ;)

About the postpositions requiring a certain case: in Toolbox, I used something like: "post+DAT" for part of speech. Could I use that in FLEx as well? (I find it very useful to have that information visible when analyzing and translating!) @Andreas: You mentioned leaving a grammar note: where exactly? Which field would you use?

Also in general: is it better to use the grammatical categories that are offered in FLEx and the Inflectional Affix Gloss Builder or my old ones (like "post+DAT", "v.refl")? If I should convert all of that to the ones used in FLEx: is there an effective way of doing it? Or would I have to do it manually? Is there anything I ought to do in Toolbox before importing the dictionary? Can I even access the Inflectional Affix Gloss Builder 

And: when I'm analyzing a word: is it possible to edit my old entry? I don't want to add a new sense or allomorph, I just want to check what's in the lexicon and maybe modify (especially now, in the beginning of using FLEx!) Is there anything like the jump path in Toolbox? I mean, can I jump from a word in a text directly to the word in the lexicon? 

I know I'll have to redo my interlinearization, I'm just trying to figure out how best to do all this and how to start! 

Thanks again...!

[Andreas_Joswig]

Dear Friederike,

In FLEx, the way data is handled is fundamentally different from the way it was done in Toolbox, and for this reason alone I think it would be good if you spend some time to familiarize yourself with the basics of your new software. There are several ways of doing this - the 'help' menu provides a lot of resources, but you'll find the "Introduction to Lexicography" and the "Introduction to Parsing" particularly helpful. I would also warmly recommend the course that Kevin mentioned in his e-mail - it is not quite finished yet, but what is there is a lot more userfriendly than most things in the help menu. In most places in FLEx you also have a context-help feature that is available on clicking to the left of any field or feature, and the help files are very well done, although sometimes a bit hard to read.

On your questions:

Postpositions can take any gloss in FLEx - you could easily call them "of+DAT" or something like that. The gloss of a morpheme is not particularly tied to its grammatical function, but of course it helps to keep the relation visible. The grammar note is a field that is available under the sense level of an entry, but usually not visible, in order to keep the screen uncluttered. You can make it visible by checking the little box "show hidden fields" in the top right corner of the entry. You can set the visibility of each field the way you want it when clicking on the blue triangle next to a field.

FLEx does not give you much flexibility in naming your categories - they always need to be chosen from a drop-down list for each lexical entry. You can determine the content of the drop-down list, though. The advantage is that your data will be a lot more consistent this way (although Toolbox also had its ways to accomplish this). The inflectional affix gloss builder only works for inflectional affixes, as the name indicates. It will suggest glosses for each morpheme, according to the Leipzig conventions, I believe, but you are free to change the gloss, as it is really not firmly tied to the grammatical function. When you imported your Toolbox project, FLEx will have kept the previous glosses. It is still good to use the inflectional affix gloss builder, as it will tag the morpheme with the correct grammatical categories, and these categories are then also included into the list of categories for this language. You'll want that, although it means a lot of manual work. Of course it is possible to edit your old entry in the lexicon edit view.

The Text and Words section of FLEx provides a number of views on the text. The baseline does not give you many viewing options, but it is the only way to edit the text as such. But once you use the gloss view, you can get from any word in the text to the word-analysis view, and from the analyze view, you can click on each morpheme with a link to its place in the lexicon. You'll find what you want.

But again, my main advice right now is that you spend some time on the help resources or with Kevin's course, and that will get you going in no time.

Andreas

To view this discussion on the web visit https://groups.google.com/d/msgid/flex-list/18f00b3f-e7e3-4b84-a410-639e06dc69een%40googlegroups.com.

[Friederike Kronauer]

Thanks a lot! I managed to add the apostrophe and by right-clicking on a word I can jump to the entry in the lexicon! That really helps! These little things can be really frustrating... (And yes, I'm slowly working through the course Kevin mentioned.)

I'll also have to work through the Leipzig conventions. I was looking for something like that!

One more question though: I'm interested in Turkic languages and at one point (in the future), it would be great if I could compare several different ones, vocabulary, grammar, everything. Maybe find a text that's been translated into all of them... In order to do that: is there anything I ought to do now to make it easier? And if I'm ready: how would I do that? Would I have to have one dictionary with several languages? Or just several different projects?

Friederike

[Tom Br]

Hi all, I'm posting this because this thread is the first hit for googling "convert Toolbox to FLEx" and there is now a way which could use some visibility :)

If you're still looking for a way to convert Toolbox interlinear texts (with full morpheme glosses) into FLEx, check out: https://github.com/git-debase-verbose/FWdata-Import which has an open source tool written in python that imports flextext XML (exported from ELAN/Toolbox) directly into FLEx projects and keeps glossing intact. I haven't tested it but I was at a low resource language linguistics workshop in Maastricht today where there was a presentation about this. For the record, here is the abstract of that presentation:

FLExtext interlinear import preserving morpheme glosses (Elena Lazarenko, Aleksandr Riaposov and Alexandre Arkhipov; University of Hamburg)

We present a tool that addresses a weakness in FieldWorks Language Explorer (SIL FLEx), a software widely used for interlinear morphological glossing of texts (IGT) in the field of language documentation, in particular that of low-resource languages. The weakness springs from FLEx being unable to import IGT texts with glosses kept intact. As a result, it reduces the interoperability with other software solutions used for IGT purposes such as ELAN, EXMARaLDA, or Field Linguist’s Toolbox.

Our tool is a Python application with a simple GUI that reads .flextext XML files and appends their contents to the FieldWorks project database file, another flavor of XML with the .fwdata extension. The tool has been successfully put to use to integrate ca. 450 Tundra and Forest Enets (iso: enh and enf; Samoyedic < Uralic) texts (ca. 140,000 tokens) originally glossed in Toolbox into an existing .fwdata file, and to import 33 Kalmyk texts (iso: xal; Mongolic) with interlinear glossing into a new FLEx project.

---

Cheers,

Tom Brand

Op donderdag 11 maart 2021 om 22:19:52 UTC+1 schreef Friederike Kronauer: