Split big words/sentences

[Amir Simantov]

Hi guys,

I am aware of a post that was active a year ago called Sandhi splitting tool but I was not sure neither if my case is the same nor whether I should reincarnate an old post.

Here is the issue. I am a webmaster (no Sanskrit knowledge) and in the database of my client he wants to compare versions of texts in manuscripts (this is called "text re-use detection"). However, all the tools that do this detection assume that words are separated by a space (or a comma, etc.). 

So I need a library that I can right on demand on the website that splits big sentences into their Sanskrit small words (I know that there are many option to split, morphologically speaking, so let's say that I will take the most probable option). Only after splitting I can use one of the libraries that do the text re-use detection.

Here is an example of texts to compare:

• iti śrīsaṃkṣiptavedāntaśāstraprakriyāṃ śrīmatparamahaṃsaparivrājakācāryaśrīmacchaṅkarabhagavataḥ kṛtau bahimukhāntaḥ praṇavamajñānabodhinī adhyātmavidyaupadesavidhi samāpta
  
• iti śrīmat paramahaṃsaparivrājakācārya(śrī)machaṅ(ṃ)karakṛtavahimakhātaḥ pravaṇamajñānavodhanī atmavidyopadeśavidhi samāptaḥ
  
• śrīmatparamahaṃsaparivrajakācāryaśrīmachaṃkarakṛtapraṇavatrayam ajñānabodhanaṃ adhyātmavidyopadeśavidhiḥ samāptaḥ
  
• śrīmatparamahamsaparivrājakācāryaśrīmachakarācāryakṛtabahirmukhāntaḥ pravaṇamajñānabodhinī // adhayātmavidyopadeśavidhiḥ samāptaḥ
  

(Taken from a work called Ajñānabodhinī).

Does anyone know about something that I can use?

Thank you very much.

Amir

[Michael Bykov]

Hi Amir,

that is complex task. In my experiments, now deprecated, (look https://github.com/mbykov/vigraha)

50-symbols word gives up to 150 000 possible chains of possible padas. From which you then have to select the chains consisting of the real words.

Nevertheless, try (based on this routine) a Chromium plugin:

https://chrome.google.com/webstore/detail/morpheus-chromium-plugin/bochopmkmeclhfljekbigjdedacgkanc

The plugin can process more or less quickly the words up to 25 syms. The longer word you have to split into parts by hand before processing.

 

Amir

--
You received this message because you are subscribed to the Google Groups "sanskrit-programmers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to sanskrit-programmers+unsub...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--

М.

http://diglossa.ru
xmpp://m.b...@jabber.ru

[Dhaval Patel]

https://pypi.python.org/pypi/sanskrit-parser/0.0.1.dev2

And

https://github.com/kmadathil/sanskrit_parser may help you.

--

[Avinash L Varna]

The sanskrit parser library indeed has similar goals as your request (disclaimer: I am one of the contributors). It is currently still under development, but can produce reasonably good results. E.g. for the following input (corresponding to the first sentence you provided):
iti śrīsaṃkṣiptavedāntaśāstraprakriyāṃ śrīmatparamahaṃsaparivrājakācāryaśrīmacchaṅkarabhagavataḥ kṛtau bahimukhāntaḥ praṇavamajñānabodhinī adhyātmavidyopadeśavidhiḥ samāptaḥ

the output in SLP1 encoding is:
[iti, SrI, saMkziptavedAntaSAstraprakriyAm, SrImat, paramahaMsaparivrAjakAcArya, SrImat, SaNkara, Bagavatas, kftO, ba, him, uKAm, tas, praRavam, ajYAnaboDinI, aDyAtmavidyA, upadeSa, viDis, samAptas]

For very long sentences, the splitting can take some time because of the graph path-finding algorithm. The above example took a couple of minutes on my laptop. Whether your web server can afford to spend that much time splitting long sentences is up to you. There are plans to add NN based capabilities in the future that may reduce the computational cost.

Another limitation is that the sanskrit_parser library currently does not handle any errors in the input. E.g. your original input had vidy*au*padesa which had to be corrected to vidy*o*padeśa, bahimukha probably needs to be bahirmukha (which is why there is a weird split in the output), etc.

However, if you are just looking for text reuse detection, would it still work if the result is not semantically/morphologically valid? If so, you might be able to use some consistent sub-unit tokenization algorithms such as sentencepiece (https://github.com/google/sentencepiece) or morphessor (http://morpho.aalto.fi/projects/morpho/). For the former, a model trained on DCS data is available as part of this project - https://github.com/cvikasreddy/skt

You will still have to deal with errors/inconsistencies in the input, and probably some other issues that I haven't thought of :)

Thanks

Avinash

To unsubscribe from this group and stop receiving emails from it, send an email to sanskrit-programmers+unsubscrib...@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

[Amir Simantov]

Hi!

Thank you very much all.

I will check all references.

You are very kind.

Amir

You received this message because you are subscribed to a topic in the Google Groups "sanskrit-programmers" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/sanskrit-programmers/MEGnEfTGKQw/unsubscribe.
To unsubscribe from this group and all its topics, send an email to sanskrit-programmers+unsub...@googlegroups.com.