enwiktionary

[Alhaitham Ibrahim]

Hello

Know that you are having a problem with scraping Wiktionary

Would like to know if you tried to use the dumps again instead of scraping

There was a discussion about the size of enterprise html dump

https://github.com/itkach/aard2-android/issues/168#issuecomment-1756079555

But now it seems that the size went up again [13.62 GB] and maybe worth a try

https://dumps.wikimedia.org/other/enterprise_html/runs/20240820/enwiktionary-NS0-20240820-ENTERPRISE-HTML.json.tar.gz

Thanks

[AardF...@web.de]

The namespaces which some people are using are not in the dumps. They need to be scraped seperately. Hence the compilation does not work without couchdb.

I deleted the couchdb setup with all databases and started from scratch.

Actually I am scraping and compiling enwiktionary-20240820 as a test.

It will take a couple of days until finished.

Would you please check the file if ready?

[Alhaitham Ibrahim]

> Would you please check the file if ready?

Not sure what file and what to check

[AardF...@web.de]

I am creating a slob file enwiktionary-20240820.slob and will upload it to

ftp.halifax.rwth-aachen.de/aarddict/enwiki

I will let you know here when it will be available.

From there you may download the file and check if it is working fine for you.

 

Let me know if you are not interested.

 

Thank you

 

Markus

--
You received this message because you are subscribed to the Google Groups "aarddict" group.
To unsubscribe from this group and stop receiving emails from it, send an email to aarddict+u...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/aarddict/324fd023-7d3a-4aab-98db-c86990c399aen%40googlegroups.com.

[Alhaitham Ibrahim]

> enwiktionary-20240820.slob

That is exactly my wish and very much interested

Thanks

[SailorVenusFan]

The 8/20 file is not there. Do you have discord too? I want that for the development of Aard 2

[AardF...@web.de]

The enwiktionary-20240820 is still in process. This takes a couple of days to be generated. 

ETA could be sometimes tonight...

Then it will be synced tomorrow morning. 

Markus Braun

---

From: SailorVenusFan <fedijus...@gmail.com>
Sent: Saturday, August 31, 2024 19:23
To: aarddict
Subject: Re: enwiktionary

To view this discussion on the web visit https://groups.google.com/d/msgid/aarddict/60699cfc-30a9-4e6a-98fe-bbed98664161n%40googlegroups.com.

[AardF...@web.de]

Nope no discord. 

All communication just on one platform only please. We use this Google platform for the time being. 

Markus

---

From: SailorVenusFan <fedijus...@gmail.com>
Sent: Saturday, August 31, 2024 19:23
To: aarddict
Subject: Re: enwiktionary

To view this discussion on the web visit https://groups.google.com/d/msgid/aarddict/60699cfc-30a9-4e6a-98fe-bbed98664161n%40googlegroups.com.

[AardF...@web.de]

it will be synced this evening

[Alhaitham Ibrahim]

Working great, thanks a lot

[JSToJestJaSam]

Does this mean that fiwiktionary can finally be updated too?

[Nikolai Yourin]

Looks promising, thanks a lot Markus! Here's a brief comparison:

20230601 => 20240820: 733183 additions / 11039 deletions
20240315 => 20240820: 575397 additions / 6093 deletions

Strangely enough, this version contains a lot of inflected forms that were not included in enwiktionary-20240315,

even though most of the articles in question are not that recent:

vulcanizássedes
    (reintegrationist norm) second-person plural imperfect subjunctive of vulcanizar
    (last edited on 11 December 2023)

zoomorfizó
    third-person singular preterite indicative of zoomorfizar
    (last edited on 9 December 2023)

On Sunday, September 1, 2024 at 11:37:44 AM UTC+3 AardF...@web.de wrote:

[AardF...@web.de]

Yes.

I will update the wiktionaries upon request. 

So far en and fi ;) 

Markus

---

From: JSToJestJaSam <joonata...@gmail.com>
Sent: Monday, September 2, 2024 20:25

To view this discussion on the web visit https://groups.google.com/d/msgid/aarddict/36d01fae-c1a0-4b1a-aa3d-ff3e13c18b48n%40googlegroups.com.

[AardF...@web.de]

Nicolai, 

This looks great. 

Yes, I included the name spaces. 

The name spaces are scraped into a couchdb and then combined with the NS0 dump. 

How did you generate those figures? 

Markus

---

From: Nikolai Yourin <n.yo...@gmail.com>
Sent: Monday, September 2, 2024 20:46

To view this discussion on the web visit https://groups.google.com/d/msgid/aarddict/3771237b-8074-4625-aa83-562333228881n%40googlegroups.com.

[Alhaitham Ibrahim]

Just found out that there are a lot of duplicated entries

Example:

[AardF...@web.de]

Interesting. Is it a duplicate article or a duplicate link to an article?

[Alhaitham Ibrahim]

Duplicate articles, as you can see, there are small differences like the beginning of Etymology

There are even triplicated articles like "farang"

[Nikolai Yourin]

For this to work, you'll need to have GoldenDict installed.
In GoldenDict, click 'Dictionary headwords' -> Export, then sort the resulting list:
    sort list1.txt >headwords1.txt

Once you have two sorted headword lists, diff them:
    diff -u headwords1.txt headwords2.txt | tail -n +3 >headwords.diff
    grep -c ^+ headwords.diff
    grep -c ^- headwords.diff


I'd prefer to do this programmatically, but 'slob' (the Python module) won't let me:

    with slob.open(sys.argv[1]) as r:
headwords = list(r.as_dict())

It just eats up all available memory no matter how tiny the input file is.

On Tuesday, September 3, 2024 at 2:53:17 AM UTC+3 AardF...@web.de wrote:

[AardF...@web.de]

Not sure where this comes from. As these are articles, the duplicates seem to reside in the NS0 dump. 

Markus

---

From: Alhaitham Ibrahim <compu...@gmail.com>
Sent: Tuesday, September 3, 2024 18:54

To view this discussion on the web visit https://groups.google.com/d/msgid/aarddict/d92b490b-56bc-4c81-82fa-59c1b25eb81bn%40googlegroups.com.

[Igor Tkach]

On Tue, Sep 3, 2024 at 4:40 PM Nikolai Yourin <n.yo...@gmail.com> wrote:

    with slob.open(sys.argv[1]) as r:
headwords = list(r.as_dict())

slob instance is already a sorted sequence, you can just iterate over it, like so:

 

for item in s:
     print(item.key)

or use list comprehension or generator expression. This works:

with slob.open(sys.argv[1]) as r:

     headwords = (item.key for item in r)

 

To view this discussion on the web visit https://groups.google.com/d/msgid/aarddict/13c549b1-8999-40b2-8856-d5118e65d95dn%40googlegroups.com.

[Nikolai Yourin]

Thank you Igor, it makes sense now.

Markus, I'm attaching a tiny script that can be run either like this:

    $ slob-headwords.py enwiktionary-20240820.slob >headwords.txt 2>duplicates.txt

or like this:

    $ slob-headwords.py enwiktionary-20240315.slob enwiktionary-20240820.slob
    enwiktionary-20240315.slob: 7,781,227 unique headwords + 130 duplicates
    enwiktionary-20240820.slob: 8,350,590 unique headwords + 264,274 duplicates
    575,427 additions / 6,064 deletions

As you can see, enwiktionary-20240820 does indeed contain a lot of duplicates
(the number of additions is still accurate though, since duplicate entries do not get counted towards that number).

For the sake of speed, the script makes no distinction between real headwords and built-in styles/scripts

such as '~/MathJax/MediaWiki.js' (because accessing 'item.content_type' seems to be quite costly).

[Alhaitham Ibrahim]

If the duplicate issue is present in the enterprise dumps then it might have been fixed in the latest one

enwiktionary-NS0-20240820-ENTERPRISE-HTML.json.tar.gz

[13.62 GB]

enwiktionary-NS0-20240901-ENTERPRISE-HTML.json.tar.gz

[13.14 GB]

It is good to know that nothing is missing from 20240820

[AardF...@web.de]

Thank you, will check it with the enwiktionary-20240901.slob when it is compiled

[Nikolai Yourin]

I think this is a clear improvement over 20240820:

    $ slob-headwords.py enwiktionary-20240820.slob enwiktionary-20240901.slob

    enwiktionary-20240820.slob: 8,350,590 unique headwords + 264,274 duplicates

    enwiktionary-20240901.slob: 8,367,203 unique headwords + 12,975 duplicates
    16,813 additions / 200 deletions

No idea what to make of the duplicates list (attached, as well as a slightly improved version of the script).
There are 12673 duplicate headwords, of which 7086 are annex entries (e.g. 'Reconstruction:Proto-Algonquian/net-') that most people wouldn't care about, so that's a relief.

13 extra copies of this?
https://en.wiktionary.org/wiki/%E2%9D%A8
(last edited on 2 September 2024, at 15:52)

[AardF...@web.de]

Thank you for the updated code.

 

I would not do anything with the duplicates. They are residing in the dumps.

If they are in the annexes, I guess they are there for a reason and the duplication is just a side effect.

… and next month we are starting over anyway.

I am glad they reduced the duplicates significantly

To view this discussion on the web visit https://groups.google.com/d/msgid/aarddict/cacc70d8-e5c7-4092-8a6e-421b5a3a0b21n%40googlegroups.com.

[Nikolai Yourin]

Absolutely.

I'm not even sure if it's possible to do any better here.

Just for the record, enwiktionary-20230601.slob, which was also made from an enterprise dump, contains 185,384 duplicates (spread over 126,306 headwords).

[JSToJestJaSam]

Wow! Thanks a million for updating fiwikt!!! You're amazing! This helps me so much!

[Alhaitham Ibrahim]

Hello, hope you are doing great

Just letting you know the latest enwiktionary folder is empty

https://ftp.halifax.rwth-aachen.de/aarddict/enwiki/enwiktionary20241101-slob/

No need to do it again, just for future reference that it can happen

[Markus Braun]

Thank you for the hint. The script did not went through. Waiting for the wikis to finish and will update the wiktionaries then. 

Both together is too heavy load for my little machine :-) 

Markus Braun

---

From: Alhaitham Ibrahim <compu...@gmail.com>
Sent: Friday, November 15, 2024 03:59

To view this discussion visit https://groups.google.com/d/msgid/aarddict/5370d034-d8bf-47e8-a181-47545840e4e8n%40googlegroups.com.