the _projects variable in the class AnnifRegistry

[Sefiane Touhami]

Hello everyone,

I have set up annif on a windows machine. I encounter a problem with the static variable "_projects" of the class "AnnifRegistry". 

When I train the data with the project "nn_ensemble" with the sources "sources=yso-tfidf-en,yso-mllm-en:2". The issue is that the static variable "_projects" becomes empty in multiprocessing when it is called in pool.imap_unordered. 

As a result the nn-emsemble algorithm crashes.

Best Regardes.

Sefiane

[Osma Suominen]

Hello Sefiane!

Thanks for your message.

We develop and test Annif using Linux. There are important differences
in the multiprocessing implementation between Linux and Windows. It's
not possible for us, the core developers, to work around these
differences and to test on Windows (but contributions are welcome!). The
main issue that Windows doesn't have a forking process model which Annif
relies on for some of the parallel processing (it allows the sharing of
already loaded models between processes).

Perhaps this should be stated more clearly in the top level README and
other documentation.

I recommend that you use the Docker images, which work on every major
platform (Linux is of course used within the container).

Best,
Osma

> --
> You received this message because you are subscribed to the Google
> Groups "Annif Users" group.
> To unsubscribe from this group and stop receiving emails from it, send
> an email to annif-users...@googlegroups.com
> <mailto:annif-users...@googlegroups.com>.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/annif-users/bcfd9c71-5e44-4a2a-9a0d-8f5bb718e664n%40googlegroups.com
> <https://groups.google.com/d/msgid/annif-users/bcfd9c71-5e44-4a2a-9a0d-8f5bb718e664n%40googlegroups.com?utm_medium=email&utm_source=footer>.

--
Osma Suominen
D.Sc. (Tech), Information Systems Specialist
National Library of Finland
P.O. Box 15 (Unioninkatu 36)
00014 HELSINGIN YLIOPISTO
Tel. +358 50 3199529
osma.s...@helsinki.fi
http://www.nationallibrary.fi

[Sefiane Touhami]

Hi Osma,

Thank you so much for your quick response. Unfortunately, unlike Linux, Windows does not have a fork() system call, which duplicates the current process. I asked hoping that someone has installed annif on windows and has encountered and solved the problem. I will let you know if I can find a solution to make this variable shared between processes on Windows.

BR.

Sefiane

To unsubscribe from this group and stop receiving emails from it, send an email to annif-users...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/annif-users/ea4682d1-3296-e521-8bd4-5bf037bbce59%40helsinki.fi.