Is hyperopt avaiable for ensemble?

[Ball]

Hi all,

I am trying to assign weight to the backends included in a simple ensemble of omikuji-bonsai, mllm, and xtransformer. Can I use the hyperopt command to get the best settings? or it's a manual process to try different combinations?

Thanks,

Lucas

[juho.i...@helsinki.fi]

Hi Lucas,

Yes, you can use hyperopt for that with the (simple) ensemble.

Just noticed that the ensemble page in Annif wiki is missing this information, I'll add it.

-Juho

[Osma Suominen]

Hi Lucas,

see also the EXTRA section here:

https://github.com/NatLibFi/Annif-tutorial/blob/main/exercises/06_ensemble_project.md

-Osma

On 13/03/2026 10:43, juho.i...@helsinki.fi wrote:
> Hi Lucas,
>
> Yes, you can use hyperopt for that with the (simple) ensemble.
>

> Just noticed that the ensemble page <https://github.com/NatLibFi/Annif/
> wiki/Backend%3A-Ensemble> in Annif wiki is missing this information,

> I'll add it.
>
> -Juho
>
> On Wednesday, 11 March 2026 at 17:46:30 UTC+2 mak.wi...@gmail.com wrote:
>
> Hi all,
>
> I am trying to assign weight to the backends included in a simple
> ensemble of omikuji-bonsai, mllm, and xtransformer. Can I use the
> hyperopt command to get the best settings? or it's a manual process
> to try different combinations?
>
> Thanks,
> Lucas
>

> --
> You received this message because you are subscribed to the Google
> Groups "Annif Users" group.
> To unsubscribe from this group and stop receiving emails from it, send
> an email to annif-users...@googlegroups.com <mailto:annif-
> users+un...@googlegroups.com>.
> To view this discussion visit https://groups.google.com/d/msgid/annif-
> users/3122a6b0-9ec8-4c2d-bdb8-624b6d5977bdn%40googlegroups.com <https://
> groups.google.com/d/msgid/annif-users/3122a6b0-9ec8-4c2d-
> bdb8-624b6d5977bdn%40googlegroups.com?utm_medium=email&utm_source=footer>.

--
Osma Suominen
D.Sc. (Tech), Information Systems Specialist
National Library of Finland
P.O. Box 15 (Unioninkatu 36)
00014 HELSINGIN YLIOPISTO
Tel. +358 50 3199529
osma.s...@helsinki.fi
http://www.nationallibrary.fi

[Ball]

Hi Osma and Juho,

Thanks for the info. We tried running the simple Ensemble for trained Omikuji-bonsai, MLLM, and Xtransformer using the Xtransformer branch image available from https://github.com/NatLibFi/Annif/pull/798 but ran into the error appended below. If we take out the Xtransformer backend from the Ensemble, the hyperopt ran without any issues.

===== Start of log ========

Looking for optimal hyperparameters using 100 trials
2026-03-27T02:20:19.027Z INFO  [omikuji::model] Loading model from data/projects/omikuji-bonsai-topic-en/omikuji-model...
2026-03-27T02:20:19.040Z INFO  [omikuji::model] Loading model settings from data/projects/omikuji-bonsai-topic-en/omikuji-model/settings.json...
2026-03-27T02:20:19.044Z INFO  [omikuji::model] Loaded model settings Settings { n_features: 2233466, classifier_loss_type: Hinge }...
2026-03-27T02:20:19.045Z INFO  [omikuji::model] Loading tree from data/projects/omikuji-bonsai-topic-en/omikuji-model/tree0.cbor...
2026-03-27T02:20:24.756Z INFO  [omikuji::model] Loading tree from data/projects/omikuji-bonsai-topic-en/omikuji-model/tree2.cbor...
2026-03-27T02:20:30.448Z INFO  [omikuji::model] Loading tree from data/projects/omikuji-bonsai-topic-en/omikuji-model/tree1.cbor...
2026-03-27T02:20:35.467Z INFO  [omikuji::model] Loaded model with 3 trees; it took 16.44s
multiprocessing.pool.RemoteTraceback:
"""
Traceback (most recent call last):
  File "/usr/local/lib/python3.12/multiprocessing/pool.py", line 125, in worker
    result = (True, func(*args, **kwds))
                    ^^^^^^^^^^^^^^^^^^^
  File "/Annif/annif/parallel.py", line 76, in suggest_batch
    suggestion_batch = project.suggest(batch, self.backend_params)
                       ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Annif/annif/project.py", line 272, in suggest
    return self._suggest_with_backend(transformed_docs, backend_params)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Annif/annif/project.py", line 151, in _suggest_with_backend
    return self.backend.suggest(docs, beparams)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Annif/annif/backend/backend.py", line 143, in suggest
    return self._suggest_batch(documents, params=beparams)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Annif/annif/backend/xtransformer.py", line 248, in _suggest_batch
    prediction = self._model.predict(
                 ^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/pecos/xmc/xtransformer/model.py", line 581, in predict
    _, embeddings = self.text_encoder.predict(
                    ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/pecos/xmc/xtransformer/matcher.py", line 642, in predict
    label_pred, embeddings = self._predict(
                             ^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/pecos/xmc/xtransformer/matcher.py", line 768, in _predict
    for batch in dataloader:
                 ^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/torch/utils/data/dataloader.py", line 494, in __iter__
    return self._get_iterator()
           ^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/torch/utils/data/dataloader.py", line 427, in _get_iterator
    return _MultiProcessingDataLoaderIter(self)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/torch/utils/data/dataloader.py", line 1170, in __init__
    w.start()
  File "/usr/local/lib/python3.12/multiprocessing/process.py", line 118, in start
    assert not _current_process._config.get('daemon'), \
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
AssertionError: daemonic processes are not allowed to have children
"""

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/usr/local/bin/annif", line 6, in <module>
    sys.exit(cli())
             ^^^^^
  File "/usr/local/lib/python3.12/site-packages/click/core.py", line 1442, in __call__
    return self.main(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/click/core.py", line 1363, in main
    rv = self.invoke(ctx)
         ^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/click/core.py", line 1830, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
                           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/click/core.py", line 1226, in invoke
    return ctx.invoke(self.callback, **ctx.params)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/click/core.py", line 794, in invoke
    return callback(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/click/decorators.py", line 34, in new_func
    return f(get_current_context(), *args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/flask/cli.py", line 400, in decorator
    return ctx.invoke(f, *args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/click/core.py", line 794, in invoke
    return callback(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Annif/annif/cli.py", line 674, in run_hyperopt
    rec = proj.hyperopt(documents, trials, jobs, metric, results_file)
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Annif/annif/project.py", line 322, in hyperopt
    return optimizer.optimize(trials, jobs, results_file)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Annif/annif/backend/hyperopt.py", line 132, in optimize
    objective_args = self._prepare(n_jobs)
                     ^^^^^^^^^^^^^^^^^^^^^
  File "/Annif/annif/backend/ensemble.py", line 132, in _prepare
    for suggestions, gold_batch in pool.imap_unordered(
                                   ^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/multiprocessing/pool.py", line 873, in next
    raise value
AssertionError: daemonic processes are not allowed to have children

====== End of log =======

I believe I have seen someone successfully ran Ensemble with Xtransformer, so I am wondering why it's failing on us.

Thanks,

Lucas

[Sven Sass]

I believe this happens if you use --jobs.

Running it with 1 will work.

Best regards

Sven