Xtransformer backend

[Ball]

Hi folks,

We tried to add and run the Xtransformer backend (https://github.com/NatLibFi/Annif/pull/798) but ran into issues. It gave us the following error message when we started a training job

warning: Could not create backend xtransformer, make sure you've installed optional dependencies
Traceback (most recent call last):
  File "/usr/local/bin/annif", line 6, in <module>
    sys.exit(cli())
             ^^^^^
  File "/usr/local/lib/python3.12/site-packages/click/core.py", line 1161, in __call__
    return self.main(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/click/core.py", line 1082, in main
    rv = self.invoke(ctx)
         ^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/click/core.py", line 1697, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
                           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/click/core.py", line 1443, in invoke
    return ctx.invoke(self.callback, **ctx.params)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/click/core.py", line 788, in invoke
    return __callback(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/click/decorators.py", line 33, in new_func
    return f(get_current_context(), *args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/flask/cli.py", line 400, in decorator
    return ctx.invoke(f, *args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/click/core.py", line 788, in invoke
    return __callback(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Annif/annif/cli.py", line 222, in run_train
    proj.train(documents, backend_params, jobs)
  File "/Annif/annif/project.py", line 284, in train
    beparams = backend_params.get(self.backend.backend_id, {})
                                  ^^^^^^^^^^^^^^^^^^^^^^^
AttributeError: 'NoneType' object has no attribute 'backend_id'

The code was able to progress a bit further after we commented out python codes mentioning "pecos". We tried to find it in the dependencies installed but all we could find was "libpecos". Any ideas what we did wrong?

Thanks,

Lucas

[Lakshmi rb]

Hi Lucas,

Thanks for testing the Xtransformer backend and reporting this.

The issue comes from a dependency mismatch in the pecos library. pecos currently imports the AdamW optimizer from the transformers library rather than from torch.optim. In recent versions of transformers, this optimizer import path has been deprecated and eventually removed, which causes the backend to fail to initialize. This has been reported upstream in pecos issue #311.

As a temporary fix,  pin transformers to a version that still supports from transformers import AdamW (≤ 4.49.0).

For pip, 

   pip install "transformers<=4.49.0"

For Poetry, add this to your pyproject.toml under  [tool.poetry.dependencies]:

   transformers = "<=4.49.0" 

Regards,

Lakshmi Rajendram Bashyam
Angewandte Forscherin// Automatisierung der Sacherschließung
Applied researcher // Automation of Subject Indexing
Neuer Jungfernstieg 21
20354 Hamburg
T: +49 176 29139271
E: l.rajendr...@zbw.eu