another error during training nn-ensemble

[Maciej Sagata]

Hi!

We've finally (at the National Library of Poland) managed to train nn-ensemble, but the success was... temporary.

When we switched to vocab in SKOS and gave it another try, we've encountered another error - maybe someone can help us this time as well.

We trained two models (mllm and omikuji-bonsai) and they work well, but the nn-ensemble on top of that has stuck at this point:

logs:

I tensorflow/compiler/mlir/mlir_graph_optimization_pass.cc:176] None of the MLIR Optimization Passes are enabled (registered 2) I tensorflow/core/platform/profile_utils/cpu_utils.cc:114] CPU Frequency: 3892750000 Hz

In the project directory there is a nn-train.mdb directory with data.mdb and lock.mdb, RAM usage is at 20,4 GB (out of 64 GB).

Configuration as follows:

[mllm-pl-lem]
name=MLLM Polish
language=pl
backend=mllm
analyzer=simple
vocab=vocab-pl-lem
limit=1000
min_samples_leaf=9
max_leaf_nodes=958
max_samples=0.9925

[omikuji-bonsai-pl-lem]
name=Omikuji Bonsai Polish
language=pl
backend=omikuji
analyzer=simple
vocab=vocab-pl-lem
cluster_balanced=False
cluster_k=10
max_depth=3

[nn-ensemble-pl-lem]
name=NN ensemble Polish
language=pl
backend=nn_ensemble
sources=mllm-pl-lem,omikuji-bonsai-pl-lem
limit=100
vocab=vocab-pl-lem
nodes=100
dropout_rate=0.2
epochs=10

Any ideas what can be the cause of such behaviour?

[juho.i...@helsinki.fi]

Hi!

The log messages seem harmless. The preprocessing might just take a long time, up to hours (on a big trainset), and without showing output. You could try training nn-ensemble with a small number of documents, like 100 or just 10. You can use the --docs option of the train command for that. If training with 10 documents does not finish in a few minutes, then there is something wrong.

Adding also the option -v DEBUG to train command will show some more log messages, which could be helpful.

-Juho