Multi-Task TDNN Training /// How to set num_epochs for each task?

205 views
Skip to first unread message

Joshua Meyer

unread,
Feb 24, 2018, 4:56:54 AM2/24/18
to kaldi-help

Dear Dan & Pegah,

I'm working with the multilingual Babel nnet3 scripts, and I've run into a problem.

The problem is the following: my tasks (ie. different languages) have different numbers of
training examples, and this results in some tasks being trained much longer than others.

The graph below shows what I'm talking about.


The task with most training examples gets the most training time, after all iterations for

other tasks have been exhausted.


Instead of 'parallel' learning I end up with 'sequential' learning, and the regularization effect

of earlier tasks is lost (as you can see in the jump in performance of train-TASK-A, after the

other tasks are exhausted).


Ideally, I'd like to be able to define the number of iterations per task, so that I can run more

iterations for smaller data sets, but I'm not sure how.


The training script I'm using is `steps/nnet3/train_raw_dnn.py`


Thank you for your time.


Best,

Josh





Daniel Povey

unread,
Feb 24, 2018, 12:42:23 PM2/24/18
to kaldi-help
What scripts are you using?   Give the full pathname from the root of the Kaldi repo.

--
Go to http://kaldi-asr.org/forums.html find out how to join
---
You received this message because you are subscribed to the Google Groups "kaldi-help" group.
To unsubscribe from this group and stop receiving emails from it, send an email to kaldi-help+unsubscribe@googlegroups.com.
To post to this group, send email to kaldi...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/kaldi-help/3a2f9d47-f7ff-4e78-a728-2ee527710fa0%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Joshua Meyer

unread,
Feb 24, 2018, 12:48:51 PM2/24/18
to kaldi...@googlegroups.com
This is the main run script I'm working of off:

egs / babel_multilang/s5/local/nnet3/run_tdnn_multilingual.sh

-josh


суббота, 24 февраля 2018 г. пользователь Daniel Povey написал:
What scripts are you using?   Give the full pathname from the root of the Kaldi repo.
You received this message because you are subscribed to a topic in the Google Groups "kaldi-help" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/kaldi-help/lNAQ7aZptDU/unsubscribe.
To unsubscribe from this group and all its topics, send an email to kaldi-help+unsubscribe@googlegroups.com.

To post to this group, send email to kaldi...@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.


--
Joshua Meyer
Ph.D. Candidate
University of Arizona

Vimal Manohar

unread,
Feb 24, 2018, 1:15:54 PM2/24/18
to kaldi...@googlegroups.com
steps/nnet3/multilingual/combine_egs.sh has lang2weight option to downscale some tasks 0.1,0.5,1 etc. The weight can be set inversely proportional to the sqrt(amount of data).

The other option is to modify combine_egs.sh to duplicate the egs.scp for a task to do more training on that and pass that to allocate_multilingual_example.py.

```
for i in ${num_copies[$lang]}; do
  awk -v i=$i '{print $1"-"i" "$2}' ${multi_egs_dir[$lang]}/egs.scp
done > new_egs.scp
```
 

On Sat, Feb 24, 2018 at 12:48 PM Joshua Meyer <joshua.ric...@gmail.com> wrote:
This is the main run script I'm working of off:

egs / babel_multilang/s5/local/nnet3/run_tdnn_multilingual.sh

-josh


суббота, 24 февраля 2018 г. пользователь Daniel Povey написал:
What scripts are you using?   Give the full pathname from the root of the Kaldi repo.
On Sat, Feb 24, 2018 at 4:56 AM, Joshua Meyer <joshua.ric...@gmail.com> wrote:

Dear Dan & Pegah,

I'm working with the multilingual Babel nnet3 scripts, and I've run into a problem.

The problem is the following: my tasks (ie. different languages) have different numbers of
training examples, and this results in some tasks being trained much longer than others.

The graph below shows what I'm talking about.


The task with most training examples gets the most training time, after all iterations for

other tasks have been exhausted.


Instead of 'parallel' learning I end up with 'sequential' learning, and the regularization effect

of earlier tasks is lost (as you can see in the jump in performance of train-TASK-A, after the

other tasks are exhausted).


Ideally, I'd like to be able to define the number of iterations per task, so that I can run more

iterations for smaller data sets, but I'm not sure how.


The training script I'm using is `steps/nnet3/train_raw_dnn.py`


Thank you for your time.


Best,

Josh





--
Go to http://kaldi-asr.org/forums.html find out how to join
---
You received this message because you are subscribed to the Google Groups "kaldi-help" group.
To unsubscribe from this group and stop receiving emails from it, send an email to kaldi-help+...@googlegroups.com.

To post to this group, send email to kaldi...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/kaldi-help/3a2f9d47-f7ff-4e78-a728-2ee527710fa0%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--
Go to http://kaldi-asr.org/forums.html find out how to join
---
You received this message because you are subscribed to a topic in the Google Groups "kaldi-help" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/kaldi-help/lNAQ7aZptDU/unsubscribe.
To unsubscribe from this group and all its topics, send an email to kaldi-help+...@googlegroups.com.


--
Joshua Meyer
Ph.D. Candidate
University of Arizona

--
Go to http://kaldi-asr.org/forums.html find out how to join
---
You received this message because you are subscribed to the Google Groups "kaldi-help" group.
To unsubscribe from this group and stop receiving emails from it, send an email to kaldi-help+...@googlegroups.com.

To post to this group, send email to kaldi...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/kaldi-help/CACTjhaSdybkmO%3D4Q%3DNvTAJRCbJb9AxQ5StgHhvrVikAMv-_wbA%40mail.gmail.com.

For more options, visit https://groups.google.com/d/optout.
--
Vimal Manohar
PhD Student
Electrical & Computer Engineering
Johns Hopkins University

Daniel Povey

unread,
Feb 24, 2018, 1:21:01 PM2/24/18
to kaldi-help
I actually think it's likely a bug in that script-- it's not how the script is supposed to work.  But I'll follow up more definitively when I get more information.

On Sat, Feb 24, 2018 at 1:15 PM, Vimal Manohar <vimal.m...@gmail.com> wrote:
steps/nnet3/multilingual/combine_egs.sh has lang2weight option to downscale some tasks 0.1,0.5,1 etc. The weight can be set inversely proportional to the sqrt(amount of data).

The other option is to modify combine_egs.sh to duplicate the egs.scp for a task to do more training on that and pass that to allocate_multilingual_example.py.

```
for i in ${num_copies[$lang]}; do
  awk -v i=$i '{print $1"-"i" "$2}' ${multi_egs_dir[$lang]}/egs.scp
done > new_egs.scp
```
 
On Sat, Feb 24, 2018 at 12:48 PM Joshua Meyer <joshua.richard.meyer@gmail.com> wrote:
This is the main run script I'm working of off:

egs / babel_multilang/s5/local/nnet3/run_tdnn_multilingual.sh

-josh


суббота, 24 февраля 2018 г. пользователь Daniel Povey написал:
What scripts are you using?   Give the full pathname from the root of the Kaldi repo.
On Sat, Feb 24, 2018 at 4:56 AM, Joshua Meyer <joshua.richard.meyer@gmail.com> wrote:

Dear Dan & Pegah,

I'm working with the multilingual Babel nnet3 scripts, and I've run into a problem.

The problem is the following: my tasks (ie. different languages) have different numbers of
training examples, and this results in some tasks being trained much longer than others.

The graph below shows what I'm talking about.


The task with most training examples gets the most training time, after all iterations for

other tasks have been exhausted.


Instead of 'parallel' learning I end up with 'sequential' learning, and the regularization effect

of earlier tasks is lost (as you can see in the jump in performance of train-TASK-A, after the

other tasks are exhausted).


Ideally, I'd like to be able to define the number of iterations per task, so that I can run more

iterations for smaller data sets, but I'm not sure how.


The training script I'm using is `steps/nnet3/train_raw_dnn.py`


Thank you for your time.


Best,

Josh





--
Go to http://kaldi-asr.org/forums.html find out how to join
---
You received this message because you are subscribed to the Google Groups "kaldi-help" group.
To unsubscribe from this group and stop receiving emails from it, send an email to kaldi-help+unsubscribe@googlegroups.com.

To post to this group, send email to kaldi...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/kaldi-help/3a2f9d47-f7ff-4e78-a728-2ee527710fa0%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--
Go to http://kaldi-asr.org/forums.html find out how to join
---
You received this message because you are subscribed to a topic in the Google Groups "kaldi-help" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/kaldi-help/lNAQ7aZptDU/unsubscribe.
To unsubscribe from this group and all its topics, send an email to kaldi-help+unsubscribe@googlegroups.com.


--
Joshua Meyer
Ph.D. Candidate
University of Arizona

--
Go to http://kaldi-asr.org/forums.html find out how to join
---
You received this message because you are subscribed to the Google Groups "kaldi-help" group.
To unsubscribe from this group and stop receiving emails from it, send an email to kaldi-help+unsubscribe@googlegroups.com.
--
Vimal Manohar
PhD Student
Electrical & Computer Engineering
Johns Hopkins University

--
Go to http://kaldi-asr.org/forums.html find out how to join
---
You received this message because you are subscribed to the Google Groups "kaldi-help" group.
To unsubscribe from this group and stop receiving emails from it, send an email to kaldi-help+unsubscribe@googlegroups.com.

To post to this group, send email to kaldi...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/kaldi-help/CAGLZPU-VfAaEQh-GeUOUztssidXqqg1_f69PD_ioN3u-iTsvNQ%40mail.gmail.com.

Joshua Meyer

unread,
Feb 24, 2018, 1:29:27 PM2/24/18
to kaldi...@googlegroups.com
For my run script, I didn't change anything essential to Pegah's original script, mostly some lines for defining variables. The pipeline is identical.

That is, I think any solution for Pegah's script should work for me.

Here's my version of Pegah's original script: 


The visualization script gets frame accuracy data formatted from this scrpt: https://github.com/JRMeyer/kaldi-mirror/blob/master/egs/kgz/kyrgyz-model/utils/format_accuracy_for_plot.sh

-josh




суббота, 24 февраля 2018 г. пользователь Joshua Meyer написал:
This is the main run script I'm working of off:

egs / babel_multilang/s5/local/nnet3/run_tdnn_multilingual.sh

-josh


суббота, 24 февраля 2018 г. пользователь Daniel Povey написал:
What scripts are you using?   Give the full pathname from the root of the Kaldi repo.
Reply all
Reply to author
Forward
0 new messages