SpecAugment

[Wonkyum Lee]

https://arxiv.org/pdf/1904.08779.pdf

Probably, it's worth replicating this augmentation with kaldi chain model. Has any one tried this approach before? 

[Daniel Povey]

Yes, we are aware of this and hope to do some work on it within the next couple of weeks.

On Mon, Apr 22, 2019 at 6:54 PM Wonkyum Lee <won...@gridspace.com> wrote:

https://arxiv.org/pdf/1904.08779.pdf

Probably, it's worth replicating this augmentation with kaldi chain model. Has any one tried this approach before? 

--
Go to http://kaldi-asr.org/forums.html find out how to join
---
You received this message because you are subscribed to the Google Groups "kaldi-help" group.
To unsubscribe from this group and stop receiving emails from it, send an email to kaldi-help+...@googlegroups.com.
To post to this group, send email to kaldi...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/kaldi-help/15e81bf7-9148-441b-afec-52b435b8f7d1%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[jianwei zhou]

Any update?

在 2019年4月23日星期二 UTC+8上午6:58:28，Dan Povey写道：

Yes, we are aware of this and hope to do some work on it within the next couple of weeks.

On Mon, Apr 22, 2019 at 6:54 PM Wonkyum Lee <won...@gridspace.com> wrote:

https://arxiv.org/pdf/1904.08779.pdf

Probably, it's worth replicating this augmentation with kaldi chain model. Has any one tried this approach before? 

--
Go to http://kaldi-asr.org/forums.html find out how to join
---
You received this message because you are subscribed to the Google Groups "kaldi-help" group.

To unsubscribe from this group and stop receiving emails from it, send an email to kaldi...@googlegroups.com.

[shaheen kader]

@Prof Dan, Eagerly looking forward to this. 

[Daniel Povey]

I believe Phani Nidadavolu might have some preliminary results on this.  He saw some improvements but they were

very small--nothing like what the Google people saw.  But he's looking into it more I think.

Dan

To unsubscribe from this group and stop receiving emails from it, send an email to kaldi-help+...@googlegroups.com.

To post to this group, send email to kaldi...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/kaldi-help/13dd04fe-45fe-4b8f-ade7-03a537640ac3%40googlegroups.com.

[Jeff Brower]

Dan-

The paper does mention "handcrafted" so warping, masking, etc might need a methodology of its own.  I wonder if they ran through a series of augmentation combinations automatically, over time, to find which ones work best.

-Jeff

[Daniel Povey]

Yeah, they must have done some kind of tuning.

With regard to the time-warping part of what they did:

I don't believe this should be fundamentally different from the speed augmentation that we routinely do.

Our initial paper did show comparisons with the simple speed modification, versus a reconstruction that

makes it faster but spectrally the same, and the simple one worked better.   The reconstruction 

version (in our old experiments) should be quite similar to their time-warping, I would think.

Dan

To unsubscribe from this group and stop receiving emails from it, send an email to kaldi-help+...@googlegroups.com.

To post to this group, send email to kaldi...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/kaldi-help/1b3227a7-f48f-4193-9098-f9cd34d4478c%40googlegroups.com.

[Rudolf A. Braun]

Apparently time warping is not important: https://github.com/Kyubyong/specAugment

I got an okay improvement with the very first implementation I tried. I just do the augmentation once before training though. I think it would be better to do the augmentation on-the-fly during training, so it never sees the same augmentation again, but I'm not sure how one would do that.

[Daniel Povey]

You're right, it would be better to do it on the fly.

It just occurred to me possibly the easiest way would be to do it as an nnet layer.  Would be a "non-simple" one so might be easier if I do it myself.

Of course you'd have to transform from MFCCs back to log-mels first, but that's something we already do in some recipes (e.g. CNN).

Dan

To unsubscribe from this group and stop receiving emails from it, send an email to kaldi-help+...@googlegroups.com.

To post to this group, send email to kaldi...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/kaldi-help/8a00c1d0-fbd5-4b7e-8acb-46f1037e0f85%40googlegroups.com.

[fei]

Hi ,Dan

    Is there the implement  related to the SpecAugment in kaldi ？

Thank you 

在 2019年5月18日星期六 UTC+8上午2:26:56，Dan Povey写道：

[reza ali]

Hi

excuse me, in pr #3119 https://github.com/kaldi-asr/kaldi/pull/3119

have some experiment to use data augmentation for asr.

I don't know to follow that paper or not?  

[Daniel Povey]

We (well, mostly Phani Sankar and Ashish Arora)  have been doing tons of experiments trying to reproduce SpecAugment but we have not got any success.  I have heard of experiments at IDIAP and at Facebook that have been done with similar results.  People have had improvements versus a non-augmented baseline, but not against standard augmentation methods.  We are still trying various things, but at this pont it doesn't look promising.

Dan

--

Go to http://kaldi-asr.org/forums.html find out how to join
---
You received this message because you are subscribed to the Google Groups "kaldi-help" group.

To unsubscribe from this group and stop receiving emails from it, send an email to kaldi-help+...@googlegroups.com.

To post to this group, send email to kaldi...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/kaldi-help/c2a4d038-e549-440b-8481-0c38bf8396db%40googlegroups.com.

[Daniel Povey]

After hearing more information, I think I may have mis-stated the results.  There are some results which are somewhat encouraging after all, I think, from at least one of these places.  We're continuing to work on it.

I don't think the results will be anything like as dramatic as what the Google people saw, but it looks like there may be some improvement over our standard augmentation setup.

Dan

[Rémi Francis]

I talked to the first author of SpecAugment, he said that they used 900 epochs for LibriSpeech for their best results, but start noticing improvements over the baseline after 200 epochs.

On Tuesday, 18 June 2019 15:30:52 UTC+1, Dan Povey wrote:

We (well, mostly Phani Sankar and Ashish Arora)  have been doing tons of experiments trying to reproduce SpecAugment but we have not got any success.  I have heard of experiments at IDIAP and at Facebook that have been done with similar results.  People have had improvements versus a non-augmented baseline, but not against standard augmentation methods.  We are still trying various things, but at this pont it doesn't look promising.

Dan

On Tue, Jun 18, 2019 at 8:57 AM reza ali <aliiire...@gmail.com> wrote:

Hi

excuse me, in pr #3119 https://github.com/kaldi-asr/kaldi/pull/3119

have some experiment to use data augmentation for asr.

I don't know to follow that paper or not?  

--
Go to http://kaldi-asr.org/forums.html find out how to join
---
You received this message because you are subscribed to the Google Groups "kaldi-help" group.

To unsubscribe from this group and stop receiving emails from it, send an email to kaldi...@googlegroups.com.

[Armando]

900 epochs? on a 1000 h corpus?

[Rémi Francis]

The paper states 32 TPUs for 7 days, that's 4 epochs per TPU per day, it seems in the right range.

[Daniel Povey]

OK.  Here at Hopkins we initially weren't getting any improvement at all from SpecAugment-style frequency masking, but did start to see some improvement after increasing  the num-epochs.  So there might be some hope, but we have to confirm.

Dan

To unsubscribe from this group and stop receiving emails from it, send an email to kaldi-help+...@googlegroups.com.

To post to this group, send email to kaldi...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/kaldi-help/beb5698e-305a-4857-aaa1-56d2660deade%40googlegroups.com.

[Shujian Liu]

"The paper states 32 TPUs for 7 days"

That is 43,000 dollars for each run!

[Jeff Brower]

Shujian-

Not if you're running on TPUs at a data center on the Columbia River.

-Jeff

[Daniel Povey]

Update:

we have managed to get some improvements out of SpecAugment, at least on a small dataset (which

is where we'd expect the most improvement).  See here:

https://github.com/kaldi-asr/kaldi/pull/3449/files

there is about 10% relative improvement on mini-Librispeech, using both CNN+TDNN-F and TDNN-F-only topologies.

(Note: on this setup, CNN+TDNN-F is about 10% or more (relative) better than TDNN-F only, but this

is atypical; on larger data, they are much closer, and I'd normally use TDNN-F only, for speed)

Dan

To unsubscribe from this group and stop receiving emails from it, send an email to kaldi-help+...@googlegroups.com.

To post to this group, send email to kaldi...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/kaldi-help/e839e985-7422-4b51-9f12-4cc781314fb5%40googlegroups.com.

[Daniel Povey]

Not sure what you're doing there, but all my changes are checked into that branch, I just confirmed, and it is working.

You may have mixed versions.

On Mon, Jul 8, 2019 at 6:03 AM Armando <armando.m...@gmail.com> wrote:

do we have to add

spec-augment-layer

to config_to_layer in parser.py?

To view this discussion on the web visit https://groups.google.com/d/msgid/kaldi-help/e839e985-7422-4b51-9f12-4cc781314fb5%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--
Go to http://kaldi-asr.org/forums.html find out how to join
---
You received this message because you are subscribed to the Google Groups "kaldi-help" group.
To unsubscribe from this group and stop receiving emails from it, send an email to kaldi-help+...@googlegroups.com.
To post to this group, send email to kaldi...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/kaldi-help/4304eb8e-8f69-49ab-b66f-9b2c0bcc8e89%40googlegroups.com.

[Armando]

yes, I thought i had deleted my previous message, it's ok

[Armando]

I see that 1i and 1h (with and without SpecAugment) have the same number of epochs. Didn't you notice imorovements by increasing the number of epochs?

On Monday, July 8, 2019 at 6:51:17 AM UTC+2, Dan Povey wrote:

[Daniel Povey]

I didn't try increasing the number of epochs.  Go ahead and try (also increasing the number of parameters), if it gives further improvements we can update the example.

To unsubscribe from this group and stop receiving emails from it, send an email to kaldi-help+...@googlegroups.com.

To post to this group, send email to kaldi...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/kaldi-help/f45f614a-8f95-4279-9c18-ffb138bb9a25%40googlegroups.com.

[Rémi Francis]

Have you got results on a bigger dataset?

[Daniel Povey]

Not yet, but we're working on it.

To unsubscribe from this group and stop receiving emails from it, send an email to kaldi-help+...@googlegroups.com.

To post to this group, send email to kaldi...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/kaldi-help/7b69692a-894f-4e83-b758-a1cfc4b7cc34%40googlegroups.com.

[Armando]

I'm training on 600h data set, so I'm using the swbd set up with specaugment, but so far, it's degrading performance with respect to the same iteration of the specaugment-less model (about 15% relative worse), but I've done not even 2 epochs, maybe I'll see something better as training goes on

[Rudolf A. Braun]

Is that without dropout (as in did you remember to turn it off)?

[Daniel Povey]

Turning off dropout may not be important; it didn't affect WER in mini-librispeech and I removed it to simplify the setup.

Ashish was also doing experiments on Switchboard with this SpecAugment setup, and also didn't see improvements.

Bear in mind that the Kaldi recipes use unusually small models and, of course, already use augmentation, so overfitting may be less of an issue than for more typical setups.

Ashish is trying larger models and more epochs to see if it starts helping then.

Dan

Dan

To unsubscribe from this group and stop receiving emails from it, send an email to kaldi-help+...@googlegroups.com.

To post to this group, send email to kaldi...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/kaldi-help/a857cbe2-b364-48ae-975c-89f08140f933%40googlegroups.com.

[orum farhang]

Hi,

Does anyone have a good news on Spec Augment using Kaldi receipes? Do you guys have any setup suggestion (model size, parameters, model type etc.) to start with it and try this?

Many Thanks

[ashish arora]

I did experiments with switchboard as training set and eval2000 as test set. The aim of these experiments was to try to make specAugmentation frequency part work for switchboard setup. The setup was similar to specAugment implementation (using utterance level mean for masking).  These experiments were done without i-vectors and used clean data for tree building. I tried 80-dimensional features, more epochs, 15 and 27 coefficients for masking. It helped in improving results from 14.1% WER to 13.4% WER. Its results were as follows:

Modification	eval2000
Speed perturbation (40 dim)	14.1
Frequency masking (40 dim) (13 coeff) (550 iter)	13.7
Frequency masking (80 dim) (27 coeff) (550 iter)	13.6
Frequency masking and speed perturbation (80 dim) (27 coeff) (550 iter)	13.4

- After there was 10% relative improvement for the minilibrispeech with specAugmentation . I started doing experiments with the new online implementation of specAugment with the default parameters for switchboard and AMI datasets. In addition to using 80 dimensional features and more epochs as above, I did experiments with larger model. I tried both recurrent networks and wider TDNN networks. TDNN model performed better than LSTM model. For Swbd the results improved from 12.9% WER to 12.6% WER and for AMI the WER improved from 35.1% WER to 34.9% WER. I am not able to get improvement on significant switchboard and AMI yet. 

But, during the experiments for AMI setup, it helped in finding a better tuned tdnn model and improved the results from 40.5% WER on test to 35.1% WER. Also, 80 dimensional features worked better as compared to 40 dimensional features for AMI dataset. The WER with 80 dimensional feature is 35.1% WER on test and WER with 40 dimensional feature is 35.7% WER. Its results are as follows:

EXP	WER dev	WER eval	Remark
0	36.6	40.5	Default model 8M parameters (40 dimension)
1	32.5	36	Baseline (34M parameters) (80 dimension)
2	-	35.7	Baseline + dropout (40 dimension)
3	31.7	35.1	Baseline + dropout (80 dimension)
4	31.3	34.9	Baseline + spec augment (80 dimension)

Thanks,

Ashish

--
Go to http://kaldi-asr.org/forums.html find out how to join
---
You received this message because you are subscribed to the Google Groups "kaldi-help" group.
To unsubscribe from this group and stop receiving emails from it, send an email to kaldi-help+...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/kaldi-help/307de052-f71a-4b29-a146-1acd488d77c3%40googlegroups.com.

[Itai Peer]

thanks Ashish for the answer and the results

I looked in RESULTS for kadli-swbd recipes and saw better resutls with several recipes , like with re-scoring , and i-vec  

assuming of-course i did not mix with training-testing sets   , there are several results with fisher mixed over-there , i ignored that.. 

for example   

https://github.com/kaldi-asr/kaldi/blob/09abddaccee68833fa28efb102f021f9b47803be/egs/swbd/s5c/RESULTS#L173

do you think that spec-augmentation can improve further even if the most heavy guns in the arsenal are used as well ? 

I'm not too familiars with SWB , so i'm not sure if i-vec for example can benefit too much here ,since it is quite "clean" corpora , all telephones , , 2 speakers ,  ( i dont remember if it was streo or mono recording , so diarization was issue there? )  

בתאריך יום שני, 9 בספטמבר 2019 בשעה 06:47:01 UTC+3, מאת ashish arora:

To unsubscribe from this group and stop receiving emails from it, send an email to kaldi...@googlegroups.com.

[ashish arora]

The results shared for the switchboard experiments were without i-vectors. The results are based on switchboard run_tdnn_7q recipe but without i-vectors. On the baseline, i-vectors help by around 10%. I haven't done the experiment with including ivectors for switchboard but AMI results include i-vectors also. Yeah, it is quite "clean" stereo recoding but I guess diarization is not major issue here as speakers are mostly taking turns while speaking (little overlapping regions).

To unsubscribe from this group and stop receiving emails from it, send an email to kaldi-help+...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/kaldi-help/5da931df-ca17-4c11-9d03-f24b89220ede%40googlegroups.com.

[Daniel Povey]

Ashish: maybe part of the issue was that my implementation masks with zeros, not with the utterance level mean.  It's after batchnorm, so it's like using he global mean.  But perhaps if it were combined with online CMVN, like what we recently merged from Karel, it would be better than the current best baseline?  (Because then the utterance-level mean would be essentially zero.) . Perhaps you could try that?  (I don't recall whether our current Swbd example uses online-CMVN.) . If not I hope you'll find out what differences between my nnet3 implementation and their paper might be responsible for the difference.

To view this discussion on the web visit https://groups.google.com/d/msgid/kaldi-help/CAGK0eznhhJ1rQCjvXjLUcbyTjCkbdU3kLrv1y-xcGxXJ4-nx8g%40mail.gmail.com.

[ashish arora]

Ok, thanks. I will do the experiments combining with online CMVN and will compare the implementation with the paper.

To view this discussion on the web visit https://groups.google.com/d/msgid/kaldi-help/CAEWAuyTOE5c%2B%2BJYjo1zP0iLr7KCkPhaG9uuy%3DaBLkOxKwJrPAA%40mail.gmail.com.

[reza ali]

This is an implement of SpecAugment using PyTorch, Is it useful?

https://github.com/zcaceres/spec_augment

https://github.com/keunwoochoi/torchaudio-contrib/issues/44

----

Do the results improve?

[mili lali]

New paper "SPECAUGMENT ON LARGE SCALE DATASETS"

https://arxiv.org/pdf/1912.05533.pdf