run_tdnn_1n.sh 실행 중 steps/combine_ali_dirs.sh 관련 에러 문의

[dboo]

안녕하세요. 제로스를 입문하는 제린이 입니다.

다름이 아니라 제가 열심히 구글그룹의 글들과 인터넷블로그와 프로젝트내의 지침서들을 읽으며 학습을 시키고 있는데 해당 에러에 대한 내용의 해결법을 찾지못해 이렇게 문의 드립니다.

일단 칼디를 컴파일하는데에는 문제없이 완료한것 같(?)구요.

제로스 프로젝트를 받아서 run_openslr.sh를 통해 GMM트레이닝을 마치고

s5/local/chain/multi_condition/run_tdnn_1n.sh 를 실행하였습니다.

맨 마지막에 콘솔에 찍힌 내용은 다음과 같습니다.

~~~

Succeeded creating CMVN stats for train_clean_sp_rvb1_hires

local/multi_condition/copy_ali_dir.sh: copied alignments from exp/tri4b_ali_train_clean_sp to exp/tri4b_ali_train_clean_sp_temp_0

local/multi_condition/copy_ali_dir.sh: copied alignments from exp/tri4b_ali_train_clean_sp to exp/tri4b_ali_train_clean_sp_temp_1

steps/combine_ali_dirs.sh data/train_clean_sp_rvb1 exp/tri4b_ali_train_clean_sp_rvb exp/tri4b_ali_train_clean_sp_temp_0 exp/tri4b_ali_train_clean_sp_temp_1

steps/combine_ali_dirs.sh: warning: Alignment lattices (lat.*.gz) are not present in exp/tri4b_ali_train_clean_sp_temp_0, not combining. Consider '--combine_lat false' to suppress this warning.

steps/combine_ali_dirs.sh: note: Temporary directory exp/tri4b_ali_train_clean_sp_rvb/temp.Ksj2Q8 will not be deleted in case of script failure, so you could examine it for troubleshooting.

steps/combine_ali_dirs.sh: Gathering alignments from each source directory.

run.pl: 40 / 40 failed, log is in exp/tri4b_ali_train_clean_sp_rvb/log/gather_alignments.*.log

~~~

고수님들 알려주시면 감사하겠습니다.

현재 개발환경은

- Ubuntu 20.04 LTS

- Tesla T4 GPU

- CPU(s): 32 , thread per core : 2

입니다.

run_tdnn_1n.sh 에서는

- nj =30

- num_jobs_initial=1

- num_jobs_final=1

으로만 변경한 상태입니다.

~~~

run.pl: 40 / 40 failed, log is in exp/tri4b_ali_train_clean_sp_rvb/log/gather_alignments.*.log

~~~

위 로그의 내용은 다음과 같았습니다.

~~~

# Started at Fri Sep 10 10:22:34 KST 2021

# copy-int-vector 'ark:gunzip -c exp/tri4b_ali_train_clean_sp_temp_1/ali.20.gz |' ark,scp:exp/tri4b_ali_train_clean_sp_rvb/temp.myFBBW/ali.40.ark,exp/tri4b_ali_train_clean_sp_rvb/temp.myFBBW/ali.40.scp

LOG (copy-int-vector[5.5.965~1-e6cc1]:main():copy-int-vector.cc:83) Copied 0 vectors of int32.

# Accounting: time=0 threads=1

# Ended (code 1) at Fri Sep 10 10:22:34 KST 2021, elapsed time 0 seconds

~~~

export -p 로 확인할 수 있는(호출 위치 : zeroth/s5/) 적용되어있는 환경변수입니다.

~~~

declare -x DBUS_SESSION_BUS_ADDRESS="unix:path=/run/user/1000/bus"

declare -x HOME="/root"

declare -x LANG="ko_KR.UTF-8"

declare -x LC_ADDRESS="ko_KR.UTF-8"

declare -x LC_ALL="ko_KR.UTF-8"

declare -x LC_IDENTIFICATION="ko_KR.UTF-8"

declare -x LC_MEASUREMENT="ko_KR.UTF-8"

declare -x LC_MONETARY="ko_KR.UTF-8"

declare -x LC_NAME="ko_KR.UTF-8"

declare -x LC_NUMERIC="ko_KR.UTF-8"

declare -x LC_PAPER="ko_KR.UTF-8"

declare -x LC_TELEPHONE="ko_KR.UTF-8"

declare -x LC_TIME="ko_KR.UTF-8"

declare -x LESSCLOSE="/usr/bin/lesspipe %s %s"

declare -x LESSOPEN="| /usr/bin/lesspipe %s"

declare -x LOGNAME="ucsit-dl"

declare -x S_COLORS=".xspf=00;36: ...." (길어서 생략)

declare -x MAIL="/var/mail/root"

declare -x MOTD_SHOWN="pam"

declare -x OLDPWD="/home/ucsit-dl/workspace/zeroth/s5/local"

declare -x PATH="/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin"

declare -x PWD="/home/ucsit-dl/workspace/zeroth/s5"

declare -x SHELL="/bin/bash"

declare -x SHLVL="2"

declare -x TERM="xterm-256color"

declare -x XDG_DATA_DIRS="/usr/local/share:/usr/share:/var/lib/snapd/desktop"

declare -x XDG_RUNTIME_DIR="/run/user/1000"

declare -x XDG_SESSION_CLASS="user"

declare -x XDG_SESSION_ID="483"

declare -x XDG_SESSION_TYPE="tty"

~~~

답변주시면 감사하겠습니다!

[이승현]

exp/tri4b_ali_train_clean_sp_temp_1/ali.*.gz 이 파일들이 정상적으로 생성이 안되면 이런 에러가 발생하는 것 같습니다.

ali.*.gz 파일을 열어보시거나 파일 크기가 0인지 확인해보세요. 만약 그렇다면 ali.*.gz가 제대로 생성이 안된 것이니

alignment를 생성하는 부분을 보시면 될 것 같습니다.

2021년 9월 10일 금요일 오후 2시 43분 35초 UTC+9에 dboo님이 작성:

[dboo]

일단 답변 감사드립니다.

말씀하신대로 해당 디렉토리에 파일 크기를 확인해 보았는데 3~4메가 정도로 나왔는데 정상적인 파일 크기일까요...?

파일 크기가 0이 아닌것을 확인하고 그냥 무지성으로 한번 더 run_tdnn_n1.sh을 돌려보았습니다.

다시 돌려서 그런지 그 전과는 로그가 달랐는데, 맨 마지막 줄에  ... not sure what to do.  Exiting. 라고 나오면서 끝이 나네요..

export.sh 를 실행하면 chain_rvb 디렉토리 등을 찾을수 없다고 나와 제대로 수행되지 않은것은 알것 같은데..

원인을 찾지 못하니 답답하네요..ㅜㅜ

로그전체를 하나하나 훑어보는데 warning이 굉장히 많이 나옵니다.. 이것때문일까요?

답변 주시면 감사하겠습니다.

~~~

./local/chain/multi_condition/run_tdnn_1n.sh

utils/perturb_data_dir_speed.sh: generated speed-perturbed version of data in data/train_clean, in data/temp1

fix_data_dir.sh: kept all 22263 utterances.

fix_data_dir.sh: old files are kept in data/temp1/.backup

utils/validate_data_dir.sh: Successfully validated data-directory data/temp1

utils/perturb_data_dir_speed.sh: generated speed-perturbed version of data in data/train_clean, in data/temp2

fix_data_dir.sh: kept all 22263 utterances.

fix_data_dir.sh: old files are kept in data/temp2/.backup

utils/validate_data_dir.sh: Successfully validated data-directory data/temp2

utils/combine_data.sh data/train_clean_tmp data/temp1 data/temp2

utils/combine_data.sh: combined utt2uniq

utils/combine_data.sh [info]: not combining segments as it does not exist

utils/combine_data.sh: combined utt2spk

utils/combine_data.sh [info]: not combining utt2lang as it does not exist

utils/combine_data.sh: combined utt2dur

utils/combine_data.sh [info]: not combining utt2num_frames as it does not exist

utils/combine_data.sh: combined reco2dur

utils/combine_data.sh [info]: not combining feats.scp as it does not exist

utils/combine_data.sh: combined text

utils/combine_data.sh [info]: not combining cmvn.scp as it does not exist

utils/combine_data.sh [info]: not combining vad.scp as it does not exist

utils/combine_data.sh [info]: not combining reco2file_and_channel as it does not exist

utils/combine_data.sh: combined wav.scp

utils/combine_data.sh: combined spk2gender

fix_data_dir.sh: kept all 44526 utterances.

fix_data_dir.sh: old files are kept in data/train_clean_tmp/.backup

utils/validate_data_dir.sh: Successfully validated data-directory data/train_clean_tmp

steps/make_mfcc.sh --cmd run.pl --mem 2G --nj 20 data/train_clean_tmp exp/make_mfcc/train_clean_tmp mfcc_perturbed

utils/validate_data_dir.sh: Successfully validated data-directory data/train_clean_tmp

steps/make_mfcc.sh: [info]: no segments file exists: assuming wav.scp indexed by utterance.

steps/make_mfcc.sh: Succeeded creating MFCC features for train_clean_tmp

steps/compute_cmvn_stats.sh data/train_clean_tmp exp/make_mfcc/train_clean_tmp mfcc_perturbed

Succeeded creating CMVN stats for train_clean_tmp

fix_data_dir.sh: kept all 44526 utterances.

fix_data_dir.sh: old files are kept in data/train_clean_tmp/.backup

utils/copy_data_dir.sh: copied data from data/train_clean to data/temp0

utils/copy_data_dir.sh: file utt2uniq exists in dest data/temp0 but not in src data/train_clean.  Moving it to

 ... data/temp0/.backup/utt2uniq

utils/validate_data_dir.sh: Successfully validated data-directory data/temp0

utils/combine_data.sh data/train_clean_sp data/train_clean_tmp data/temp0

utils/combine_data.sh: combined utt2uniq

utils/combine_data.sh [info]: not combining segments as it does not exist

utils/combine_data.sh: combined utt2spk

utils/combine_data.sh [info]: not combining utt2lang as it does not exist

utils/combine_data.sh: combined utt2dur

utils/combine_data.sh: combined utt2num_frames

utils/combine_data.sh: combined reco2dur

utils/combine_data.sh: combined feats.scp

utils/combine_data.sh: combined text

utils/combine_data.sh: combined cmvn.scp

utils/combine_data.sh [info]: not combining vad.scp as it does not exist

utils/combine_data.sh [info]: not combining reco2file_and_channel as it does not exist

utils/combine_data.sh: combined wav.scp

utils/combine_data.sh: combined spk2gender

fix_data_dir.sh: kept all 66789 utterances.

fix_data_dir.sh: old files are kept in data/train_clean_sp/.backup

fix_data_dir.sh: kept all 66789 utterances.

fix_data_dir.sh: old files are kept in data/train_clean_sp/.backup

steps/align_fmllr.sh --nj 20 --cmd run.pl --mem 2G data/train_clean_sp data/lang_nosp exp/tri4b exp/tri4b_ali_train_clean_sp

steps/align_fmllr.sh: feature type is lda

steps/align_fmllr.sh: compiling training graphs

steps/align_fmllr.sh: aligning data in data/train_clean_sp using exp/tri4b/final.alimdl and speaker-independent features.

steps/align_fmllr.sh: computing fMLLR transforms

steps/align_fmllr.sh: doing final alignment.

steps/align_fmllr.sh: done aligning data.

steps/diagnostic/analyze_alignments.sh --cmd run.pl --mem 2G data/lang_nosp exp/tri4b_ali_train_clean_sp

analyze_phone_length_stats.py: WARNING: optional-silence SIL is seen only 60.6164280332% of the time at utterance end.  This may not be optimal.

steps/diagnostic/analyze_alignments.sh: see stats in exp/tri4b_ali_train_clean_sp/log/analyze_alignments.log

steps/diagnostic/analyze_alignments.sh --cmd run.pl --mem 2G data/lang_nosp exp/tri4b_ali_train_clean_sp

analyze_phone_length_stats.py: WARNING: optional-silence SIL is seen only 60.6164280332% of the time at utterance end.  This may not be optimal.

steps/diagnostic/analyze_alignments.sh: see stats in exp/tri4b_ali_train_clean_sp/log/analyze_alignments.log

2606 warnings in exp/tri4b_ali_train_clean_sp/log/align_pass1.*.log

1 warnings in exp/tri4b_ali_train_clean_sp/log/analyze_alignments.log

698 warnings in exp/tri4b_ali_train_clean_sp/log/fmllr.*.log

2755 warnings in exp/tri4b_ali_train_clean_sp/log/align_pass2.*.log

local/nnet3/multi_condition/run_ivector_common.sh: creating reverberated MFCC features

utils/combine_data.sh data/train_clean_sp_rvb1 data/train_clean_sp/split20/1_rvb1 data/train_clean_sp/split20/2_rvb1 data/train_clean_sp/split20/3_rvb1 data/train_clean_sp/split20/4_rvb1 data/train_clean_sp/split20/5_rvb1 data/train_clean_sp/split20/6_rvb1 data/train_clean_sp/split20/7_rvb1 data/train_clean_sp/split20/8_rvb1 data/train_clean_sp/split20/9_rvb1 data/train_clean_sp/split20/10_rvb1 data/train_clean_sp/split20/11_rvb1 data/train_clean_sp/split20/12_rvb1 data/train_clean_sp/split20/13_rvb1 data/train_clean_sp/split20/14_rvb1 data/train_clean_sp/split20/15_rvb1 data/train_clean_sp/split20/16_rvb1 data/train_clean_sp/split20/17_rvb1 data/train_clean_sp/split20/18_rvb1 data/train_clean_sp/split20/19_rvb1 data/train_clean_sp/split20/20_rvb1

utils/combine_data.sh: combined utt2uniq

utils/combine_data.sh [info]: not combining segments as it does not exist

utils/combine_data.sh: combined utt2spk

utils/combine_data.sh [info]: not combining utt2lang as it does not exist

utils/combine_data.sh [info]: not combining utt2dur as it does not exist

utils/combine_data.sh [info]: not combining utt2num_frames as it does not exist

utils/combine_data.sh [info]: not combining reco2dur as it does not exist

utils/combine_data.sh [info]: not combining feats.scp as it does not exist

utils/combine_data.sh: combined text

utils/combine_data.sh [info]: not combining cmvn.scp as it does not exist

utils/combine_data.sh [info]: not combining vad.scp as it does not exist

utils/combine_data.sh [info]: not combining reco2file_and_channel as it does not exist

utils/combine_data.sh: combined wav.scp

utils/combine_data.sh [info]: not combining spk2gender as it does not exist

fix_data_dir.sh: kept all 133578 utterances.

fix_data_dir.sh: old files are kept in data/train_clean_sp_rvb1/.backup

utils/copy_data_dir.sh: copied data from data/train_clean_sp_rvb1 to data/train_clean_sp_rvb1_hires

utils/copy_data_dir.sh: file feats.scp exists in dest data/train_clean_sp_rvb1_hires but not in src data/train_clean_sp_rvb1.  Moving it to

 ... data/train_clean_sp_rvb1_hires/.backup/feats.scp

utils/copy_data_dir.sh: file cmvn.scp exists in dest data/train_clean_sp_rvb1_hires but not in src data/train_clean_sp_rvb1.  Moving it to

 ... data/train_clean_sp_rvb1_hires/.backup/cmvn.scp

utils/copy_data_dir.sh: file utt2dur exists in dest data/train_clean_sp_rvb1_hires but not in src data/train_clean_sp_rvb1.  Moving it to

 ... data/train_clean_sp_rvb1_hires/.backup/utt2dur

utils/copy_data_dir.sh: file utt2num_frames exists in dest data/train_clean_sp_rvb1_hires but not in src data/train_clean_sp_rvb1.  Moving it to

 ... data/train_clean_sp_rvb1_hires/.backup/utt2num_frames

utils/copy_data_dir.sh: file frame_shift exists in dest data/train_clean_sp_rvb1_hires but not in src data/train_clean_sp_rvb1.  Moving it to

 ... data/train_clean_sp_rvb1_hires/.backup/frame_shift

utils/validate_data_dir.sh: Successfully validated data-directory data/train_clean_sp_rvb1_hires

utils/data/perturb_data_dir_volume.sh: added volume perturbation to the data in data/train_clean_sp_rvb1_hires

steps/make_mfcc.sh --nj 20 --mfcc-config conf/mfcc_hires.conf --cmd run.pl --mem 2G data/train_clean_sp_rvb1_hires exp/make_hires/train_clean_sp_rvb1 mfcc_rvb

utils/validate_data_dir.sh: Successfully validated data-directory data/train_clean_sp_rvb1_hires

steps/make_mfcc.sh: [info]: no segments file exists: assuming wav.scp indexed by utterance.

steps/make_mfcc.sh: Succeeded creating MFCC features for train_clean_sp_rvb1_hires

steps/compute_cmvn_stats.sh data/train_clean_sp_rvb1_hires exp/make_hires/train_clean_sp_rvb1 mfcc_rvb

Succeeded creating CMVN stats for train_clean_sp_rvb1_hires

local/multi_condition/copy_ali_dir.sh: copied alignments from exp/tri4b_ali_train_clean_sp to exp/tri4b_ali_train_clean_sp_temp_0

local/multi_condition/copy_ali_dir.sh: copied alignments from exp/tri4b_ali_train_clean_sp to exp/tri4b_ali_train_clean_sp_temp_1

steps/combine_ali_dirs.sh data/train_clean_sp_rvb1 exp/tri4b_ali_train_clean_sp_rvb exp/tri4b_ali_train_clean_sp_temp_0 exp/tri4b_ali_train_clean_sp_temp_1

steps/combine_ali_dirs.sh: warning: Alignment lattices (lat.*.gz) are not present in exp/tri4b_ali_train_clean_sp_temp_0, not combining. Consider '--combine_lat false' to suppress this warning.

steps/combine_ali_dirs.sh: note: Temporary directory exp/tri4b_ali_train_clean_sp_rvb/temp.oSLvxv will not be deleted in case of script failure, so you could examine it for troubleshooting.

steps/combine_ali_dirs.sh: Gathering alignments from each source directory.

steps/combine_ali_dirs.sh: Splitting combined alignments into 4 archives on speaker boundary.

steps/combine_ali_dirs.sh: Combined 132700 alignments for 133578 utterances. There were 878 utterances (1%) without alignments, and 0 alignments not matching any utterance.

steps/combine_ali_dirs.sh: Stored combined alignments in exp/tri4b_ali_train_clean_sp_rvb

utils/subset_data_dir.sh: reducing #utt from 133578 to 100000

utils/subset_data_dir.sh: reducing #utt from 133578 to 30000

steps/train_lda_mllt.sh --cmd run.pl --mem 2G --num-iters 13 --realign-iters  --splice-opts --left-context=3 --right-context=3 3000 10000 data/train_100k_hires data/lang_nosp exp/tri4b_ali_train_clean_sp_rvb exp/nnet3_rvb/tri2b

steps/train_lda_mllt.sh: Accumulating LDA statistics.

steps/train_lda_mllt.sh: Accumulating tree stats

steps/train_lda_mllt.sh: Getting questions for tree clustering.

steps/train_lda_mllt.sh: Building the tree

steps/train_lda_mllt.sh: Initializing the model

WARNING (gmm-init-model[5.5.965~1-e6cc1]:InitAmGmm():gmm-init-model.cc:55) Tree has pdf-id 1 with no stats; corresponding phone list: 6 7 8 9 10

This is a bad warning.

steps/train_lda_mllt.sh: Converting alignments from exp/tri4b_ali_train_clean_sp_rvb to use current tree

Training pass 1

Training pass 2

steps/train_lda_mllt.sh: Estimating MLLT

Training pass 3

Training pass 4

steps/train_lda_mllt.sh: Estimating MLLT

Training pass 5

Training pass 6

steps/train_lda_mllt.sh: Estimating MLLT

Training pass 7

Training pass 8

Training pass 9

Training pass 10

Training pass 11

Training pass 12

steps/train_lda_mllt.sh: Estimating MLLT

steps/diagnostic/analyze_alignments.sh --cmd run.pl --mem 2G data/lang_nosp exp/nnet3_rvb/tri2b

analyze_phone_length_stats.py: WARNING: optional-silence SIL is seen only 60.6164280332% of the time at utterance end.  This may not be optimal.

steps/diagnostic/analyze_alignments.sh: see stats in exp/nnet3_rvb/tri2b/log/analyze_alignments.log

1 warnings in exp/nnet3_rvb/tri2b/log/analyze_alignments.log

981 warnings in exp/nnet3_rvb/tri2b/log/lda_acc.*.log

11772 warnings in exp/nnet3_rvb/tri2b/log/acc.*.*.log

12 warnings in exp/nnet3_rvb/tri2b/log/update.*.log

1 warnings in exp/nnet3_rvb/tri2b/log/init_model.log

1 warnings in exp/nnet3_rvb/tri2b/log/questions.log

1 warnings in exp/nnet3_rvb/tri2b/log/build_tree.log

exp/nnet3_rvb/tri2b: nj=4 230.89h data log-like=-52.31 states=2432 gauss=6080 tree-impr=3.57 lda-sum=12.00 mllt:impr,logdet=0.80,0.95

steps/train_lda_mllt.sh: Done training system with LDA+MLLT features in exp/nnet3_rvb/tri2b

steps/online/nnet2/train_diag_ubm.sh --cmd run.pl --mem 2G --nj 20 --num_threads 30 --num-frames 700000 data/train_30k_hires 512 exp/nnet3_rvb/tri2b exp/nnet3_rvb/diag_ubm

filter_scps.pl: warning: some input lines did not get output

steps/online/nnet2/train_diag_ubm.sh: Directory exp/nnet3_rvb/diag_ubm already exists. Backing up diagonal UBM in exp/nnet3_rvb/diag_ubm/backup.KcB

steps/online/nnet2/train_diag_ubm.sh: initializing model from E-M in memory,

steps/online/nnet2/train_diag_ubm.sh: starting from 256 Gaussians, reaching 512;

steps/online/nnet2/train_diag_ubm.sh: for 20 iterations, using at most 700000 frames of data

Getting Gaussian-selection info

steps/online/nnet2/train_diag_ubm.sh: will train for 4 iterations, in parallel over

steps/online/nnet2/train_diag_ubm.sh: 20 machines, parallelized with 'run.pl --mem 2G'

steps/online/nnet2/train_diag_ubm.sh: Training pass 0

steps/online/nnet2/train_diag_ubm.sh: Training pass 1

steps/online/nnet2/train_diag_ubm.sh: Training pass 2

steps/online/nnet2/train_diag_ubm.sh: Training pass 3

steps/online/nnet2/train_ivector_extractor.sh --cmd run.pl --mem 2G --nj 10 data/train_clean_sp_rvb1_hires exp/nnet3_rvb/diag_ubm exp/nnet3_rvb/extractor

steps/online/nnet2/train_ivector_extractor.sh: Directory exp/nnet3_rvb/extractor already exists. Backing up iVector extractor in exp/nnet3_rvb/extractor/backup.P7c

steps/online/nnet2/train_ivector_extractor.sh: doing Gaussian selection and posterior computation

Accumulating stats (pass 0)

Summing accs (pass 0)

Updating model (pass 0)

Accumulating stats (pass 1)

Summing accs (pass 1)

Updating model (pass 1)

Accumulating stats (pass 2)

Summing accs (pass 2)

Updating model (pass 2)

Accumulating stats (pass 3)

Summing accs (pass 3)

Updating model (pass 3)

Accumulating stats (pass 4)

Summing accs (pass 4)

Updating model (pass 4)

Accumulating stats (pass 5)

Summing accs (pass 5)

Updating model (pass 5)

Accumulating stats (pass 6)

Summing accs (pass 6)

Updating model (pass 6)

Accumulating stats (pass 7)

Summing accs (pass 7)

Updating model (pass 7)

Accumulating stats (pass 8)

Summing accs (pass 8)

Updating model (pass 8)

Accumulating stats (pass 9)

Summing accs (pass 9)

Updating model (pass 9)

utils/data/modify_speaker_info.sh: copied data from data/train_clean_sp_rvb1_hires to data/train_clean_sp_rvb1_hires_max2, number of speakers changed from 630 to 66936

utils/validate_data_dir.sh: Successfully validated data-directory data/train_clean_sp_rvb1_hires_max2

steps/online/nnet2/extract_ivectors_online.sh --cmd run.pl --mem 2G --nj 20 data/train_clean_sp_rvb1_hires_max2 exp/nnet3_rvb/extractor exp/nnet3_rvb/ivectors_train_clean_sp_rvb1_hires

steps/online/nnet2/extract_ivectors_online.sh: extracting iVectors

steps/online/nnet2/extract_ivectors_online.sh: combining iVectors across jobs

steps/online/nnet2/extract_ivectors_online.sh: done extracting (online) iVectors to exp/nnet3_rvb/ivectors_train_clean_sp_rvb1_hires using the extractor in exp/nnet3_rvb/extractor.

./local/chain/multi_condition/run_tdnn_1n.sh: creating lang directory data/lang_chain with chain-type topology

./local/chain/multi_condition/run_tdnn_1n.sh: data/lang_chain already exists and seems to be older than data/lang...

 ... not sure what to do.  Exiting.

~~~

2021년 9월 10일 금요일 오후 3시 7분 38초 UTC+9에 이승현님이 작성:

[이승현]

alignment 파일의 크기가 0이 아니라면 정상적인 것입니다.

...not sure what to do. Exiting. 으로 끝나면서 스크립트가 종료된 것은

data/lang_chain 디렉터리가 이미 존재하기 때문입니다.

그냥 data/lang_chain 디렉터리만 삭제하고 다시 학습을 진행하시면 될 것 같습니다.

로그에서 warning이 눈에 띄어도 대부분은 크리티컬한 warning은 아니니 크게 신경 쓰지 않으셔도 됩니다.

보통 데이터에 관한 로그인데 크리티컬한 정도를 넘어서면 아예 중단이 되니 중단이 되지 않는다면 그냥 참고만 하시면 됩니다.

2021년 9월 13일 월요일 오후 5시 33분 7초 UTC+9에 dboo님이 작성:

[dboo]

답변 감사합니다.

말씀 주신대로 data/lang_chain 디렉토리 삭제 후 다시 run_openslr.sh 실행했더니 최종단계까지 완료되었습니다.

감사합니다.!

2021년 9월 14일 화요일 오전 10시 42분 17초 UTC+9에 이승현님이 작성: