Bad FST header : data/lang/tmp/CLG_3_1.fst

[Pranay Shah]

Hi,

With triphone decoding I am getting an error as shown above.

The following are the things I have tried which were recommended for similar issues in other posts:

1) Increased memory upto 150GB (of a virtual instance)

2) Reduced $nj to 2

3) Ran validate_lang.pl (got all 'Ok's); validate_data_dir and validate_dict_dir (again no warnings)

4) Checked the existence of files lang/G.fst and lang/tmp/LG.fst (both existed with sizes above a few MBs)

Also running fstinfo G.fst gives;

fst type                                          vector
arc type                                          standard
input symbol table                                none
output symbol table                               none
# of states                                       37133
# of arcs                                         139664
initial state                                     1
# of final states                                 3892
# of input/output epsilons                        0
# of input epsilons                               0
# of output epsilons                              37132
input label multiplicity                          1
output label multiplicity                         1
# of accessible states                            37133
# of coaccessible states                          37133
# of connected states                             37133
# of connected components                         1
# of strongly conn components                     776
input matcher                                     y
output matcher                                    n
input lookahead                                   n
output lookahead                                  n
expanded                                          y
mutable                                           y
error                                             n
acceptor                                          n
input deterministic                               y
output deterministic                              y
input/output epsilons                             n
input epsilons                                    n
output epsilons                                   y
input label sorted                                y
output label sorted                               n
weighted                                          y
cyclic                                            y
cyclic at initial state                           n
top sorted                                        n
accessible                                        y
coaccessible                                      y
string                                            n
weighted cycles                                   y

while fstinfo L.fst gives:

fst type                                          vector
arc type                                          standard
input symbol table                                none
output symbol table                               none
# of states                                       521469
# of arcs                                         967571
initial state                                     0
# of final states                                 1
# of input/output epsilons                        1
# of input epsilons                               1
# of output epsilons                              738446
input label multiplicity                          236.505
output label multiplicity                         1.46111
# of accessible states                            521469
# of coaccessible states                          521469
# of connected states                             521469
# of connected components                         1
# of strongly conn components                     2
input matcher                                     n
output matcher                                    y
input lookahead                                   n
output lookahead                                  n
expanded                                          y
mutable                                           y
error                                             n
acceptor                                          n
input deterministic                               n
output deterministic                              n
input/output epsilons                             y
input epsilons                                    y
output epsilons                                   y
input label sorted                                n
output label sorted                               y
weighted                                          y
cyclic                                            y
cyclic at initial state                           n
top sorted                                        n
accessible                                        y
coaccessible                                      y
string                                            n
weighted cycles                                   y

Any help would be appreciated.
Thank you.

[Daniel Povey]

Try removing data/lang/tmp and rerunning; also check if your disk is nearly full.

--
Go to http://kaldi-asr.org/forums.html find out how to join
---
You received this message because you are subscribed to the Google Groups "kaldi-help" group.
To unsubscribe from this group and stop receiving emails from it, send an email to kaldi-help+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Pranay Shah]

Thank you so much Dan sir. My disk has still 8+ GBs left. After removing data/lang/tmp and rerunning, I got the same error. Also the file CLG_3_1.fst is always of 0 bytes. Any other ideas, sir? 

To unsubscribe from this group and stop receiving emails from it, send an email to kaldi-help+...@googlegroups.com.

[Daniel Povey]

Most likely your machine has very little memory, but I can't be 100% sure.

I would have expected that fstcomposecontext command to have printed std::bad_alloc if it was a memory issue.  Try running just that command, with output to a temporary file, and see what happens.

Next time don't attach a screenshot, paste the text.

[Pranay Shah]

Sir, You were absolutely correct. Our total RAM size is 156GB and all of it is being used up as can be seen in the snippet below. 

fstcomposecontext --context-size=3 --central-position=1 --read-disambig-syms=data/lang/phones/disambig.int --write-disambig-syms=data/lang/tmp/disambig_ilabels_3_1.int data/lang/tmp/ilabels_3_1.4809 

ERROR: FstHeader::Read: Bad FST header: standard input

mv: cannot stat 'data/lang/tmp/ilabels_3_1.4809': No such file or directory

fstisstochastic data/lang/tmp/CLG_3_1.fst 

ERROR: FstHeader::Read: Bad FST header: data/lang/tmp/CLG_3_1.fst

ERROR (fstisstochastic[5.1.0~1-eba4]:ReadFstKaldi():kaldi-fst-io.cc:35) Reading FST: error reading FST header from data/lang/tmp/CLG_3_1.fst

[ Stack-Trace: ]

kaldi::MessageLogger::HandleMessage(kaldi::LogMessageEnvelope const&, char const*)

kaldi::MessageLogger::~MessageLogger()

fst::ReadFstKaldi(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >)

main

__libc_start_main

_start

[info]: CLG not stochastic.

make-h-transducer --disambig-syms-out=exp/tri1/graph/disambig_tid.int --transition-scale=1.0 data/lang/tmp/ilabels_3_1 exp/tri1/tree exp/tri1/final.mdl 

ERROR (make-h-transducer[5.1.0~1-eba4]:Input():kaldi-io.cc:742) Error opening input stream data/lang/tmp/ilabels_3_1

[ Stack-Trace: ]

kaldi::MessageLogger::HandleMessage(kaldi::LogMessageEnvelope const&, char const*)

kaldi::MessageLogger::~MessageLogger()

kaldi::Input::Input(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, bool*)

main

__libc_start_main

_start

Command exited with non-zero status 1

Command being timed: "./run_tri_graph.sh"

User time (seconds): 31809.43

System time (seconds): 199.38

Percent of CPU this job got: 105%

Elapsed (wall clock) time (h:mm:ss or m:ss): 8:24:43

Average shared text size (kbytes): 0

Average unshared data size (kbytes): 0

Average stack size (kbytes): 0

Average total size (kbytes): 0

Maximum resident set size (kbytes): 159112304 

Average resident set size (kbytes): 0

Major (requiring I/O) page faults: 2978

Minor (reclaiming a frame) page faults: 34686328

Voluntary context switches: 1874307

Involuntary context switches: 46521

Swaps: 0

File system inputs: 860768

File system outputs: 2286432

Socket messages sent: 0

Socket messages received: 0

Signals delivered: 0

Page size (bytes): 4096

 Corpus size : 6.8 GB (14897 files totalling to 24.66 hours) 

 Sir, we are running this on a virtual instance with 24 cpus and memory 156 GB. Increasing the memory beyond this is not an option for us. I realise an option is to connect the machines using slurm/sge but i want to know if there is any other work around?

Thank you sir.

[Daniel Povey]

That is a huge amount of memory and it normally wouldn't take more than a few gigs.  Most likely it's because your setup has a very large number of phones and a lot of short words of one or two phones, so that too many combinations of phones are allowed.

Either that or something very wrong is happening.  Your setup must be quite unusual.  If you want you can send to me an archive with enough info to replicate the problem (but should include the original dict-dir from which you generated the lexicon, as it's more human readable than the L.fst).  But don't bother with that if it's just an obvious case of a very large phone set with many short words.

Dan