cmudict.0.7a file format
Carl K
is a derivative of CMU Dict v.6 - I want to incorporate cmudict.0.7a.
I am not 100% sure how I am going to do that. I am not 100% sure what all the
v.6 derivation process is, but my instinct says I am just going to merge the v.7
with the VoxForgeDict, and maybe look for 'corrections' between v.6 and .7 and
try to back out anything that smells like bad data.
Whatever I do, I would like to start with some code to parse the cmudict.0.7a.
I have yet to find a definitive spec on it's format - either in english or some
program code.
I can understand that maybe it doesn't exist. If this is the case, who has the
'authority' to make that call, and to designate a 'location' for a spec to be
developed: pile up all the things existing code relies on, and once the dust
settles, bless it the current spec.
and regardless of who/where, I know there are at least 3 things that need to be
consolidated, so if anyone has any more, add them to this thread.
1. http://htk.eng.cam.ac.uk/prot-docs/HTKBook/node174_mn.html
WORD [ '['OUTSYM']' ] [PRONPROB] P1 P2 P3 P4 ....
2. http://en.wikipedia.org/wiki/CMU_Pronouncing_Dictionary
subsequent entries are followed by an index in parentheses.
3. ken says: "(I am not sure if this the case anymore) Julius required acoustic
models trained with a pronunciation dictionary that used an OUTSYM." (so OUTSYM
is not optional as described in #1.
4. The entries need to be in alphabetical order. I have 0.0 clue what would
actually require this, and from looking at the data, it seems the alpha of 's
and #2 "subsequent..index in parentheses" conflict:
AFRICA [AFRICA] ae f r ax k ax
AFRICA'S [AFRICA'S] ae f r ax k ax z
AFRICA'S(2) [AFRICA'S] ae f r ix k ax z
AFRICA(2) [AFRICA] ae f r ix k ax
AFRICA(3) [AFRICA] ae f er k ax
AFRICAN [AFRICAN] ae f r ax k ax n
AFRICAN(2) [AFRICAN] ae f r ix k ax n
Carl K
def read_dict(dict_filename):
"""
Read dictionary file into a python dictionary of the following format:
"""
# dict to return
vfd={}
for row in open(dict_filename):
row = row.strip('\n')
if row: # (ignores the blank rows at start/end of file
# create a list from row data delimited by space
row=row.split()
word = row.pop(0)
# dump the (n) from word(n)
if word.endswith(')'): word = word[:word.find('(')]
# current words's data dict
wd = {}
# look for outsym
if row[0].startswith('['):
# [1:-1] is everthing between the first and last []
wd['outsym'] = row.pop(0)[1:-1]
# report weirdness
if word != wd['outsym']: print word, wd['outsym'], row
# look for pronprob
if row[0].startswith(('0','1')):
wd['pronprob'] = row.pop(0)
# rest of the row is the phones
wd['phones'] = ' '.join(row)
# add/append word:data
l=vfd.get(word,[])
l.append(wd)
vfd[word]=l
return vfd
def write_dict(vfd, dict_filename):
"""
Write the python dictionary to a text file
"""
fout=open(dict_filename,'w')
fout.write('\n')
# create an aplpha list of the dict:
l=vfd.items()
l.sort()
for word,wds in l:
# print word, wds
for i,wd in enumerate(wds):
# print wd
# build the output row in a list
row =[]
# "subsequent entries are followed by an index in parentheses."
if i:
row.append("%s(%s)" % (word,i+1))
else:
row.append(word)
if wd.has_key('outsym'): row.append( "[%s]" % wd['outsym'] )
if wd.has_key('pronprob'): row.append( wd['pronprob'] )
row.append(wd['phones'])
row = "%s\n" % ' '.join(row)
# if len(row)!=3: print row
# row = "%-15s %-15s %s\n" % (tuple(row))
# print row
fout.write(row)
fout.write('\n')
fout.close()
if __name__=='__main__':
dict_filename='VoxForge/VoxForgeDict'
d=read_dict(dict_filename)
write_dict(d,dict_filename+".new")