DTMF second harmonic test
How Chee
I am trying to reduce talk off in a DTMF detector. Most of the
application notes and advices suggest testing the second harmonic of
the detected DTMF. However, I find that this test does not do much in
rejecting talk off. Statistics from my test shows that this test
triggerred only maybe 5% of the time in a two hour recording. Other
test like the row relative and column relative test triggered more
than 50%. I did the test years ago and I did not check whether the
second harmonic test and the row/column test fail at the same time.
(which means the second harmonic test is redundant). In practice, what
I see is that when speech travel through the phone lines (or maybe
even before), the higher frequencies will be attenuated and
therefore second harmonic test would most probably pass. In addition,
I find this test (depending on how they test it) can fail a valid DTMF
when the frequency of the DTMF is at the +/- 1.5% point.
My question:
1) any papers which justify the use of this test ?
2) is it safe to remove such test since everybody is using this test
(I guess) ?
3) anybody has statistics justifying the use of second harmonic test ?
Thank you,
Hew
Clay S. Turner
Instead of the 2nd harmonic test, just find the time domain energy and
see how it relates to the sum of the loudest row and column energies.
In a perfect world this will exactly match. A threshold can be set at
70 to 80 percent purity. The theorem that supports this is Parseval's
(actually the special case here is due to Bessel - Parseval's is more
general). I've use this with Goertzel based decoders quite sucessfully.
But for your original question, an early Goerzel based decoder with 2nd
harmonic detected was described in a TI app note. It was written by
Patrick Mock. In it he describes how the decoder passes the Mitel test
for talk off immunity. The energy purity approach I described cuts the
needed Goertzel sections in half.
Clay
Steve Underwood
> Hello How Chee,
> Instead of the 2nd harmonic test, just find the time domain energy and
> see how it relates to the sum of the loudest row and column energies.
> In a perfect world this will exactly match. A threshold can be set at
> 70 to 80 percent purity. The theorem that supports this is Parseval's
> (actually the special case here is due to Bessel - Parseval's is more
> general). I've use this with Goertzel based decoders quite sucessfully.
> But for your original question, an early Goerzel based decoder with 2nd
> harmonic detected was described in a TI app note. It was written by
> Patrick Mock. In it he describes how the decoder passes the Mitel test
> for talk off immunity. The energy purity approach I described cuts the
> needed Goertzel sections in half.
I think the success of this approach depends a lot on the application.
What you describe doesn't work too well in the traditional dialing
application. There, the DTMF decoder needs to work in the prescence of
strong dial tone reflected back from lousy hybrids. There are a number
of application notes, from TI and others, using 16 Goertzels (8 tone and
8 second harmonics) and these give good results in the prescence of
dial-tone. You can, of course filter out the dial-tone, but its
harmonics run across the DTMF band and mess things up. I believe a
number of the old switch capacitor detectors used the second harmonic
test, but its a long time since I looked at one.
Now, if your application is something like CTI, what you describe is
great. You need an echo cancellor to remove the voice prompt energy
being spilled back into the DTMF decoder, so the only thing that should
be in the channel (apart from the residue from the cancellor, and some
noise) are the DTMF tones. Its easy to get the BellCore test tapes down
to around 20 hits in total with the strategy you describe. I don't think
the 2nd harmonic approach and no echo cancellor can match that.
I'm not clear why you need to resort to Parseval or Bessel. Surely it is
just obvious that if the only things in the channel are 2 DTMF tones,
the sum of their energies should roughly equal the total. If it doesn't
there is something else present.
Regards,
Steve
Clay S. Turner
my comments below
Steve Underwood wrote:
>
> Clay S. Turner wrote:
> > Hello How Chee,
> > Instead of the 2nd harmonic test, just find the time domain energy and
> > see how it relates to the sum of the loudest row and column energies.
> > In a perfect world this will exactly match. A threshold can be set at
> > 70 to 80 percent purity. The theorem that supports this is Parseval's
> > (actually the special case here is due to Bessel - Parseval's is more
> > general). I've use this with Goertzel based decoders quite sucessfully.
> > But for your original question, an early Goerzel based decoder with 2nd
> > harmonic detected was described in a TI app note. It was written by
> > Patrick Mock. In it he describes how the decoder passes the Mitel test
> > for talk off immunity. The energy purity approach I described cuts the
> > needed Goertzel sections in half.
>
> I think the success of this approach depends a lot on the application.
> What you describe doesn't work too well in the traditional dialing
> application.
Actually I beg to differ. This is used in a lot of commercial systems
that use MF-R1, MFC-R2 and DTMF signalling. I have it deployed in
systems in many places in the world.
The general app is the terminal acts as a central office where another
exchange dials in with clear channel signalling. Once the connection is
made, the user then is allowed to use DTMF overdial.
> There, the DTMF decoder needs to work in the prescence of
> strong dial tone reflected back from lousy hybrids.
My applications were/are mostly T1/E1 environments. But I also have it
implemented with POTS lines.
> There are a number
> of application notes, from TI and others, using 16 Goertzels (8 tone and
> 8 second harmonics) and these give good results in the prescence of
> dial-tone.
My experience is that dial tone is essentially a non issue. Again in my
apps, the system acts as a central office, so there is no inbound dial
tone. Any tone that leaks in from other lines is hardly a problem.
But if one extends the system to include supervisory tones, for U.S.
systems there are only 4 extra frequencies. (no need to decode the quad
freq ROH) The valid tone pairs can still be examined and compared to the
temporal energy. In U.K. type systems just add the 400 Hz (In China I
remember 440 Hz being used) tone detector. This still results in fewer
tone decoders than using 2nd harmonic detectors.
> You can, of course filter out the dial-tone, but its
> harmonics run across the DTMF band and mess things up. I believe a
> number of the old switch capacitor detectors used the second harmonic
> test, but its a long time since I looked at one.
>
> Now, if your application is something like CTI, what you describe is
> great. You need an echo cancellor to remove the voice prompt energy
> being spilled back into the DTMF decoder, so the only thing that should
> be in the channel (apart from the residue from the cancellor, and some
> noise) are the DTMF tones. Its easy to get the BellCore test tapes down
> to around 20 hits in total with the strategy you describe. I don't think
> the 2nd harmonic approach and no echo cancellor can match that.
>
> I'm not clear why you need to resort to Parseval or Bessel. Surely it is
> just obvious that if the only things in the channel are 2 DTMF tones,
> the sum of their energies should roughly equal the total. If it doesn't
> there is something else present.
Actually the DFT (by Goertzel or any other method) performs an N/2 gain
of the signal. This is reflected in Parseval's theorem. What you think
is obvious is mathematically supported by Parseval's theorem. Plus it
gives you a way of figuring out the lower purity expected when a tone is
off frequency and the energy leaks to the other DFT bins.
Clay
>
> Regards,
> Steve
Jim Thomas
>
> My experience is that dial tone is essentially a non issue. Again in my
> apps, the system acts as a central office, so there is no inbound dial
> tone. Any tone that leaks in from other lines is hardly a problem.
I'd think you'd want to eliminate DC before employing Parseval, no?
--
Jim Thomas E-mail: jth...@bittware.com
Principal Applications Engineer Web: http://www.bittware.com
Bittware, Inc Tel: (703) 779-7770
I thought I was wrong once, but I was mistaken.
Steve Underwood
> My experience is that dial tone is essentially a non issue. Again in my
> apps, the system acts as a central office, so there is no inbound dial
> tone. Any tone that leaks in from other lines is hardly a problem.
I'm not sure I follow you here. Surely being a central office is exactly
the case where dial tone is a problem (unless you echo cancel, of
course). You put out dial tone, and wait for the caller to respond. If
you don't echo cancel you must tolerate dial-tone spilling back.
The other situations you described are like the CTI application I
described. You either have a naturally clean channel, or you clean it up
with cancellation.
As far as I can tell we are not at odds here. With a clean channel, the
straightforward "do the two biggest tones equate to a high percentage of
the total energy?" test is hard to beat in performance or complexity.
Throw in a few other sanity tests (e.g twist) and you can get down to
about 20 hits on the BellCore test tapes. I don't know anyone who does
better. Have you ever seem someone get zero?
I found last year that Mitel no longer supply their DTMF test tape. Gone
are the days of "we get zero hits on the Mitel test tape", which wasn't
that hard to achieve. You *have* to be a real (wo)man, and say what you
get with the BellCore tapes these days. Its a pity those are not
available on CD.
Regards,
Steve
Steve Underwood
> "Clay S. Turner" wrote:
>
>>My experience is that dial tone is essentially a non issue. Again in my
>>apps, the system acts as a central office, so there is no inbound dial
>>tone. Any tone that leaks in from other lines is hardly a problem.
>>
>
> I'd think you'd want to eliminate DC before employing Parseval, no?
Sure, but the only issue there is the settling time of the zero-DC
restorer. You better not damp it so hard it takes ages to recover from
the DC kick caused by going off-hook. Otherwise you might miss the first
digit or two :)
Regards,
Steve
How Chee
> Clay S. Turner wrote:
>
> > My experience is that dial tone is essentially a non issue. Again in my
> > apps, the system acts as a central office, so there is no inbound dial
> > tone. Any tone that leaks in from other lines is hardly a problem.
>
> I'm not sure I follow you here. Surely being a central office is exactly
> the case where dial tone is a problem (unless you echo cancel, of
> course). You put out dial tone, and wait for the caller to respond. If
> you don't echo cancel you must tolerate dial-tone spilling back.
>
detector, unless of course if the dial tone is 13 dB higher than the
DTMF tones and you are using rectangular window. Using the tests
describe only in TI notes can cause false detection in case a strong
tone appears in the region of 1000 Hz to 1150 Hz. e.g fax tone
detected as a low level "*"
> The other situations you described are like the CTI application I
> described. You either have a naturally clean channel, or you clean it up
> with cancellation.
>
> As far as I can tell we are not at odds here. With a clean channel, the
> straightforward "do the two biggest tones equate to a high percentage of
> the total energy?" test is hard to beat in performance or complexity.
> Throw in a few other sanity tests (e.g twist) and you can get down to
> about 20 hits on the BellCore test tapes. I don't know anyone who does
> better. Have you ever seem someone get zero?
>
The total energy test is sure hard to beat but perform badly in the
presence of echo as in a IVR application. I always worry that the echo
cancellor did not converge well enough before a digit is pressed. By
the way, if the echo cancellor did not fully converge, wil it 'screw'
up the dtmf level ? The echo cancellor that I have changes the level
of the 2 peaks before fully converged. This will cause the received
signal to fail the the twist test. Is my echo cancellor not working
properly or is this normal ?
> I found last year that Mitel no longer supply their DTMF test tape. Gone
> are the days of "we get zero hits on the Mitel test tape", which wasn't
> that hard to achieve. You *have* to be a real (wo)man, and say what you
> get with the BellCore tapes these days. Its a pity those are not
> available on CD.
>
Wish I can get hold of this tape, to test out the DTMF detector that I
have.
> From Jim,
> You can get better immunity by testing that the DTMF tone is not the
> second harmonic of a non-DTMF SIGNAL (in other words, test for 0.5f
> instead of 2f).
This technique of testing frequencies lower than 697 Hz seems to work
well in no echo environment. Instead of 0.5f, you can analysis the
speech section that causes talk off and find the peak within the range
of 300 to 600 Hz. (I think it is only a few of them). Use the frequecy
of these peaks instead of the second harmonic frequencies and you will
surely get less talk off. But once again , this technique perform
badly in echo. (DTMF embedded in echo which are speech will get a lot
of talk down).
Regards,
Hew
Steve Underwood
> Dial tone is generally not a problem for the TI's 16 goertzel DTMF
> detector, unless of course if the dial tone is 13 dB higher than the
> DTMF tones and you are using rectangular window. Using the tests
> describe only in TI notes can cause false detection in case a strong
> tone appears in the region of 1000 Hz to 1150 Hz. e.g fax tone
> detected as a low level "*"
Some designs are excessively complex, but I think the 16 goertzel design
it probably well chosen for dial-tone tolerant environments. There are
actually several TI DTMF designs in app. notes, but I think all are
variations on the 16 goertzel theme. Also they have three in each app
note. - the description, the source code, and the source code comments
which all differ in little ways :)
I am not clear why FAX tone should cause you problems. Is it hitting the
highest of the low group and the lowest of the high group at the same
time? I haven't suffered problems like this.
> The total energy test is sure hard to beat but perform badly in the
> presence of echo as in a IVR application. I always worry that the echo
> cancellor did not converge well enough before a digit is pressed. By
> the way, if the echo cancellor did not fully converge, wil it 'screw'
> up the dtmf level ? The echo cancellor that I have changes the level
> of the 2 peaks before fully converged. This will cause the received
> signal to fail the the twist test. Is my echo cancellor not working
> properly or is this normal ?
I don't see how you could ever get reasonable performance in an IVR
application without echo cancellation. There are CTI cards which try,
but they perform poorly - unless you keep the voice prompt level
unreasonably low, and have little attenuation of the DTMF tones. Echo
cancellation works well with DTMF - a million Dialogic cards can't be
wrong :) It isn't the most demanding cancellation problem, as the echo
loss enhancement doesn't need to be extreme - only stable.
> Wish I can get hold of this tape, to test out the DTMF detector that I
> have.
The BellCore tapes are over US$200, and its a pain to find the right
place to order them from. However, someone got me a fresh set for
something last year, and we transcribed them as well as we could to 8kHz
16 bit PCM on a hard disk. Now I don't need to worry about tape stretch.
I don't think the demise of the Mitel tape is too much of a loss. The
tests on the first side are better generated digitally, and the talk-off
test is far easier to pass than the BellCore one.
>>From Jim,
>>You can get better immunity by testing that the DTMF tone is not the
>>second harmonic of a non-DTMF SIGNAL (in other words, test for 0.5f
>>instead of 2f).
>>
>
> This technique of testing frequencies lower than 697 Hz seems to work
> well in no echo environment. Instead of 0.5f, you can analysis the
> speech section that causes talk off and find the peak within the range
> of 300 to 600 Hz. (I think it is only a few of them). Use the frequecy
> of these peaks instead of the second harmonic frequencies and you will
> surely get less talk off. But once again , this technique perform
> badly in echo. (DTMF embedded in echo which are speech will get a lot
> of talk down).
I tried looking for 1/2 the lower group frequencies and twice the upper
group frequencies once. I thought it might be better looking for things
well separated from the tones themselves, rather than looking for the
2nd harmonics of the low group mixed in amongst the high group. It
seemed to hurt the dial-tone tolerance though. Maybe I wasn't persistent
enough with the tweaking.
Regards,
Steve
Clay S. Turner
Actually with T1/E1 circuits, the off hook is performed via th AB (ABCD)
bits, so there is no DC ringing problems.
Clay
Steve Underwood
> Steve & Jim,
>
> Actually with T1/E1 circuits, the off hook is performed via th AB (ABCD)
> bits, so there is no DC ringing problems.
Or tones (MFC/R2, R1, etc), or messages (SS7, ISDN, etc.), but yes what
you say is correct. However, when someone lifts the receiver, and loop
current starts to flow, many line cards give a huge kick in the PCM
channel. Any DC restorer must behave well if that occurs.
Despite the supposed 300Hz-3400Hz bandwidth of telephone lines, finding
a substantial DC offset in a uLaw or ALaw signal is pretty common. You
need to remove that.
Regards,
Steve
Clay S. Turner
Steve Underwood wrote:
>
> Clay S. Turner wrote:
>
> > My experience is that dial tone is essentially a non issue. Again in my
> > apps, the system acts as a central office, so there is no inbound dial
> > tone. Any tone that leaks in from other lines is hardly a problem.
>
> I'm not sure I follow you here. Surely being a central office is exactly
> the case where dial tone is a problem (unless you echo cancel, of
> course). You put out dial tone, and wait for the caller to respond. If
> you don't echo cancel you must tolerate dial-tone spilling back.
In my app being like a central office, I'm talking about a paging
terminal. We don't supply dial tone to the subscriber. However, when
the subscriber dials the terminal, he actually DTMF's his local
exchange, which in turn will signal me via MF-R1, MFC-R2, or sometimes
high speed DTMF. These occur on a clear channel. We then provide our own
supervisory tones, but then I have control of my hybrid.
In some cases the calls get to me via a two hop situation, so the twist
must be expanded greater than the standard Bellcore spec. They assume a
single hop from subscriber to exchange.
>
> The other situations you described are like the CTI application I
> described. You either have a naturally clean channel, or you clean it up
> with cancellation.
>
> As far as I can tell we are not at odds here. With a clean channel, the
> straightforward "do the two biggest tones equate to a high percentage of
> the total energy?" test is hard to beat in performance or complexity.
> Throw in a few other sanity tests (e.g twist) and you can get down to
> about 20 hits on the BellCore test tapes. I don't know anyone who does
> better. Have you ever seem someone get zero?
No I haven't seen a score of zero and we agree on the effectiveness of
the approach. One of the biggest problems is the use of a signalling
method that was designed for use on a clear channel being used on non
clear ones. (when a digit is pressed the mike is muted.) Thus I think
there will always be either false digits or missed digits. It is tweaked
until it operates best. The customer is given a standard set of
parameters for the DTMF code, and he is given the opportunity to adjust
them if he desires. Most are happy with the default params.
One neat algo I saw (patented by ATT) uses a high-low filter splitter
followed with two 2nd order LPC analyzers. The code is computationally
simple and efficient, plus it allows for easy configuration. They just
solve for the root's position for each tone and see if it is in the
correct position.
Clay
J. Shima
thought I was ahead of the curve way back then! LOL
At any rate, in my decoder I also used a highpass and lowpass filter
bank. Although my LPC was larger than 2nd order and used an adaptive
filter scheme. I let the LPC converge and simply did a peak search on
the result. It works pretty well and can also meet the Bellcore specs
for minimum tone duration. If you did use a 2nd order LPC, a simply
root solver would give you the correct freq, which is what I was
trying to do. I was also thinking of using the modified covariance
method. The alg worked great for talk off situations, since the
predictor wouldnt converge to anything discernible.
How Chee
>
> I am not clear why FAX tone should cause you problems. Is it hitting the
> highest of the low group and the lowest of the high group at the same
> time? I haven't suffered problems like this.
Correct me if my theory is wrong. A high level tone (let say -6 dBm)
in the region of 1000-1100 Hz can cause the goertzel filter (equiv. to
102 point DFT) for frequency 941 Hz and 1209 Hz to give some output
(energy present).
Energy is detected here since Goertzel algorithm is a DFT with
rectangular window. Rectangular window give only about 13 dB second
side lobe attenuation.
Therefore , the 941 Hz and 1209 Hz output seems to see 941 Hz and
1209 Hz signal that is transmitted at a lower level. Let say we have
18 db of attenuation instead of 13 dB. -6 dBm - 18 gives -24 dBm and
at this signal level, a DTMF can still be considered present. In my
test, it will pass the one row frequency and one column frequency
test, twist test, 2nd harmonic test and level test. The peak energy to
total energy of course will fail it as a DTMF.
The basic idea is to use the second side lobe level to create a false
digit. I have never try it before but it might be possible to give a
false digit "A" by a dual tone with frequency around 540 Hz and 1800
Hz.(assuming detection is base on the 16 Goertzel output only and
nothing else)
Regards,
Hew