Chasing Bugs in the Fog

[rickman]

I have a bug in a test fixture that is FPGA based. I had thought it was
in the software which controls it, but after many hours of chasing it
around I've concluded it must be in the FPGA code.

I didn't think it was in the VHDL because it had been simulated well and
the nature of the bug is an occasional dropped character on the receive
side. Who can't design a UART? Well, it could be in the handshake with
the state machine, but still...

So I finally got around to adding some debug signals which I would
monitor on an analyzer and guess what, the bug is gone! I *hate* when
that happens. I can change the code so the debug signals only appear
when a control register is set to enable them, but still, I don't like
this. I want to know what is causing this DURN THING!

Anyone see this happen to them before?

Oh yeah, someone in another thread (that I can't find, likely because I
don't recall the group I posted it in) suggested I add synchronizing FFs
to the serial data in. Sure enough I had forgotten to do that. Maybe
that was the fix... of course! It wasn't metastability, I bet it was
feeding multiple bits of the state machine! Durn, I never make that
sort of error. Thanks to whoever it was that suggested the obvious that
I had forgotten.

--

Rick

[Rob Gaddi]

On Mon, 17 Jun 2013 20:00:01 -0400
rickman <gnu...@gmail.com> wrote:

> So I finally got around to adding some debug signals which I would
> monitor on an analyzer and guess what, the bug is gone! I *hate* when
> that happens. I can change the code so the debug signals only appear
> when a control register is set to enable them, but still, I don't like
> this. I want to know what is causing this DURN THING!
>
> Anyone see this happen to them before?
>
> Oh yeah, someone in another thread (that I can't find, likely because I
> don't recall the group I posted it in) suggested I add synchronizing FFs
> to the serial data in. Sure enough I had forgotten to do that. Maybe
> that was the fix... of course! It wasn't metastability, I bet it was
> feeding multiple bits of the state machine! Durn, I never make that
> sort of error. Thanks to whoever it was that suggested the obvious that
> I had forgotten.
>
> --
>
> Rick

Not metastability, a race condition. Asynchronous external input
headed to multiple clocked elements, each of which it reaches via a
different path with a different delay.

When you added debugging signals you changed the netlist, which changed
the place and route, making unpredictable changes to those delays. In
this case, it happened to push it into a place where _as far as you
tested_, it seems happy. But it's still unsafe, because as you change
other parts of the design, the P&R of that section will still change
anyhow, and you start getting my favorite situation, the problem that
comes and goes based on entirely unrelated factors.

The fix you fixed fixes it. When you resynchronized it on the same
clock as you're running around the rest of the logic, you forced that
path to become timing constrained. As such, the P&R takes it upon
itself to make sure that the timing of that route is irrelevant with
respect to the clock period, and your problem goes away for good.

--
Rob Gaddi, Highland Technology -- www.highlandtechnology.com
Email address domain is currently out of order. See above to fix.

[jt_eaton]

>So I finally got around to adding some debug signals which I would
>monitor on an analyzer and guess what, the bug is gone! I *hate* when
>that happens. I can change the code so the debug signals only appear
>when a control register is set to enable them, but still, I don't like
>this. I want to know what is causing this DURN THING!
>
>Anyone see this happen to them before?
>

>--
>
>Rick
>

Yes, This is called a "Heisenbug". Usually involves a clock domain
crossing
mistake.


John Eaton


---------------------------------------
Posted through http://www.FPGARelated.com

[Kevin Neilson]

One mistake that is not too hard to make is forgetting to put a synchronizer flop on the input of an edge detector, like you might have on a UART input (so that the edge detector has two flops, total). Depending on the routing delays, this can cause you to miss a sizable percentage of edges. (Not just delayed, but missed completely.) Using only a single flop is sometimes known as using the "greedy path".

(Actually, to mitigate metastability as well, an edge detector ought to have three flops and an AND gate. Using two is sometimes known as using the "sneaky path".)

[rickman]

On 6/17/2013 8:14 PM, Rob Gaddi wrote:
> On Mon, 17 Jun 2013 20:00:01 -0400
> rickman<gnu...@gmail.com> wrote:
>
>> So I finally got around to adding some debug signals which I would
>> monitor on an analyzer and guess what, the bug is gone! I *hate* when
>> that happens. I can change the code so the debug signals only appear
>> when a control register is set to enable them, but still, I don't like
>> this. I want to know what is causing this DURN THING!
>>
>> Anyone see this happen to them before?
>>
>> Oh yeah, someone in another thread (that I can't find, likely because I
>> don't recall the group I posted it in) suggested I add synchronizing FFs
>> to the serial data in. Sure enough I had forgotten to do that. Maybe
>> that was the fix... of course! It wasn't metastability, I bet it was
>> feeding multiple bits of the state machine! Durn, I never make that
>> sort of error. Thanks to whoever it was that suggested the obvious that
>> I had forgotten.
>>
>> --
>>
>> Rick
>
> Not metastability, a race condition. Asynchronous external input
> headed to multiple clocked elements, each of which it reaches via a
> different path with a different delay.
>
> When you added debugging signals you changed the netlist, which changed
> the place and route, making unpredictable changes to those delays.

No, when changing the debug output I added the synchronization FFs which
fixed the problem.

My point was that when the other poster suggested that I need to sync to
the clock I mistook that for metastability forgetting that the input
went to multiple sections of logic. So actually I made the same mistake
twice... lol

> In
> this case, it happened to push it into a place where _as far as you
> tested_, it seems happy. But it's still unsafe, because as you change
> other parts of the design, the P&R of that section will still change
> anyhow, and you start getting my favorite situation, the problem that
> comes and goes based on entirely unrelated factors.
>
> The fix you fixed fixes it. When you resynchronized it on the same
> clock as you're running around the rest of the logic, you forced that
> path to become timing constrained. As such, the P&R takes it upon
> itself to make sure that the timing of that route is irrelevant with
> respect to the clock period, and your problem goes away for good.

Just to make sure of what was what (it has been two years since I last
worked with this design) I pulled the FFs out and added back just one.
Sure enough the bug reappears with no FFs, but goes away with just one.
The added debug info available allowed me to see exactly the error and
sure enough, when a start bit comes in there is a chance that the two
counters are not properly set and the error shows up in the center of
the bit where the current contents of the shift register are moved into
the holding register as a new char.

I guess what most likely happened is that when I wrote the UART code I
assumed the sync FFs would be external and when I wrote the wrapper code
I assumed the FFs were inside the UART. In other words, I didn't have a
proper spec and never gave this problem proper consideration.

I will revisit this design and look at the other inputs. No reason to
assume I didn't make the same mistake elsewhere.

--

Rick

[rickman]

On 6/18/2013 3:16 PM, Kevin Neilson wrote:
> One mistake that is not too hard to make is forgetting to put a synchronizer flop on the input of an edge detector, like you might have on a UART input (so that the edge detector has two flops, total). Depending on the routing delays, this can cause you to miss a sizable percentage of edges. (Not just delayed, but missed completely.) Using only a single flop is sometimes known as using the "greedy path".
>
> (Actually, to mitigate metastability as well, an edge detector ought to have three flops and an AND gate. Using two is sometimes known as using the "sneaky path".)

Everyone is saying the same thing, so I guess I didn't explain clearly.
Someone had already pointed out to me that I needed a synchronizer on
the received data signal in another thread that I can't find now. I
took them at their word, but was thinking they meant it was about
metastability which I figured was not a problem at these speeds (yes,
the speeds do make a difference for metastability since you never chase
it away, you just minimize it). I wasn't thinking about the serial in
signal feeding the state machine, just the shift register.

So when I made the changes, which included the synchronizer, it worked.
Because I didn't expect the synchronizer to do anything, I had
forgotten about it until I was typing the post here. I remembered at
the end of the message and realized that was what fixed the problem...

Sorry for the confusion. Still, thanks to all who replied and
especially the mystery person who suggested it in the other thread
wherever that was.

--

Rick

[Nicolas Matringe]

Le 18/06/2013 23:45, rickman a �crit :

> I guess what most likely happened is that when I wrote the UART code I
> assumed the sync FFs would be external and when I wrote the wrapper code
> I assumed the FFs were inside the UART. In other words, I didn't have a
> proper spec and never gave this problem proper consideration.

Several years ago a young engineer reused my long proven UART code and
modified it, carelessly removing the synchronizing FF. He came to see me
and complained that my UART didn't work, it hung after some
unpredictable time.
I thought for a few minutes, guessed he probably had removed the FF and
fixed his problem right away.

Nicolas

[Kevin Neilson]

That's the same thing that happened to me when I had the problem last. I had an edge detector connected to a big synchronizer module that was in turn connected to all the input pins. When I had problems I looked inside the synchronizer module and found that it didn't have a flop on that line; it was just wired straight through.

[Theo Markettos]

rickman <gnu...@gmail.com> wrote:
> Everyone is saying the same thing, so I guess I didn't explain clearly.
> Someone had already pointed out to me that I needed a synchronizer on
> the received data signal in another thread that I can't find now. I
> took them at their word, but was thinking they meant it was about
> metastability which I figured was not a problem at these speeds (yes,
> the speeds do make a difference for metastability since you never chase
> it away, you just minimize it). I wasn't thinking about the serial in
> signal feeding the state machine, just the shift register.

There's 3 things that could have gone wrong (and might still be doing
wrong):

You failed to synchronise between the clock domain of the input serial link
and the clock of your system (sounds like you fixed this one)

You failed to constrain the clocks and other inputs so the synthesis tool
knows what timing budget it has to meet

You failed timing analysis and didn't notice - in other words the synthesis
tool says the design it produced doesn't meet your supplied timing
constraints, despite its best efforts. If the failure is small it may still
work in some voltage/temperature/silicon situations, but it isn't guaranteed
in all cases.


Normally the last one will raise big red flags in the tool, assuming the
timing analyser does get run as part of the build. However the first two
are easy to overlook and you get no warning from the tools.

Theo

[Jon Elson]

rickman wrote:


> I didn't think it was in the VHDL because it had been simulated well and
> the nature of the bug is an occasional dropped character on the receive
> side.  Who can't design a UART?  Well, it could be in the handshake with
> the state machine, but still...
>

Any time you recompile an FPGA and the problem disappears or
changes, it is a STRONG indication it is a timing problem.
Regenerating the place & route changes timings subtly between
sections, and may eliminate a marginal setup or hold time
problem. You should make sure all signals that cross clock
boundaries are properly synchronized, and that you are giving the
right clock specification to your clocks in the ucf file.
If there are tricky timings on parts connected to the FPGA,
then you need to define the timings in the ucf file.

Jon