Google Groups no longer supports new Usenet posts or subscriptions. Historical content remains viewable.
Dismiss
How to gain deep knowledge of Vector CANoe to solve missed CAN messages problem ?
135 views
Skip to first unread message
learn
Dec 4, 2015, 4:21:13 PM12/4/15
to
Our Electronic Control Unit (ECU) is based on Renesas RH850F1L
Microcontroller. We basically have closed loop control system.
Outputs of this ECU power DC Motors to move Third-Row Minivan Seats.
ECU inputs are coming from DC Motor. This feedback signal is hall pulses
indicating whether motor is turning or not or whether seat is moving or
not.
The software running in the ECU talks to PC Based CAN tool. They talk
over simple proprietary protocol. They basically talk over 11-bit CAN.
When the seat is moving, every 10ms, software running in the ECU sends
seat position message to PC Based CAN Tool. When seat is at position 1,
position 1 message is sent to the tool. After 10ms, position 2 message is
sent to the tool. After next 10ms, position 3 message is sent to the tool,
and so on.
If everything is working correctly, over 1000 consecutive position
messages will be sent to the tool. Time between these messages will be
about 10ms.
The problem is that from time to time messages are getting missed. The
message that don't make it to the tool also don't show up in the CAN log.
First 80 position CAN messages are correctly sent by ECU and correctly
received by the tool.
Position messages 81 through 84 don't show up in the tool. Also, they
don't show up in the CAN log.
Position messages 85 to 122 are correctly sent by ECU and correctly
received by the tool.
Position message 123 doesn't show up in the tool. Also, it's missing
from the CAN log.
Position messages 124 to 177 will be correctly sent by ECU and correctly
received by the tool.
Position message 178 to 181 don't show up in the tool. Also, they don't
show up in the CAN log.
And so on.
I'm using very complex CANoe Configuration File that was created by
someone else. It's overwhelming. There are so many nodes, so many
message in CAN Trace.
From where do I begin to debug this problem.
11-bit proprietary protocol messages are exchanged between ECU and tool.
Also, CANoe is sending so many messages. I don't even know what all
these messages are. I have a hint as follows:
Hint: CANoe sends messages to ECU in bursts. Instead, it needs to send
them at slower rate. This might be the reason low priority position
messages are missed from ECu. Turn off some nodes in CANoe.
This shouldn't be ECU software problem because it's not sending messages
in bursts.
This is challenging problem for me because I don't have sufficient CANoe
knowledge. Please help debug this problem.
---------------------------------------
Posted through http://www.EmbeddedRelated.com
Microcontroller. We basically have closed loop control system.
Outputs of this ECU power DC Motors to move Third-Row Minivan Seats.
ECU inputs are coming from DC Motor. This feedback signal is hall pulses
indicating whether motor is turning or not or whether seat is moving or
not.
The software running in the ECU talks to PC Based CAN tool. They talk
over simple proprietary protocol. They basically talk over 11-bit CAN.
When the seat is moving, every 10ms, software running in the ECU sends
seat position message to PC Based CAN Tool. When seat is at position 1,
position 1 message is sent to the tool. After 10ms, position 2 message is
sent to the tool. After next 10ms, position 3 message is sent to the tool,
and so on.
If everything is working correctly, over 1000 consecutive position
messages will be sent to the tool. Time between these messages will be
about 10ms.
The problem is that from time to time messages are getting missed. The
message that don't make it to the tool also don't show up in the CAN log.
First 80 position CAN messages are correctly sent by ECU and correctly
received by the tool.
Position messages 81 through 84 don't show up in the tool. Also, they
don't show up in the CAN log.
Position messages 85 to 122 are correctly sent by ECU and correctly
received by the tool.
Position message 123 doesn't show up in the tool. Also, it's missing
from the CAN log.
Position messages 124 to 177 will be correctly sent by ECU and correctly
received by the tool.
Position message 178 to 181 don't show up in the tool. Also, they don't
show up in the CAN log.
And so on.
I'm using very complex CANoe Configuration File that was created by
someone else. It's overwhelming. There are so many nodes, so many
message in CAN Trace.
From where do I begin to debug this problem.
11-bit proprietary protocol messages are exchanged between ECU and tool.
Also, CANoe is sending so many messages. I don't even know what all
these messages are. I have a hint as follows:
Hint: CANoe sends messages to ECU in bursts. Instead, it needs to send
them at slower rate. This might be the reason low priority position
messages are missed from ECu. Turn off some nodes in CANoe.
This shouldn't be ECU software problem because it's not sending messages
in bursts.
This is challenging problem for me because I don't have sufficient CANoe
knowledge. Please help debug this problem.
---------------------------------------
Posted through http://www.EmbeddedRelated.com
Hans-Bernhard Bröker
Dec 4, 2015, 5:44:00 PM12/4/15
to
Am 04.12.2015 um 22:21 schrieb learn:
> The problem is that from time to time messages are getting missed. The
> message that don't make it to the tool also don't show up in the CAN log.
If that poses a serious problem to your usage of those data, that means
> The problem is that from time to time messages are getting missed. The
> message that don't make it to the tool also don't show up in the CAN log.
that proprietary protocol of yours is ill-suited for use over CAN.
CAN, like any multi-master shared-medium protocol, cannot guarantee
successful delivery of every single message. That means protocols that
are run across it must be able to tolerate the occasional missed
message, and either request a re-transmission, or just cope with not
having that particular packet at all.
> I'm using very complex CANoe Configuration File that was created by
> someone else. It's overwhelming. There are so many nodes, so many
> message in CAN Trace.
and b) you're only interested in those few message of your own protocol,
why are you even using that big simulation? I'll assume you do that
because the real work environment of your ECU has all those nodes, too,
sending just as many messages. So turning off some of the simulated
nodes is not going to actually solve anything. It'll just make the
simulation lose some similarity to the real environment.
Unless you have some way of guaranteeing that similar message burst
patterns cannot ever happen in real life, with multiple real ECUs on the
bus instead of a single CANoe simulating them all, your problem is _not_
how to keep CANoe from producing those message bursts --- it's how to
keep your protocol from breaking apart if they do happen.
In a nutshell: this test run has exposed a flaw in your proprietary
little protocol. You need to repair that flaw, not the test run.
learn
Dec 5, 2015, 1:33:19 PM12/5/15
to
How to find out what is ill suited about our protocol? And then how can
we make it robust so messages aren't missed.
What source(s) will provide me more information on lack of CAN robustness
(missed messages can occur)?
Yes, I'm using complex CAN configuration because it simulates real world
environment.
Tauno Voipio
Dec 5, 2015, 2:08:07 PM12/5/15
to
protocols that messages occasionally get lost. Even UDP in the
TCP/IP protocol suite has to accept lost datagrams, although
it does not always use broadcast addressing.
--
-TV
Les Cargill
Dec 5, 2015, 2:59:08 PM12/5/15
to
or who it's from.
--
Les Cargill
upsid...@downunder.com
Dec 5, 2015, 5:00:57 PM12/5/15
to
On Fri, 4 Dec 2015 23:44:03 +0100, Hans-Bernhard Bröker
<HBBr...@t-online.de> wrote:
>Am 04.12.2015 um 22:21 schrieb learn:
>
>> The problem is that from time to time messages are getting missed. The
>> message that don't make it to the tool also don't show up in the CAN log.
The problem with the OP's case seems to be that the messages are lost
<HBBr...@t-online.de> wrote:
>Am 04.12.2015 um 22:21 schrieb learn:
>
>> The problem is that from time to time messages are getting missed. The
>> message that don't make it to the tool also don't show up in the CAN log.
somewhere between the receiver station terminal and the final
application.
>If that poses a serious problem to your usage of those data, that means
>that proprietary protocol of yours is ill-suited for use over CAN.
>CAN, like any multi-master shared-medium protocol, cannot guarantee
>successful delivery of every single message.
the few protocols in which the transmitting station knows when the
correct signal is delivered to each receiving station terminals.
The other question is can that signal be transferred to the
application.
>That means protocols that
>are run across it must be able to tolerate the occasional missed
>message, and either request a re-transmission, or just cope with not
>having that particular packet at all.
1.) you can have some time specific unreliable information transfer
(e.g. UDP)
2.) you can have reliable connections but no real time predictability
(e.g.. TCP)
There is no "correct" solution, but you have to select case by case
what to use.
David Brown
Dec 6, 2015, 6:30:07 AM12/6/15
to
On 05/12/15 23:00, upsid...@downunder.com wrote:
> On Fri, 4 Dec 2015 23:44:03 +0100, Hans-Bernhard Bröker
> <HBBr...@t-online.de> wrote:
>
>> Am 04.12.2015 um 22:21 schrieb learn:
>>
>>> The problem is that from time to time messages are getting missed. The
>>> message that don't make it to the tool also don't show up in the CAN log.
>
> The problem with the OP's case seems to be that the messages are lost
> somewhere between the receiver station terminal and the final
> application.
>
>> If that poses a serious problem to your usage of those data, that means
>> that proprietary protocol of yours is ill-suited for use over CAN.
>
> That is the real issue.
>
>> CAN, like any multi-master shared-medium protocol, cannot guarantee
>> successful delivery of every single message.
>
> Sorry to be pedantic but as you know that CAN is actually only one of
> the few protocols in which the transmitting station knows when the
> correct signal is delivered to each receiving station terminals.
>
That's not /quite/ true. CAN can see that the telegram has been
> On Fri, 4 Dec 2015 23:44:03 +0100, Hans-Bernhard Bröker
> <HBBr...@t-online.de> wrote:
>
>> Am 04.12.2015 um 22:21 schrieb learn:
>>
>>> The problem is that from time to time messages are getting missed. The
>>> message that don't make it to the tool also don't show up in the CAN log.
>
> The problem with the OP's case seems to be that the messages are lost
> somewhere between the receiver station terminal and the final
> application.
>
>> If that poses a serious problem to your usage of those data, that means
>> that proprietary protocol of yours is ill-suited for use over CAN.
>
> That is the real issue.
>
>> CAN, like any multi-master shared-medium protocol, cannot guarantee
>> successful delivery of every single message.
>
> Sorry to be pedantic but as you know that CAN is actually only one of
> the few protocols in which the transmitting station knows when the
> correct signal is delivered to each receiving station terminals.
>
successfully transmitted to at least one other node - but you have no
guarantee that it is received by /all/ nodes. It takes an unusual bad
bus topology to get a situation where some nodes can receive a telegram
correctly while others get errors, but it is possible - and may be worth
considering if there are lots of nodes on the bus, or it is poorly
terminated, or laid out in something other than a linear bus.
Also note that all CAN controllers on the bus will send an acknowledge
bit if the telegram is seen in the correct format, with the correct CRC
check - it says nothing about the CAN controller being set up correctly
to receive the message.
> The other question is can that signal be transferred to the
> application.
>
>> That means protocols that
>> are run across it must be able to tolerate the occasional missed
>> message, and either request a re-transmission, or just cope with not
>> having that particular packet at all.
>
> As I have said many times regarding real time control systems:
>
> 1.) you can have some time specific unreliable information transfer
> (e.g. UDP)
>
> 2.) you can have reliable connections but no real time predictability
> (e.g.. TCP)
>
> There is no "correct" solution, but you have to select case by case
> what to use.
>
Hans-Bernhard Bröker
Dec 6, 2015, 7:09:45 AM12/6/15
to
>> In a nutshell: this test run has exposed a flaw in your proprietary
>> little protocol. You need to repair that flaw, not the test run.
>> little protocol. You need to repair that flaw, not the test run.
> How to find out what is ill suited about our protocol?
I already told you that: it's the fact that it appears to assume that
every single message will successfully make it onto the bus. That
assumption has turned out to be incorrect.
> And then how can we make it robust so messages aren't missed.
be missed. You'll have to build a protocol that can withstand missed
messages to some extent, or at least notice that messages didn't arrive.
Hans-Bernhard Bröker
Dec 6, 2015, 7:28:35 AM12/6/15
to
Am 05.12.2015 um 21:03 schrieb Les Cargill:
unresolved collisions between two nodes sending messages bearing the
same ID, and starting simultaneously.
> Tauno Voipio wrote:
>> CAN is a broadcast datagram protocol. It in the nature of such
>> protocols that messages occasionally get lost. Even UDP in the
>> TCP/IP protocol suite has to accept lost datagrams, although
>> it does not always use broadcast addressing.
> The addressing in CAN is about what's in the PDU, not where it's going
> or who it's from.
It has to imply who it's from, if only because otherwise there would be
>> CAN is a broadcast datagram protocol. It in the nature of such
>> protocols that messages occasionally get lost. Even UDP in the
>> TCP/IP protocol suite has to accept lost datagrams, although
>> it does not always use broadcast addressing.
> The addressing in CAN is about what's in the PDU, not where it's going
> or who it's from.
unresolved collisions between two nodes sending messages bearing the
same ID, and starting simultaneously.
Hans-Bernhard Bröker
Dec 6, 2015, 7:41:56 AM12/6/15
to
Am 05.12.2015 um 23:00 schrieb upsid...@downunder.com:
> On Fri, 4 Dec 2015 23:44:03 +0100, Hans-Bernhard Bröker
> <HBBr...@t-online.de> wrote:
>>> The problem is that from time to time messages are getting missed. The
>>> message that don't make it to the tool also don't show up in the CAN log.
>
> The problem with the OP's case seems to be that the messages are lost
> somewhere between the receiver station terminal and the final
> application.
I don't think so. If they had gotten lost inside the receiver, they
> On Fri, 4 Dec 2015 23:44:03 +0100, Hans-Bernhard Bröker
> <HBBr...@t-online.de> wrote:
>>> The problem is that from time to time messages are getting missed. The
>>> message that don't make it to the tool also don't show up in the CAN log.
>
> The problem with the OP's case seems to be that the messages are lost
> somewhere between the receiver station terminal and the final
> application.
would at least have shown up in the CAN log, which I'm assuming is made
by CANoe, separate from the receiver itself.
>> CAN, like any multi-master shared-medium protocol, cannot guarantee
>> successful delivery of every single message.
> Sorry to be pedantic but as you know that CAN is actually only one of
> the few protocols in which the transmitting station knows when the
> correct signal is delivered to each receiving station terminals.
1) The CAN ACKknowledge field is essentially a wired OR, i.e. the
transmitter gets a successful ACK if any other active node saw a
correctly formed message --- there's no requirement for every node to
agree on that.
2) ACK does not depend on actual delivery of the received message. Even
if a message is blocked by a node's input filters, it will still be
ACKed by that node.
upsid...@downunder.com
Dec 6, 2015, 7:51:58 AM12/6/15
to
On Sun, 06 Dec 2015 12:30:00 +0100, David Brown
<david...@hesbynett.no> wrote:
>> Sorry to be pedantic but as you know that CAN is actually only one of
>> the few protocols in which the transmitting station knows when the
>> correct signal is delivered to each receiving station terminals.
>>
>
>That's not /quite/ true. CAN can see that the telegram has been
>successfully transmitted to at least one other node - but you have no
>guarantee that it is received by /all/ nodes.
You are right, if the node is physically disconnected from the bus or
<david...@hesbynett.no> wrote:
>> Sorry to be pedantic but as you know that CAN is actually only one of
>> the few protocols in which the transmitting station knows when the
>> correct signal is delivered to each receiving station terminals.
>>
>
>That's not /quite/ true. CAN can see that the telegram has been
>successfully transmitted to at least one other node - but you have no
>guarantee that it is received by /all/ nodes.
powered down, the intended receiver is not getting the message.
>It takes an unusual bad
>bus topology to get a situation where some nodes can receive a telegram
>correctly while others get errors, but it is possible - and may be worth
>considering if there are lots of nodes on the bus, or it is poorly
>terminated, or laid out in something other than a linear bus.
error, it should garble the bus and every node (including the
transmitter) on the bus should detect this garbling.
Of course, lots of garbled messages will finally kill the throughput
and cause unacceptable latencies.
I have worked with CanOpen systems with 50-75 nodes, 100-250 m cable
lengths, carefully observing branch cable limitations and optoisolator
delays. Sometimes it works reliably on the calculated speed, sometimes
we had to drop the line speed.
>Also note that all CAN controllers on the bus will send an acknowledge
>bit if the telegram is seen in the correct format, with the correct CRC
>check - it says nothing about the CAN controller being set up correctly
>to receive the message.
other node. Garbling/jamming tells that at least on node on the net is
"hard of hearing".
upsid...@downunder.com
Dec 6, 2015, 8:11:35 AM12/6/15
to
On Fri, 04 Dec 2015 15:21:07 -0600, "learn" <102059@EmbeddedRelated>
wrote:
>Our Electronic Control Unit (ECU) is based on Renesas RH850F1L
>Microcontroller. We basically have closed loop control system.
>
> Outputs of this ECU power DC Motors to move Third-Row Minivan Seats.
>
> ECU inputs are coming from DC Motor. This feedback signal is hall pulses
>indicating whether motor is turning or not or whether seat is moving or
>not.
>
> The software running in the ECU talks to PC Based CAN tool. They talk
>over simple proprietary protocol. They basically talk over 11-bit CAN.
>
> When the seat is moving, every 10ms, software running in the ECU sends
>seat position message to PC Based CAN Tool. When seat is at position 1,
>position 1 message is sent to the tool. After 10ms, position 2 message is
>sent to the tool. After next 10ms, position 3 message is sent to the tool,
>and so on.
>
> If everything is working correctly, over 1000 consecutive position
>messages will be sent to the tool. Time between these messages will be
>about 10ms.
>
> The problem is that from time to time messages are getting missed. The
>message that don't make it to the tool also don't show up in the CAN log.
>
> First 80 position CAN messages are correctly sent by ECU and correctly
>received by the tool.
> Position messages 81 through 84 don't show up in the tool. Also, they
>don't show up in the CAN log.
What is "position", is it just a serial number or a CAN identifiers
(which determines the telegram priority) ?
Have you calculated the buss throughput at your chosen speed with
maximum payload and maximum transparencies (worst case about 150 bits
in a telegram) ?
How long is your transmission queue, does it overflow when a lot of
messages are sent in a burst ? Do you have a priority (CAN-ID) based
transmit queue system ?
What is your transmit timeout ? Can a message be deleted, before it is
actually transmitted due to low priority ?
wrote:
>Our Electronic Control Unit (ECU) is based on Renesas RH850F1L
>Microcontroller. We basically have closed loop control system.
>
> Outputs of this ECU power DC Motors to move Third-Row Minivan Seats.
>
> ECU inputs are coming from DC Motor. This feedback signal is hall pulses
>indicating whether motor is turning or not or whether seat is moving or
>not.
>
> The software running in the ECU talks to PC Based CAN tool. They talk
>over simple proprietary protocol. They basically talk over 11-bit CAN.
>
> When the seat is moving, every 10ms, software running in the ECU sends
>seat position message to PC Based CAN Tool. When seat is at position 1,
>position 1 message is sent to the tool. After 10ms, position 2 message is
>sent to the tool. After next 10ms, position 3 message is sent to the tool,
>and so on.
>
> If everything is working correctly, over 1000 consecutive position
>messages will be sent to the tool. Time between these messages will be
>about 10ms.
>
> The problem is that from time to time messages are getting missed. The
>message that don't make it to the tool also don't show up in the CAN log.
>
> First 80 position CAN messages are correctly sent by ECU and correctly
>received by the tool.
> Position messages 81 through 84 don't show up in the tool. Also, they
>don't show up in the CAN log.
(which determines the telegram priority) ?
Have you calculated the buss throughput at your chosen speed with
maximum payload and maximum transparencies (worst case about 150 bits
in a telegram) ?
How long is your transmission queue, does it overflow when a lot of
messages are sent in a burst ? Do you have a priority (CAN-ID) based
transmit queue system ?
What is your transmit timeout ? Can a message be deleted, before it is
actually transmitted due to low priority ?
upsid...@downunder.com
Dec 6, 2015, 8:34:48 AM12/6/15
to
On Sun, 6 Dec 2015 13:41:51 +0100, Hans-Bernhard Bröker
The problem is too systematic to be attributed to some electrical
problems on the network.
Of course having a monitoring device separate from the CANoe would be
helpful.
It more sounds like an overload problem on the transmission side, too
much or too long data bursts into the buss compared to average
throughput would also explain the lost messages.
>>> CAN, like any multi-master shared-medium protocol, cannot guarantee
>>> successful delivery of every single message.
>
>> Sorry to be pedantic but as you know that CAN is actually only one of
>> the few protocols in which the transmitting station knows when the
>> correct signal is delivered to each receiving station terminals.
>
>Except that it doesn't, for at least two reasons:
>
>1) The CAN ACKknowledge field is essentially a wired OR, i.e. the
>transmitter gets a successful ACK if any other active node saw a
>correctly formed message --- there's no requirement for every node to
>agree on that.
>
>2) ACK does not depend on actual delivery of the received message. Even
>if a message is blocked by a node's input filters, it will still be
>ACKed by that node.
That might explain why some specific messages are always lost if
"position" refers to the CAN identifier.
<HBBr...@t-online.de> wrote:
>Am 05.12.2015 um 23:00 schrieb upsid...@downunder.com:
>> On Fri, 4 Dec 2015 23:44:03 +0100, Hans-Bernhard Bröker
>> <HBBr...@t-online.de> wrote:
>
>>>> The problem is that from time to time messages are getting missed. The
>>>> message that don't make it to the tool also don't show up in the CAN log.
>>
>> The problem with the OP's case seems to be that the messages are lost
>> somewhere between the receiver station terminal and the final
>> application.
>
>I don't think so. If they had gotten lost inside the receiver, they
>would at least have shown up in the CAN log, which I'm assuming is made
>by CANoe, separate from the receiver itself.
After rereading the original message, I agree with you.
>Am 05.12.2015 um 23:00 schrieb upsid...@downunder.com:
>> On Fri, 4 Dec 2015 23:44:03 +0100, Hans-Bernhard Bröker
>> <HBBr...@t-online.de> wrote:
>
>>>> The problem is that from time to time messages are getting missed. The
>>>> message that don't make it to the tool also don't show up in the CAN log.
>>
>> The problem with the OP's case seems to be that the messages are lost
>> somewhere between the receiver station terminal and the final
>> application.
>
>I don't think so. If they had gotten lost inside the receiver, they
>would at least have shown up in the CAN log, which I'm assuming is made
>by CANoe, separate from the receiver itself.
The problem is too systematic to be attributed to some electrical
problems on the network.
Of course having a monitoring device separate from the CANoe would be
helpful.
It more sounds like an overload problem on the transmission side, too
much or too long data bursts into the buss compared to average
throughput would also explain the lost messages.
>>> CAN, like any multi-master shared-medium protocol, cannot guarantee
>>> successful delivery of every single message.
>
>> Sorry to be pedantic but as you know that CAN is actually only one of
>> the few protocols in which the transmitting station knows when the
>> correct signal is delivered to each receiving station terminals.
>
>Except that it doesn't, for at least two reasons:
>
>1) The CAN ACKknowledge field is essentially a wired OR, i.e. the
>transmitter gets a successful ACK if any other active node saw a
>correctly formed message --- there's no requirement for every node to
>agree on that.
>
>2) ACK does not depend on actual delivery of the received message. Even
>if a message is blocked by a node's input filters, it will still be
>ACKed by that node.
"position" refers to the CAN identifier.
Les Cargill
Dec 6, 2015, 3:14:53 PM12/6/15
to
With CAN, you must have seperate media for two devices transmitting
with the same CAN id. There can be only one...
I suppose there is some way in which to be clever about it, but it's
an indefensible sort of cleverness.
--
Les Cargill
Les Cargill
Dec 6, 2015, 3:15:41 PM12/6/15
to
--
Les Cargill
Jack
Dec 7, 2015, 3:22:33 AM12/7/15
to
Il giorno venerdì 4 dicembre 2015 22:21:13 UTC+1, learn ha scritto:
> The problem is that from time to time messages are getting missed. The
> message that don't make it to the tool also don't show up in the CAN log.
so the messages don't go through the bus OR the tool/log can't read them for some reason?
> The problem is that from time to time messages are getting missed. The
> message that don't make it to the tool also don't show up in the CAN log.
>
> I'm using very complex CANoe Configuration File that was created by
> someone else. It's overwhelming. There are so many nodes, so many
> message in CAN Trace.
you can also create your own config file, or ask the guy who did it for some help.
> From where do I begin to debug this problem.
Connect a oscilloscope to the bus and find out.
> 11-bit proprietary protocol messages are exchanged between ECU and tool.
> Also, CANoe is sending so many messages. I don't even know what all
> these messages are. I have a hint as follows:
>
> Hint: CANoe sends messages to ECU in bursts. Instead, it needs to send
> them at slower rate. This might be the reason low priority position
> messages are missed from ECu. Turn off some nodes in CANoe.
> This shouldn't be ECU software problem because it's not sending messages
> in bursts.
>
> This is challenging problem for me because I don't have sufficient CANoe
> knowledge. Please help debug this problem.
0 new messages