Realtime loop used too much time in the last 30 seconds

[Zdeněk Materna]

Hi all,

we (Brno University of Technology) have experienced problems with halted motors before, but now it started to be really painful problem. Our PR2 runs on Indigo, Ubuntu 14.04 / lowlatency kernel. There are various error messages:

• Halting, realtime loop only ran at 743.3333 Hz
  
• Motors halted (service request); Dropped packets in last 10 seconds
• Realtime loop used too much time in the last 30 seconds.; Halting, realtime loop only ran at 743.3333 Hz.
• Safety Lockout: WATCHDOG (Name: r_elbow_flex_motor)
• !!! Broke Realtime, used more than 1000 micro seconds in update loop

C1 has 16 cores, average load is approximately 8, memory usage looks fine. Any idea how to fix it? Many thanks!

[Michael Görner]

Hi there,

did you check whether you exhaust the bandwidth on the robot-internal
network?
We found connecting C1 to the kinect2 on the head can add a lot of
traffic and result in overall delays.

Michael / v4hn

On Tue, 17 Sep 2019 06:59:10 -0700 (PDT)
Zdeněk Materna <zdenek....@gmail.com> wrote:

> Hi all,
>
> we (Brno University of Technology) have experienced problems with
> halted motors before, but now it started to be really painful
> problem. Our PR2 runs on Indigo, Ubuntu 14.04 / lowlatency kernel.
> There are various error messages:
>

> - Halting, realtime loop only ran at 743.3333 Hz
> -
> - Motors halted (service request); Dropped packets in last 10
> seconds
> - Realtime loop used too much time in the last 30 seconds.;

> Halting, realtime loop only ran at 743.3333 Hz.

> - Safety Lockout: WATCHDOG (Name: r_elbow_flex_motor)
> - !!! Broke Realtime, used more than 1000 micro seconds in update

> loop
>
> C1 has 16 cores, average load is approximately 8, memory usage looks
> fine. Any idea how to fix it? Many thanks!
>

--
Michael 'v4hn' Görner, M.Sc. Cognitive Science, PhD Student
Universität Hamburg
Faculty of Mathematics, Informatics and Natural Sciences
Department of Informatics
Group Technical Aspects of Multimodal Systems

Vogt-Kölln-Straße 30
D-22527 Hamburg

Room: F-315
Phone: +49 40 42883-2432
Website: https://tams.informatik.uni-hamburg.de/people/goerner/

[Zdeněk Materna]

Hello,

well, we don't have Kinect2 attached but too much network traffic could definitely be the reason - I will try to use some monitoring tool (any tips? :-)). Actually, there are for sure some network problems - we are running roscore on an external PC and usually ping from C2 to that PC works for some limited time and then suddenly stops. Routes seems to be ok, everything, it just don't work... My original hypotheses was overheating.

Best,

Zdenek

Dne čtvrtek 19. září 2019 7:32:24 UTC+2 Michael Görner napsal(a):

[Zdeněk Materna]

Hello,

I tried to install some network monitoring tools - bmon shows a high number of errors (Frame Error, CRC Error) for ecat0 interface - see attached screenshot. Any ideas?

Right now  I'm not able to enable motors at all (robot worked fine for approx. half an hour before that). There are also some strange messages in the terminal:

readMailboxInternal : ERROR asking for read repeat after dropping 1 packets

readMailboxInternal : ERROR asking for read repeat after dropping 1 packets

readMailboxInternal : ERROR asking for read repeat after dropping 2 packets

[ERROR] [1569332931.924796342]: No data have arrived for more than one second. Assuming that camera is no longer streaming.

[ERROR] [1569332931.927171245]: No data have arrived for more than one second. Assuming that camera is no longer streaming.

[WARN] [WallTime: 1569332933.830863] Caught exception <urlopen error [Errno 113] No route to host>

[ERROR] [1569332936.305849692]: Matching URL name://narrow_stereo_r#la...@10.69.0.44 : No cameras matched the URL.

[ERROR] [1569332936.306653971]: Matching URL name://wide_stereo_r#la...@10.69.0.46 : No cameras matched the URL.

[ERROR] [1569332936.327056989]: Matching URL name://narrow_stereo_l#la...@10.69.0.43 : No cameras matched the URL.

[ERROR] [1569332936.335725482]: Matching URL name://wide_stereo_l#la...@10.69.0.45 : No cameras matched the URL.

The air-intake filter was just replaced and output of "sensors" command looks ok - so it's not related to overheating.

Best,

Zdeněk

Dne čtvrtek 19. září 2019 13:35:51 UTC+2 Zdeněk Materna napsal(a):