Re: [beagleboard] I2C timeouts

[Gerald Coley]

I2C0 is not available on the headers as indicated in the schematic. The other ones are available, without any pullups, which need to be provided in order to work.

Gerald

On Wed, May 29, 2013 at 11:27 AM, Kevin McCarthy <sign...@gmail.com> wrote:

Greetings, 

I've recently purchased a BBB to use as a low-power, silent, network music streamer. But, I've had some troubles with it. My basic setup is that the BBB (running the latest update to the default Angstrom distro) mounts a directory of FLAC files over NFS and runs mpd to play them through an asynchronous USB 2.0 DAC. Software wise, everything seems to work fine, but only for a short time. After some seemingly random time (sometimes seconds, sometimes minutes) of playback, the audio gets distorted and starts to stutter. It's as though the BBB can't push the audio data fast enough to the DAC. It will recover in a matter of seconds and resume playback, but only for a short time before it happens again. Along with the audible problem, I see this in the log:

May 29 15:57:18 bone kernel: omap_i2c 44e0b000.i2c: controller timed out

May 29 15:57:19 bone kernel: omap_i2c 44e0b000.i2c: controller timed out

May 29 15:57:19 bone kernel: dummy 0-0034: Error -110 reading from cec:0xfe

Although usually I only see the "controller timed out" without the "Error -110" to accompany it.

So, I've been trying to figure out what's going on, but I'm no expert in this stuff, and I haven't had much luck. One curious thing I have found is that if I do this:

while true; do i2cdetect -y -r 0; done > /dev/null

While mpd is playing, things seem to go a lot smoother. I never get the timeout error in the logs, or the complete stop in playback, but I do get a little bit of distortion occasionally. It sounds like the same problem, but massively reduced in scale; like it's only hanging up for a few milliseconds here and there instead of seconds at a time.

The only lead I've been able to find is the hardware documentation that says the lack of pullup resistors on the i2c bus can cause timeouts. I was going to add pullup resistors, thinking the internal pullups weren't doing the job, but it doesn't look like i2c0 is available on the headers. So, I decided to come here and see if any experts had any ideas...

Has anyone seen this, or even if you haven't, does anybody have any ideas?

Thanks.

-Kevin

--
For more options, visit http://beagleboard.org/discuss
---
You received this message because you are subscribed to the Google Groups "BeagleBoard" group.
To unsubscribe from this group and stop receiving emails from it, send an email to beagleboard...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.
 
 

--

Gerald

 

ger...@beagleboard.org

g-co...@ti.com

http://beagleboard.org/

http://circuitco.com/support/

[Kevin McCarthy]

Ok, after considerable screwing around, I think I've found a solution to the i2c timeout problem. It occurred to me that maybe it wasn't actually anything to do with i2cdetect that fixed the problem, and that maybe it was just load. So I wrote a "busy the CPU" program and it worked just as well to tame the problem as my i2cdetect loop.

So, thinking that maybe CPU scaling was causing the problem, I switched from the "ondemand" to the "performance" scaling_governor:

echo "performance" > /sys/devices/system/cpu/cpu0/cpufreq/scaling_governor

All of my i2c timeout problems vanished and everything is working as it should now. I'm guessing that the CPU speed changing during heavy i2c traffic was causing some sort of timing issue or something.

Figured I'd better follow up in case someone else runs into the same problem.

-Kevin

[monzie]

Thanks Kevin your tip seems to have solved my problems after a day of troubleshooting.
I was experiencing I2C timeouts and kernel panics at random intervals when streaming video.
I am not connected to or using any I2C devices.
Switching to performance has resolved the issues.
This command has a similar effect and is a little shorter:
sudo cpufreq-set -f 1GHZ
cpufreq-info reports the actual current clock frequency.

Does anyone have a good explanation as to why changing the cpu speed eliminates the i2c timeouts?
I would have thought the kernel would service the i2c bus regardless of other activity.

[Kevin McCarthy]

Careful of the cpu-ondemand.timer systemd unit. I had to disable that too, or the problem would return 10 minutes after reboot. For some reason it's set up to wait until 10 minutes after it boots then to change the cpufreq governor back to ondemand. Still has me a little peeved because I can't think of a good reason for it to be set up that way, unless it's to confuse the end user who is trying to change the cpufreq governor permanently.

 

Does anyone have a good explanation as to why changing the cpu speed eliminates the i2c timeouts?
I would have thought the kernel would service the i2c bus regardless of other activity.

I'm pretty sure it's not changing the CPU speed that fixes it. It's actually NOT changing the CPU speed. With ondemand, it's constantly trying to scale the CPU speed to save energy. I suspect the CPU is interrupted just long enough to miss an I2C clock or something like that when the frequency changes. Forcing it to ondemand keeps the clock speed from changing, and the CPU manages to catch all of the data on the I2C bus. But, that's just a guess. 

It could also be that "ondemand" lowers the CPU speed to a point where it can't keep up with the I2C bus speed (unlikely). Or it could be something wonky where the SOC pulls the I2C pins low for an instant when it resets after a clock speed change or something like that. Someone smarter than I will probably have to investigate to find out for sure.

-Kevin

[Daniel Nilsson]

On Wednesday, August 14, 2013 2:11:31 PM UTC+2, Kevin McCarthy wrote:

Does anyone have a good explanation as to why changing the cpu speed eliminates the i2c timeouts?
I would have thought the kernel would service the i2c bus regardless of other activity.

I'm pretty sure it's not changing the CPU speed that fixes it. It's actually NOT changing the CPU speed. With ondemand, it's constantly trying to scale the CPU speed to save energy. I suspect the CPU is interrupted just long enough to miss an I2C clock or something like that when the frequency changes. Forcing it to ondemand keeps the clock speed from changing, and the CPU manages to catch all of the data on the I2C bus. But, that's just a guess. 

I've seen this issue as well, but haven't had a chance yet to investigate further. In my case though I was using the "performance" scaling governor which keeps the CPU running at full speed, but a user-space application that used 80-90% CPU would still trigger this issue after a while (1 minute or so). The user-space application ran with only normal SCHED_OTHER threads, so no issue with SCHED_FIFO threads starving kernel irq-threads should be possible. The application (linphone) is also an audio application, but audio is going on through an I2S codec and not USB so I'm not sure if it is just coincidence that we seem to have been able to trigger this more than once using audio applications. There is no I2C traffic at all when my testcase starts failing.

Regards
Daniel

[William C Bonner]

I've been seeing this same behavior, and believe that it is contributing to my board locking up while I'm running an app that is simply streaming the webcam output over WiFi using a multicast port. 

Once it gets to this point, It's continuously printing junk to the serial console.

[ 2695.793131] omap_i2c 44e0b000.i2c: controller timed out

[ 2696.799063] omap_i2c 44e0b000.i2c: controller timed out

[ 2697.806895] omap_i2c 44e0b000.i2c: controller timed out

[ 2697.812991] tps65217 0-0024: Read from reg 0xf failed

[ 2698.820553] omap_i2c 44e0b000.i2c: controller timed out

[ 2698.826543] tps65217 0-0024: Read from reg 0x11 failed

[ 2698.832365] cpufreq_cpu0: failed to scale voltage down: -110

[ 2700.146663] omap_i2c 44e0b000.i2c: controller timed out

[ 2701.152639] omap_i2c 44e0b000.i2c: controller timed out

[ 2702.162376] omap_i2c 44e0b000.i2c: controller timed out

[ 2702.168381] tps65217 0-0024: Read from reg 0xf failed

[ 2703.176036] omap_i2c 44e0b000.i2c: controller timed out

[ 2703.182062] tps65217 0-0024: Read from reg 0x11 failed

[ 2703.187967] cpufreq_cpu0: failed to scale voltage down: -110

[ 2704.511886] omap_i2c 44e0b000.i2c: controller timed out

[ 2705.517833] omap_i2c 44e0b000.i2c: controller timed out

[ 2706.525624] omap_i2c 44e0b000.i2c: controller timed out

[ 2706.535484] tps65217 0-0024: Read from reg 0xf failed

[ 2707.543204] omap_i2c 44e0b000.i2c: controller timed out

[ 2707.549184] tps65217 0-0024: Read from reg 0x11 failed

[ 2707.555006] cpufreq_cpu0: failed to scale voltage down: -110

I have been getting it to stop by manually issuing the command

/usr/bin/cpufreq-set -g performance

but I'd prefer if I can have that command issues as part of the boot sequence. I knew the old style linux completely, but am still learning the systemctl version. Can someone tell me the simplest way to have the above command run at boot?

[Gerald Coley]

This looks to me like it is having trouble changing the voltages on the fly, performed during on demand performance scaling. Add the CPU Frequency Scaling Monitor to the panel and set it to 1GHZ. See if the issue goes away.

Gerald

--

[William C Bonner]

I'm not certain I understand what you mean by "Add the CPU Frequency Scaling Monitor to the panel" 

I'm running this unit purely via SSH or Serial access, and have gone so far as to disable the HDMI on boot, which seems to automatically disable several services. I've got a MicroSD card in the slot, but am running primarily from the eMMC, with extra storage mounted from the MicroSD. Here's my uEnv.txt file on the MicroSD card.

mmcdev=1

bootpart=1:2

mmcroot=/dev/mmcblk1p2

optargs=quiet capemgr.disable_partno=BB-BONELT-HDMI,BB-BONELT-HDMIN

and I've added these items to /etc/fstab:

/dev/mmcblk0p2       /home                auto       defaults              0  2

/dev/mmcblk0p1       /media/BONEBOOT      auto       defaults              0  2

/dev/mmcblk1p1       /media/BEAGLEBONE    auto       ro                    0  2

Is there a kernel command I can add to uEnv.txt that will put the governor in performance mode, or fixed frequency? (I know I've seen a pointer to the source for U-Boot where it does the option processing, but I can't seem to find it right now.)

[Koen Kooi]

Op 7 sep. 2013, om 05:41 heeft William C Bonner <wimb...@gmail.com> het volgende geschreven:

> I'm not certain I understand what you mean by "Add the CPU Frequency Scaling Monitor to the panel"
>
> I'm running this unit purely via SSH or Serial access, and have gone so far as to disable the HDMI on boot, which seems to automatically disable several services. I've got a MicroSD card in the slot, but am running primarily from the eMMC, with extra storage mounted from the MicroSD. Here's my uEnv.txt file on the MicroSD card.
>
> mmcdev=1
> bootpart=1:2
> mmcroot=/dev/mmcblk1p2
> optargs=quiet capemgr.disable_partno=BB-BONELT-HDMI,BB-BONELT-HDMIN
>
> and I've added these items to /etc/fstab:
>
> /dev/mmcblk0p2 /home auto defaults 0 2
> /dev/mmcblk0p1 /media/BONEBOOT auto defaults 0 2
> /dev/mmcblk1p1 /media/BEAGLEBONE auto ro 0 2
>
> Is there a kernel command I can add to uEnv.txt that will put the governor in performance mode, or fixed frequency? (I know I've seen a pointer to the source for U-Boot where it does the option processing, but I can't seem to find it right now.)

It's not uboot related, but it switches from 'performance' to 'ondemand' after 10 minutes, you can play around with

/lib/systemd/system/cpu-ondemand.service
/lib/systemd/system/cpu-ondemand.timer

To change that behaviour.

[William C Bonner]

Should the command:

systemctl disable cpu-ondemand.timer

work to stop the service from being run for future boots?

I issued that command, powered off my board, and now when I log I think it's running in performance mode, which is what I want.

root@beaglebone:~# cpufreq-info 

cpufrequtils 008: cpufreq-info (C) Dominik Brodowski 2004-2009

Report errors and bugs to cpu...@vger.kernel.org, please.

analyzing CPU 0:

  driver: generic_cpu0

  CPUs which run at the same hardware frequency: 0

  CPUs which need to have their frequency coordinated by software: 0

  maximum transition latency: 300 us.

  hardware limits: 300 MHz - 1000 MHz

  available frequency steps: 300 MHz, 600 MHz, 800 MHz, 1000 MHz

  available cpufreq governors: conservative, ondemand, userspace, powersave, performance

  current policy: frequency should be within 300 MHz and 1000 MHz.

                  The governor "performance" may decide which speed to use

                  within this range.

  current CPU frequency is 1000 MHz (asserted by call to hardware).

  cpufreq stats: 300 MHz:nan%, 600 MHz:nan%, 800 MHz:nan%, 1000 MHz:nan%

[kavitha bk]

Hi

I also get this Issue
[ 1275.297180] omap_i2c omap_i2c.2: controller timed out
[ 1276.297149] omap_i2c omap_i2c.2: controller timed out
[ 1282.296966] omap_i2c omap_i2c.2: controller timed out
[ 1283.296936] omap_i2c omap_i2c.2: controller timed out
[ 1284.296905] omap_i2c omap_i2c.2: controller timed out

Keeping it userspace and running at HIgher frequency reduces the FRequency of occurance

Thanks,
Kavitha