BeagleBone Black: v3.8.13-bone37 kernel flash slowly fills up

[Craig Mehan]

I am new in the software arena, so forgive me if I am missing something obvious.  I have a BeagleBone Black.  I have loaded Debian  v3.8.13-bone37 from https://rcn-ee.net/deb/microsd/wheezy/bone-debian-7.3-2014-01-24-2gb.img.xz .  I believe my flash slowly fills up without additional software being loaded.

After a fresh "flashing" of the eMMC a df -h shows:

 

Filesystem                                                                      Size  Used     Avail      Use%   Mounted on

rootfs                                                                              1.7G  837M    753M      53%     /

udev                                                                               10M         0       10M         0%   /dev

tmpfs                                                                              50M  220K       50M         1%   /run

/dev/disk/by-uuid/55f7d188-d284-4832-8d00-f68a649c8364  1.7G  837M     753M       53%   /

tmpfs                                                                             5.0M        0      5.0M        0%    /run/lock

tmpfs                                                                            100M        0      100M        0%    /run/shm

/dev/mmcblk0p1                                                              96M    72M       25M      75%    /boot/uboot

With or without user activity the rootfs and /dev/disk/... grows to fill the available space.  As shown:

Filesystem                                                                      Size  Used   Avail   Use%    Mounted on

rootfs                                                                             1.7G  1.7G         0    100%    /

udev                                                                               10M     0       10M       0%     /dev

tmpfs                                                                              50M  220K    50M       1%    /run

/dev/disk/by-uuid/55f7d188-d284-4832-8d00-f68a649c8364  1.7G  1.7G        0     100%    /

tmpfs                                                                             5.0M     0     5.0M        0%    /run/lock

tmpfs                                                                            100M     0     100M        0%    /run/shm

/dev/mmcblk0p1                                                             96M   72M      25M       75%   /boot/uboot

I have tried executing apt-get clean, autoclean, remove and autoremove.  Nothing seems to reduce the used flash.  I have reflashed the eMMC with the above mentioned SD image three times.  It has acted the same each time.  I first noticed this problem when I tried to do an apt-get upgrade and received error messages of out of disk space.  I then have reflashed two more times to make sure the problem is consistent. 

Help would be greatly appreciated.

Craig

[Robert Nelson]

Hi Craig,

It's /var/log/ but i've never seen it fill up so fast!

can you do a:

du -sh /var/log/* | pastebinit
dmesg | pastebinit

Did you enable systemd based boot or is this still the traditional one?

Yesterday i added a tweak so systemd only uses 8Mb's..
https://github.com/RobertCNelson/omap-image-builder/blob/master/scripts/chroot.sh#L368

sudo sed -i -e 's:#SystemMaxUse=:SystemMaxUse=8M:g'
/etc/systemd/systemd-journald.conf

Regards,

--
Robert Nelson
http://www.rcn-ee.com/

[Craig Mehan]

Hi Robert,

I will perform your suggestions tonight.  I took another troubleshooting step last night of loading the previous version (3.8.13-bone32).  I am ashamed I did not think of it before posting.  I have let it run all night and the older version has a more muted memory growth of 37 MB in ~12 hrs.

Your question: Did you enable systemd based boot or is this still the traditional one?  I am sorry, I am not knowledgeable enough to answer your question (I have some google'ing to do).  I am not aware of actively changing the booting process.  I only copied the image to an SD card and then flashed the eMMC.

I will reload the newer (v3.8.13-bone37) system (from my PC harddrive) and post your suggestions tonight.

Thank you for your attention,

Craig

[Robert Nelson]

On Wed, Jan 29, 2014 at 11:21 AM, Craig Mehan <craigl...@gmail.com> wrote:
> Hi Robert,
>
> I will perform your suggestions tonight. I took another troubleshooting
> step last night of loading the previous version (3.8.13-bone32). I am
> ashamed I did not think of it before posting. I have let it run all night
> and the older version has a more muted memory growth of 37 MB in ~12 hrs.
>
> Your question: Did you enable systemd based boot or is this still the
> traditional one? I am sorry, I am not knowledgeable enough to answer your
> question (I have some google'ing to do). I am not aware of actively
> changing the booting process. I only copied the image to an SD card and
> then flashed the eMMC.

Okay, by default systemd isn't enabled on that particular image you
downloaded. It would be nice to catch what is causing the /var/log/
overload, do you have a specific usb device connected?

[Craig Mehan]

Ok I have done the pastebinits that you requested.  Appears they sent ok.  The memory usage was a bit more under control this flashing session, but still high and growing.

On usb devices connected, what started all this is trying to get the Asus USB-BT211 bluetooth dongle to work with the Beaglebone Black.  I don't really remember if the dongle was plugged in for my original email. The pastebinits that were sent, do not have the dongle plugged in.  After I did the pastebinits, I did plug in the dongle and did not see any real change in the memory used or rate of memory used.  Although with both the first email I sent and this email I have not installed the bluetooth or atheros modules.

On the bluetooth dongle subject, it appears I chose poorly from both experience and from the Google posts on this dongle.  I did finally get the dongle to light up and pair with my phone after I did an aptitude install firmware-atheros.  But, then the memory problem started rearing its head and I have not been back to that problem.

I appreciate your attention to this and am happy to provide anymore data or run any tests needed.

Regards,

Craig Mehan

[Robert Nelson]

On Wed, Jan 29, 2014 at 11:06 PM, Craig Mehan <craigl...@gmail.com> wrote:
> Ok I have done the pastebinits that you requested. Appears they sent ok.
> The memory usage was a bit more under control this flashing session, but
> still high and growing.

A little miss-communication. You kinda need to share the http links...

> On usb devices connected, what started all this is trying to get the Asus
> USB-BT211 bluetooth dongle to work with the Beaglebone Black. I don't
> really remember if the dongle was plugged in for my original email. The
> pastebinits that were sent, do not have the dongle plugged in. After I did
> the pastebinits, I did plug in the dongle and did not see any real change in
> the memory used or rate of memory used. Although with both the first email
> I sent and this email I have not installed the bluetooth or atheros modules.
>
> On the bluetooth dongle subject, it appears I chose poorly from both
> experience and from the Google posts on this dongle. I did finally get the
> dongle to light up and pair with my phone after I did an aptitude install
> firmware-atheros. But, then the memory problem started rearing its head and
> I have not been back to that problem.
>
> I appreciate your attention to this and am happy to provide anymore data or
> run any tests needed.

Regards,

[Craig Mehan]

Oops, sorry, I should have mentioned I am a newbie. After some googling:

http://paste.debian.net/79272/
http://paste.debian.net/79273/

Thank you for your help,

Craig Mehan

[Robert Nelson]

On Thu, Jan 30, 2014 at 11:14 AM, Craig Mehan <craigl...@gmail.com> wrote:
> Oops, sorry, I should have mentioned I am a newbie. After some googling:
>
> http://paste.debian.net/79272/
> http://paste.debian.net/79273/

Ah crap.. i fixed that a year ago!

gen_ndis_query_resp: RNDIS_OID_GEN_RCV_NO_BUFFER

And then just unfixed it a couple weeks ago..
https://github.com/RobertCNelson/linux-dev/commit/3f61fc35d46cce0ee1a7543b666b2e9a99822b60

[Robert Nelson]

On Thu, Jan 30, 2014 at 12:42 PM, Robert Nelson <robert...@gmail.com> wrote:
> On Thu, Jan 30, 2014 at 11:14 AM, Craig Mehan <craigl...@gmail.com> wrote:
>> Oops, sorry, I should have mentioned I am a newbie. After some googling:
>>
>> http://paste.debian.net/79272/
>> http://paste.debian.net/79273/
>
> Ah crap.. i fixed that a year ago!
>
> gen_ndis_query_resp: RNDIS_OID_GEN_RCV_NO_BUFFER
>
> And then just unfixed it a couple weeks ago..
> https://github.com/RobertCNelson/linux-dev/commit/3f61fc35d46cce0ee1a7543b666b2e9a99822b60

So the fix is on the way.. (few hours of building..) to fix it now..

cd /opt/scripts/
git pull
./tools/update_kernel.sh --kernel v3.8.13-bone35

Then tomorrow just run:
cd /opt/scripts/
./tools/update_kernel.sh

[tir...@gmail.com]

Hi Robert 

I've (of course ) the same problem with my distro that is a 3.8.13-bone37 

I did the bone35 kernel update and it worked perfectly.  yet I'm wondering If I will have to "suffer" from any regression from 37 to 35 ? 

cheers

 Eric

[Robert Nelson]

On Wed, Feb 5, 2014 at 5:43 AM, <tir...@gmail.com> wrote:
> Hi Robert
> I've (of course ) the same problem with my distro that is a 3.8.13-bone37
> I did the bone35 kernel update and it worked perfectly. yet I'm wondering
> If I will have to "suffer" from any regression from 37 to 35 ?

Well, bone40 has been out with the /var/log/ overflow from driver fix.

[tir...@gmail.com]

update done ! 

all fine :-) 

 

Thank you for your quick reply ;-) 

cheers 

Eric

[godsf...@gmail.com]

Can someone tell me how to apply the built kernel on debian? I followed the above instructions to download from git and then compiled. I don't know how to apply the built kernel. Thanks

[Robert Nelson]

On Sat, Feb 8, 2014 at 8:29 AM, <godsf...@gmail.com> wrote:

Can someone tell me how to apply the built kernel on debian? I followed the above instructions to download from git and then compiled. I don't know how to apply the built kernel. Thanks

If your image is on the microSD, just stick it in and run "./tools/install_kernel.sh"

If you've flashed it to the eMMC, well it's not that trival.

Regards,

[Brad Andersen]

Robert, when might you be releasing a bone40 flasher for the BBB?  From this discussion it appears the bone37 has a memory leak (I'm using the bone37 flasher from your http://elinux.org/BeagleBoardDebian page.

Thanks,

Brad

[Robert Nelson]

On Thu, Feb 13, 2014 at 12:32 PM, Brad Andersen <anderse...@gmail.com> wrote:

Robert, when might you be releasing a bone40 flasher for the BBB?  From this discussion it appears the bone37 has a memory leak (I'm using the bone37 flasher from your http://elinux.org/BeagleBoardDebian page.

I'd like to get an image out today/tomorrow.. testing a few things..

But, you can upgrade the kernel today via:

cd /opt/script/

git pull

./tools/update_kernel.sh (it'll pull in bone40)

Regards,

[jglos...@gmail.com]

I think this memory leak still exists in bone50.

[Robert Nelson]

On Wed, Sep 24, 2014 at 9:10 PM, <jglos...@gmail.com> wrote:
> I think this memory leak still exists in bone50.

Latest stable is bone67, so "update"..

I'll let you review:
https://github.com/RobertCNelson/bb-kernel/commits/am33x-v3.8

to find out when i actually fixed it..

[jglos...@gmail.com]

Thanks, but I'm not really sure how to update from that. Can you advise me? The latest release on the beaglebone.org site is still the 5-14-2014 version with bone50.

Also, I can't see in your github history that it is actually fixed, but I'll be happy to test bone67 for you if you can tell me how to update it.

Thank you.

James

[Robert Nelson]

On Thu, Sep 25, 2014 at 12:21 PM, <jglos...@gmail.com> wrote:
> Thanks, but I'm not really sure how to update from that. Can you advise me?
> The latest release on the beaglebone.org site is still the 5-14-2014 version
> with bone50.
>
> Also, I can't see in your github history that it is actually fixed, but I'll
> be happy to test bone67 for you if you can tell me how to update it.

cd /opt/scripts/tools/
git pull
sudo ./update_kernel.sh

(reboot)

[jglos...@gmail.com]

Thank you. That does seem pretty simple. Unfortunately, I get this error message:

debian@beaglebone:/opt/scripts/tools$ sudo ./update_kernel.sh 

info: checking archive

--2014-05-18 00:24:33--  https://rcn-ee.net/deb/wheezy-armhf/LATEST-omap-psp

Resolving rcn-ee.net (rcn-ee.net)... 69.163.222.213

Connecting to rcn-ee.net (rcn-ee.net)|69.163.222.213|:443... connected.

ERROR: The certificate of `rcn-ee.net' is not trusted.

The certificate has not yet been activated

Is there a way to make it ignore the certificate?

Thanks again,

James

[Robert Nelson]

On Thu, Sep 25, 2014 at 2:25 PM, <jglos...@gmail.com> wrote:
> Thank you. That does seem pretty simple. Unfortunately, I get this error
> message:
>
>
> debian@beaglebone:/opt/scripts/tools$ sudo ./update_kernel.sh
> info: checking archive
> --2014-05-18 00:24:33-- https://rcn-ee.net/deb/wheezy-armhf/LATEST-omap-psp
> Resolving rcn-ee.net (rcn-ee.net)... 69.163.222.213
> Connecting to rcn-ee.net (rcn-ee.net)|69.163.222.213|:443... connected.
> ERROR: The certificate of `rcn-ee.net' is not trusted.
> The certificate has not yet been activated
>
> Is there a way to make it ignore the certificate?

update your clock:

sudo ntpdate pool.ntp.org

The certificate was updated in August 2014 (the image if the time
isn't updated starts in May 2014)

[James Glossinger]

That did it. Thanks. The update went fine.

I'm still seeing the disk filling up at the same rate as before. I delete log files before checking disk space. There is data in a temporary file that fills up at a rate of about 1mb/day. The space gets freed on a reboot. 

Can you think of what may be doing this? Is this the same problem that others have reported?

--
For more options, visit http://beagleboard.org/discuss
---
You received this message because you are subscribed to a topic in the Google Groups "BeagleBoard" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/beagleboard/OXmp8bi6X1I/unsubscribe.
To unsubscribe from this group and all its topics, send an email to beagleboard...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Robert Nelson]

On Thu, Sep 25, 2014 at 3:37 PM, James Glossinger <jglos...@gmail.com> wrote:
> That did it. Thanks. The update went fine.
>
> I'm still seeing the disk filling up at the same rate as before. I delete
> log files before checking disk space. There is data in a temporary file that
> fills up at a rate of about 1mb/day. The space gets freed on a reboot.

1mb/day!!!! This original thread was about 1x'smb/minute.. ;)

> Can you think of what may be doing this? Is this the same problem that
> others have reported?

I've capped systemd around 8mb max via:

SystemMaxUse=8M

[James Glossinger]

I agree that 1mb/day is a slowish leak. I'm glad it has improved. Until recently, my system only had 40mb free at all, so this would fill up in a month. I hope it is capped at 8mb as you say.

But is the 8mb limit relevant if I'm deleting the logs before checking space? /var/log is empty when I check space. Are you sure this will halt at 8mb, or is there something else I should check for?

[jglos...@gmail.com]

I've been thinking more about this slow leak. I do believe it has something to do with logging, and it seems that logs are being stored in a temporary file until reboot even after being written to /var/log. But perhaps the more important question is, "why are there so many errors being logged?"

I see constant error messages to the console even when no applications are being run. These are the repeating errors I see:

[   21.170242] libphy: PHY 4a101000.mdio:01 not found

[   21.177302] net eth0: phy 4a101000.mdio:01 not found on slave 1

[   26.880502] libphy: PHY 4a101000.mdio:01 not found

[   26.886371] net eth0: phy 4a101000.mdio:01 not found on slave 1

[   27.335371] libphy: PHY 4a101000.mdio:01 not found

[   27.341350] net eth0: phy 4a101000.mdio:01 not found on slave 1

[  602.052044] tilcdc 4830e000.fb: timeout waiting for framedone

What is going on here? Do you have any suggestions?

Thanks for your help.

James

[jglos...@gmail.com]

The errors I posted recently don't show up as often as I claimed, so that's not the problem. This is a mystery to me. Sorry for all the emails about this. If you have any suggestions on things to test for, I'd love to hear. Thanks again.

On Thursday, September 25, 2014 1:58:52 PM UTC-7, RobertCNelson wrote:

[James Glossinger]

Ok, I figured this all out. There is no filling up of the disk from the operating system. My method of investigating was causing the problem. I started with a real disk use issue in my own software that I fixed. Then I noticed a remaining and much slower disk use that turned out to be caused by the way I was looking into the problem. No, I wasn't logging to the beaglebone itself, I was logging data to another machine, but .... poorly.

Both bone67 and bone50 are fine.

Feel free to delete my comments about this non issue.

Thanks again for your time and for showing me how to upgrade the kernel.

James

[William Hermans]

Either way, this still would not be "a leak" This is standard Linux behavior. Logging that is.

You received this message because you are subscribed to the Google Groups "BeagleBoard" group.
To unsubscribe from this group and stop receiving emails from it, send an email to beagleboard...@googlegroups.com.

[James Glossinger]

If logging was the problem, then you are correct. Logging was only one of the possible reasons I was looking into. 

[William Hermans]

Well you're not the first to experience this "issue" and maybe not the last. However with that said I've yet to experience it myself. The only thing I can think of is that I use a custom rootsfs based off of Roberts barefs image. I also compile my own kernel, but I would think that should have little to do with it.