BUG, TERRIBLE BUG!!! Re: [android-developers] Re: HELP! (was Re: Low on space)

[Stoyan Damov]

Well, game over for me, I MUST do a factory reset and I guess I need
to pray that this WILL work. I just browsed the Market, searching for
system tools, tried downloading System Monitor (which just slept at
"starting download"), and it Market's data grew to 2MB (which I can't
clear).

Now I only have 1 (*ONE* MB) free memory and can't install my app -
I'm getting "Installation error: INSTALL_FAILED_INSUFFICIENT_STORAGE"
and that's it :(
Memory just went lower and lower and lower no matter whether I install
or delete apps, that's fucking great :(

Is there *anyone* who'll dare to tell me there's no bug in Android?
How the fuck I don't have *anything* installed (besides the
pre-installed G1 apps) and my memory is only ONE MB???????
Where are the ~70 MBs??? :(((


On Mon, Jan 12, 2009 at 1:40 AM, Mark Murphy <mmu...@commonsware.com> wrote:
>
> Stoyan Damov wrote:
>> but hang on... apparently I have misunderstood something.
>>
>> Are you saying that based on what you read I won't have to re-activate
>> the phone (i.e. have to enter the SIM's unlock code, which presumably,
>> based on numerous posts I can buy for ~$25) but merely add my
>> carrier's EDGE's APN so I can sign in to Google?
>
> It's more that I am saying that a factory data reset does not appear to
> require activation of the phone (i.e., phone company work). It does
> require going through the whole GMail thing, which was a problem for
> ADP1 buyers, which is why people wrote up the ways to deal with that.
>
> Since I don't own an ADP1, I have not paid attention to the details of
> exactly what those instructions were, just that they existed.
>
> --
> Mark Murphy (a Commons Guy)
> http://commonsware.com
> _The Busy Coder's Guide to Android Development_ Version 2.0 Available!
>
> >
>

[Dianne Hackborn]

On Sun, Jan 11, 2009 at 5:04 PM, Stoyan Damov <stoyan...@gmail.com> wrote:

Is there *anyone* who'll dare to tell me there's no bug in Android?

Why would anyone claim there are no bugs in -any- piece of complicated software?  (Except I guess TeX).
 

How the fuck I don't have *anything* installed (besides the
pre-installed G1 apps) and my memory is only ONE MB???????
Where are the ~70 MBs??? :(((

Just to clarify, none of the installed applications are shown to be using a large amount of data?  If that is the case, then there are very few places the storage could have gone.  First be sure to check in I believe /data/local (may be something else, it's the directory under /data owned by the shell allowing it to place files there).  If that is not large, it could be in data/system, but unlikely.

--
Dianne Hackborn
Android framework engineer
hac...@android.com

Note: please don't send private questions to me, as I don't have time to provide private support.  All such questions should be posted on public forums, where I and others can see and answer them.

[Joel Knighton]

Do you have busybox on the device?  If so, I can help you find where the space is gone.  With only the included android shell binaries, however, I don't know if I can.

--
Joel Knighton

[Joel Knighton]

Okay, Stoyan, if you want to do a ls -l -R and then email it to me (joel.k...@gmail.com) I'll read through it and see if I notice a problem.  (You can email straight from the terminal app)

--
Joel Knighton

[Stoyan Damov]

Joel, thanks for your help, but I unfortunately I can't help you help
me because I don't have root, so I can't install busybox and I can't
give you a meaningful file listing with so many directories being
denied access. I didn't see anything interesting in the log, and btw
ls hung while (or after) it was listing ./sys/bus/platform/devices. I
guess I'll have to bite the bullet and do a factory reset :(

Cheers,
Stoyan

[Stoyan Damov]

On Mon, Jan 12, 2009 at 4:33 AM, Dianne Hackborn <hac...@android.com> wrote:
> On Sun, Jan 11, 2009 at 5:04 PM, Stoyan Damov <stoyan...@gmail.com>
> wrote:
>>
>> Is there *anyone* who'll dare to tell me there's no bug in Android?
>
> Why would anyone claim there are no bugs in -any- piece of complicated
> software? (Except I guess TeX).

Actually Dianne I am *stunned* how few bugs I've found during my 1
month usage of Android.
Hail to Google engineers (I'm *not* joking)!
This one, however, is a showstopper for me in the verbatim sense.

>
>>
>> How the fuck I don't have *anything* installed (besides the
>> pre-installed G1 apps) and my memory is only ONE MB???????
>> Where are the ~70 MBs??? :(((
>
> Just to clarify, none of the installed applications are shown to be using a
> large amount of data?

Yes, none.

> If that is the case, then there are very few places
> the storage could have gone. First be sure to check in I believe
> /data/local (may be something else, it's the directory under /data owned by
> the shell allowing it to place files there). If that is not large, it could
> be in data/system, but unlikely.

Well, the over-the-air patch @#$%ed root access so I can't look anywhere.

Cheers

[bx]

I'm having the same problem too now, of sorts.. Market claims to be
using 19MB of storage (and grows with every app I download), and there
doesn't appear to be any way to clear it.

On Jan 12, 1:00 am, "Stoyan Damov" <stoyan.da...@gmail.com> wrote:
> On Mon, Jan 12, 2009 at 4:33 AM, Dianne Hackborn <hack...@android.com> wrote:
> > On Sun, Jan 11, 2009 at 5:04 PM, Stoyan Damov <stoyan.da...@gmail.com>

[HaleRazor]

I am also seeing similar behavior. Market now shows using 21MB. I
thought it might be related to the apps that I had installed, so I
uninstalled three applications of varying sizes...the biggest being
3MB. Market still showed using 21MB. I have formed the suspicion
that there is a bug in the Market application. I am concerned that
the Market size is related to the number of applications available in
the Market. As a result, I do not plan on opening the Market
application any more until I have read something that convinces me
that my suspicions are wrong or that they were correct and the problem
has been fixed.

[Mark Murphy]

This has been written up on a few occasions, including me on this thread.

The Android Market apparently caches market-related information (e.g.,
icons and descriptions). This cache is not proactively released.
However, when free storage space drops far enough, the Market will free
up the cache.

The core Android team has indicated that they need to try to free up
that cache more proactively.

The Market has nothing to do with what apps you have installed or
uninstalled, AFAIK, so uninstalling an app will not affect the Market size.

[Disconnect]

What is your reference for the market actually freeing cache?

The devs I talked to about this indicated it did not do that. (Ditto for the occasional lost webpage screenshot, but at least 'delete all data' can clear that at the expense of bookmarks, etc)

[Stoyan Damov]

On Mon, Jan 12, 2009 at 8:43 PM, Mark Murphy <mmu...@commonsware.com> wrote:
>
> The Android Market apparently caches market-related information (e.g.,
> icons and descriptions). This cache is not proactively released.
> However, when free storage space drops far enough, the Market will free
> up the cache.
>
> The core Android team has indicated that they need to try to free up
> that cache more proactively.
>
> The Market has nothing to do with what apps you have installed or
> uninstalled, AFAIK, so uninstalling an app will not affect the Market size.

Mark, this doesn't make sense - a .png icon is 5K. Let's say that you
have browsed 1000 apps - that's ~5M.
I very much doubt descriptions will increase this number to more than
6MB. Let's put 4 MB of comments (ahem).
Now, compare 10MB to 21 - my guess is Market does a little bit more
than expected.

Cheers

[HaleRazor]

>
> The Market has nothing to do with what apps you have installed or
> uninstalled, AFAIK, so uninstalling an app will not affect the Market size.
>

The Market may not have anything to do with what apps are installed,
but the Market application on the device does. I agree with Stoyan
about that app keeping track of more than expected.

[Dianne Hackborn]

On Mon, Jan 12, 2009 at 1:00 AM, Stoyan Damov <stoyan...@gmail.com> wrote:

Actually Dianne I am *stunned* how few bugs I've found during my 1

month usage of Android.
Hail to Google engineers (I'm *not* joking)!

Wow, that's nice to hear. :)
 

This one, however, is a showstopper for me in the verbatim sense.

I understand.
 

> Just to clarify, none of the installed applications are shown to be using a
> large amount of data?
Yes, none.

Another place to look -- there is a filesystem bug that can sometimes happen where unlinked files are not recovered.  Here is the comment from an engineer who knows more about it:

"They can easily tell by looking at the number of unlinked files for the user partition in /proc/yaffs. If that number is very large, then they can reboot the device, wait a few second after they see the android, then pull the battery again. That should make the number of unlinked files drop back down. If that number isn't very large, then it is probably something else."

Unfortunately it looks like only root cat read /proc/yaffs (though that seems a little overly restrictive).  However you can try the pulling the battery trick and see if that helps.
 

Well, the over-the-air patch @#$%ed root access so I can't look anywhere.

The /data/local directory is owned by the shell user, so you don't need root for that -- just "cd /data/local" and look at what is there.  There is a chance that some temp .apk files have been left there from "adb install", or some other files created by other shell sessions.

[Stoyan Damov]

On Mon, Jan 12, 2009 at 10:55 PM, Dianne Hackborn <hac...@android.com> wrote:
>
> Another place to look -- there is a filesystem bug that can sometimes happen
> where unlinked files are not recovered. Here is the comment from an
> engineer who knows more about it:
>
> "They can easily tell by looking at the number of unlinked files for the
> user partition in /proc/yaffs. If that number is very large, then they can
> reboot the device, wait a few second after they see the android, then pull
> the battery again. That should make the number of unlinked files drop back
> down. If that number isn't very large, then it is probably something else."

I read about this on the net -- I thought it was some sort of a dark
joke or something -- apparently not :)
I did pull the battery though - nothing (good) happened.

This developer you're talking about - can he elaborate on how the
"battery pull trick" actually works -- I'm genuinely interested.

>
> Unfortunately it looks like only root cat read /proc/yaffs (though that
> seems a little overly restrictive). However you can try the pulling the
> battery trick and see if that helps.
>
>>
>> Well, the over-the-air patch @#$%ed root access so I can't look anywhere.
>
> The /data/local directory is owned by the shell user, so you don't need root
> for that -- just "cd /data/local" and look at what is there. There is a
> chance that some temp .apk files have been left there from "adb install", or
> some other files created by other shell sessions.

I already reset the phone but if I encounter this again I'll check
there (+ I'll have root this time :)

Thanks,
Stoyan

[Jean-Baptiste Queru]

Second hand information about the battery trick: yaffs2 has some
sanity-checking code that can detect and recover from unlinked files,
but that code is only run when the filesystem wasn't cleanly
unmounted.

JBQ

--
Jean-Baptiste M. "JBQ" Queru
Android Engineer, Google.

[Stoyan Damov]

Thanks.

A little bit more info about that bug - I *am* experiencing it again.
It *is* related to re-installs of one and the same application over
and over again.
I reinstalled my app maybe ~20 times today and slowly my ~70 went to 63 MB.
My app is 2MB so I have to have ~68MB but I don't. I noticed the
browser took 2MB and deleted them.
However, the free memory increased by ONE MB.

WTF is going on here?

Hurray! :) I did the battery pull and apparently I've hit the right
moment to pull the battery.
My memory increased from 64 to 69 MB. Now, how the device pulled that
off is a mystery to me because my app is 2MB (perhaps the ~70 MB are
close to ~71) but what the heck, the good thing is that the bug is
indeed *this* one, and not another which I'm the only one
experiencing!

Problem "solved", THANKS to everybody!

Cheers

[Jean-Baptiste Queru]

There is a bug somewhere (it's assigned to me for investigation) where
the system process keeps apk files open after they get unlinked in
some scenario close to what you mention (install, launch, uninstall),
which can then trigger the yaffs2 leak bug.

JBQ

[Stoyan Damov]

You might want to check whether this is related to having a shortcut
of the app on the home screen. I have a hunch it is.

Cheers

[Jean-Baptiste Queru]

When I tried to reproduce it a few months ago I think that I was able
to reproduce it without such a shortcut, but I might be wrong.

JBQ

[Joel Knighton]

Okay, someone who can replicate this problem, can you perform a "cat /proc/yaffs" and then post the output here.  Curious to see the YAFFS debugging info.

--
Joel Knighton

[Joel Knighton]

In fact, if you have root access, a "$su #echo all > /proc/yaffs #cat /proc/yaffs" would be optimal.  This should give a fair amount of debugging info for system, userdata, and cache.  If you post that up here, I should be able to give it a shot.

--
Joel Knighton

[Stoyan Damov]

I guess by "you" you don't mean me - I don't have root access. BTW, I
just found another bug, which is very weird but I'll send it in a new
post.

[Joel Knighton]

Just anybody.  If you'd like, you could still do the first one for me and help out a bit.  

--
Joel Knighton

[Disconnect]

Why not? Its available to anyone with a g1 these days (if you are willing to wipe out your data, which it sounds like you have to do occasionally anyway)

http://tinyurl.com/g1rc30

[Stoyan Damov]

Find it below, I'll do it again when the memory drops so you can compare:

YAFFS built:Oct 6 2008 14:13:20
$Id$
$Id$

Device 0 "system"
startBlock......... 0
endBlock........... 539
totalBytesPerChunk. 2048
nDataBytesPerChunk. 2048
chunkGroupBits..... 0
chunkGroupSize..... 1
nErasedBlocks...... 9
nReservedBlocks.... 5
blocksInCheckpoint. 1
nTnodesCreated..... 2700
nFreeTnodes........ 55
nObjectsCreated.... 500
nFreeObjects....... 2
nFreeChunks........ 2558
nPageWrites........ 0
nPageReads......... 36630
nBlockErasures..... 0
nGCCopies.......... 0
garbageCollections. 0
passiveGCs......... 0
nRetriedWrites..... 0
nShortOpCaches..... 10
nRetireBlocks...... 0
eccFixed........... 0
eccUnfixed......... 0
tagsEccFixed....... 0
tagsEccUnfixed..... 0
cacheHits.......... 0
nDeletedFiles...... 42
nUnlinkedFiles..... 342
nBackgroudDeletions 0
useNANDECC......... 1
isYaffs2........... 1
inbandTags......... 0

Device 1 "userdata"
startBlock......... 0
endBlock........... 597
totalBytesPerChunk. 2048
nDataBytesPerChunk. 2048
chunkGroupBits..... 0
chunkGroupSize..... 1
nErasedBlocks...... 12
nReservedBlocks.... 5
blocksInCheckpoint. 0
nTnodesCreated..... 2500
nFreeTnodes........ 44
nObjectsCreated.... 2400
nFreeObjects....... 22
nFreeChunks........ 32854
nPageWrites........ 543
nPageReads......... 3394
nBlockErasures..... 1
nGCCopies.......... 0
garbageCollections. 1
passiveGCs......... 1
nRetriedWrites..... 0
nShortOpCaches..... 10
nRetireBlocks...... 0
eccFixed........... 0
eccUnfixed......... 0
tagsEccFixed....... 0
tagsEccUnfixed..... 0
cacheHits.......... 265
nDeletedFiles...... 2166
nUnlinkedFiles..... 5152
nBackgroudDeletions 0
useNANDECC......... 1
isYaffs2........... 1
inbandTags......... 0

Device 2 "cache"
startBlock......... 0
endBlock........... 539
totalBytesPerChunk. 2048
nDataBytesPerChunk. 2048
chunkGroupBits..... 0
chunkGroupSize..... 1
nErasedBlocks...... 536
nReservedBlocks.... 5
blocksInCheckpoint. 0
nTnodesCreated..... 100
nFreeTnodes........ 81
nObjectsCreated.... 100
nFreeObjects....... 84
nFreeChunks........ 34414
nPageWrites........ 6
nPageReads......... 24
nBlockErasures..... 0
nGCCopies.......... 0
garbageCollections. 0
passiveGCs......... 0
nRetriedWrites..... 0
nShortOpCaches..... 10
nRetireBlocks...... 0
eccFixed........... 0
eccUnfixed......... 0
tagsEccFixed....... 0
tagsEccUnfixed..... 0
cacheHits.......... 0
nDeletedFiles...... 3
nUnlinkedFiles..... 6
nBackgroudDeletions 0
useNANDECC......... 1
isYaffs2........... 1
inbandTags......... 0

[Stoyan Damov]

On Tue, Jan 13, 2009 at 12:05 AM, Disconnect <dc.dis...@gmail.com> wrote:
> Why not? Its available to anyone with a g1 these days (if you are willing to
> wipe out your data, which it sounds like you have to do occasionally anyway)
>
> http://tinyurl.com/g1rc30
>
> On Mon, Jan 12, 2009 at 5:00 PM, Stoyan Damov <stoyan...@gmail.com>
> wrote:
>>
>> I guess by "you" you don't mean me - I don't have root access. BTW, I
>> just found another bug, which is very weird but I'll send it in a new
>> post.

Until today I didn't have the need to have root access :)
Even now I don't *need* to have one, but *will* downgrade to RC29 when
I have some spare time.

Cheers,
Stoyan

[Joel Knighton]

Beautiful.  It does appear to be nUnlinkedFiles in userdata causing the problem after continual uninstall/reinstall (my phone is at 30k after infrequent installations, I notice yours is at 5k after a fresh install/battery trick).  Working on a shell script that could potentially remedy this problem, but this is obviously not an optimal solution.

--
Joel Knighton

[Greg White]

On Mon, Jan 12, 2009 at 4:54 PM, Joel Knighton <joel.k...@gmail.com> wrote:

Okay, someone who can replicate this problem, can you perform a "cat /proc/yaffs" and then post the output here.  Curious to see the YAFFS debugging info.

I don't have the output handy, but I went from ~12K unlinked files to ~2K.  This makes a difference.

[Justin (Google Employee)]

Here is some more information on what's happening. There is a problem
in yaffs2 where unlinked files are not removed from a check-pointed
filesystem. Whenever Android sleeps, it check-points the filesystem.
When you uninstall an app the files are unlinked. If the device then
goes to sleep before the unlinked files are actually removed, they
won't be.

The proper way to work around this it to shut down your phone by
holding down the end call button until you get a dialog asking to shut
down your phone. Turn your phone back on. BEFORE your phone can sleep
after it reboot, shut it down again. This will clean your unlinked
files.

This works because when the system starts it has no check-points in
the filesystem and one won't be created if the system doesn't sleep.
The unlinked files are actually removed then when the file system is
mounted or unmounted (I'm not sure which it actually is).

Cheers,
Justin
Android Team @ Google

On Jan 12, 1:56 pm, "Greg White" <debauchedsl...@gmail.com> wrote:

[Stoyan Damov]

On Tue, Jan 13, 2009 at 1:46 AM, Justin (Google Employee)
<j...@google.com> wrote:
>
> Here is some more information on what's happening. There is a problem
> in yaffs2 where unlinked files are not removed from a check-pointed
> filesystem. Whenever Android sleeps, it check-points the filesystem.
> When you uninstall an app the files are unlinked. If the device then
> goes to sleep before the unlinked files are actually removed, they
> won't be.
>
> The proper way to work around this it to shut down your phone by
> holding down the end call button until you get a dialog asking to shut
> down your phone. Turn your phone back on. BEFORE your phone can sleep
> after it reboot, shut it down again. This will clean your unlinked
> files.

Come again, I'm not sure understand that part.
Are these the steps for the workaround?
1. Shutdown phone
2. Turn on phone
3. Unlock SIM card, wait for home screen to appear (do I need that?)
4. Shutdown phone
5. Turn on phone, and files will be deleted

Thanks,
Stoyan

[Justin (Google Employee)]

Yes, your understanding is correct. Yes, you will need to unlock the
screen and wait for the home scree to appear before shutting the phone
down the second time. Shutting the phone down is better than pulling
the battery.

Cheers,
Justin
Android Team @ Google

[Stoyan Damov]

On Tue, Jan 13, 2009 at 2:25 AM, Justin (Google Employee)
<j...@google.com> wrote:
>
> Yes, your understanding is correct. Yes, you will need to unlock the
> screen and wait for the home scree to appear before shutting the phone
> down the second time. Shutting the phone down is better than pulling
> the battery.
>

Justin, God bless you!
Now go and find that dev who introduced the bug and make something
painful to him :)

Can't thank you enough - beats the battery pull! :)

Cheers,
Stoyan