/sdcard vs. /sdcard-ext

[Zamslam]

The droid razr and a few others have /mnt/sdcard map to a finite and
small internal storage area, while the real, expandable, swappable sd
card is at /mnt/sdcard-ext. Some apps map end user ui elements that
specify an apps storage with this in mind, but alas not yet with my
most often used app - NewsRob.

Can you allow local caches to be stored there so I can go back to
storing more than just the basic articles?

[Mariano Kamp]

Is what you describe somewhat official? Can you point me to documentation that states that those mount points are standard?

I think what you ask is possible today with a debug setting. Would you mind having a look through the archives? I don't know if there is a "canonical" introduction, but the info is there per se.

http://groups.google.com/group/newsrob/search?hl=en_US&group=newsrob&q=databasePath&qt_g=Search+this+group

--
You received this message because you are subscribed to the Google Groups "NewsRob User Group" group.
To post to this group, send email to new...@googlegroups.com.
To unsubscribe from this group, send email to newsrob+u...@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/newsrob?hl=en.

[Tim Wood]

This is how my Verizon Droid RAZR Maxx is configured as well. I'm on Motorola's stock ROM. But I've yet to run into any space issues.

[Matthew S.]

I do not think there is a Android documented standard mapping between internal and external storage mount points.

Samsung Galaxy Note (and S I believe) for Android 2.3.x mounts internal at /mnt/sdcard & external at /mnt/sdcard/external_sd

TLDR;

I also have a AC Ryan Veolo (Android 2.2 based media player) with USB interfaces and internal memory. It mounts the different storage areas differently also. I'm not at home currently to confirm the mount points, I think there are multiple mount points. but /mnt/nand, /mnt/sdcard, /mnt/usb/??? come into play. The external (USB) devices can be FAT32 or EXT4 (haven't tried exFAT).

It would make a lot of sense for Google to standardize mount points, attributes, and preferences (sda, sdb, sdc for Storage Device X, along with attributes such as fixed (phone internal), removable (SD CARD) or semi-volatile (USB) or volatile (e.g. comms link associated such as cloud or network), plus a way to define 'default' devices for specific scenarios, e.g. app data, app cache data etc that the user can manage.

I liked the way AmigaDOS use to manage storage device mappings with things like HDA:, HDB: for hard drive, and FDA:, FDB: etc for floppy drives, maybe something similar could be standardized for OS managed mount points, or API calls to ID and map storage.

[Mariano Kamp]

That's much easier on iOS. They only have one linear storage area ;)

I also think that Google should add some virtualization layer that is standard and at best would also include methods to move content from one storage area to the other.

In absence of that I fear that I need to maintain a list where I configure for each new device what the mount points are. Not a good solution. 

When I rework that part, I'll have another look. There is work to do in that area anyway.

--

You received this message because you are subscribed to the Google Groups "NewsRob User Group" group.

To view this discussion on the web visit https://groups.google.com/d/msg/newsrob/-/62A566xQxm4J.

[Matthew S.]

Maybe an Advanced or Experimental option where the user can manually spec a path? Probably with URL to a discussion where people can share device specific path info :-) Oh, yeah, and a big fat warning stating the setting is experimental/advanced and could cause data-loss or or damage to the phone (just to scare people, cos a bad path could cause grief)

This is a true case of fragmentation that Google could eliminate - I'm all for differentiation, but a developer having to codify something like this on a per device/OS level is not very supportive, especially since the flaw (in UX terms) is obvious and inherent in the system features/design.

I don't think you can set mount points on FAT32 partitions, otherwise a rooted device you could manually move the NewRob directory, and set a mount point to it at the original location - would have to manual add the mount command to rc.local.

To unsubscribe from this group, send email to newsrob+unsubscribe@googlegroups.com.

[Mariano Kamp]

On Tue, Apr 17, 2012 at 10:07 AM, Matthew S. <mas...@gmail.com> wrote:

Maybe an Advanced or Experimental option where the user can manually spec a path? Probably with URL to a discussion where people can share device specific path info :-) Oh, yeah, and a big fat warning stating the setting is experimental/advanced and could cause data-loss or or damage to the phone (just to scare people, cos a bad path could cause grief)

Well, I think for those advanced users they could use the debug setting via newsrob.debug.

This is a true case of fragmentation that Google could eliminate - I'm all for differentiation, but a developer having to codify something like this on a per device/OS level is not very supportive, especially since the flaw (in UX terms) is obvious and inherent in the system features/design.

Amen.

[Skylar]

If you ask me, the root problem is that NewsRob is going after the
storage directory directly and has to know the device specific
implementations. That's the whole reason Android added
Context.getExternalFilesDir(), it fetches a directory specific to the
app on the external storage device.

http://developer.android.com/reference/android/content/Context.html#getExternalFilesDir(java.lang.String)

Adding experimental settings, etc. etc. sounds like over engineering a
problem that already has a solution.

On Apr 17, 4:12 am, Mariano Kamp <mariano.k...@gmail.com> wrote:

[Mariano Kamp]

Well, it's a long standing issue on my TODO list to use this API call (It wasn't there when I implemented the storage provider initially). It would be good for other reasons too. 

But are you speculating that this will always resolve to the correct mount point or are you sure that this is the case?

--
You received this message because you are subscribed to the Google Groups "NewsRob User Group" group.

To post to this group, send email to new...@googlegroups.com.

To unsubscribe from this group, send email to newsrob+u...@googlegroups.com.

[Skylar]

I develop for Android as well, and I have yet to see
getExternalFilesDir() fail to resolve to a valid mount (unless the SD
card is not present). On my NookColor tab (running CM9) there is the
standard /data, an internal storage mount of around 1 GB, and the SD
card mount... this method resolves to the SD Card, not the extra
internal storage, which I would expect.

That said, I did a quick search for Razr and getExternalFilesDir and
found one thread where someone says it didn't work... but I'm not sure
if that's improper implementation, incorrect permissions, or an issue
with the -ext setup.

http://community.developer.motorola.com/t5/Android-App-Development-for/Help-with-the-Moto-RAZR-and-Video/td-p/23790

On Apr 27, 4:29 am, Mariano Kamp <mariano.k...@gmail.com> wrote:
> Well, it's a long standing issue on my TODO list to use this API call (It
> wasn't there when I implemented the storage provider initially). It would
> be good for other reasons too.
>
> But are you speculating that this will always resolve to the correct mount
> point or are you sure that this is the case?
>
>
>
>
>
>
>
> On Fri, Apr 27, 2012 at 7:20 AM, Skylar <skylar.sut...@gmail.com> wrote:
> > If you ask me, the root problem is that NewsRob is going after the
> > storage directory directly and has to know the device specific
> > implementations. That's the whole reason Android added
> > Context.getExternalFilesDir(), it fetches a directory specific to the
> > app on the external storage device.
>

> >http://developer.android.com/reference/android/content/Context.html#g...)

[Mariano Kamp]

Quick correction. I thought, for whatever reason, that I directly access /mnt/sdcard. But I was wrong, I never have. 

That's what I do for the location resolution.

...

public class SdCardStorageAdapter extends AbstractStorageAdapter {

    private static final String TAG = SdCardStorageAdapter.class.getName();

    private static final String BASE = Environment.getExternalStorageDirectory().getPath() + "/newsrob";

    private static final File BASE_DIR = new File(BASE);

    private boolean readOnly;

    private boolean mounted;

    private boolean shared;

    BroadcastReceiver myBroadcastReceiver;

...

So I should already get the right path, but when using the Android 2.2. API I would get the "data/newsrob" prefix and the cleanup automatically.

Anyway. The documentation of the newer method is vague. Not quite sure what to make of it. In particular I am wondering about devices like the XOOM who, to the best of my knowledge, have external storage that is the built-in sd card, but also external external storage, which is the removable SD card.

[Skylar]

See my previous post. My NookColor has a similar storage setup as the
XOOM and it resolves to the removable storage.

On Apr 28, 10:10 am, Mariano Kamp <mariano.k...@gmail.com> wrote:
> Quick correction. I thought, for whatever reason, that I directly access
> /mnt/sdcard. But I was wrong, I never have.
>
> That's what I do for the location resolution.
>
> ...
>
> public class SdCardStorageAdapter extends AbstractStorageAdapter {
>
>     private static final String TAG = SdCardStorageAdapter.class.getName();
>

>    * private static final String BASE =
> Environment.getExternalStorageDirectory().getPath() + "/newsrob";*
>
> *    private static final File BASE_DIR = new File(BASE);*

>
>     private boolean readOnly;
>
>     private boolean mounted;
>
>     private boolean shared;
>
>     BroadcastReceiver myBroadcastReceiver;
> ...
>
> So I should already get the right path, but when using the Android 2.2. API
> I would get the "data/newsrob" prefix and the cleanup automatically.
>
> Anyway. The documentation of the newer method is vague. Not quite sure what
> to make of it. In particular I am wondering about devices like the XOOM
> who, to the best of my knowledge, have external storage that is the
> built-in sd card, but also external external storage, which is the
> removable SD card.
>
>
>
>
>
>
>
> On Fri, Apr 27, 2012 at 2:13 PM, Skylar <skylar.sut...@gmail.com> wrote:
> > I develop for Android as well, and I have yet to see
> > getExternalFilesDir() fail to resolve to a valid mount (unless the SD
> > card is not present). On my NookColor tab (running CM9) there is the
> > standard /data, an internal storage mount of around 1 GB, and the SD
> > card mount... this method resolves to the SD Card, not the extra
> > internal storage, which I would expect.
>
> > That said, I did a quick search for Razr and getExternalFilesDir and
> > found one thread where someone says it didn't work... but I'm not sure
> > if that's improper implementation, incorrect permissions, or an issue
> > with the -ext setup.
>

> >http://community.developer.motorola.com/t5/Android-App-Development-fo...

[Mariano Kamp]

Didn't happen that way for my XOOM. Maybe because there wasn't an external sd card at the beginning.

Anyway, could you please remove the "newsrob" directory on your sd card, uninstall the app and then try this version?

https://sites.google.com/site/newsrobapp/beta-program/newsrob489-1.apk

[Mariano Kamp]

And indeed, also the Android 2.2+ approach resolves on the XOOM to /mnt/sdcard, not /mnt/external1, whereby the later is the removable SD card and the former the internal memory. 

So it doesn't seem to be in line with your rooted NookColor.

Not an ideal solution ;(

If nothing else breaks I'll keep the Android 2.2+ approach in though.

For custom ROM users it may still make more sense to directly create an empty "newsrob" directory on the "official" external storage, because that directory will then be used by NewsRob and can be used again after you re-install NewsRob after installing a new ROM. NewsRob will then pickup your per-feed settings from the old directory. Clearing the cache would still make sense in that case.

[Skylar]

I haven't had a chance to check out the beta build - I'll try to do
that tonight.

getExternalFilesDir() has a subtle benefit though: it actually
resolves to /mnt/sdcard/Android/data/com.yourappname.whatever

Right now, newsrob (and many other apps) are barfing files all over
the root directory of my SD Card... making it very hard to keep a
clean house. The Android Dev Guide recommends all app files and cache
go under the /Android/data/ folder to a) keep things clean, b) let the
OS manage them. If you uninstall the app, the files go away. If you
install a new ROM, the files stay and will get picked up when you
reinstall the app.

[alexander...@googlemail.com]

I'd second this request; I've got a Samsung GT-I9100 and it has a /sdcard (the 11.5GB left over from the 16GB eMMC internal flash, after 4.5GB has been used for /system, /data, /cache etc) and a /sdcard/external_sd (my micro SDHC card). Unfortunately, I regularly experience corruption of the VFAT filesystem on /sdcard, and whenever I do, it's always files within /sdcard/NewsRob (probably merely because NewsRob is the most prolific writer to that filesystem). I'd like to work out whether I've got a hardware fault or a hardware bug, and being able to move the NewsRob article cache to /sdcard/external_sd/NewsRob would be a good first step in doing so.

It also strikes me, given that NewsRob is a write-heavy application, that if it wears out some flash storage, it's better that it's the removable, replaceable micro SD card!

I've tried the debug setting that Mariano mentions, but all it moved was the newsrob.db :-(

Best Regards,
Alex

[Mariano Kamp]

Alex, 

I use NewsRob since the end of 2008 and besides a single card in my G1 I have not had an issue with an SD card at all. I think that I used it extensively on more than a dozen devices though.

But of course, yes, it does write a lot to the sd card.

Anyway I could add another debug option, given the time (read after the tablet version), but have you tried the current beta? When you start without having a /sdcard/newsrob in place it should store it's own directory in the place the OS assigns to it, if I remember correctly - it's a little time ago that I thought about it. Could you please try that, if you haven't. 

Managing the use of internal memory and external memories is the job of the OS. Unfortunately the OS does a pretty bad job at that and let's the fragmentation of the platform surface right to the developer and its users. I am not aware of a proper way to deal with that, other than maintaining a list of devices and what sd card themes they support. And even if I would do that, this is a lot of work, dealing with sd cards, that are not there anymore and moving stuff around when the user changes his mind where to store stuff, and making sure along the way that this operation is as atomic as possible, even when the OS kills the process in between ... and on and on and on and on. 

FWIW, from my understanding it is a good thing to not fill sd cards to the brim, because the way they operate they become very inefficient in terms of write speed and creating more wear when there is not much space left and there needs to be a consolidation before writing new blocks.

--

You received this message because you are subscribed to the Google Groups "NewsRob User Group" group.

To view this discussion on the web visit https://groups.google.com/d/msg/newsrob/-/BlauWzi9OegJ.

[Mark Otway]

Second that. I've used Newsrob for years on a Magic, then over 2 years on an N1, and now on my S3 and never had an issue, so wouldn't worry about it if I were you.

[Alex B]

Mariano -

I began using NewsRob on my Hero at the beginning of 2010, and I don't remember having problems there (maybe once, possibly). Of course, that was a single core, lower performance handset, from a different manufacturer, and with a real PNY micro SD card providing /sdcard.

On the Samsung Galaxy S 2, the mounts look like this:

$ mount                                                        

rootfs / rootfs ro,relatime 0 0

tmpfs /dev tmpfs rw,nosuid,relatime,mode=755 0 0

devpts /dev/pts devpts rw,relatime,mode=600 0 0

proc /proc proc rw,relatime 0 0

sysfs /sys sysfs rw,relatime 0 0

none /acct cgroup rw,relatime,cpuacct 0 0

tmpfs /mnt/asec tmpfs rw,relatime,mode=755,gid=1000 0 0

tmpfs /mnt/obb tmpfs rw,relatime,mode=755,gid=1000 0 0

none /dev/cpuctl cgroup rw,relatime,cpu 0 0

/dev/block/mmcblk0p9 /system ext4 ro,noatime,barrier=1,data=ordered 0 0

/dev/block/mmcblk0p7 /cache ext4 rw,nosuid,nodev,noatime,barrier=1,data=ordered 0 0

/dev/block/mmcblk0p1 /efs ext4 rw,nosuid,nodev,noatime,barrier=1,data=ordered 0 0

/dev/block/mmcblk0p10 /data ext4 rw,nosuid,nodev,noatime,barrier=1,data=ordered,noauto_da_alloc 0 0

/dev/block/mmcblk0p4 /mnt/.lfs j4fs rw,relatime 0 0

/sys/kernel/debug /sys/kernel/debug debugfs rw,relatime 0 0

/dev/block/vold/259:3 /mnt/sdcard vfat ro,dirsync,nosuid,nodev,noexec,noatime,nodiratime,uid=1000,gid=1015,fmask=0002,dmask=0002,allow_utime=0020,codepage=cp437,iocharset=iso8859-1,shortname=mixed,utf8,errors=remount-ro 0 0

tmpfs /mnt/sdcard/external_sd tmpfs rw,dirsync,nosuid,nodev,noexec,noatime,nodiratime,size=0k,mode=755,gid=1000 0 0

tmpfs /mnt/sdcard/usbStorage tmpfs rw,dirsync,nosuid,nodev,noexec,noatime,nodiratime,size=0k,mode=755,gid=1000 0 0

/dev/block/vold/179:25 /mnt/sdcard/external_sd vfat rw,dirsync,nosuid,nodev,noexec,noatime,nodiratime,uid=1000,gid=1023,fmask=0002,dmask=0002,allow_utime=0020,codepage=cp437,iocharset=iso8859-1,shortname=mixed,utf8,errors=remount-ro 0 0

/dev/block/vold/179:25 /mnt/secure/asec vfat rw,dirsync,nosuid,nodev,noexec,noatime,nodiratime,uid=1000,gid=1023,fmask=0002,dmask=0002,allow_utime=0020,codepage=cp437,iocharset=iso8859-1,shortname=mixed,utf8,errors=remount-ro 0 0

tmpfs /mnt/sdcard/external_sd/.android_secure tmpfs ro,relatime,size=0k,mode=000 0 0

/dev/block/vold/259:3 is also known as /dev/block/mmcblk0p11, i.e. a partition of the SGS2's onboard eMMC (which, incidentally is known to have at least one bug in a number of Samsung handsets, including the example I own - see http://forum.xda-developers.com/showthread.php?t=1693704 ). Alternatively, it could be a bug in the Samsung's kernel's MMC or FAT code. Here's an example of how the NewsRob article directory looks after there's been corruption:

$ tar -zcvf /wxpd/Samsung/Internal-sdcard-20120802.tar.gz . ... ./newsrob/a7ab40c797b58d8ff/a7ab40c797b58d8ff_16nr ./newsrob/a7ab40c797b58d8ff/a7ab40c797b58d8ff_11nr ./newsrob/a7ab40c797b58d8ff/a7ab40c797b58d8ff_10nr ./newsrob/a7ab40c797b58d8ff/a7ab40c797b58d8ff_7nr ./newsrob/a7ab40c797b58d8ff/preview.pngnr ./newsrob/a4649bf1dcc10abf2/ ./newsrob/a4649bf1dcc10abf2/a4649bf1dcc10abf2_s.htmlnr tar: ./newsrob/a4649bf1dcc10abf2/ .inner-.tex: Cannot stat: Input/output error tar: ./newsrob/a4649bf1dcc10abf2/ter\:alph.a(o: Cannot stat: Input/output error tar: ./newsrob/a4649bf1dcc10abf2/-ccob .s.lan: Cannot stat: Input/output error ./newsrob/a4649bf1dcc10abf2/d-positi.on: tar: ./newsrob/a4649bf1dcc10abf2/d-positi.on\:: Read error at byte 0, while reading 3584 bytes: Input/output error ./newsrob/a4649bf1dcc10abf2/ine .lin.k-c tar: ./newsrob/a4649bf1dcc10abf2/ine .lin.k-c: Read error at byte 0, while reading 8704 bytes: Input/output error tar: ./newsrob/a4649bf1dcc10abf2/backgrou.nd-: Cannot stat: Input/output error tar: ./newsrob/a4649bf1dcc10abf2/reviews-.fea: Cannot stat: Input/output error tar: ./newsrob/a4649bf1dcc10abf2/rm\: uppe.rca: Cannot stat: Input/output error tar: ./newsrob/a4649bf1dcc10abf2/ font-we.igh: Cannot stat: Input/output error tar: ./newsrob/a4649bf1dcc10abf2/ 5px 2px.;}.: File removed before we read it tar: ./newsrob/a4649bf1dcc10abf2/ws-featu.re-: Cannot stat: Input/output error ./newsrob/a4649bf1dcc10abf2/decorati.on: tar: ./newsrob/a4649bf1dcc10abf2/decorati.on\:: Read error at byte 0, while reading 9728 bytes: Input/output error ./newsrob/a4649bf1dcc10abf2/feature-.sin tar: ./newsrob/a4649bf1dcc10abf2/feature-.sin: Read error at byte 0, while reading 9216 bytes: Input/output error tar: ./newsrob/a4649bf1dcc10abf2/; width\:. 10: Cannot stat: Input/output error ./newsrob/a4649bf1dcc10abf2/single l.i{ tar: ./newsrob/a4649bf1dcc10abf2/single l.i{: Read error at byte 0, while reading 2048 bytes: Input/output error ...

Sometimes I need to manually fsck the filesystem one or more times in a single day. I've also seen it go from fscking as clean in the morning to corrupt and remounted read-only by the afternoon without any intervening crash, reset, power cycle, or battery failure. Every block of the eMMC reads fine with dd though, so I don't think it's wearing out, at least.

As I said originally, I'm sure this is not a NewsRob bug (frankly, I think the implementation of the GS2 suffers from awful firmware QA and some nasty hardware cost engineering), but the amount of writing NewsRob does is obviously provoking a problem elsewhere, and I'd like to be able to figure out where, and maybe work around it.

Best Regards,

Alex

On Saturday, 4 August 2012 17:56:26 UTC+1, Mariano Kamp wrote:

Alex, 

I use NewsRob since the end of 2008 and besides a single card in my G1 I have not had an issue with an SD card at all. I think that I used it extensively on more than a dozen devices though.

But of course, yes, it does write a lot to the sd card.

Anyway I could add another debug option, given the time (read after the tablet version), but have you tried the current beta? When you start without having a /sdcard/newsrob in place it should store it's own directory in the place the OS assigns to it, if I remember correctly - it's a little time ago that I thought about it. Could you please try that, if you haven't. 

Managing the use of internal memory and external memories is the job of the OS. Unfortunately the OS does a pretty bad job at that and let's the fragmentation of the platform surface right to the developer and its users. I am not aware of a proper way to deal with that, other than maintaining a list of devices and what sd card themes they support. And even if I would do that, this is a lot of work, dealing with sd cards, that are not there anymore and moving stuff around when the user changes his mind where to store stuff, and making sure along the way that this operation is as atomic as possible, even when the OS kills the process in between ... and on and on and on and on. 

FWIW, from my understanding it is a good thing to not fill sd cards to the brim, because the way they operate they become very inefficient in terms of write speed and creating more wear when there is not much space left and there needs to be a consolidation before writing new blocks.

On Wed, Aug 1, 2012 at 3:41 PM, Alex B wrote:

On Saturday, 14 April 2012 00:57:15 UTC+1, Zamslam wrote:

The droid razr and a few others have /mnt/sdcard map to a finite and
small internal storage area, while the real, expandable, swappable sd
card is at /mnt/sdcard-ext.  Some apps map end user ui elements that
specify an apps storage with  this in mind, but alas not yet with my
most often used app - NewsRob.

Can you allow local caches to be stored there so I can go back to
storing more than just the basic articles?

I'd second this request; I've got a Samsung GT-I9100 and it has a /sdcard (the 11.5GB left over from the 16GB eMMC internal flash, after 4.5GB has been used for /system, /data, /cache etc) and a /sdcard/external_sd (my micro SDHC card). Unfortunately, I regularly experience corruption of the VFAT filesystem on /sdcard, and whenever I do, it's always files within /sdcard/NewsRob (probably merely because NewsRob is the most prolific writer to that filesystem). I'd like to work out whether I've got a hardware fault or a hardware bug, and being able to move the NewsRob article cache to /sdcard/external_sd/NewsRob would be a good first step in doing so.

It also strikes me, given that NewsRob is a write-heavy application, that if it wears out some flash storage, it's better that it's the removable, replaceable micro SD card!

I've tried the debug setting that Mariano mentions, but all it moved was the newsrob.db :-(

Best Regards,
Alex

--
You received this message because you are subscribed to the Google Groups "NewsRob User Group" group.

To view this discussion on the web visit https://groups.google.com/d/msg/newsrob/-/BlauWzi9OegJ.

To post to this group, send email to new...@googlegroups.com.

To unsubscribe from this group, send email to newsrob+unsubscribe@googlegroups.com.

[Alex B]

On Saturday, 4 August 2012 17:56:26 UTC+1, Mariano Kamp wrote:

Alex,

[H]ave you tried the current beta? When you start without having a /sdcard/newsrob in place it should store it's own directory in the place the OS assigns to it, if I remember correctly - it's a little time ago that I thought about it. Could you please try that, if you haven't.

OK, I've just tried uninstalling and reinstalling the latest release with no pre-existing /sdcard/newsrob directory; on the first sync, it creates /sdcard/newsrob.

Uninstalling the latest release, deleting /sdcard/newsrob and installing the latest beta ( https://sites.google.com/site/newsrobapp/beta-program/newsrob489-1.apk?attredirects=0&d=1 ) results in it creating /sdcard/Android/data/com.newsrob (just like a bunch of other apps, including Opera, Words with Friends, VideoLan VLC, Google Earth, YouTube, Google Docs, Google Books, Facebook and so on). That suggests that the latest beta of NewsRob is indeed doing things "the right way", but Samsung's implementation of getExternalFilesDir is perhaps rather stupid.

Perhaps bind-mounting /sdcard/external_sd/newsrob as /sdcard/newsrob...

Thanks, and best regards,
Alex

[Mariano Kamp]

Thanks for the verification Alex. I will see if I can add a debug setting for the location without too much work.

--

You received this message because you are subscribed to the Google Groups "NewsRob User Group" group.

To view this discussion on the web visit https://groups.google.com/d/msg/newsrob/-/cW7F0BPIp1EJ.

[Alex B]

Hi Mariano -

Whilst you're there, it might be worth adding some options that might help with the filesystem corruption problem. Someone on #android has also seen the same problem on two different, unrooted, Samsung devices. Like me, he attributes it to the kernel (mmc driver, FAT filesystem, etc) or Samsung's hardware. I'd suggest some kind of user-configurable write rate throttling (e.g. wait for Xms after closing a file) and regular fsync() calls (which I realise will also have the side effect of slowing things down as well). FYI, gReader doesn't seem to provoke the same problem, even though I've configured it to store its article cache in the same place as NewsRob.

Is your article cache writing code multi-threaded? I wonder if the problem only manifests on multi-core devices, like the SGS 2.

As a fellow programmer, I have every sympathy with you; application programmers should not have to clutter their code with workarounds like those I suggest. :-(

Best Regards,
Alex

[Mariano Kamp]

And I am not doing any fsyncs or other low level interactions with the fs. I only use plain Java. 

Yes, downloading web pages happens in parallel. I could add a debug setting for that.

My experience is that other newsreaders are not easily comparable with NewsRob. In particular when it comes to offline.

To view this discussion on the web visit https://groups.google.com/d/msg/newsrob/-/VodLWI9icVMJ.

[Alex B]

Hi Mariano -

Yes, a debug option to throttle and/or make article downloading/writing single-threaded would be handy for those of us with suspected-buggy devices.

FYI, having created an app to do getExternalFilesDir() etc and run it on my Samsung GT-I9100 running ICS 4.0.3, I see your dilemma:

getExternalStorageDirectory() returns /mnt/sdcard
getDataDirectory() returns /data
getDownloadCacheDirectory() returns /cache
getRootDirectory() returns /system
getExternalStoragePublicDirectory(Environment.DIRECTORY_MOVIES) returns /mnt/sdcard/Movies
getExternalCacheDir() returns null (!)
getExternalFilesDir(null) returns null (!)
getExternalFilesDir(Environment.DIRECTORY_MOVIES) returns null (!)

Android (or Samsung, at least) ought to implement an API like Motorola's http://developer.motorola.com/docs/motorola-external-storage-api/

Best Regards,
Alex

[Alex B]

FYI, it seems I didn't wait long enough. I get the same problems with gReader, just less frequently, it seems.

On Thursday, 9 August 2012 16:03:03 UTC+1, Alex B wrote:

Someone on #android has also seen the same problem on two different, unrooted, Samsung devices. Like me, he attributes it to the kernel (mmc driver, FAT filesystem, etc) or Samsung's hardware. [...] FYI, gReader doesn't seem to provoke the same problem, even though I've configured it to store its article cache in the same place as NewsRob.

[Knowthenazz (Samsung Galaxy S2)]

I think the debug options, to prevent file corruption, discussed here
would be a great addition.

This file corruption has happened to me a few times, and most recently
when I was out of town for a week. I lost the use of both my internal
and external SD cards for most of my trip as I didn't have access to a
PC to repair the problem.

I'm running stock 2.3.6 on my Samsung Galaxy S2.

[Mariano Kamp]

NewsRob downloads articles in parallel and after parsing the html and css then the images for an article in parallel as well.

In newsrob.debug:

downloadArticlesInParallel=0 -> should disable downloading articles in parallel. Setting it to 1 will download 3 articles in parallel.

downloadImagesInParallel=0 -> should disable downloading images in parallel. When set to 1, up to 15 images are downloaded in parallel.

Those parallel downloaded from above will only be done when the screen is off. This test, however, is only done when initiating the downloads, i.e. a running download will not be interrupted when you turn on the screen.

This should work (disabling them by settings those values to 0), but I haven't tested it lately. If this doesn't work, let me know. Use on your own risk. Maybe just set the image download to 0. The article download includes parsing etc., which takes CPU and does not write to disk.

--
You received this message because you are subscribed to the Google Groups "NewsRob User Group" group.

[Alex Butcher]

Thanks, Mariano - have downloaded 489-4 and have set both to 0 and
will be monitoring the situation.

Cheers,
Alex

[Alex Butcher]

Further to my previous message, something I have noticed is that
tapping on the 'Update Running' notification seems to cause NewsRob to
go to a black screen. Leaving my device on that screen causes a
soft-reboot (i.e. manufacturer logo, but not startup sound).

[Mariano Kamp]

No idea why this happen and haven't seen it myself, but have heard of it. Maybe this is due to hitting a lock that somebody else holds? Really no idea, sorry.

[Alex Butcher]

Hi Mariano -

This might be relevant:

# logcat | grep -i "sqlite\|newsrob"
D/com.newsrob.storage.SdCardStorageAdapter( 4958): setupMedium
status=mounted mounted false->true, read-only false->false, shared
false -> false.
D/com.newsrob.EntriesRetriever( 4958): Accessing Google Reader service.
D/com.newsrob.EntriesRetriever( 4958): totalFetchedArticleCount=1000
D/com.newsrob.EntriesRetriever( 4958): totalSeenArticleCount=1000
D/com.newsrob.EntriesRetriever( 4958): Accessing Google Reader service.
D/com.newsrob.EntriesRetriever( 4958): totalFetchedArticleCount=1018
D/com.newsrob.EntriesRetriever( 4958): totalSeenArticleCount=1018
D/com.newsrob.EntriesRetriever( 4958): Accessing Google Reader service.
D/com.newsrob.EntriesRetriever( 4958): Accessing Google Reader service.
D/com.newsrob.EntriesRetriever( 4958): Accessing Google Reader service.
I/SqliteDatabaseCpp( 4958): sqlite returned: error code = 1, msg = no
such table: temp_ids_PINNED, db=xxx
W/SQLiteDatabase( 4958): database lock has not been available for 30
sec. Current Owner of the lock is 7112. Continuing to wait in thread:
7113
W/SQLiteDatabase( 4958): database lock has not been available for 30
sec. Current Owner of the lock is 7112. Continuing to wait in thread:
7113
V/AlarmManager( 2024): trigger WAKEUP Alarm{424521c0 type 0 com.newsrob}
W/SQLiteDatabase( 4958): database lock has not been available for 30
sec. Current Owner of the lock is 7112. Continuing to wait in thread:
7113
W/SQLiteDatabase( 4958): database lock has not been available for 30
sec. Current Owner of the lock is 7112. Continuing to wait in thread:
7117
W/SQLiteDatabase( 4958): database lock has not been available for 30
sec. Current Owner of the lock is 7112. Continuing to wait in thread:
7113
W/SQLiteDatabase( 4958): database lock has not been available for 30
sec. Current Owner of the lock is 7112. Continuing to wait in thread:
7117
W/SQLiteDatabase( 4958): database lock has not been available for 30
sec. Current Owner of the lock is 7112. Continuing to wait in thread:
7113
W/SQLiteDatabase( 4958): database lock has not been available for 30
sec. Current Owner of the lock is 7112. Continuing to wait in thread:
7117
W/SQLiteDatabase( 4958): database lock has not been available for 30
sec. Current Owner of the lock is 7112. Continuing to wait in thread:
7113
W/SQLiteDatabase( 4958): database lock has not been available for 30
sec. Current Owner of the lock is 7112. Continuing to wait in thread:
7117
W/SQLiteDatabase( 4958): database lock has not been available for 30
sec. Current Owner of the lock is 7112. Continuing to wait in thread:
7113
W/SQLiteDatabase( 4958): database lock has not been available for 30
sec. Current Owner of the lock is 7112. Continuing to wait in thread:
7117
W/SQLiteDatabase( 4958): database lock has not been available for 30
sec. Current Owner of the lock is 7112. Continuing to wait in thread:
7113
W/SQLiteDatabase( 4958): database lock has not been available for 30
sec. Current Owner of the lock is 7112. Continuing to wait in thread:
7117
W/SQLiteDatabase( 4958): database lock has not been available for 30
sec. Current Owner of the lock is 7112. Continuing to wait in thread:
7113
W/SQLiteDatabase( 4958): database lock has not been available for 30
sec. Current Owner of the lock is 7112. Continuing to wait in thread:
7117
W/SQLiteDatabase( 4958): database lock has not been available for 30
sec. Current Owner of the lock is 7112. Continuing to wait in thread:
7113
W/SQLiteDatabase( 4958): database lock has not been available for 30
sec. Current Owner of the lock is 7112. Continuing to wait in thread:
7117
W/SQLiteDatabase( 4958): database lock has not been available for 30
sec. Current Owner of the lock is 7112. Continuing to wait in thread:
7113
W/SQLiteDatabase( 4958): database lock has not been available for 30
sec. Current Owner of the lock is 7112. Continuing to wait in thread:
7117
W/SQLiteDatabase( 4958): database lock has not been available for 30
sec. Current Owner of the lock is 7112. Continuing to wait in thread:
7113
W/SQLiteDatabase( 4958): database lock has not been available for 30
sec. Current Owner of the lock is 7112. Continuing to wait in thread:
7117
W/SQLiteDatabase( 4958): database lock has not been available for 30
sec. Current Owner of the lock is 7112. Continuing to wait in thread:
7113
W/SQLiteDatabase( 4958): database lock has not been available for 30
sec. Current Owner of the lock is 7112. Continuing to wait in thread:
7117
W/SQLiteDatabase( 4958): database lock has not been available for 30
sec. Current Owner of the lock is 7112. Continuing to wait in thread:
7113
W/SQLiteDatabase( 4958): database lock has not been available for 30
sec. Current Owner of the lock is 7112. Continuing to wait in thread:
7117
W/SQLiteDatabase( 4958): database lock has not been available for 30
sec. Current Owner of the lock is 7112. Continuing to wait in thread:
7113
W/SQLiteDatabase( 4958): database lock has not been available for 30
sec. Current Owner of the lock is 7112. Continuing to wait in thread:
7117
W/SQLiteDatabase( 4958): database lock has not been available for 30
sec. Current Owner of the lock is 7112. Continuing to wait in thread:
7113
W/SQLiteDatabase( 4958): database lock has not been available for 30
sec. Current Owner of the lock is 7112. Continuing to wait in thread:
7117

> --
> You received this message because you are subscribed to the Google Groups
> "NewsRob User Group" group.

> To unsubscribe from this group and stop receiving emails from it, send an
> email to newsrob+u...@googlegroups.com.

>
> To post to this group, send email to new...@googlegroups.com.

> Visit this group at http://groups.google.com/group/newsrob?hl=en.
> For more options, visit https://groups.google.com/groups/opt_out.
>
>

[Mariano Kamp]

"atabase lock has not been available for 30
sec. Current Owner of the lock is 7112. Continuing to wait in thread:
7113"

Yes, that would explain the black screens. Can your reproduce this at will?

[Alex Butcher (googlemail)]

It seems to happen during the "exact syncing states" phase of a sync (possibly every time?)

--
Alex Butcher
Apologies for brevity and any typos: sent from my phone

[Mariano Kamp]

That's definitively a long lasting database operation. It would make sense.

Not sure if I can make it faster though. I'll have a look. At least I could tell you what to look for in the log to pinpoint the issue. 

Do you happen to be a developer?

--

Alex Butcher
Apologies for brevity and any typos: sent from my phone

--

[Alex Butcher]

Hi Mariano -

Try doing a sync, then tap on the spinner to get the progress bar,
then wait for it to be on 'Exact syncing states' and try to read, go
back, etc.

I have downloadArticlesInParallel=1 and downloadImagesInParallel=0.
Behaviour seemed the same with 0 for both, though.

I am a developer, but not for Android (yet). I'm familiar with Linux,
and therefore somewhat familiar with Android internals, reverse
engineering, etc.

HTH,
Alex

[Mariano Kamp]

Weren't able to reproduce it myself so far and can't spent much time on it at the moment.

Could you send me a log file where I can see that behaviour?

The download* parameters will not affect it.

Access to the database happens using a single lock, if I recall correctly.

Exact syncing does long operations on the database, so I could see that happening, but 30 seconds seems strange. Will check your log for which db operation is actually taking that long.

Maybe in the log it looks like this: "DB.updateStatesFromTempTable state.*took" ... or "DB.populateTempIds.*took" ...

Do you have many unread articles? This will make exact syncing slower. Same goes for pinned articles and starred articles.

Those numbers, btw., are outputted to the log at the beginning of a sync as part of a semi formatted block, easy to spot. It starts with "-- Time: ".

[Alex Butcher]

I've attached an annotated log file; look for ## AJB for my annotations.

[Mariano Kamp]

Thanks very much Alex.

Regarding the annotations. Hmmmh. Doesn't make sense to me. NewsRob doesn't say it is doing anything at that point. So I **suspect** it may have been hanging on a lock for quite some time, maybe a dead lock.

The actual actions, the exact syncing, happens directly above your annotations:

<02 12 07:51:13:610> Timing: DB.populateTempIds STARRED count=8943 took 63774 ms.

<02 12 07:51:13:614> ER.differentialUpdateOfArticleStates for STARRED. (4E)

<02 12 07:51:13:666> ER.differentialUpdateOfArticleStates(pinned) for STARRED. (4.2E) articlesInTemp_Ids_READ=10000

<02 12 07:51:13:720> ER.differentialUpdateOfArticleStates(pinned) for STARRED. (4.2E) articlesInTemp_Ids_PINNED=-1

<02 12 07:51:13:724> DB.updateStatesFromTempTable state=STARRED

<02 12 07:51:13:754> Timing: DB.updateStatesFromTempTable - mark existing read took 6 ms.

<02 12 07:51:13:785> Timing: DB.updateStatesFromTempTable - mark as x took 26 ms.

<02 12 07:51:14:20> Timing: DB.updateStatesFromTempTable state=STARRED took 296 ms.

<02 12 07:51:14:66> ER.differentialUpdateOfArticleStates for STARRED. (5E)

<02 12 07:51:14:97> ER.differentialUpdateOfArticleStates for STARRED. (6)

<02 12 07:51:14:543> ER.differentialUpdateOfArticleStates for READ. (3E)

<02 12 07:51:14:616> Executing sql=CREATE TABLE IF NOT EXISTS temp_ids_READ (atom_id TEXT PRIMARY KEY);

<02 12 07:51:14:689> Timing: DB.clearTempTable STARRED took 572 ms.

<02 12 07:51:17:784> Timing: DB.clearTempTable READ took 3099 ms.

<02 12 07:51:17:865> DB.populateTempIds(READ): number of article ids=10000

<02 12 07:51:18:00> clearTempTable

<02 12 07:51:18:24> Timing: DB.clearTempTable PINNED took 3331 ms.

<02 12 07:51:18:29> Timing: ER.differentialUpdateOfArticleStates for STARRED. took 77069 ms.

<02 12 07:51:18:52> Waiting for future java.util.concurrent.FutureTask@41575a08 done.

<02 12 07:51:18:58> Waiting for future java.util.concurrent.FutureTask@4157add8

Regarding the log per se, is this a typical sync for you? 

<02 12 07:49:46:949> Timing: Delete Articles From Db took 109762 ms.

<02 12 07:49:46:983> No of articles affected=933

That's a pretty long delete, but also a lot of articles and their assets to get rid of.

So, still no clue. Will try to set time aside to replicate on the emulator.

[Alex Butcher]

Hi Mariano -

On Tue, Feb 12, 2013 at 2:35 PM, Mariano Kamp <marian...@gmail.com> wrote:
> Regarding the log per se, is this a typical sync for you?
> <02 12 07:49:46:949> Timing: Delete Articles From Db took 109762 ms.
> <02 12 07:49:46:983> No of articles affected=933
> That's a pretty long delete, but also a lot of articles and their assets to
> get rid of.

I don't usually pay much attention to it whilst it syncs and cleans
up. The only data point I can offer is that sometimes the entire
process takes 30+ minutes, given how many articles I sync (and that
some of my feeds are full web pages, in order to be readable offline).

HTH,
Alex

[Mariano Kamp]

Alex,

given the latest news it doesn't probably matter all that much, but still, I am not able to reproduce it on any of my devices, including the emulator.

When you get this black screen can you press a hw button or swipe over the screen to produce an "application not responding dialog" and then chose "force close". The OS will create the file: /data/anr/traces.txt. Could you send this to me. It contains stack traces from all running threads and I can likely see where the app was hanging. Maybe this will help.