Multiple external storage devices

[Al Sutton]

I was wondering if there is any scope for a contribution to support multiple external storage devices?

The use case is originates from devices with a USB host mode (such as tablets) where there can be an SD card and a USB storage device available along with an large internal storage pool. An application may wish to see the amount of space available and possibly connection type so it can make decisions on data storage/caching.

For example; faced with a device which has 16GB of available internal storage, an 8GB available on an SD card, and a 250GB available on a USB hard disk the application may want to use the 250GB hard disk for caching larger data (e.g. images, data downloads) and the SD card for more frequently accessed information (e.g. HTML source, result sets). My current thinking is that the connection type would be useful for making decisions about availability (e.g. the more useful the data the less likely you'd store it on a more frequently removed device such as a USB disk), but I fully understand there is no guaranteed probability of availability for any storage area (who's to say the user won't pop the SD and leave the USB drive connected).

What are peoples thoughts?

Al.
======
Funky Android Limited is registered in England & Wales with the company number 6741909.

The views expressed in this email are those of the author and not necessarily those of Funky Android Limited, it's associates, or it's subsidiaries.

[Dan Morrill]

This is on the edge of our radar, at the moment. We haven't yet seen any devices actually request or need this, but it does seem likely to be something we'll have to deal with eventually. The hard part is coming up with a characterization of the many forms of media, both in terms of temporality (i.e. how likely it is to be removed) and capacity.

For instance, the SD card on a phone is nearly always a micro-SD card and thus fairly difficult to insert or remove (it would be a challenge for 6 year old), meaning it's sort of semi-permanent. But on a tablet, it might be using full-size SD which is much easier to remove. And then a USB thumb drive is trivial to add and remove (and that 6 year old can easily grab a USB stick and pull it out). So these 3 devices have different transient-ness, but could have identical capacity.

The other issue that needs to be addressed is how system apps should behave. Should the MediaScanner scan any storage device connected automatically? Or should it use criteria? For instance, putting an SD card in a phone should definitely trigger the MediaScanner automatically; but should inserting a thumb drive? If I insert an SD card from my camera, do I really want to add all those images to Gallery with all the scanning time that takes, or do I really just want to have the ability to launch a slideshow directly from the SD card?

What I've been kicking around in my head is a sort of "storage manager" API like the LocationManager, where you can request a storage path based on criteria, or be notified of inserts/removes based on criteria. So we might, for instance, create constants for StorageManager.INTERNAL (un-removable), StorageManager.SEMI_PERMANENT (like an SD card behind the battery), or TRANSIENT (like a thumb drive or SD edge slot that can be trivially added and removed.) Today apps say "hand me the location of shared storage where I can write some data"; with this model they might say "find me a shared storage volume that is SEMI_PERMANENT or better."

A music-caching app like that doubleTwist one, or Spotify might be able to tolerate TRANSIENT, since their copies are just cache and they can re-download. But the MediaScanner wants things to be a bit more permanent before adding to the database, so it might stipulate SEMI_PERMANENT or better. And then if someone builds a draconian app with full-on DRM or something, they might require INTERNAL or something.

I am not currently convinced that we would want to characterize capacity in this way though. For one thing, Moore's Law makes it difficult to define buckets: 16GB might be high capacity today, but it will be mid capacity next year, for instance. And then hypothetical devices with hard drives or large SSDs might be 100GB+, making "large capacity" have a different meaning on phones vs. TVs or something. And then finally, putting capacity under app control removes a degree of control from the system. For instance, if the system has a 64GB volume with 2GB free and a 16GB volume with 8GB free, the system would probably want to hand the 16GB volume to an app in most cases; but if we are letting apps specify that they need a large-capacity device, the system couldn't do that.

And if we're going to do this, we should do it right and consider things like encryption -- should there be a StorageManager.SECURE?  Anything else?

So yeah, this is coming, but we don't have a clear solution for it yet. I'm sure Dianne's been thinking about it too (and I think we talked about it once even), so maybe she'll chime in here with her thoughts.

After others have a chance to comment, I think the next thing to do would be to have someone (did you just volunteer, Al? ;) write up a design doc from the conversation to formalize it a bit more.

- Dan

[Al Sutton]

My initial thoughts were that the transient-ness of an storage device would be highly removable by default and then up to porting teams to define it down appropriately for a particular storage connector (be it micro-SD drive, SD-drive, or USB Hub).

I would suggest media scanner, by default, only scans permanent or semi-permanent (e.g. phone microSD). There could be a setting to decrease the transient-ness threshold, and an option to manually scan specific devices so a user can trigger a scan of a USB thumb drive without a 2TB USB disk being scanned as well.

The storage manager API sounds like it's a good solution, my initial thought was extending android.os.Environment to have a series of methods which would return an Array or List of storage areas and their characteristics. The existing external storage methods in Environment could then feed off these for backwards compatibility, but a storage manager API would take this a step further as it could be extended to included non-physically local storage locations (e.g. cloud storage, NAS, etc.) and include a StorageManager.NETWORK value to show there is a potentially unreliable transport medium between the device and the physical storage area.

I'd avoid trying to bucket storage capacities and allow a real value to be specified. What comes to mind is OTA update and mapping software which can determine the size a data download needs and so would want to say "Give me the SEMI_PERMANENT storage capable of holding x MB". I also can't see a benefit to bucketing it when applications are most likely going to be dealing in solid values, but if you've got a use-case for it I'm all ears :).

Do we want to make secure storage accessible to applications as a separate entity? Apps already have access to generic storage through Context.openFileOutput / openFileInput. I can see confusion being caused by having two private storage areas so I'd suggest an initial implementation which deals with purely insecure storage and allow applications to implement their own security if they require it.

I'm happy to write up a design doc. I'll plan to start mid/late next week to give the conversation a chance to run and hopefully have something to circulate for people to read in the oh-so-quiet time before xmas :).

Al.

======
Funky Android Limited is registered in England & Wales with the company number  6741909. 

The views expressed in this email are those of the author and not necessarily those of Funky Android Limited, it's associates, or it's subsidiaries.

[Dianne Hackborn]

There is stuff coming in HC that I think this should be building on.

Also I think this needs to be designed from the UI down.  The very last thing I want to do is just throw in an API and have apps start doing stuff with it, and suddenly the user is confronted with dealing with "oh where do you want to put or get these files" all over the place.

In fact just getting the current SD card mess (it being a big dumping ground of crud from apps) cleaned up would be a first good step...  and some of that is coming in HC.

The user experience we have often talked about around more transient removable storage is as "import/export", but not a place that applications put their working data on.  For example if you have pictures on a thumb drive that you plug into your tablet, you could browse through them, copy selected ones to your device's internal storage, or copy from stuff in internal storage to the thumb drive.  But you wouldn't be directly working on images there in the same way as internal storage.

Also an early version of StorageManager already exists in Froyo (though I think it is all hidden), and there has been more development of it in GB and HC.

--
Dianne Hackborn
Android framework engineer
hac...@android.com

Note: please don't send private questions to me, as I don't have time to provide private support, and so won't reply to such e-mails.  All such questions should be posted on public forums, where I and others can see and answer them.

[Al Sutton]

Don't suppose you want to give me a sneaky look at HC so work on a usable design :).

Al.

======
Funky Android Limited is registered in England & Wales with the company number  6741909. 

The views expressed in this email are those of the author and not necessarily those of Funky Android Limited, it's associates, or it's subsidiaries.

[Mark Carter]

I was going to suggest having a ContentProvider which returns the
various storage options. System-wide settings could restrict which
storage options are returned, based on some kind of criteria (for
example, to always use a certain storage option for the Media Player).
So often just one item would be returned, making it less likely the
user would need to make a choice.

Would also be nice to provide an icon for each type of storage (e.g.
microSD, SD, thumb drive) so that (when there are multiple storage
options) the returned Cursor can easily be plugged into a ListView
with minimum fuss.

Agree with Dianne that you don't want to have a load of these choices
popping up for the user everywhere.

[Al Sutton]

I think a ContentProvider would be too heavyweight, but after Diannes comments I don't think this is something that can be designed by the community.

She mentioned there are changes in GB & HC, neither of which the public community currently has access to, so I can see there is a high risk that any design and/or implementation decisions could easily be out of step with what's already been/being done and thus be unusable.

If someone can come up with a way of working on this given the access constraints to GB & HC I'd be happy to work on the design document, but as things currently stand I don't want to spend time on something that could quite easily end up as useless due to work already done that we can't see.

Al.

======
Funky Android Limited is registered in England & Wales with the company number 6741909.

The views expressed in this email are those of the author and not necessarily those of Funky Android Limited, it's associates, or it's subsidiaries.

[Jean-Baptiste Queru]

From where I'm sitting, I think that it's important to focus on the
various aspects, possibly more or less in this order although it's all
intertwined:

-What kind of user experience do we want to have? (this part of the
discussion will eventually need to include UX people, not just
engineers).

-How do we deal with backward and forward compatibility for all the
apps that currently access shared storage (and especially for the ones
that write to it)?

-How do the APIs need to evolve to reflect that?

-Implementation details...

-Side question: how about remote filesystems and remote file access,
e.g. CIFS/SMB, UPnP, DLNA...?

-Side question: how about read-only media, e.g. write-protected SD cards?

JBQ

--
Jean-Baptiste M. "JBQ" Queru
Software Engineer, Android Open-Source Project, Google.

Questions sent directly to me that have no reason for being private
will likely get ignored or forwarded to a public forum with no further
warning.

[Dianne Hackborn]

On Fri, Dec 10, 2010 at 11:40 AM, Al Sutton <a...@funkyandroid.com> wrote:

Don't suppose you want to give me a sneaky look at HC so work on a usable design :).

Basically there will be a facility to put the "external" storage and /data directory on the same partition.  That is, if you have a device with 16GB of internal storage, the complete 16GB is available to both /data and Environment.getExternalStorage(), without having to define a partitioning between the two.

This pretty deeply changes the semantics of what Environment.getExternalStorage() is, limiting it in scope.  For devices built for HC with this design, the only reason to put things on external storage is if you want to globally share them, such as music or pictures.  It is no longer needed as a secondary location for "large stuff" to avoid filling up /data.  For example, "Apps on SD card" is no longer useful for external storage.

This for the most part will greatly simplify the user experience around the internal storage of their device, though it does raise some new edge cases that need to be addressed.  (The big one is what if you fill up all of your storage with stuff in Environment.getExternalStorage(), how do you present the data contained there to the user for them to clean it up?)

This also means we should sit down and completely rethink what an "SD card" means and how it is managed by the platform.  For these devices, it is no longer a central piece of storage that is required by some applications for them to be able to run.

And of course, this is also going to be a gradual evolution, so we need to keep in mind how we move towards this new model while still maintaining a reasonable situation for devices that have the old /data vs /sdcard separation.

Anyway, it is true that this is a difficult area to make AOSP contributions to, because it has so many repercussions to the overall UI.

[Al Sutton]

Thanks for the explanation, but it doesn't seem to address the problem I'm aiming to solve for multiple external storage devices.

As I understand things from your email it sounds like the HC changes are actually going in the wrong direction for this. It seems that it's going to become impossible for an application to know if "external" storage really is external and treat it appropriately, and there seems to be no consideration that applications may want to make intelligent choices and adapt to the space storage areas available.

I full agree ExternalStorage has become a dumping ground, but I'm pretty sure that in many cases this because there is a need for a dumping ground for large objects and external storage offers the best fit for the requirement. As you rightly say if it's a dumping ground and the /data and /sdcard are on the same partition there is a risk of resource depravation for one caused by an applications use of the other, so my question would be why not offer multiple dumping grounds to the application and let it choose which is the most suitable rather than making it take pot luck as to whether using external storage will cause problems for /data?

I think you're right in that thinking of external storage as an sdcard has got to be moved away from, but it seems the current move is towards a design which creates a single opaque pool of storage will cause more problems than it solves, whereas I'm proposing the opposite direction of making storage a transparent collection of areas an application can choose from which would allow developers to write their applications to use the areas appropriately, and any that don't will get the same treatment any application which uses resources poorly will; it'll get relegated to obscurity by a better performing application.

Al.

======

Funky Android Limited is registered in England & Wales with the company number  6741909. 

The views expressed in this email are those of the author and not necessarily those of Funky Android Limited, it's associates, or it's subsidiaries.

[Al Sutton]

I can see the need for UX guys to be involved, but I think this is something which will benefit from a first iteration which presents the functionality to apps. My view is that if we make multiple external storage areas available to apps we can see how they use it, so what works and what doesn't, and see what we need to change, because I'd be surprised if any group of us could predict all of the use cases we'll see.

I'm tempted to suggest writing some form of dummy StorageManager library which developers could use during the development of the API so they can thrash it and give feedback. That way we can spark peoples imagination and see what UX/APi issues it throws up.

Al .

======
Funky Android Limited is registered in England & Wales with the company number 6741909.

The views expressed in this email are those of the author and not necessarily those of Funky Android Limited, it's associates, or it's subsidiaries.

[Al Sutton]

This thread has been quiet for a bit now so I guess it's crunch time.

I'm grateful for Diannes explanation of the plans, but it did seem that the amount of work being done in the storage area in GB & HC is significant and so, at the moment, I get the impression there's no value to continuing on to a design document because any design which didn't take the GB/HC changes into account would need a re-design prior to implementation.

Does anyone see the situation differently?

Al.

======
Funky Android Limited is registered in England & Wales with the company number 6741909.

The views expressed in this email are those of the author and not necessarily those of Funky Android Limited, it's associates, or it's subsidiaries.

[Dan Morrill]

Yeah it's crunch time, we're buried til EOY at this point. Let's revisit in January. I at least don't consider the conversation over, just yet. :)  But it probably is premature for a design doc.

- Dan

[Dianne Hackborn]

On Thu, Dec 16, 2010 at 2:51 PM, Dan Morrill <morr...@google.com> wrote:

Yeah it's crunch time, we're buried til EOY at this point. Let's revisit in January. I at least don't consider the conversation over, just yet. :)  But it probably is premature for a design doc.

You mean we won't be in crunch time in January?? :)

Sorry for the slow response; I also don't consider this over, but it will be easier to talk when we can all look at things in terms of where HC is at.

[Al Sutton]

OK, I'll wait until someone pings me before continuing with this.

Thanks,

Al.

======
Funky Android Limited is registered in England & Wales with the company number  6741909. 

The views expressed in this email are those of the author and not necessarily those of Funky Android Limited, it's associates, or it's subsidiaries.

[Al Sutton]

Given the recent comments relating to Honeycombs source not being made available I'm going to be holding off on this until I can see what's currently in place to ensure that any suggested design does fit well with releases which are already shipping.

Al.

-- 

T: @alsutton W: www.funkyandroid.com

The views expressed in this email are those of the author and not necessarily those of Funky Android Limited, it's associates, or it's subsidiaries.