Replacing data with bundle_data?

[Andrew Grieve]

While trying to figure out how best to use runtime_deps to generate a .isolate file for Android, I'm now questioning whether we should redesign how we handle app data entirely! Here's what we currently have:

1. Desktop platforms: data / data_deps produce a list of files required at runtime.

2. Android: android_assets() specify files to be packed into the .apk's as assets/ directory

3. iOS: bundle_data() specify files to be copied into a final .bundle directory

4. Android unit tests: Use a checked in .isolate file to list files that should be side-loaded to the device before tests can be run.

#4 is what I've been working / thinking about lately, and it's just not going well. Right now I've hacked together a change to GN to write out runtime_deps, and then a python script which blacklists 90% of the entries in it. Ugly!

I'm wondering if we could come up with an actual x-platform system for specifying runtime deps?

Some design considerations I think are true (please let me know if I'm wrong):

1. Assets are compressed by default on Android, but we specify that some be uncompressed in order to mmap them

2. Both iOS and Android need to be able to control packaging of these deps at build-time

   - This includes ability to rename files (Android v8 snapshots do this for _32 and _64)

3. All runtime deps are read-only

4. Runtime deps do not need to support enumeration at runtime

Here's the strawman:

1. Get rid of data / data_deps in favour of bundle_data

2. Make the existing "runtime_deps" logic of GN apply only to shared libs (aka, the part of data that isn't explicitly listed in .gn files)

3. Mark files as needing to be uncompressed via a special {{marker}}: 

      E.g. outputs=["{{uncompressed_bundle_resources_dir}}/{{source_file_part}}"]

4. Create an API in base that fetches a file from bundle_data in an x-platform way.

Option A for an API:

class AssetManager {
  Asset Open(FilePath asset_path)
}
class Asset {
  bool IsValid();  // Whether the asset exists.
  MemoryMappedFile Map();
  int64_t GetSize();
  int Read(int64_t offset, char* data, int size); // Works even when compressed.
}

There are ~200 references to DIR_TEST_DATA, and following just a few led me to net/base/file_stream_context.cc. Such a change looks to be *months* of work.

Option B for an API:

class AssetManager {
  File Open(FilePath asset_path)
  MemoryMappedFile OpenMapped(FilePath asset_path)
  FilePath PathForAsset(FilePath asset_path) --> app-asset://$asset_path
}

To make File handle this, we'd need it to add "if (IsAsset())" to almost every function, at least for OS_ANDROID where it's not actually a file we're dealing with.

Option C for an API:

- Same as B, except we use pipes to create an fd that represents a portion of the .apk

- This kind of sucks since each pipe requires a dedicated thread to feed it.

Option D for an API:

- Same as A, except for tests we extract all assets on launch and set DIR_TEST_DATA accordingly.

Any other options? Opinions as to whether this is even a problem worth solving?

[Sylvain Defresne]

I'm not so sure about the x-platform API to access files in bundle_data.

On iOS, the application can only access files that are in its own application bundle (and in a shared location if it wants to communicate with its extension, but this is iOS specific and does not need a x-platform API). This means that the existing base/ API only gives access to files in the bundle_data already on iOS.

On Mac, we already have system specific way to access to files in the application bundle when required.

So the only remaining platform that would need to access the bundle data is Android (since Windows, Linux and Chrome OS won't use bundle_data). I guess that on Android the application already has a way to access the files in its .apk. So, I guess that you want on Android to use bundle_data for other files outside of the .apk, test files maybe?

I do not really understand how Android currently run its tests. Does it expect to find the files in the .apk? Outside of it? On iOS (and OS X) you just use standard open/read/write to access files in the application bundle since a bundle is just a simple directory (not an archive as on Android).

My recommendation, would be to not have a x-platform API to access resources from bundle but have system specific one if needed (e.g., on iOS we use currently NSBundle to access most of the resources in the bundle when they are iOS specific). For resources that are x-platform (mostly tests), then we have a platform specific way to make them available through the regular base::File API.

For iOS, this is just correctly initializing the DIR_TEST_DATA (already implemented), and for Android, it could be to extract the resources on launch and correctly set the DIR_TEST_DATA.

Regarding compressing the resources or not, instead of using a special {{marker}}, I would just add some flags to bundle_data and propagate it to the create_bundle target.

-- Sylvain

[Andrew Grieve]

On Wed, Mar 23, 2016 at 5:52 AM, Sylvain Defresne <sdef...@chromium.org> wrote:

I'm not so sure about the x-platform API to access files in bundle_data.

On iOS, the application can only access files that are in its own application bundle (and in a shared location if it wants to communicate with its extension, but this is iOS specific and does not need a x-platform API). This means that the existing base/ API only gives access to files in the bundle_data already on iOS.

On Mac, we already have system specific way to access to files in the application bundle when required.

So the only remaining platform that would need to access the bundle data is Android (since Windows, Linux and Chrome OS won't use bundle_data). I guess that on Android the application already has a way to access the files in its .apk. So, I guess that you want on Android to use bundle_data for other files outside of the .apk, test files maybe?

This is exactly my point. Android and iOS do not use GN's data / data_deps, but capture the same information via android_assets() and bundle_data(). It's a burden to have 3 ways to capture the same use-case.

 

I do not really understand how Android currently run its tests. Does it expect to find the files in the .apk? Outside of it? On iOS (and OS X) you just use standard open/read/write to access files in the application bundle since a bundle is just a simple directory (not an archive as on Android).

It currently sets  DIR_TEST_DATA to the external storage location, and the test runner is responsible for putting the files there before running the test.

My recommendation, would be to not have a x-platform API to access resources from bundle but have system specific one if needed (e.g., on iOS we use currently NSBundle to access most of the resources in the bundle when they are iOS specific). For resources that are x-platform (mostly tests), then we have a platform specific way to make them available through the regular base::File API.

Definitely don't want to remove the platform-specific APIs. They're always useful when more power is required.

[Brett Wilson]

My gut says that trying to refactor the way Chrome distributes bundle data will be quite difficult and not super high benefit. I'm not intimately familiar enough with the Android packaging to have a specific opinion on that.

There is some merit to considering whether there should be one concept of "data" and that we delete data and data deps in favor of a slightly generalized version of bundle data and the Android packaging thingy. Dirk and I have discussed this before in person, along with whether the setup apps should use runtime deps instead of hand-coded lists.

We've so far concluded that it's better to separate out the testing/isolate-type dependencies from the build packaging. Historically the isolate stuff has been pretty loose and it's easy to add data annotations in leaf nodes that unknowingly get propagated up the dependency tree and end up being distributed. We want the test environment to be by-default permissive and include things you need to avoid problems, and we want build packaging to be restrictive so you know you're opting in to shipping something to all users.

Brett

[Andrew Grieve]

Hmm, didn't know desktop didn't use data for specifying which files should be sent to users.

Sounds like maybe data / data_deps might only be suitable for desktop unit tests. Sending files to devices is really slow, so a permissive model isn't really what we want for Android / iOS. I'll experiment with adding android_assets() to tests that need files and having them extracted on start-up & see how annoying that is.

[Dirk Pranke]

We don't really want to be specifying data_deps that aren't actually needed to run tests, period; isolating them and sending them to the swarm is expensive too, even if it's not quite as expensive as loading them onto a device (and, of course, in swarmed android tests you have to use both).

I think the idea of finding a generic way to specify these types of dependencies is a good one, but I don't know enough of the details to know how to do so at the moment.

I was even floating the idea to Brett that really data_deps needed to go away and we might need to create a parallel set of targets like bundle_data even for desktop tests, in order

to have more flexibility for doing things differently across desktop/android/ios.

Perhaps the thing to do is to do something android-specific for now that is as tailored to android and close to ideal as possible (in much the same way that bundle_data is currently very mac-specific), and

then we can step back and see what the commonalities, if any, are between the three.

-- Dirk

[Andrew Grieve]

+jbudorick (as fyi)

[Nico Weber]

Running tests on swarming, Android, iOS all require "pack up all files needed by this test" logic. I think fixing this somehow is important. One of gn's points is to unify our build story across systems more. I think it's worth coming up with something here.

I'd try to do something simpler than the AssertManager thing. I don't know what exactly, but I think this is something that should be figured out.

On Wed, Mar 23, 2016 at 7:30 PM, Brett Wilson <bre...@chromium.org> wrote:

[Dirk Pranke]

Yes, I think we all agree that this is important. 

However, it also appears that android device / ios device / swarming need three different things, and so we're still working out what commonality does exist and what the best way to implement things is. It looks like data/data_deps by itself isn't quite good enough, and bundle_data probably isn't much better, and now we're exploring ways to get the runtime_deps exposed programmatically in the build itself (or, at least, indicate that we need the runtime_deps for a given target to be generated so that a python script can manipulate it as part of the build).

-- Dirk

[Sylvain Defresne]

Created https://bugs.chromium.org/p/chromium/issues/detail?id=599319 to track this (please close if this is a duplicate).

-- Sylvain