Generic data update service?

[Gervase Markham]

We keep hitting cases where we would like Firefoxes in the field to have
some data updated using a process which is much lighter in expended
effort than shipping a security release. Here are some examples of the
data Firefox stores that I know of which might benefit from this:

- The Public Suffix List (more important with new TLDs coming)
- The Root CA store, or trust lists in general
- Addon and plugin blacklists
- The default prefs file
- The UA override list (needed for B2G, at least)

(In addition, the IDN TLD whitelist would have been on that list until
it was recently obsoleted, and there are possible future additions -
intermediate CA whitelists being one I know some people want. Safe
Browsing already has its own dedicated service.)

It was the last of these which has triggered this conversation. Given
that we need a system for dynamically updating the UA override list on
B2G and maybe Android too, and that I know Brian Smith is thinking about
how we can have dynamic updates to the Root CA store, is it worth
spending a little extra effort to build something which can be used for
these other cases too, and any others which come along?

The question for webdev is: do we have existing services we can
repurpose or tweak for this?

v1 requirements:

- Firefox polls server every 24 hours
- Secure connection
- Server sends names of files to be replaced and new data
- File on disk at Firefox end gets replaced (and would be reloaded on
restart)
- Ability to send different data to different clients
- Ability to cope and fall-back to profile-stored per-user info if
Firefox directory is read-only

Future stuff:

- Ability to trigger data reload without restart
- Ability to integrate data not stored in separate file
- Send only deltas
- Intelligent scheduling
- Server-side ability to send different data to different versions of
the same client

Gerv

[Benjamin Smedberg]

On 7/12/2013 12:49 PM, Gervase Markham wrote:
> We keep hitting cases where we would like Firefoxes in the field to have
> some data updated using a process which is much lighter in expended
> effort than shipping a security release. Here are some examples of the
> data Firefox stores that I know of which might benefit from this:
>

snip

> - Addon and plugin blacklists

These are already loaded dynamically.

> - The default prefs file

Unless there is a much clearer use case for this, I strongly oppose
trying to lump default prefs in with any of the rest of these. In the
case of a specific release issue where a feature needs to be disabled,
we can accomplish this using hotfix addons (at least for desktop; B2G
may not have hotfix capabilities but probably should grow them in 1.2 or
after).

>
> The question for webdev is: do we have existing services we can
> repurpose or tweak for this?

AMO serves the plugin/addon blocklists. Since the service is pretty
simple, though, I'm not sure that we would want to piggyback new data
types onto existing services.

An important aspect of this which was not mentioned is that if we're
going to start polling for more data, we should do a single poll, rather
than having separate pings for each data type. And since some of these
lists have the potential to be large, we may need to be careful to only
download them if there is actually an update, and decide on the policy
for downloading updates over paid (cellular) data versus wifi.
>
> v1 requirements:

>

> - File on disk at Firefox end gets replaced (and would be reloaded on
> restart)

I object to this requirement. This should be a per-profile item and we
should not be writing any files in the application directory. This is
necessary for many reasons including security and avoiding unwanted UAC
prompts, but also because altering installation files voids incremental
updates.

> - Ability to send different data to different clients

This needs to be fleshed out. On what data do you make this
determination, and how do you manage the backend for deciding which
clients get which pieces of data?

I think the general concept of making more of our "lists" be dynamic is
sound, but I'm very skeptical of the technical solution that you appear
to be outlining.

--BDS

[Nicholas Nethercote]

On Fri, Jul 12, 2013 at 9:49 AM, Gervase Markham <ge...@mozilla.org> wrote:
> We keep hitting cases where we would like Firefoxes in the field to have
> some data updated using a process which is much lighter in expended
> effort than shipping a security release.

Would such an update increment the version number? I suspect you'd
want to be able to easily determine if an update has been applied, and
having to distinguish e.g. "Firefox 30 without update 1" vs. "Firefox
30 with update 1" could be annoying (and makes me think of Windows
service packs).

Nick

[Gervase Markham]

On 12/07/13 18:20, Benjamin Smedberg wrote:
> I think the general concept of making more of our "lists" be dynamic is
> sound, but I'm very skeptical of the technical solution that you appear
> to be outlining.

The technical solution was 3 minutes on the back of an envelope. Feel
free to tear it apart entirely. But for this discussion, can we focus on
the general rightness or wrongness of such a thing?

Gerv

[Joe Drew]

On 2013-07-12 12:49 PM, Gervase Markham wrote:
> The question for webdev is: do we have existing services we can
> repurpose or tweak for this?

When I needed to implement downloadable graphics driver/system
blacklists, I piggybacked on the existing AMO blocklist ping. The
blocklist already has the ability to dispatch arbitrary blocklist items;
you simply observe blocklist-items-YOURITEMNAME like
<https://mxr.mozilla.org/mozilla-central/source/widget/xpwidgets/GfxInfoBase.cpp#514>
has. Then you get called when the blocklist encounters eg <gfxItems>.

In general, yes, we absolutely need a generalized downloaded-update
service. Benoit Jacob has talked about extending the graphics blacklist
to be such a thing, but we might already have it in the form of the
XML-based AMO blocklist.

joe

[Robert Strong]

On 7/12/2013 1:12 PM, Nicholas Nethercote wrote:
> On Fri, Jul 12, 2013 at 9:49 AM, Gervase Markham <ge...@mozilla.org> wrote:

>> We keep hitting cases where we would like Firefoxes in the field to have
>> some data updated using a process which is much lighter in expended
>> effort than shipping a security release.

> Would such an update increment the version number? I suspect you'd
> want to be able to easily determine if an update has been applied, and
> having to distinguish e.g. "Firefox 30 without update 1" vs. "Firefox
> 30 with update 1" could be annoying (and makes me think of Windows
> service packs).
>
> Nick

It wouldn't update all of the places where version numbers are stored
today since that would break incremental updates but there is no reason
that the places in the UI couldn't read in this information and show it
in the UI and thereby avoid breaking incremental updates.

Robert

[Clint Talbert]

On 7/12/2013 9:49 AM, Gervase Markham wrote:
> We keep hitting cases where we would like Firefoxes in the field to have
> some data updated using a process which is much lighter in expended
> effort than shipping a security release. Here are some examples of the
> data Firefox stores that I know of which might benefit from this:
>
> - The Public Suffix List (more important with new TLDs coming)
> - The Root CA store, or trust lists in general
> - Addon and plugin blacklists
> - The default prefs file
> - The UA override list (needed for B2G, at least)
>
>

This is all good stuff, and I want to support us being nimble. We also
need to balance that against security and quality in our builds. We go
through the release process for a reason, and we exert the energy to QA
these builds and ensure we can update them incrementally, reliably, and
repeatably. I think that such a service like this can be fine, but I'd
want to be very certain we only change certain, safe items in the
profile directory, and we stay away from items in the application directory.

My two cents,
Clint

[Mark Finkle]

Couldn't the add-on hotfix approach be used for some of these items?

> _______________________________________________
> dev-platform mailing list
> dev-pl...@lists.mozilla.org
> https://lists.mozilla.org/listinfo/dev-platform

[Gervase Markham]

On 12/07/13 21:12, Nicholas Nethercote wrote:
> Would such an update increment the version number? I suspect you'd
> want to be able to easily determine if an update has been applied, and
> having to distinguish e.g. "Firefox 30 without update 1" vs. "Firefox
> 30 with update 1" could be annoying (and makes me think of Windows
> service packs).

Good question; don't know. Perhaps each resource needs to be separately
versioned, and have info about that tucked away somewhere less obvious
than Help|About where it can be referenced for the small subset of bugs
where it's useful?

Gerv

[Gervase Markham]

On 13/07/13 00:36, Clint Talbert wrote:
> This is all good stuff, and I want to support us being nimble. We also
> need to balance that against security and quality in our builds. We go
> through the release process for a reason, and we exert the energy to QA
> these builds and ensure we can update them incrementally, reliably, and
> repeatably. I think that such a service like this can be fine, but I'd
> want to be very certain we only change certain, safe items in the
> profile directory, and we stay away from items in the application
> directory.

If that's the general feeling, then we'd need to move any of these data
items which are currently not profile-specific (e.g. the PSL) to be
profile-specific. (Which itself suggests that updating them should rev.
the Firefox version number.)

Gerv

[Ben Hearsum]

Updating any of the things in the original would break incremental
updates as things stand now. We could figure out a way around it (eg,
forcing complete updates for those files), but it's something that we
should work out up front.

[Benjamin Smedberg]

Or it means that we need to be willing to issue dot-releases to update
these items. We're pretty nimble with the desktop release cycle already.
We should definitely measure this tradeoff before doing a bunch of
engineering on this. As I understand it, the major factor here is that
we are not nearly as nimble for FxOS updates, and so this is more of an
issue, correct?

--BDS

[Gervase Markham]

On 15/07/13 14:57, Benjamin Smedberg wrote:
> Or it means that we need to be willing to issue dot-releases to update
> these items. We're pretty nimble with the desktop release cycle already.
> We should definitely measure this tradeoff before doing a bunch of
> engineering on this. As I understand it, the major factor here is that
> we are not nearly as nimble for FxOS updates, and so this is more of an
> issue, correct?

Certainly the original motivation for this discussion was a desire to be
able to update the UA override list on Firefox OS after shipping, and I
assume there was an implied "without shipping a full update to the
device" in there somewhere.

Gerv