Heads Up: <profile>/storage upgraded from version 1.0 to 2.0

[Jan Varga]

Since the integration of bug 1339081 [1] ​in Nightly, the storage has
been upgraded from version 1.0 to 2.0
This means that if you run an already upgraded profile (by current
Nightly) in an older version of Firefox, then any storage APIs that
use Quota Manager (especially IndexedDB and DOM cache) won't work.
If you think you see any problems related to this upgrade, please report
in the bug [1] or file a new bug under DOM: Quota Manager.

​[1] https://bugzilla.mozilla.org/show_bug.cgi?id=1339081

​Thanks
Jan

[Robert Strong]

Are there any problems experienced by clients that downgrade to an older
version after their profile has been upgraded?

Thanks,
Robert

> _______________________________________________
> dev-platform mailing list
> dev-pl...@lists.mozilla.org
> https://lists.mozilla.org/listinfo/dev-platform
>

[Jan Varga]

On 07/03/17 23:43, Robert Strong wrote:
> Are there any problems experienced by clients that downgrade to an
> older version after their profile has been upgraded?
>

This major version change is downgrade-incompatible, so IndexedDB and
DOM cache won't work in an older version if their profile has been upgraded.
IndexedDB is also used internally, so stuff that depends on it likely
won't work too.
There's bug 1246615 [2] where you can find a discussion related to this
issue.

[2] https://bugzilla.mozilla.org/show_bug.cgi?id=1246615

> Thanks,
> Robert
>
>
> On Tue, Mar 7, 2017 at 2:32 PM, Jan Varga <jva...@mozilla.com
> <mailto:jva...@mozilla.com>> wrote:
>
> Since the integration of bug 1339081 [1] ​in Nightly, the storage has
> been upgraded from version 1.0 to 2.0
> This means that if you run an already upgraded profile (by current
> Nightly) in an older version of Firefox, then any storage APIs that
> use Quota Manager (especially IndexedDB and DOM cache) won't work.
> If you think you see any problems related to this upgrade, please
> report in the bug [1] or file a new bug under DOM: Quota Manager.
>
> ​[1] https://bugzilla.mozilla.org/show_bug.cgi?id=1339081
> <https://bugzilla.mozilla.org/show_bug.cgi?id=1339081>
>
> ​Thanks
> Jan
> _______________________________________________
> dev-platform mailing list

> dev-pl...@lists.mozilla.org <mailto:dev-pl...@lists.mozilla.org>
> https://lists.mozilla.org/listinfo/dev-platform
> <https://lists.mozilla.org/listinfo/dev-platform>
>
>

[Xidorn Quan]

On Wed, Mar 8, 2017, at 09:59 AM, Jan Varga wrote:
> On 07/03/17 23:43, Robert Strong wrote:
> > Are there any problems experienced by clients that downgrade to an
> > older version after their profile has been upgraded?
> >
> This major version change is downgrade-incompatible, so IndexedDB and
> DOM cache won't work in an older version if their profile has been
> upgraded.
> IndexedDB is also used internally, so stuff that depends on it likely
> won't work too.
> There's bug 1246615 [2] where you can find a discussion related to this
> issue.

It would probably be a good idea to backup the old files when upgrading,
so that old version can at least pick the backup file to use.

It is important especially given that we don't use a separate profile
for Nightly and Beta (which I think we really should!), but we sometimes
ask people to test them. It is very bad that we put users who want to
help us in risk of destroying their daily profile.

- Xidorn

[Mike Hommey]

While talking about this... I think it's about time we had an actual
plan for data cleanup.

Last week, when the cloudflare thing happened, I went through the files
in my profile looking for all my password-manager-managed passwords and
the domains associated with them. They used to be in a sqlite db, and
are now in json.

The change happened 3 years ago, in Firefox 32.

Guess what? the sqlite db is still there in my profile, with old, stale,
data in it.

Mike

[Ben Kelly]

On Tue, Mar 7, 2017 at 6:09 PM, Xidorn Quan <m...@upsuper.org> wrote:

> > This major version change is downgrade-incompatible, so IndexedDB and
> > DOM cache won't work in an older version if their profile has been
> > upgraded.
> > IndexedDB is also used internally, so stuff that depends on it likely
> > won't work too.
> > There's bug 1246615 [2] where you can find a discussion related to this
> > issue.
>
> It would probably be a good idea to backup the old files when upgrading,
> so that old version can at least pick the backup file to use.
>

Maintaining downgrade compatibility is a lot more complex than people think
and we have zero test support for it. It hasn't been safe for *years*
(look at IDB db migration code history) and its not going to be safe unless
we want to invest a lot in it. We just need to do bug 1246615 to make FF
warn when a power user gets themselves into this unsupported mode.
Per-channel profiles like dev-edition would be great too.

As an example of why "backup the db" is harder than it sounds, you would
need to backup the entire storage subsystem. If you lose data in IDB, but
don't lose data in other sub-systems like cache API, then an origin can
find itself in a corrupted state. If we allows writes at this point it can
permanently wedge itself in a corrupted state. From the perspective of
preserving user data its much safer to simply refuse to open the storage
system when we detect an incompatibility like this.

[Kris Maglione]

On Tue, Mar 07, 2017 at 08:02:47PM -0500, Ben Kelly wrote:
>As an example of why "backup the db" is harder than it sounds, you would
>need to backup the entire storage subsystem. If you lose data in IDB, but
>don't lose data in other sub-systems like cache API, then an origin can
>find itself in a corrupted state.

Not to mention the fact that when switching between versions
that use different schemas, we'd necessarily lose the changes
made by one version or the other when restoring or switching
between backups.

[Matthew N.]

On 2017-03-07 7:09 PM, Mike Hommey wrote:
> While talking about this... I think it's about time we had an actual
> plan for data cleanup.
>
> Last week, when the cloudflare thing happened, I went through the files
> in my profile looking for all my password-manager-managed passwords and
> the domains associated with them. They used to be in a sqlite db, and
> are now in json.
>
> The change happened 3 years ago, in Firefox 32.
>
> Guess what? the sqlite db is still there in my profile, with old, stale,
> data in it.

I'm not sure why you're using this as an example of a need for a "plan
for data cleanup": I don't see this as a general problem and it
shouldn't be causing you harm. A bug was filed for that
(https://bugzilla.mozilla.org/show_bug.cgi?id=1013947) which is the
usual behaviour so it's not like we forgot about this. We intentionally
gave a grace period to give people time to downgrade if necessary. The
problem is that password manager hasn't been getting properly resourced
other than a bit in the first half of 2015 and this bug is not
considered a priority since like I said above it doesn't cause harm,
it's a small file and that old file is using the same encryption as the
new file. If you disagree about the priority then feel free to comment
in the bug or find an assignee.

Matthew

[Ehsan Akhgari]

On 2017-03-07 8:02 PM, Ben Kelly wrote:
> On Tue, Mar 7, 2017 at 6:09 PM, Xidorn Quan <m...@upsuper.org> wrote:
>
>>> This major version change is downgrade-incompatible, so IndexedDB and
>>> DOM cache won't work in an older version if their profile has been
>>> upgraded.
>>> IndexedDB is also used internally, so stuff that depends on it likely
>>> won't work too.
>>> There's bug 1246615 [2] where you can find a discussion related to this
>>> issue.
>>
>> It would probably be a good idea to backup the old files when upgrading,
>> so that old version can at least pick the backup file to use.
>>
>
> Maintaining downgrade compatibility is a lot more complex than people think

Yes.

> and we have zero test support for it.

This part isn't entirely true.

The full picture is a bit more complex. Some components have supported
full downgrades with automated tests running on the infrastructure for
years. Examples are the cookie manager and the permission manager.
Other components have intentionally decided that it's OK to not maintain
backwards compatibility, for example, IndexedDB. In other places,
people (myself included) have just not been careful enough and have
(either intentionally or unintentionally) landed code that isn't
downgrade compatible.

It's certainly true that in general there is no tests that would catch
you if you land code that breaks a downgrade scenario, and over the
years it has become quite clear that maintaining a downgrade-compatible
browser is impractical.

I agree with Ben, Jan and others that this situation is unsustainable,
and we have to do something about it. I also agree with Xidorn about
how bad it is that we risk destroying data from the very subset of our
user population who go out of their way to help us by testing changes on
newer builds and risk losing their data when going back to an older build.

To be perfectly honest, the situation in bug 1246615 is quite
frustrating. I no longer understand what we are waiting for in that
bug. It seems like that bug is being scope creeped into a larger plan
(based on comment 29 there) and still unclear what that exactly means,
and here we are now: while Firefox 55 rides the trains, we *will*
destroy our community's browsing data as they help us test Firefox.

Benjamin, can we please address this with the urgency it deserves?
Thank you for your attention.

Cheers,
Ehsan

[rne...@mozilla.com]

> As an example of why "backup the db" is harder than it sounds, you would
> need to backup the entire storage subsystem.

Mossop suggested I chime in on this thread, but I think Ben has covered much of what I'd say!

Some set of schema changes is safe or possible to downgrade: rearranging deckchairs, reversible optimizations, etc.

Some additional set can be downgraded thanks to semantic restrictions in place: adding a purely optional field, for example. (That doesn't mean that it's trivial to make SQLite or another disk format behave correctly, of course, and one can't always tell when an optional field isn't really optional in every circumstance….)

A broader set is very difficult to downgrade. Code to get the downgrade right is hard to fix in the wild and hard to test. Systems that spread data across multiple data sources (multiple files, prefs, caches…) get really hairy.

Leaving old files on disk is almost never the correct solution: think of the effect that this has on Sync, for example. If you're thinking about leaving files on disk for anything other than disaster recovery, you should reconsider.

In short: it's hard to make non-trivial changes to a schema, particularly in SQLite, and expect to be able to hop between versions. The tradeoff in effort and bugs versus benefit is slanted. If we're communicating to our users that they can safely do so — "just try this out on Nightly with your main profile!" — we should stop doing so.

Two alternatives to consider:

- Support two storage versions in parallel for a while: ship the code and use the old one until the switch is flipped. This is acceptable if you're not in a rush to ship. Hands up if you're not in a rush to ship.
- Don't allow downgrades.

The difficulty of evolving storage was one of the motivations for Project Mentat. Schema fragments in Mentat can evolve additively without a version bump if the additions are safe. Clients can introspect the schema and operate in a degraded mode if they can get what they need. Additive fragments don't affect older fragments at all. And a higher-level schema makes some of the DB churn we see in our Firefoxes less likely.

Storage is hard.

[Benjamin Smedberg]

I apologize for the delays in bug 1246615. It fell off my radar with a
series of trips. I've set up a meeting tomorrow to at least identify who is
responsible for this decision, because it's not exactly me. I analyzed some
data in the bug which I'll repeat here.

Here is an analysis of the patterns of people downgrading:
https://gist.github.com/bsmedberg/1af70857106bfe29128a0e8d0b656804 - this
analysis was reviewed by chutten

users that switched channels at all: 0.48%
users that reverted to an older version: 2.55%
users that reverted to an older version, staying on the release channel:
2.19%

Downgrades are more prevalent than I thought, and channel-switching is not
a primary factor in downgrades. So the ideas that we would mostly solve
this problem by giving channels separate profiles appears to be false.

I still believe that we should do our best to make some set of downgrades
seamless for the user (if perhaps with some reversion to prior data or
duplicating). And that showing people a banner warning is user-hostile in
the sense that if they downgraded it was probably for a reason and so they
aren't going "back". What I'd like is some clearer guidance about how much
we should support downgraders and how much engineering effort we should put
into making downgrade as painless as possible.

--BDS


On Wed, Mar 8, 2017 at 9:40 AM, Ehsan Akhgari <ehsan....@gmail.com>
wrote:

[Justin Dolske]

It would be useful to know if most of those users are just downgrading the
previous version (N-1 for release, N-7 for ESR), and how quickly they
return to a current release (assuming they do!).

This would help inform to what degree we need to support (and test!) the
myriad combinations of upgrade-downgrade-reupgrade combinations.

I also wonder how much of user-reversion is driven by addon incompatibility
(which will improve in Firefox 57 with WebExtensions), but I'm not sure how
we would answer that.

Justin

On Thu, Mar 16, 2017 at 12:15 PM, Benjamin Smedberg <benj...@smedbergs.us>
wrote:

> _______________________________________________
> dev-platform mailing list
> dev-pl...@lists.mozilla.org

> https://lists.mozilla.org/listinfo/dev-platform
>

[rne...@mozilla.com]

On Thursday, March 16, 2017 at 12:16:24 PM UTC-7, Benjamin Smedberg wrote:

> users that reverted to an older version, staying on the release channel:
> 2.19%

Benjamin, two further questions, one of which I think you can answer, and one you probably can't:

- As Dolske asked: what proportion downgraded more than a single version (e.g., hopping from 51 back to 47)?
- What proportion downgrade across a storage version boundary at all?

Supporting downgrades of a single version has possible solutions that supporting multiple version downgrades does not.

And if it turns out that -- whether by luck or design -- we tend not to ship features that annoy/break users at the same time as storage version changes, then that changes the math.