Backups for Firefox OS

[Frederik Braun]

Hi list,

I am planning to spend a bit of my Q3 time to sketch out how to do
proper backups for Firefox OS. I can't believe nobody else has already
looked into this, so I'm looking to synchronize with you :)

I guess most people currently either use the backup_restore_profile.sh
from the TWQA team[1] or pull selective files [2]. Which is sort-of
reliable as long as you don't move to previous versions[3].

I had a small chat with Fabrice last week and we thought about data
exfiltration and serialization. But doing that in a storage agnostic way
(for session/localeStorage, IndexedDB and what not) for every app could
be very cumbersome. Even more so if this would mean for the backup
service to execute script in arbitrary apps[4].

But let's step back for a moment and think about use cases.
* back-up the whole system. i.e., everything that is required to do a
full recovery from whatever broken state the phone is in.
* all apps and settings, not the system.
* apps selectively? (this might conflict with the intent not to execute
scripts within other apps. Or maybe through a "give me your backup
data"-call the app is responsible to support?)


Maybe you can up with some more?


Thanks,
Freddy




[1] <https://github.com/Mozilla-TWQA/B2G-flash-tool/>
[2] What I've seen so far is "adb pull /data/local; adb pull /data/b2g;
adb pull /system/b2g/webapps"
[3] IndexedDB schemas usually don't go back.
[4] This is a security nightmare. We really don't want to go there.

[Rewarp]

As someone dogfooding the latest dev branch, and not being a developer, I think backing up all apps (with settings) as well as phone settings as far as is reasonable would be most useful.

Thanks for taking the lead.

--
Bitmessage me BM-2D97QV3Y4VRfjayrWahqE61XKuQw6dJvj6

_______________________________________________
dev-b2g mailing list
dev...@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-b2g

[Natalia Martinez-Winter]

not a developer either, I thought back-up&restore would be part of
"find my device" type of service.
was hoping that all previous installed apps could be retrieved thanks
to Firefox account ?

On Thu Jul 24 15:29:05 2014, Rewarp wrote:
> As someone dogfooding the latest dev branch, and not being a
> developer, I think backing up all apps (with settings) as well as
> phone settings as far as is reasonable would be most useful.
>
> Thanks for taking the lead.
>

> /--
> Bitmessage <http://bitmessage.org/> me/
> *BM-2D97QV3Y4VRfjayrWahqE61XKuQw6dJvj6*

>
>
> On 24 July 2014 07:31, Frederik Braun <fbr...@mozilla.com

> dev...@lists.mozilla.org <mailto:dev...@lists.mozilla.org>
> https://lists.mozilla.org/listinfo/dev-b2g

>
>
>
>
> _______________________________________________
> dev-b2g mailing list
> dev...@lists.mozilla.org
> https://lists.mozilla.org/listinfo/dev-b2g

--
Mozilla Foundation <https://www.mozilla.org/en-US/firefox/os/>
*Natalia Martinez-Winter*
EMEA Marketing Manager
Firefox OS

nat...@mozilla.com <mailto:#>
+33 6 888 11 959 <tel:+33688811959>
natalia...@skype.com
twitter : @martinezwinter

[Jonas Sicking]

On Thu, Jul 24, 2014 at 5:31 AM, Frederik Braun <fbr...@mozilla.com> wrote:
> But let's step back for a moment and think about use cases.
> * back-up the whole system. i.e., everything that is required to do a
> full recovery from whatever broken state the phone is in.
> * all apps and settings, not the system.
> * apps selectively? (this might conflict with the intent not to execute
> scripts within other apps. Or maybe through a "give me your backup
> data"-call the app is responsible to support?)

This sound like great usecases.

I don't think we'd need to run scripts within any apps in order to
allow the user to pick which apps to back up. Currently the
application registry lives as a JSON file which means that you can
read it fairly easily as long as you have file access.

Are you looking at some solution for end-user usable backup, or
something just for developers?

/ Jonas

[Fabrice Desré]

On 07/24/2014 07:10 AM, Natalia Martinez-Winter wrote:
> not a developer either, I thought back-up&restore would be part of "find
> my device" type of service.

It could absolutely be part of the remote commands you can send to a
lost phone, but that's not the only situation you may want to backup.

> was hoping that all previous installed apps could be retrieved thanks to
> Firefox account ?

No, because you need a firefox account on the marketplace only to
install payed apps. And of course you can install apps that are not from
the marketplace.

Fabrice
--
Fabrice Desr�
b2g team
Mozilla Corporation

[Frederik Braun]

Long-term, I think we need to provide a backup solution that is usable
for end-users. But I think the first iterations could be a developer thing.

> / Jonas
>

[Frederik Braun]

On 05.08.2014 06:42, Jonas Sicking wrote:
> On Thu, Jul 24, 2014 at 5:31 AM, Frederik Braun <fbr...@mozilla.com> wrote:
>> But let's step back for a moment and think about use cases.
>> * back-up the whole system. i.e., everything that is required to do a
>> full recovery from whatever broken state the phone is in.
>> * all apps and settings, not the system.
>> * apps selectively? (this might conflict with the intent not to execute
>> scripts within other apps. Or maybe through a "give me your backup
>> data"-call the app is responsible to support?)
>
> This sound like great usecases.
>
> I don't think we'd need to run scripts within any apps in order to
> allow the user to pick which apps to back up. Currently the
> application registry lives as a JSON file which means that you can
> read it fairly easily as long as you have file access.

Sorry, I meant to comment here as well: Are there docs or source files
you want to recommend me looking into?

>
> Are you looking at some solution for end-user usable backup, or
> something just for developers?
>

> / Jonas
>

[Paul Theriault]

On 5 Aug 2014, at 5:42 pm, Frederik Braun <fbr...@mozilla.com> wrote:

> On 05.08.2014 06:42, Jonas Sicking wrote:
>> On Thu, Jul 24, 2014 at 5:31 AM, Frederik Braun <fbr...@mozilla.com> wrote:
>>> But let's step back for a moment and think about use cases.
>>> * back-up the whole system. i.e., everything that is required to do a
>>> full recovery from whatever broken state the phone is in.
>>> * all apps and settings, not the system.
>>> * apps selectively? (this might conflict with the intent not to execute
>>> scripts within other apps. Or maybe through a "give me your backup
>>> data"-call the app is responsible to support?)
>>
>> This sound like great usecases.
>>
>> I don't think we'd need to run scripts within any apps in order to
>> allow the user to pick which apps to back up. Currently the
>> application registry lives as a JSON file which means that you can
>> read it fairly easily as long as you have file access.
>
> Sorry, I meant to comment here as well: Are there docs or source files
> you want to recommend me looking into?

Pretty sure its: /data/local/webapps/webapps.json

>
>>
>> Are you looking at some solution for end-user usable backup, or
>> something just for developers?
>>
>> / Jonas
>>
>

[Julien Wajsberg]

Le 24/07/2014 14:31, Frederik Braun a �crit :

> Or maybe through a "give me your backup
> data"-call the app is responsible to support?)
>
>

I like this idea.

[Frederik Braun]

On 05.08.2014 11:49, Julien Wajsberg wrote:
>
> Le 24/07/2014 14:31, Frederik Braun a �crit :

>> Or maybe through a "give me your backup
>> data"-call the app is responsible to support?)
>>
>>
>
> I like this idea.
>

But do you think it's likely, third-party apps will support this?

[Julien Wajsberg]

Le 05/08/2014 12:01, Frederik Braun a �crit :

We'll need to check how other OS are doing this, but I fail to see how
to do it differently. Either the backup application knows about the
internal format (and then it's application specific) or the application
can save/restore its data in a generic way. I'm not saying it's easy,
especially for forward compatibility...

[Paul Rouget]

Frederik Braun wrote:
> On 05.08.2014 06:42, Jonas Sicking wrote:
> > On Thu, Jul 24, 2014 at 5:31 AM, Frederik Braun <fbr...@mozilla.com> wrote:
> >> But let's step back for a moment and think about use cases.
> >> * back-up the whole system. i.e., everything that is required to do a
> >> full recovery from whatever broken state the phone is in.
> >> * all apps and settings, not the system.
> >> * apps selectively? (this might conflict with the intent not to execute

> >> scripts within other apps. Or maybe through a "give me your backup

> >> data"-call the app is responsible to support?)
> >

> > This sound like great usecases.
> >
> > I don't think we'd need to run scripts within any apps in order to
> > allow the user to pick which apps to back up. Currently the
> > application registry lives as a JSON file which means that you can
> > read it fairly easily as long as you have file access.
> >

> > Are you looking at some solution for end-user usable backup, or
> > something just for developers?
> >
>

> Long-term, I think we need to provide a backup solution that is usable
> for end-users. But I think the first iterations could be a developer thing.

If the audience for now is mostly developers, maybe you want to build
the backup mechanism within WebIDE. See:
https://groups.google.com/d/msg/mozilla.dev.gaia/wX_3Db1q2fg/2J01fhfExJgJ

We already have buttons to unlock devices (http://i.imgur.com/TxEWB0t.png).

-- Paul

[Fabrice Desré]

On 08/05/2014 04:12 AM, Julien Wajsberg wrote:
>
> We'll need to check how other OS are doing this, but I fail to see how
> to do it differently. Either the backup application knows about the
> internal format (and then it's application specific) or the application
> can save/restore its data in a generic way. I'm not saying it's easy,
> especially for forward compatibility...

I thought a bit more abut that since I discussed with Frederik and I now
believe that we should try to do something that is 1) user oriented
(devs can already use the various scripts laying around) and 2) that
doesn't need any application specific code.

The way I see it is that all apps that are saving data locally do it
through a storage mechanism that is known to the platform, be it
cookies, localStorage or indexedDB. So we could provide platform-level
application backup by exporting/importing data relevant to a given app.
Anything else is likely managed "in the cloud" by the app itself.

We also have some non-app content: chrome databases (like contacts, sms)
and any data that is saved by web pages. It seems clear we want to
backup the chrome ones when doing a "system backup", but I'm not sure
about the web ones.

We should expose that trough a new webapi that would let 3rd party write
backup/restore apps, and for instance use it in find my device.

[Julien Wajsberg]

Le 05/08/2014 17:15, Fabrice Desr� a �crit :

> On 08/05/2014 04:12 AM, Julien Wajsberg wrote:
>> We'll need to check how other OS are doing this, but I fail to see how
>> to do it differently. Either the backup application knows about the
>> internal format (and then it's application specific) or the application
>> can save/restore its data in a generic way. I'm not saying it's easy,
>> especially for forward compatibility...
> I thought a bit more abut that since I discussed with Frederik and I now
> believe that we should try to do something that is 1) user oriented
> (devs can already use the various scripts laying around) and 2) that
> doesn't need any application specific code.
>
> The way I see it is that all apps that are saving data locally do it
> through a storage mechanism that is known to the platform, be it
> cookies, localStorage or indexedDB. So we could provide platform-level
> application backup by exporting/importing data relevant to a given app.
> Anything else is likely managed "in the cloud" by the app itself.
>
> We also have some non-app content: chrome databases (like contacts, sms)
> and any data that is saved by web pages. It seems clear we want to
> backup the chrome ones when doing a "system backup", but I'm not sure
> about the web ones.

So, say, the user selects the "Contacts" app. How do you decide you need
to take away the datastore for facebook contacts and the idb for
mozContacts?

Another question is: how do you manage restoring to an earlier version ?
I don't think it would generally work without help from the app itself...

[Fabrice Desré]

On 08/05/2014 08:21 AM, Julien Wajsberg wrote:

>
> So, say, the user selects the "Contacts" app. How do you decide you need
> to take away the datastore for facebook contacts and the idb for
> mozContacts?

The contacts app only comes with its own datastores. We could see that
this app needs the contacts permission and use that to hint the user
that he should also opt-in to backup the chrome mozContacts.

> Another question is: how do you manage restoring to an earlier version ?
> I don't think it would generally work without help from the app itself...

I'm not sure to understand the use case here. Can you elaborate? In
general we don't support downgrades, but starting from scratch with an
older version will work thanks to the indexedDB upgrades.

[Julien Wajsberg]

Le 05/08/2014 18:42, Fabrice Desr� a �crit :

>
>> Another question is: how do you manage restoring to an earlier version ?
>> I don't think it would generally work without help from the app itself...
> I'm not sure to understand the use case here. Can you elaborate? In
> general we don't support downgrades, but starting from scratch with an
> older version will work thanks to the indexedDB upgrades.
>

I thought this is in Frederik's initial requirements for restoring. I
don't really mind.

--
Julien