Roaming Profile Architecture
John Keiser
general architecture. We've put up a prelim doc at
http://mozilla.freezope.org/PRM/PRM02 for perusal. Following is a
summary and some design ideas that are not all in the doc yet.
In general we're trying to support roaming that allows changes to be
synced during the use of the application but also does wholesale
transfer at the beginning and the end. To do this we'll keep a list of
persistent resources around and where they come from--you could
conceivably add more persistent resources to the list.
The basic architecture is, we have a "sync master" that keeps a list of
resources (the actual in-memory data or pointers to it) and the data
sources they must be mirrored to (the user's FTP server, for example).
On startup each of these resources are brought down; on shutdown they
are sent up; in between the user can choose to synchronize them. As to
storage format, the resources and data sources decide amongst themselves
how the data will be stored in the server. The syncmaster just gets
them together and tells them which direction to transfer. This allows
us to work generically with FTP or HTTPS while storing things very
specifically in ACAP.
The way the syncmaster stores the list of resources and how they are
identified remotely and locally is an important point--you have to be
able to mirror resources on the disk even if they come from ACAP, so
that the user can still work offline. So here's how things are stored
and identified.
The resource broker is a list of resource entries, each with a name,
pointer to resource object, and pointer to data source, and a flags
identifying whether the resource should be mirrored to disk. The name
uniquely identifies the resource and is also the name that will be used
as the filename on the disk or on file-based protocols (ftp and http).
"prefs.js" or "bookmarks.html" would be names of two of our more
prominent resources. Dynamic resources can be created and given any
name they want--my dynamic bookmark folder idea could do
"bookmarks/folder_name.html" or some such thing--and they can be written
to disk or not depending on the flag mentioned earlier.
This last info is not yet in the doc, but this is the idea :)
What do y'all think?
--John
Alec Flett
man of profiles) dougt (who has visions for nsIFile) and john gaunt (who
is taking on profile sharing)
I can't seem to reach mozilla.freezope.org, but I don't see any mention
below of the directory service - I think that is a key part of roaming -
the directory service needs to have some mechanism to provide remote
files to a consumer, and some means by which you can plug into the
directory service to actually provide those files.
if this is being taken up in any serious way, I suggest you create some
pages under http://www.mozilla.org/projects/roaming/
Also, can I object to the "PRM" - the "Profile Roaming Manager" - if
we're giving it a name in english, we can at least follow english
conventions where the adjective (Roaming) comes before the noun
(Profile/Profile Manager) - Roaming Profile Manager seems reasonable to
me. Even the subject of this message is "Roaming Profile Architecture"
:) since RPM is already taken, how about some joke on roaming, such as
some nomadic animal...(According to some page I found on google, the
Peregrine falcon is used by the nomadic peoples of Mongolia?)
Alec
Doug Turner
In either case, the idea would be to implement a nsIFile interface that
instead of knowning about local file systems, it would know about how to
access a file on a ftp, http, ldap, ect. server. In theory, it would be
possible to populate the directory service with these nsIFile's and have
existing code know nothing about where the actual file comes from. Of
course, hitting a nsIFile that references a slow resource may block, so
this solution may be less than ideal.
I also think that there may be one gotcha which wasn't fully thought
out. We should have moved the open() into the nsIFile interface. If it
was, we wouldn't need the nsILocalFile or a new interface to support his.
As for a name.... how about Transitional Profiles? Imagine the logo for it:
http://thefatties.com/images/TP/wide.gif :-)
--
Doug Turner
do...@netscape.com
John Keiser
--John
Doug Turner wrote:
Visions or hallucinations? :-)
In either case, the idea would be to implement a nsIFile interface that instead of knowning about local file systems, it would know about how to access a file on a ftp, http, ldap, ect. server. In theory, it would be possible to populate the directory service with these nsIFile's and have existing code know nothing about where the actual file comes from. Of course, hitting a nsIFile that references a slow resource may block, so this solution may be less than ideal.
This is why we are considering more specific "roaming protocol" implementations that handle synchronization--they know about the protocol and about resources and they figure out how to store them on the server. They can instantiate nsIFile underneath when on local file, FTP or HTTP. On ACAP they can ask the resource to give them name/value pairs instead of a big file representing all their data (and this is where they can ask the resource for only their changed data, BTW).
I also think that there may be one gotcha which wasn't fully thought out. We should have moved the open() into the nsIFile interface. If it was, we wouldn't need the nsILocalFile or a new interface to support his.
As for a name.... how about Transitional Profiles? Imagine the logo for it:
Alec: the directory service is entirely file-based, and frozen. It may be good to extend for FTP and HTTP as well, and use it. But for ACAP, again, we run into problems. At its base, that is not a file-based protocol--and even if one considers it file-based, it is not the same granularity as our profile files. You store individual prefs and individual bits of information on the ACAP server. It would be a waste to require ACAP to export that data as a virtual file and take data in as a file--it would have to do unnecessary parsing and thinking. This is why we would have (for example) an nsIACAPSerializable interface on the resource that ACAP can call and get back the list of namespace/name/value tuples that it will dump on the server.
In this architecture so far, the roaming protocol object is similar to the directory service in that you can ask it to bring down or send up multiple resources (which often correspond to files)--on file-based implementations, the roaming protocol represents a directory on the server the way nsIDirectoryService does. It could, again, use nsIDirectoryService underneath.
--John Keiser
Alec Flett
You're right, we should extend nsIFile for FTP and HTTP, and I think we should use that as part of the implementation. However, ACAP is not a file-based architecture and it is not advantageous to store all our stuff as files there. On that protocol there are a number of reasons to store things piecemeal, not the least of which is you can share configuration with other apps if you store things with specific semantic knowledge of what they are.
However, I think that bending roaming around ACAP and forcing all roaming clients (i.e. a client such as 'bookmarks' or 'history') into this model now is going to be very hard to do right the first time. I think that we can successfully design a file-based roaming system, always keeping ACAP in the back of our minds, and reach the finer granularity in the next rev.
As Phil Peterson used to say, "Crawl, walk, run"
Here's the way I look at this (I've been honestly thinking about this 'problem' for years) - A first cut is file-based - a client (i.e. bookmarks) says "give me the bookmarks file" - and through one form or another this file is retrieved (either pre-fetched, or retrieved in some non-blocking manner) - and then it is consumed. This wouldn't be too bad and the work would be mostly confined to the roaming project/module.
Down the line, there needs to be a mechanism by which specific clients can retrieve fragments of data from their datastore, and store them in a similar manner. The question is, does the roaming code become aware of these clients data, feed it to them, and observe changes, or does each client get tweaked to speak a common protocol back to the roaming service. If that's the case, what 'protocol' do you use? We have some good ones - RDF, nsIProperties, you name it.
But no matter what happens, it means dumping the 'file' implementation for that client, and also providing a local non-roaming implementation of this granular service. This means tweaking each client that stores profile-specific data.
Now if you look at the bookmarks and history code, you'll see that restructuring them to provide granular access is no easy task, let alone the work of providing a consistent local and roaming data storage mechanism for each of these clients. It really is a lot of work.
Here's my point: you can spend 9 months hacking away at making a roaming service that supports HTTP, FTP, and ACAP at the end of those 9 months, or you can spend 3 months getting a file-based roaming service (HTTP/FTP) and then spend another 6 months hooking up ACAP.
Besides, the knowledge you gain bringing the file-based solution through to a working project will be invaluble when you go to the ACAP work. It may help you answer the questions I posed above about what 'protocol' to use, what mechanism you will have already used to provide pre-fetching or non-blocking retrieval, etc.
Basically, don't over-engineer this thing before you've done some implementation... besides, you may find that if you get the file-based implementation working well the first time, then the owners of some of the potential clients might help you out when it comes to adding ACAP support, because you've given them a working proof of concept.
Methinks a good way to do this would be to have a factory method like CreateAndOpen() that takes a URL. Perhaps we should do further and allow an open connection to be passed in--perhaps a directory service could do this--so that one does not have to create a new HTTP or FTP connection for every file.
Alec
Scott
and I'm going to call it :
ZillaVilla
I would like to be able to offer different levels of service, from just
backing up bookmarks or address book, up to everything that can live in
the prefs folder like chrome and messages, browsing prefs... the whole
ball of wax.
I want to be able to provide a usage cap and make it easy for users to
see how much space syncing bookmarks would take vs email messages for
example.
To keep it simple, I'd like to just give users an FTP folder that they
can sync from, to keep it secure, I'd like Mozilla to be able to encrypt
data client side so only the user could unlock it even if someone else
gets the data.
Alas, I'm not a programmer, but would like to contribute in any way I
can. I've made some notes on my site regarding related bugs and issues
with profile backup but it's nothing new just combining what I've found
so far.
http://zillavilla.com
let me know if I can setup perhaps a test server or ftp acount if
someone has a need for that.
John Keiser
run. Build room in your architecture to accomodate future steps, even
if you don't build them yet. And in this case building that room costs
us pretty much nothing, so I think we ought to do it :)
Even without ACAP, though, there is need for some new interfaces and
architecture we don't already have.
I believe nsIDirectoryService and nsIFile are a really big part of the
job (a large portion of the work for FTP and HTTP/HTTPS, the two most
prominent initial targets, can and should be done by making nsIFile work
with those protocols and using that). But I don't think it handles
everything we need. Let's consider some use cases for roaming. I will
deliberately leave off any mention of fine-granular storage :)
(I) User starts browser with roaming profile
For each resource that does roaming (bookmarks, addressbook, ...):
1. Read the data for this resource from the server
1a. The data may be encrypted, decrypt it if so (ask for password)
1b. The data may be from a too-old version of Netscape and the
serialization format has changed, try to read using a format reader of
that type if so
2. IF we got the data from the server, mirror it to the local disk
so we have a local copy (1b could potentially apply)
3. ELSE look for the data on the local disk
For profile resources that do not do roaming:
1. Read the data from the local disk
(II) User shuts down browser from roaming profile
For each resource that does roaming:
1. Write the data for the resource to the server (IF the data has
changed)
1a. Encrypt the data if the user has given a password to encrypt
(a requirement for keeping profile data on servers administered by other
people, which will be the norm rather than the exception)
1b. Write out the format version for the data so that future
versions can check and be backwards-compatible (NOTE: writing data out
when an older version already exists is a sticky problem)
2. Write a copy of the data to the local disk (again, check out 1b)
Other use cases exist: user hits Synchronize or some like button; user
changes where his profile roams from, makes it roaming, makes it
un-roaming. But we'll leave them off for now (I have thought about them
but this here is enough food for thought).
It seems to me that the mirroring requires some other kind of roaming
class so that the mirroring logic is not repeated all over the place.
Further, it seems to me that encryption requires a separate sort of
class as well--the most natural one to me would automatically wrap the
nsIFile's inputstream with decryption and gives that result to the
resource to parse like it normally does. (Most likely the decryption
would be some sort of decrypting input stream so that the consumer could
still use nsIInputStream to read.)
This is what the roaming manager / roaming protocol / roaming resource
objects afford us (these are the objects pointed out in the document
posted earlier). Attached to this message are some *preliminary* IDLs,
with some changes coming down the pipe--most notably mirroring will be
gone from nsIRoamingManager, and I suspect the lastRefreshTime stuff
will be moved or changed (needs more thought, the prime dilemma--or
perhaps not--is the fact that you want to store the refresh time from
the server for later comparison with files on the server, for Synchronize).
Regarding ACAP and fine-grained implementations: the restructuring of
the data sources to provide fine-grained data will be the hard part, not
allowing for it in the architecture (since the way we have chosen to
allow for it is to be agnostic about graininess and let the protocol and
resource handle it together). There are ways we can do that
restructuring gradually, too. In fact, it should be easy.
I have a preliminary cut of this with a fake counter datasource and a
local file implementation, to prove to myself that this could work (it
does). I have a few changes I want to make that will simplify it
further and build as much obviosity into the system as possible, and I
will post it soon for perusal. The IDLs are attached with this message.
When this all gets ironed out, the real task begins: converting existing
datasources (some amount of conversion will be necessary one way or
another) and making nsIFile do FTP and HTTP. We could use help with all
that.
--John
Ed Riede
The Roming Profile would be a very nice feature to add. It would be
nice to be able to store the prefs in a mailbox to syncronize the
configuration files. I do not believe that start up and shutdown are
the times to send the files up or down though. I think that the
'syncmaster' should monitor the files that it was configured to monitor
for changes. When a configuration file changes, the syncmaster should
diff the file and send it to the configred URI with a time stamp. I
believe that most operating systems can send a signal to a task when a
certain file has changed.
What do you think?
John Keiser
If a protocol chooses to monitor for changes, it can do it. Some
protocols, such as ACAP, are designed for this. But startup *has* to be
the time you bring down the entire file, and for protocols that don't
monitor their changes shutdown is an ideal time to send them up.
The major difficulty is that most of our resources are probably unable
to refresh changes--they read in once and that's it. They're not
designed for that. Hackers that want to fix this are welcome, but I
don't see it as a prime requirement for roaming--it's just a Nice Thing
To Have.
--John
John Keiser
when we've got roaming bookmarks up and running and you can help people
be early adopters and test the service :)
Encryption is part of the plan too, yeah. I don't see Roaming Profiles
being as configurable as you are looking for, for a while--at least in
the UI. The ability to decide which things you want to sync is entirely
up to the client, and I don't see us complicating things like that at
first rev. We'll see at first cut how syncing a standard set of stuff goes.
--John
Thomas Dodd
John Keiser wrote:
>The major difficulty is that most of our resources are probably unable
>to refresh changes--they read in once and that's it. They're not
>designed for that. Hackers that want to fix this are welcome, but I
>don't see it as a prime requirement for roaming--it's just a Nice Thing
>
Any pref that can be changed while running (through the
Edit->Preferences dialog)
need to be propagated to other clients. When the OK button is pressed to
exit that dialog, the changes should be stored on the server, and then
proagated to
other clients. just like changes in one browser window propigate to
other browser windows.
The one big problem for me now, is with a share $HOME,
I cannot start mozilla on a second workstation and have both
instances stay in sync. I use mozilla for email filtering, and the
filter rulls, and mailbox status get screwed up, if I leave mozilla
running at my desk (so the filters stay active) and check my mail
on another machine (like the one in the lab, or my home system).
-Thomas