How to partly update a managed resource store?
philt
I was trying to bring the LocalServer to my mediawiki install but I'm
facing the following problem when I want to update my offline store
(e.g. when saving a page):
I wrote a manifest php script such that it sends me only the list of
pages modified since my last visit (I send it my current version as a
GET parameter).
But then locally all the other pages are deleted, only the new ones
are there.
The other option, downloading all the wiki pages every time, is
overkilling.
How can I only refresh a part of the resource store without losing
what's already there?
Thanks!
Phil
Aaron Boodman
> I wrote a manifest php script such that it sends me only the list of
> pages modified since my last visit (I send it my current version as a
> GET parameter).
> But then locally all the other pages are deleted, only the new ones
> are there.
>
> The other option, downloading all the wiki pages every time, is
> overkilling.
> How can I only refresh a part of the resource store without losing
> what's already there?
Hi Phil, very cool idea to offline enable your wiki.
Gears is designed for the server to send all pages in the manifest
each time. For each URL in the manifest, it issues an HTTP conditional
GET. Only the changed pages will be downloaded.
So long as the server supports if-modified-since (which I assume
MediaWiki does), updates should be much faster than the first capture.
- a
philt
> Gears is designed for the server to send all pages in the manifest
> each time. For each URL in the manifest, it issues an HTTP conditional
> GET. Only the changed pages will be downloaded.
> So long as the server supports if-modified-since (which I assume
> MediaWiki does), updates should be much faster than the first capture.
By default Mediawiki honors properly If-Modified-Since except that for
logged users it invalidates the cache for all the pages every time one
page is edited :-(
This is supposedly to refresh all the things around a wikipage
(headers, menus, footers,...) but for our offline version we prefer
not to fetch the whole wiki just because we saved one single page.
So the ugly hack (ugly because not possible from the extension hooks):
modify a function in includes/User.php:
function validateCache( $timestamp ) {
$this->load();
if (isset($_SERVER['HTTP_X_GEARS_GOOGLE']))
return true;
return ($timestamp >= $this->mTouched);
}
This still requires about 30s to query 200 of my wiki pages (instead
of 85s + some more bandwidth for the initial download)
That means after every single edition I've to wait 30s then to reload
the current page to see my changes :-(
Imagine now a wiki of 1000, 2000 pages...
What is crudely missing in the Google Gears API is a way to avoid to
fetch all the pages at each new revision of the manifest.
What I'd like to see:
* One version per URL instead of one per manifest
o Very clean solution, manifest independent of the client
state
* Or sth like that: given the version of the client Store the
manifest could be generated on-the-fly with some:
{ "url": "/path/to/page","refresh": false }
For those interested in the results so far: http://wiki.yobi.be/wiki/Mediawiki_LocalServer
Anyway, thanks for your reply Aaron!
Phil
Brad Neuberg
Best,
Brad
bradn...@google.com
http://codinginparadise.org
Aaron Boodman
> Hi Aaron, before it does a conditional GET though does Gears check the
> manifest file to see if the version string has changed? I thought it does
> that to avoid having to do a bunch of conditional GETs.
Yes, it does, but that won't help here I don't think. The crux of the
problem is that everytime any page on the wiki has changed the entire
wiki has to be recaptured because the server updates the last update
time across the board.
One way to solve this (I suppose the way that this API was meant to be
used) is to separate the truly static resources (images, css HTML
templates, etc) from the data (the contents of the wiki pages). This
is obviously a major refactor of MediaWiki itself, and probably not a
useful answer to Phil.
Longer term, I think one interesting thing to consider for
LocalServer2 is some functionality like what Phil is describing that
would make LS more useful to multi-page sites. Basically: allow a
server to return just the changed pages since a given version. Perhaps
this could be something that could be done programattically: that way
you could call it after pressing the edit button and wait for it to
complete. It is sort of like a very basic sync protocol for static
resources.
There might be something that could be done today using ResourceStore
and hand-rolling this protocol using PHP on the server and
XHR+ResourceStore on the client. This would have the downside that
pages captured using ResourceStore do not automatically get updated,
so if there was an error, that page could get wedged,. But since the
complexity of JS on mediawiki is pretty low, I think this is less of a
risk. Also, you could have a ManagedResourceStore too that just
contains the homepage. That way, if a user ever got wedged, a request
for the homepage would unwedge you.
Does this seem like something that would work, Phil?
Interesting use case. To be honest, we designed Gears more for the
single-page style AJAX applications, where there are only a few HTML
files and the content is generated by JS. But I think your use case is
important too, and you shouldn't have to dramatically redesign things
like MediaWiki to do things like this.
We'll keep it in mind for the next version of LocalServer and
propagate this to the discussions in WhatWG for offline support in
browsers.
- a
philt
> problem is that everytime any page on the wiki has changed the entire
> wiki has to be recaptured because the server updates the last update
> time across the board.
> One way to solve this (I suppose the way that this API was meant to be
> used) is to separate the truly static resources (images, css HTML
> templates, etc) from the data (the contents of the wiki pages). This
> is obviously a major refactor of MediaWiki itself, and probably not a
> useful answer to Phil.
which were not modified) but it wouldn't solve the problem that one
change of one file of the resources will trigger a new Manifest file
version and therefore a query of all the resources.
So meanwhile I prefer my 2 lines hack than rewriting all Mediawiki :-)
> Longer term, I think one interesting thing to consider for
> LocalServer2 is some functionality like what Phil is describing that
> would make LS more useful to multi-page sites. Basically: allow a
> server to return just the changed pages since a given version. Perhaps
> this could be something that could be done programattically: that way
> you could call it after pressing the edit button and wait for it to
> complete. It is sort of like a very basic sync protocol for static
> resources.
having saved a page.
I wrote some proposals for v2 with pros&cons here:
http://wiki.yobi.be/wiki/Mediawiki_LocalServer#LocalServer_v2
I see 2 main options, a delta manifest file or a versioned manifest
file.
> There might be something that could be done today using ResourceStore
> and hand-rolling this protocol using PHP on the server and
> XHR+ResourceStore on the client. This would have the downside that
> pages captured using ResourceStore do not automatically get updated,
> so if there was an error, that page could get wedged,. But since the
> complexity of JS on mediawiki is pretty low, I think this is less of a
> risk. Also, you could have a ManagedResourceStore too that just
> contains the homepage. That way, if a user ever got wedged, a request
> for the homepage would unwedge you.
> Does this seem like something that would work, Phil?
not AJAX programmer, I'm even not much a programmer ;-) )
so I cannot judge the feasibility but if you think that would be a way
to go I can try, with some help...
> Interesting use case. To be honest, we designed Gears more for the
> single-page style AJAX applications, where there are only a few HTML
> files and the content is generated by JS. But I think your use case is
> important too, and you shouldn't have to dramatically redesign things
> like MediaWiki to do things like this.
wikis, cms & blogs around.
Offline functionality is sth I see a lot of people asked for in the
world of wikis, ultimately with local edition capabilities too but
that's another story...
The only solutions we have today are wikis using a real version
management behind, such as git.
> We'll keep it in mind for the next version of LocalServer and
> propagate this to the discussions in WhatWG for offline support in
> browsers.
I hope it'll continue.
Phil
PS: right now my wiki is featuring Google Gears support and you can
try if you want.
But I've still a few TODOs so if it looks broken when you come, that's
because I've my hands in the engine, try a few minutes later ;-)
Scott Hess
If a server modifies a file's lastmod even if the file is not changed,
then I think it's broken. I'm not sure we should spend a ton of
effort working around this, because this kind of thing can be broken
in many different ways. [That said, I can completely understand _why_
things work this way, it is generally easier to just touch everything
than to find the minimal set of things that changed.]
Does Gears support ETag? Of course, it might suffer from the same problems :-).
I think adding a way for the server to return just the modified pages
is merely moving the problem around, because you'll have to teach the
server to do this. I suspect it wouldn't be less complicated than
teaching the server to use Last-Modified or ETag correctly!
IMHO, I'm not sure that a manifest is really the right way to offline
a wiki. To me, a wiki seems like a core set of shared resources which
could be in a ManagedResourceStore, plus a bunch of individual pages
which are managed using a ResourceStore. Then your current manifest
of changes becomes a JavaScript file which edits the appropriate
ResourceStore.
-scott