Lightning and ical on http:// shares
Ulrich Spoerlein
we are using eGroupware 1.4 (eGW), which has rudimentary ical support
and encountered weird problems with a recent nightly version of
Lightning (2007-07-31) on Thunderbird 2.x.
Perhaps our understanding on what *should* happen is totally wrong, I
hope someone can enlighten me.
This is what happens:
Lightning periodically requests (GET) the ical URL and gets a complete
listing with an ETag containing the cal-last-modified value of eGW.
Lets assume ETag=10 and there are four events.
Now if someone enters an event through the web frontend, the eGW
database stores the last modified value of, eg, 20. There are now five
events in the DB, only four of which Lightning is aware of.
If I now delete an event from within lightning, it's doing:
- GET -> eGW returns five events and ETag 20
- PUT -> Lightning sends three (!!) events with an If-Header
containing ETag 10. Why on earth is it running a GET just before the
put, but ignores the new data?
- PROPFIND -> eGW doesn't support this and will send a 501 error.
We hacked the eGroupware to block such PUTs, where the IF header is
unequal to the cal-last-modified value, but I think Lightning is
unnecessarily sending the wrong data here.
I can provide traces of the communication, if required.
Is my assumption flawed? Or is it just a plain old bug in Lightning?
Cheers,
Uli
Michiel van Leeuwen
> - GET -> eGW returns five events and ETag 20
The GET is used to retrieve the most recent version from the server, in
order to make a backup of that. Nothing else. (why not? because nobody
wrote code to do anything with it)
> - PUT -> Lightning sends three (!!) events with an If-Header
> containing ETag 10. Why on earth is it running a GET just before the
> put, but ignores the new data?
> - PROPFIND -> eGW doesn't support this and will send a 501 error.
The problem with the etag is that you need to do a propfind to get a
proper etag from apache+mod_dav. If eGW doesn't support propfind,
lightning doesn't know a new etag. How could one get a new etag from eGW
(preferably in a way that doesn't conflict with the way to get an etag
from apache)
Michiel
Ulrich Spoerlein
> Ulrich Spoerlein wrote:
> > - GET -> eGW returns five events and ETag 20
>
> The GET is used to retrieve the most recent version from the server, in
> order to make a backup of that. Nothing else. (why not? because nobody
> wrote code to do anything with it)
How hard would that be? And do you think this would solve my problem
nicely, or are there additional pitfalls?
> > - PUT -> Lightning sends three (!!) events with an If-Header
> > containing ETag 10. Why on earth is it running a GET just before the
> > put, but ignores the new data?
> > - PROPFIND -> eGW doesn't support this and will send a 501 error.
>
> The problem with the etag is that you need to do a propfind to get a
> proper etag from apache+mod_dav. If eGW doesn't support propfind,
> lightning doesn't know a new etag. How could one get a new etag from eGW
> (preferably in a way that doesn't conflict with the way to get an etag
> from apache)
Hmm, but lightning *is* sending the old ETag 10. So it is groking the
ETag even without PROPFIND working. It's just using an ETag of a
"normal" GET instead of the ETag of the GET immediately before the PUT.
That's what's weird, at least to me.
Cheers,
Ulrich Spoerlein
--
It is better to remain silent and be thought a fool,
than to speak, and remove all doubt.
Michiel van Leeuwen
> On Sat, 04.08.2007 at 11:42:04 +0200, Michiel van Leeuwen wrote:
>> The GET is used to retrieve the most recent version from the server, in
>> order to make a backup of that. Nothing else. (why not? because nobody
>> wrote code to do anything with it)
>
> How hard would that be? And do you think this would solve my problem
> nicely, or are there additional pitfalls?
Dunno. Never tried it. But it would involve adding more async operations
in the provider code. And the code is already a spaghetti mess of async
operations, so i'm not really keen to try it.
Besides, the GET before uploading is used by the backup system, which is
pretty independent of the rest of the provider.
> Hmm, but lightning *is* sending the old ETag 10. So it is groking the
> ETag even without PROPFIND working. It's just using an ETag of a
> "normal" GET instead of the ETag of the GET immediately before the PUT.
> That's what's weird, at least to me.
Looking at the code, it seems the lightning does try to get an etag from
the result of the PUT. does eGW send that etag? If not, how else could
an etag be found? We need the etag right after the first put, not right
before the second put.
In fact, thinking about that, using the etag of the get before the put
would be useless, and in fact be wrong: it would mean that any changes
in the etag (and thus in the file!) are not detected anymore when trying
to upload: dataloss when a third party changed the file.
Michiel
PS: No need to Cc my personal address on replies. I will see the reply
anyway.
Ulrich Spoerlein
> > Hmm, but lightning *is* sending the old ETag 10. So it is groking the
> > ETag even without PROPFIND working. It's just using an ETag of a
> > "normal" GET instead of the ETag of the GET immediately before the PUT.
> > That's what's weird, at least to me.
Oops, s/sending/seeing/
> Looking at the code, it seems the lightning does try to get an etag from
> the result of the PUT. does eGW send that etag? If not, how else could
> an etag be found? We need the etag right after the first put, not right
> before the second put.
Ok, I'm confused. What second PUT are you referring to? Lightning needs
to know that something has changed by a third party *before* it is
PUTting the modified ics, no? Or is it the responsibility of the server
to DTRT?
I really don't know anything about WebDAV or how this synchronization is
supposed to work. I'm only faced with the task of hacking eGW to get
this stuff working on a basic level.
> In fact, thinking about that, using the etag of the get before the put
> would be useless, and in fact be wrong: it would mean that any changes
> in the etag (and thus in the file!) are not detected anymore when trying
> to upload: dataloss when a third party changed the file.
But the GET is immediately followed by the PUT. The timeframe of lost
updates is very small and, for me, acceptable.
Telling people to manually fetch the calendar, before they make updates
to it is not a very appealing notion. Setting the fetch interval to one
minute is also not a good solution, IMHO.
What I'm really looking for is for Lightning to refresh its calendar
immediately before it is PUtting something. I thought the GET mentioned
above was responsible for it, but if it is only doing backups, oh well.
How should the server send/update ETags and what is it supposed to do,
if the PUT has an old ETag set? Btw, is using
time-of-last-cal-modification suitable for the ETag?
> PS: No need to Cc my personal address on replies. I will see the reply
> anyway.
Sorry.
Bruno Browning
Looks to me like Lightning is collecting the etag from the PUT response,
and PROPFINDing to get the etag only if the PUT response doesn't provide
one. Which probably never happens: Mozilla PROPFIND was broken for a
good bit of last year and ICS-provider users never noticed. We probably
could use a HEAD request there instead of a PROPFIND, for interop with
non-DAV servers such as eGW seems to be. But again, that catch() is
seldom if ever going to be executed. So it seems to me that the issue is
not the PROPFIND, nor even really the etag. It's the lack of code to
merge deltas from those periodic GETs. Bug 329570.
bb
Michiel van Leeuwen
> Ok, I'm confused. What second PUT are you referring to? Lightning needs
> to know that something has changed by a third party *before* it is
> PUTting the modified ics, no? Or is it the responsibility of the server
> to DTRT?
It's indeed the job of the server to do that. ETag in webdav works like
this: at some point, the client gets an ETag of a document from the
server, aither as result of a GET, PUT or PROPFIND. That ETag is some
string, indicating the version of the document. It might be a hash of
it, for example.
Then, some time later, the client tries to PUT a new version of the
document on the server. If it wants to be sure that it doesn't overwrite
any changes made on the server, it can send the old ETag with the PUT.
If that ETag still matches the document on the server, nothing was
changed between the time the client got the ETag and the time the
clients wants to upload a new document: no dataloss. If the ETag does
not match, the document will not be put on the server, and the PUT
request fails.
This is what lightning does (except that it doesn't react nicely on a
failed PUT: it just puts the entire calendar in read-only. Safe, but not
really user-friendly. But that is what should happen in the case you are
describing)
> But the GET is immediately followed by the PUT. The timeframe of lost
> updates is very small and, for me, acceptable.
What matters is the ETag of the old ics file, the file that was last
parsed by lightning. And that's the version after the previous upload.
Lightning needs to know that (old) ETag, not the ETag that the file
currently has on the server.
What seems to be missing in your eGW case seems to be the ETag of the
document after the first time the ics file is uploaded. Lightning
currently tries to get that etag by looking at the result headers of the
PUT, and if there is no ETag there, it will do a PROPFIND. But if eGW
doesn't return the ETag on put, and does not support propfind, there is
no ETag for lightning to use, and things can go wrong.
> What I'm really looking for is for Lightning to refresh its calendar
> immediately before it is PUtting something. I thought the GET mentioned
> above was responsible for it, but if it is only doing backups, oh well.
Well, that's something of a synchronization problem. Lightning has a
change (otherwise it won't upload), and the remote might have a change
too. Those changes need to merged in the right way. And that's not
trivial, because there might be conflicts. Simply downloading the file
won't do it. Work is going on to make that work, but that's going slowly
lately. There is so much work to so, and so little time...
Michiel
Michiel van Leeuwen
> Looks to me like Lightning is collecting the etag from the PUT response,
> and PROPFINDing to get the etag only if the PUT response doesn't provide
> one. Which probably never happens: Mozilla PROPFIND was broken for a
> good bit of last year and ICS-provider users never noticed. We probably
afaik, it was only broken on trunk, not on branch. And most users are
using branch builds, so it just worked for them. Or am i missing something?
(I'm thinking of the check in
http://lxr.mozilla.org/seamonkey/source/calendar/providers/ics/calICSCalendar.js#925)
> could use a HEAD request there instead of a PROPFIND, for interop with
> non-DAV servers such as eGW seems to be.
Iirc, there was some problem with apache in some cases. That's why the
propfind was needed. But i can't find the details of that anymore :(
Michiel
Bruno Browning
> afaik, it was only broken on trunk, not on branch. And most users are
> using branch builds, so it just worked for them. Or am i missing something?
> (I'm thinking of the check in
> http://lxr.mozilla.org/seamonkey/source/calendar/providers/ics/calICSCalendar.js#925)
I was thinking of
https://bugzilla.mozilla.org/show_bug.cgi?id=355813
But you're right: that breakage looks to have been trunk only. My mistake.
> Iirc, there was some problem with apache in some cases. That's why the propfind was
> needed. But i can't find the details of that anymore :(
Not sure what that would have been, either. The issue wrt Apache
If-None-Match and PUT still seems to be around. :( But that's hardly
apropos. My highly non-rigorous (one HEAD and one PROPFIND) testing
showed both delivering the same etag from Apache 2.2.
bb