Complete and warning free AtomPub implementation
Michael C. Harris
APE, but there are still warnings and it is still partially
incomplete. This email is a starting point for discussion as to how
things should be finished, and by sending it I'm really saying I'm
happy to do what work I can. I'll post something similar on my blog
because the wider the discussion, the better Habari's implemenation
will be.
As I see things, these are the outstanding issues that I know about,
in no particular order. Each of these should probably be entered as a
separate issue in the tracker. As I'm not super familiar with the
spec, I've probably missed bits. If any of this is wrong, or it's
incomplete, or you think my solution is off, please let me know.
* Atom and AtomPub share a URL
While this works okay, because Habari uses the presence of
authentication details to tell the difference between a request for a
feed and a request for an AtomPub service document, it relies on the
client sending a correct username and password with the initial
request. If the client wants a service document and they don't send
authentication, they'll get a feed instead of a 401 authentication
challenge.
Solution: Provide a separate URL for the service document, probably
/atom/service.
* PHP as a CGI
When PHP is running as a CGI the HTTP Authorization header is not
passed on to the script. Currently this means that authorisation will
fail.
Solution: There is a simply (but ugly) workaround that uses
mod_rewrite to write variables that the script can read. I have a
patch ready for this one.
References:
http://joseph.randomnetworks.com/archives/2007/09/19/http-basic-authentication-a-tale-of-atompub-wordpress-php-apache-cgi-and-ssltls/
* Support WSSE authentication
Habari currently supports only HTTP Basic Authentication. Some clients
and devices only implement WSSE Authentication, such as the Nokia N73.
Solution: As a proof of concept in the past, I implemented WSSE in
WordPress. One slight catch, and the reason it didn't work in
WordPress, is that WSSE requires plain text passwords to be
retrievable on the server. Are they available in Habari? If so, it
should be trivial for me to write a patch.
References:
http://www.xml.com/pub/a/2003/12/17/dive.html
http://www.twofishcreative.com/michael/blog/2007/09/24/implementing-wsse-authentication-in-wordpress/
* AtomPub categories support
Atom entries can contain category elements. While Habari doesn't
support categories, it does support tags, so categories supplied in an
Atom entry should be written as tags on the post.
Solution: I've submitted a patch to do this.
References:
http://tools.ietf.org/html/rfc4287#section-4.2.2
http://code.google.com/p/habari/issues/detail?id=455
* AtomPub draft support
Habari does not currently support the AtomPub protocol provision for a
client to send a request for an atom entry to be considered a draft.
Solution: Detect the app:control element in atom entries and honour
the app:draft setting, where a value of 'yes' means the entry is a
draft.
References:
http://bitworking.org/projects/atom/draft-ietf-atompub-protocol-17.html#pub-control
* AtomPub summary support
I'm not sure I've got a handle on this one completely. In the section
entitled Media Link Entries, the AtomPub spec says, "[RFC4287]
specifies that Atom Entries MUST contain an atom:summary element" but
the Atom spec clearly says in both the (normative) text and
(non-normative) RelaxNG fragments that atom:summary is optional. The
Atom spec says, "It is advisable that each atom:entry element contain
... a non-empty atom:summary element when the entry contains no
atom:content element." So, given an uploaded media entry, Habari may
create a summary element for the media link entry associated with the
media entry.
Solution: I don't really see any way of producing a summary
automatically. I think the only thing we can do is to support summary
elements provided by the client. As file uploads has not yet been
implemented, it's trivial to add a summary column when it is.
References:
http://tools.ietf.org/html/rfc4287#section-4.2.13
http://tools.ietf.org/html/rfc4287#section-4.1.1.1
http://bitworking.org/projects/atom/draft-ietf-atompub-protocol-17.html#media-link-entries
* Support foreign markup delivered by AtomPub
I'm a bit hazy on this one too. The Ape produces a warning when Habari
drops a subject element that is a child of the entry element. The
relevant section seems to be section 6 of the Atom spec, but it
doesn't specifically say that foreign markup should be preserved.
Section 6.2 of the AtomPub spec says, "Unrecognized markup in an Atom
Publishing Protocol document is considered "foreign markup" [...]
Clients SHOULD preserve foreign markup when transmitting such
documents." While this isn't talking about servers, it implies foreign
markup in legal locations should be preserved. I might be missing a
reference here, but it seems to me the spec isn't clear on this point.
Solution: Explore the issue more.
References:
http://tools.ietf.org/html/rfc4287#section-6
http://bitworking.org/projects/atom/draft-ietf-atompub-protocol-17.html#appcatsex
* Decouple app:edited and atom:updated
Atom entries can have a client provided atom:updated element. The
purpose of this element is to signify that a _significant_ change has
been made to the entry. AtomPub provides the app:edited element, which
is a timestamp representing the last time that the entry was changed
in any way. An AtomPub server can override the atom:updated value and
use its own, which Habari currently does. Habari's Post class's
updated column is written on all edits and is used to fill app:edited
elements.
Solution: Create an edited column for the Post class, and use this to
populate app:edited, and use the client provided atom:updated value
(or a timestamp if it isn't provided) to fill the updated column. This
would have the side effect that with a small UI change in admin, users
could mark things as minor edits, such as fixing typos or adding tags.
However, changing Post would also have far-reaching consequences. We
could also sort the atom feed by atom:updated, as this makes semantic
sense and order is not mandated in the Atom spec, and still have
AtomPub sending in app:edited order.
References:
http://bitworking.org/projects/atom/draft-ietf-atompub-protocol-17.html#listing-collections
http://code.google.com/p/habari/issues/detail?id=445
* Atom media collection support
Habari does not currently support the upload of media collections.
Solution: This is contingent upon Habari implementing file uploads,
and will be slightly more complex than publishing of simple atom
entries because there's a level of indirection between uploaded
resources and their atom entries.
References:
http://bitworking.org/projects/atom/draft-ietf-atompub-protocol-17.html#media-link-entries
* get_collection and get_entry currently violate DRY
Solution: Refactor the code.
Andrew da Silva
At this point, we should split both protocols since the class is
getting big, confusing and contains methods not related to the Atom or
AtomPub.
Where could we put RSD though? Is RSD considered a microformat? Maybe
we could create a microformathandler.php? (And implement other
microformats)
As for the URLs, we current use /atom for the service document
(introspection).
I dislike the use of /1 for our main feed. You suggest /service for
the service document, I think /atom is fine. But we should rename /1
to /atom/feed.
As for the AtomPub "API" URL, we could use /atompub or /atom/api? /
atom/app? /app?
2. PHP as a CGI
Back when I revised the AtomHandler class, I noticed that too, the
alternative solution I found at the time was to use a RewriteRule - we
require mod_rewrite so it's all good - to save authentication
variables in an array which can then be exploded and treated normally.
Reference:
http://www.besthostratings.com/articles/http-auth-php-cgi.html
3. Support WSSE Authentication
Like I told you last night, I do not know WSSE, at all. That is why I
left the code as-is and did not even bother reviewing it.
You told me you implemented (or worked on) WSSE in Wordpress. Would
you be ready to implement it in Habari?
4. AtomPub categories support
Processing categories as tags is the way I would implement that
feature.
We could work on a patch to implement that ASAP.
5. AtomPub draft support
We could work on a patch to implement that ASAP.
6. AtomPub summary support
I believe we once received a feature request to implement "excerpts"
for posts. If we were to add that feature, that could be used as the
summary. If no excerpts exist, just use the "more" function to
retrieve the first 200 words or so?
7. Support foreign markup delivered by AtomPub
This is out of my league for now. Geoffrey may be a resourceful
person?
8. Decouple app:edited and atom:updated
I fully support the idea of adding an option on the publish page to
specify if the editing is minor. Adding the "edited" column to the
schema should not be a problem if we see it fit.
But I still find it weird to store 3 different timestamps, just to
clear my mind:
- Publication date
- Last any type of update (current :updated)
- Last major update (would be :edited)
Couldn't we use "updated" for both, just have the option "minor edit"
decide if we update or not that value/timestamp?
9. Atom media collection support
Since this depends of our media implementation, we should wait for
Owen to complete his work on that feature and then work out the best
solution. Let's focus on the other issues we can fix now.
10. get_collection and get_entry currently violate DRY
Indeed, it does violate the "do not repeat yourself" rule. One way
could be to have get_entry support a parameter $return.
If $return, do not add response struct and headers. Return the "entry"
struct only.
If not $return, add the response struct and headers. Output the
response.
On Oct 18, 7:09 pm, "Michael C. Harris" <michael.twof...@gmail.com>
wrote:
> References:http://joseph.randomnetworks.com/archives/2007/09/19/http-basic-authe...
>
> * Support WSSE authentication
> Habari currently supports only HTTP Basic Authentication. Some clients
> and devices only implement WSSE Authentication, such as the Nokia N73.
>
> Solution: As a proof of concept in the past, I implemented WSSE in
> WordPress. One slight catch, and the reason it didn't work in
> WordPress, is that WSSE requires plain text passwords to be
> retrievable on the server. Are they available in Habari? If so, it
> should be trivial for me to write a patch.
> References:http://www.xml.com/pub/a/2003/12/17/dive.htmlhttp://www.twofishcreative.com/michael/blog/2007/09/24/implementing-w...
>
> * AtomPub categories support
> Atom entries can contain category elements. While Habari doesn't
> support categories, it does support tags, so categories supplied in an
> Atom entry should be written as tags on the post.
>
> Solution: I've submitted a patch to do this.
> References:http://tools.ietf.org/html/rfc4287#section-4.2.2http://code.google.com/p/habari/issues/detail?id=455
>
> * AtomPub draft support
> Habari does not currently support the AtomPub protocol provision for a
> client to send a request for an atom entry to be considered a draft.
>
> Solution: Detect the app:control element in atom entries and honour
> the app:draft setting, where a value of 'yes' means the entry is a
> draft.
> References:http://bitworking.org/projects/atom/draft-ietf-atompub-protocol-17.ht...
>
> * AtomPub summary support
> I'm not sure I've got a handle on this one completely. In the section
> entitled Media Link Entries, the AtomPub spec says, "[RFC4287]
> specifies that Atom Entries MUST contain an atom:summary element" but
> the Atom spec clearly says in both the (normative) text and
> (non-normative) RelaxNG fragments that atom:summary is optional. The
> Atom spec says, "It is advisable that each atom:entry element contain
> ... a non-empty atom:summary element when the entry contains no
> atom:content element." So, given an uploaded media entry, Habari may
> create a summary element for the media link entry associated with the
> media entry.
>
> Solution: I don't really see any way of producing a summary
> automatically. I think the only thing we can do is to support summary
> elements provided by the client. As file uploads has not yet been
> implemented, it's trivial to add a summary column when it is.
>
> * Support foreign markup delivered by AtomPub
> I'm a bit hazy on this one too. The Ape produces a warning when Habari
> drops a subject element that is a child of the entry element. The
> relevant section seems to be section 6 of the Atom spec, but it
> doesn't specifically say that foreign markup should be preserved.
> Section 6.2 of the AtomPub spec says, "Unrecognized markup in an Atom
> Publishing Protocol document is considered "foreign markup" [...]
> Clients SHOULD preserve foreign markup when transmitting such
> documents." While this isn't talking about servers, it implies foreign
> markup in legal locations should be preserved. I might be missing a
> reference here, but it seems to me the spec isn't clear on this point.
>
> Solution: Explore the issue more.
> References:http://tools.ietf.org/html/rfc4287#section-6http://bitworking.org/projects/atom/draft-ietf-atompub-protocol-17.ht...
>
> * Decouple app:edited and atom:updated
> Atom entries can have a client provided atom:updated element. The
> purpose of this element is to signify that a _significant_ change has
> been made to the entry. AtomPub provides the app:edited element, which
> is a timestamp representing the last time that the entry was changed
> in any way. An AtomPub server can override the atom:updated value and
> use its own, which Habari currently does. Habari's Post class's
> updated column is written on all edits and is used to fill app:edited
> elements.
>
> Solution: Create an edited column for the Post class, and use this to
> populate app:edited, and use the client provided atom:updated value
> (or a timestamp if it isn't provided) to fill the updated column. This
> would have the side effect that with a small UI change in admin, users
> could mark things as minor edits, such as fixing typos or adding tags.
> However, changing Post would also have far-reaching consequences. We
> could also sort the atom feed by atom:updated, as this makes semantic
> sense and order is not mandated in the Atom spec, and still have
> AtomPub sending in app:edited order.
>
> * Atom media collection support
> Habari does not currently support the upload of media collections.
>
> Solution: This is contingent upon Habari implementing file uploads,
> and will be slightly more complex than publishing of simple atom
> entries because there's a level of indirection between uploaded
> resources and their atom entries.
> References:http://bitworking.org/projects/atom/draft-ietf-atompub-protocol-17.ht...
Michael C. Harris
> On Oct 18, 7:09 pm, "Michael C. Harris" <michael.twof...@gmail.com>
> > While this works okay, because Habari uses the presence of
> > authentication details to tell the difference between a request for a
> > feed and a request for an AtomPub service document, it relies on the
> > client sending a correct username and password with the initial
> > request. If the client wants a service document and they don't send
> > authentication, they'll get a feed instead of a 401 authentication
> > challenge.
> > Solution: Provide a separate URL for the service document, probably
> > /atom/service.
>
> getting big, confusing and contains methods not related to the Atom or
> AtomPub.
I don't see a problem in splitting Atom and AtomPub, but there will
likely be some things that are common to both, see my comment about
DRY. Something like atomhandler.php, atompubhandler.php and
atomutils.php? We'll have to have a close look at this.
> Where could we put RSD though? Is RSD considered a microformat? Maybe
> we could create a microformathandler.php? (And implement other
> microformats)
It's not a microformat, it's just a simple XML format (microformats
are vocabularies designed to be embedded in other XML documents,
usually XHTML). I don't know where it should go, but I think it's
important. Hmm, maybe it shouldn't be atomutils.php, it should be
serviceutils.php and RSD should go in there.
> As for the URLs, we current use /atom for the service document
> (introspection).
>
> I dislike the use of /1 for our main feed. You suggest /service for
> the service document, I think /atom is fine. But we should rename /1
> to /atom/feed.
I think you mentioned that the 1 was the blog id for multisites? I
don't know enough about that, but maybe it is important. When we get
deeper into things I'll test against multisites. More investigation
and thinking required here I think.
> As for the AtomPub "API" URL, we could use /atompub or /atom/api? /
> atom/app? /app?
I would vote for /atompub. I'm very strongly opposed to anything that
uses API, because the Atom API was basically a previous version of
AtomPub and it just creates confusion.
> > * PHP as a CGI
Patch attached to issue 467.
> > * Support WSSE authentication
> > Habari currently supports only HTTP Basic Authentication. Some clients
> > and devices only implement WSSE Authentication, such as the Nokia N73.
> >
> > Solution: As a proof of concept in the past, I implemented WSSE in
> > WordPress. One slight catch, and the reason it didn't work in
> > WordPress, is that WSSE requires plain text passwords to be
> > retrievable on the server. Are they available in Habari? If so, it
> > should be trivial for me to write a patch.
>
> Like I told you last night, I do not know WSSE, at all. That is why I
> left the code as-is and did not even bother reviewing it.
>
> You told me you implemented (or worked on) WSSE in Wordpress. Would
> you be ready to implement it in Habari?
Does Habari store user passwords in a retrievable form? If I can get
a user's password, I could implement WSSE pretty quickly.
> > * AtomPub categories support
It's done, there's a patch attached to issue 455.
> > * AtomPub draft support
> > Habari does not currently support the AtomPub protocol provision for a
> > client to send a request for an atom entry to be considered a draft.
> >
> > Solution: Detect the app:control element in atom entries and honour
> > the app:draft setting, where a value of 'yes' means the entry is a
> > draft.
>
> We could work on a patch to implement that ASAP.
This is now issue 466.
> > * AtomPub summary support
> > I'm not sure I've got a handle on this one completely. In the section
> > entitled Media Link Entries, the AtomPub spec says, "[RFC4287]
> > specifies that Atom Entries MUST contain an atom:summary element" but
> > the Atom spec clearly says in both the (normative) text and
> > (non-normative) RelaxNG fragments that atom:summary is optional. The
> > Atom spec says, "It is advisable that each atom:entry element contain
> > ... a non-empty atom:summary element when the entry contains no
> > atom:content element." So, given an uploaded media entry, Habari may
> > create a summary element for the media link entry associated with the
> > media entry.
> >
> > Solution: I don't really see any way of producing a summary
> > automatically. I think the only thing we can do is to support summary
> > elements provided by the client. As file uploads has not yet been
> > implemented, it's trivial to add a summary column when it is.
>
> I believe we once received a feature request to implement "excerpts"
> for posts. If we were to add that feature, that could be used as the
> summary. If no excerpts exist, just use the "more" function to
> retrieve the first 200 words or so?
That sounds reasonable, but the Atom spec says, "It is not advisable
for the atom:summary element to duplicate atom:title or atom:content
because Atom Processors might assume there is a useful summary when
there is none." When I get a chance, I'll get on #atom and ask whether
this is a wise idea, or whether we should just not include a summary
element when there is none specifically assigned.
> 7. Support foreign markup delivered by AtomPub
> This is out of my league for now. Geoffrey may be a resourceful
> person?
I think we keep this on the backburner until we've sorted out the
other issues.
> > * Decouple app:edited and atom:updated
> > Atom entries can have a client provided atom:updated element. The
> > purpose of this element is to signify that a _significant_ change has
> > been made to the entry. AtomPub provides the app:edited element, which
> > is a timestamp representing the last time that the entry was changed
> > in any way. An AtomPub server can override the atom:updated value and
> > use its own, which Habari currently does. Habari's Post class's
> > updated column is written on all edits and is used to fill app:edited
> > elements.
> >
> > Solution: Create an edited column for the Post class, and use this to
> > populate app:edited, and use the client provided atom:updated value
> > (or a timestamp if it isn't provided) to fill the updated column. This
> > would have the side effect that with a small UI change in admin, users
> > could mark things as minor edits, such as fixing typos or adding tags.
> > However, changing Post would also have far-reaching consequences. We
> > could also sort the atom feed by atom:updated, as this makes semantic
> > sense and order is not mandated in the Atom spec, and still have
> > AtomPub sending in app:edited order.
>
> I fully support the idea of adding an option on the publish page to
> specify if the editing is minor. Adding the "edited" column to the
> schema should not be a problem if we see it fit.
>
> But I still find it weird to store 3 different timestamps, just to
> clear my mind:
> - Publication date
> - Last any type of update (current :updated)
> - Last major update (would be :edited)
Not quite, :updated would be last major updated and :edited would be
last any type of update.
> Couldn't we use "updated" for both, just have the option "minor edit"
> decide if we update or not that value/timestamp?
No, we can't. According to the spec, app:edited MUST be updated on all
edits. So, to implement minor edits, and to support sorting the feed
differently than the collection, we need to have separate columns.
> 9. Atom media collection support
> Since this depends of our media implementation, we should wait for
> Owen to complete his work on that feature and then work out the best
> solution. Let's focus on the other issues we can fix now.
Agreed 100%.
> 10. get_collection and get_entry currently violate DRY
> Indeed, it does violate the "do not repeat yourself" rule. One way
> could be to have get_entry support a parameter $return.
>
> If $return, do not add response struct and headers. Return the "entry"
> struct only.
> If not $return, add the response struct and headers. Output the
> response.
We can sort this out when we split the atom and atompub
implementations.
cheers, Michael
--
Michael C. Harris
School of CS&IT, RMIT University
Scott Merrill
>>> * Support WSSE authentication
>>> Habari currently supports only HTTP Basic Authentication. Some clients
>>> and devices only implement WSSE Authentication, such as the Nokia N73.
>>>
> Does Habari store user passwords in a retrievable form? If I can get
> a user's password, I could implement WSSE pretty quickly.
No, Habari does not store plaintext copies of passwords. I'm generally
opposed to the idea of storing them in plaintext, though I'm not opposed
to additional discussion of the issue.
Utils.php has a wsse() method I created some time ago, and the comments
link to the documents I used to create the method:
http://www.xml.com/pub/a/2003/12/17/dive.html
http://www.sixapart.com/developers/atom/protocol/atom_authentication.html
I'm not sure if that's the same thing you're discussing when you say
WSSE or not.
Cheers,
Scott
--
GPG 9CFA4B35 | ski...@skippy.net | http://skippy.net/
Michael C. Harris
>
> Michael C. Harris wrote:
> >>> * Support WSSE authentication
> >>> Habari currently supports only HTTP Basic Authentication. Some clients
> >>> and devices only implement WSSE Authentication, such as the Nokia N73.
> >
> > a user's password, I could implement WSSE pretty quickly.
>
> No, Habari does not store plaintext copies of passwords. I'm generally
> opposed to the idea of storing them in plaintext, though I'm not opposed
> to additional discussion of the issue.
We're not talking about storing passwords in plaintext, but storing
retrievable passwords (ie encrypted), but of course I'm nitpicking and
it is a definite security issue. It's certainly a weakness in WSSE.
> Utils.php has a wsse() method I created some time ago, and the comments
> link to the documents I used to create the method:
> http://www.xml.com/pub/a/2003/12/17/dive.html
> http://www.sixapart.com/developers/atom/protocol/atom_authentication.html
>
> I'm not sure if that's the same thing you're discussing when you say
> WSSE or not.
It is the same thing, but it's not usable without retrievable
passwords.
Caius Durling
On 21 Oct 2007, at 14:46, Michael C. Harris wrote:
>> I'm not sure if that's the same thing you're discussing when you say
>> WSSE or not.
>
> It is the same thing, but it's not usable without retrievable
> passwords.
I'm -1 to implementing this. Retrievable passwords are a bad idea as
we don't have control over who has access to the file system (for
sqlite, etc) or the database server/table (for mysql, etc.)
C
____________________________________
Caius Durling
UK Student -- +44 (0) 7960 268100
ca...@caius.name -- nemo...@mac.com
http://caius.name/ -- http://hentan.eu
____________________________________
Geoffrey Sneddon
On 20 Oct 2007, at 23:56, Andrew da Silva wrote:
> 7. Support foreign markup delivered by AtomPub
> This is out of my league for now. Geoffrey may be a resourceful
> person?
No, APP is not my area (though the syndication format is).
- Geoffrey Sneddon