PATH_INFO spec (with regard to ";")

[Eric Wong]

Hi all,

I've been notified privately that my changes for PATH_INFO in Unicorn
0.95.2 (which also got into Thin) may not be completely kosher, but I'm
also asking for the Rack team to clarify PATH_INFO for HTTP parser
implementers.


Upon further reading (and also of the
related-but-not-necessarily-true-for-Rack RFC 3875 section 4.1.5),
I came across this:

Unlike a URI path, the PATH_INFO is not URL-encoded, and cannot
contain path-segment parameters.

First off, Rack already directly contradicts the "the PATH_INFO is not
URL-encoded" part, so Unicorn conforms to Rack specs over RFC 3875.

*But* Rack does not address the "cannot contain path-segment parameters"
part at all. So I (and probably a few other people) would like
clarification on how to handle PATH_INFO when it comes to ";"


Things to keep in mind:

* URI.parse keeps ";" in URI::HTTP#path
This point may not be relevant to us, as PATH_INFO and
URI::HTTP#path should not necessarily be treated as equals

* WEBrick keeps ";" in PATH_INFO

* PEP333 (which Rack is based on) does not go into this level of
detail regarding PATH_INFO and path segments

* PATH_INFO in Rack appears to be based on CGI/1.1 (RFC 3875)

* Again, Rack already contradicts the URL encoding rules of RFC 3875
for PATH_INFO, so there is precedence for Rack contradicting more
of RFC 3875...

* Rack::Request#full_path only looks at PATH_INFO + QUERY_STRING,
this means many Rack applications may never see the ";" parts
if Thin and Unicorn revert to old behavior.

* Rack does not require REQUEST_URI, this is an extension Unicorn
and Thin both carried over from Mongrel.

* None of the official rack/rack-contrib middleware use REQUEST_URI


Of course, in the grand scheme of things, hardly anybody uses ";" in
paths. Yay for rare corner cases making our lives difficult.

--
Eric Wong

[James Tucker]

* http://twistedmatrix.com/trac/ticket/1361

* http://jira.codehaus.org/browse/JETTY-980 (although not directly relevant, may be food for thought)

* http://ksi.cpsc.ucalgary.ca/archives/WWW-TALK/www-talk-1994q1.messages/3.html (ancient history)

* http://code.google.com/p/googleappengine/issues/detail?id=1340

> Of course, in the grand scheme of things, hardly anybody uses ";" in
> paths. Yay for rare corner cases making our lives difficult.

Amen, because I feel like I want to say "be configurable", given all of the above.

>
> --
> Eric Wong

[Iñaki Baz Castillo]

El Viernes, 11 de Diciembre de 2009, James Tucker escribió:
> > Of course, in the grand scheme of things, hardly anybody uses ";" in
> > paths. Yay for rare corner cases making our lives difficult.
>
> Amen, because I feel like I want to say "be configurable", given all of the
> above.

I'm coding a XCAP server using Rack. XCAP is basically XPath over HTTP in
which there are HTTP URI's like the following:

/auid/global/mydoc.xml/~~/ruleset/rule%5b@id=%22sip:dom.org;p=abc%22%5d

After reading RFC for URI and HTTP I think that the ";" is valid into the
above URI. I also asked in a XCAP maillist and received same conclusion.

Please tell me if you consider the ";" in the above URI wrong.

Regards.



--
Iñaki Baz Castillo <i...@aliax.net>

[James Tucker]

"Wrong" is hard to define with archaic protocol specifications that do not match "common" usage patterns, which is why I suggested it should be configurable. I would probably suggest to you that you make your client configurable to escape it's semi-colon characters in the uri that are not part of query syntax.

[Iñaki Baz Castillo]

Yes, modifications in the client could be an option. However I cannot force to
every XCAP clients to escape ";" when it's allowed as per URI BNF grammar.

[macournoyer]

"not completely kosher" as in security issue? Should I revert this
change in Thin?

[Eric Wong]

macournoyer <macou...@gmail.com> wrote:
> "not completely kosher" as in security issue? Should I revert this
> change in Thin?

Not a security issue, but it violates the CGI/1.1 RFC which Rack is
somewhat based on (but also not tied to).

--
Eric Wong

[Iñaki Baz Castillo]

El Domingo, 13 de Diciembre de 2009, James Tucker escribió:
> "Wrong" is hard to define with archaic protocol specifications that do not
> match "common" usage patterns

I've re-checked URI/HTTP grammar and the following URI (containing semicolon)
is valid for me (perhaps I'm wrong):

/auid/global/mydoc.xml/~~/ruleset/rule%5b@id=%22sip:dom.org;p=abc%22%5d

So, why do you say "archaic protocol specifications that do not match 'common'
usage patterns"?
I expect that HTTP is not just for web servers.

But as I said I could be wrong. Please if you (or someone) can verify me that
the semicolon is invalid according to URI/HTTP BNF grammar, then I would
accept it.

[James Tucker]

On 14 Dec 2009, at 00:44, Iñaki Baz Castillo wrote:

> El Domingo, 13 de Diciembre de 2009, James Tucker escribió:
>> "Wrong" is hard to define with archaic protocol specifications that do not
>> match "common" usage patterns
>
> I've re-checked URI/HTTP grammar and the following URI (containing semicolon)
> is valid for me (perhaps I'm wrong):
>
> /auid/global/mydoc.xml/~~/ruleset/rule%5b@id=%22sip:dom.org;p=abc%22%5d

This appears to be legal according the aggregate grammars in Appendix A of both 2396 and 3986. I don't have time to track down all of the relevant HTTP and CGI rfcs right now, but bug me later and I'll check those too.

> So, why do you say "archaic protocol specifications that do not match 'common'
> usage patterns"?

Things change, indeed between 2396 and 3986 there were quite some changes to the way segments are split in URIs, however, it does not affect this particular issue.

> I expect that HTTP is not just for web servers.

No comment.

> But as I said I could be wrong. Please if you (or someone) can verify me that
> the semicolon is invalid according to URI/HTTP BNF grammar, then I would
> accept it.

According to the URI grammars in RFCs 2396 and 3986 it's fine. As I said above, I have not checked the HTTP and CGI grammars, which may well be different / more restrictive.

[Eric Wong]

I�aki Baz Castillo <i...@aliax.net> wrote:
> I'm coding a XCAP server using Rack. XCAP is basically XPath over HTTP in
> which there are HTTP URI's like the following:
>
> /auid/global/mydoc.xml/~~/ruleset/rule%5b@id=%22sip:dom.org;p=abc%22%5d
>
> After reading RFC for URI and HTTP I think that the ";" is valid into the
> above URI. I also asked in a XCAP maillist and received same conclusion.
>
> Please tell me if you consider the ";" in the above URI wrong.

Hi I�aki,

You're missing the point, it's not that ";" is right or wrong. It's
"right" in that it's always been _accepted_ by the Mongrel family of
HTTP parsers. That's not the issue.

The issue is just whether or not the PATH_INFO variable can contain ";"
(a path-segment parameter) in it. RFC 3875 section 4.1.5 states it
cannot:

Unlike a URI path, the PATH_INFO is not URL-encoded, and cannot
contain path-segment parameters.

However, Rack is not governed by RFC 3875 and already deviates from
it when it comes to the "PATH_INFO is not URL-encoded".

--
Eric Wong

[Iñaki Baz Castillo]

El Lunes, 14 de Diciembre de 2009, Eric Wong escribió:

> Iñaki Baz Castillo <i...@aliax.net> wrote:
> > I'm coding a XCAP server using Rack. XCAP is basically XPath over HTTP in
> > which there are HTTP URI's like the following:
> >
> > /auid/global/mydoc.xml/~~/ruleset/rule%5b@id=%22sip:dom.org;p=abc%22%5d
> >
> > After reading RFC for URI and HTTP I think that the ";" is valid into the
> > above URI. I also asked in a XCAP maillist and received same conclusion.
> >
> > Please tell me if you consider the ";" in the above URI wrong.
>

> Hi Iñaki,

>
> You're missing the point, it's not that ";" is right or wrong. It's
> "right" in that it's always been _accepted_ by the Mongrel family of
> HTTP parsers. That's not the issue.
>
> The issue is just whether or not the PATH_INFO variable can contain ";"
> (a path-segment parameter) in it. RFC 3875 section 4.1.5 states it
> cannot:
>
> Unlike a URI path, the PATH_INFO is not URL-encoded, and cannot
> contain path-segment parameters.
>
> However, Rack is not governed by RFC 3875 and already deviates from
> it when it comes to the "PATH_INFO is not URL-encoded".

Ok, thanks a lot for the clarification.