Param containing semicolon not parsed properly by perl
ds...@memeticcandiru.com
I'm attempting to submit some data, via a url, but it isn't being parsed
properly by Perl.
Example:
The data: thequick;brownfox
The URL : www.sample.com?keyword=thequick;brownfox
Processing the input: $keyword=param('keyword');
What perl Sees: $keyword="thequick"
Can anyone explain this behavior?
--
............................................................................
"If you can't change the world... Change yourself"
-Matt Johnson
............................................................................
www.geocities.com/pentagon/bunker/1022 dan...@swan.com
Alan J. Flavell
> I'm attempting to submit some data, via a url, but it isn't being parsed
> properly by Perl.
You mean "by CGI.pm"?
> The URL : www.sample.com?keyword=thequick;brownfox
> What perl Sees: $keyword="thequick"
ITYM "what CGI.pm sees". Maybe you should check RFC1866/HTML2.0
and subsequent HTML specs including HTML4.01 in regard to the use of
semicolon as an alternative parameter separator.
> Can anyone explain this behavior?
I'm pretty sure the CGI.pm documentation deals with it too.
Maybe now that you're alerted to the issue you'll find it.
If you want & or ; as data in a hand-coded query string, and a whole
lot of other characters that are listed in the spec, then you better
%xx-encode them. CGI.pm (which I heartily recommend) implements what
the specifications document. So reading the specs is always a useful
move, IMHO.
--
PLEASE NOTE: comp.infosystems.www.authoring.cgi is a
SELF-MODERATED newsgroup. aa.net and boutell.com are
NOT the originators of the articles and are NOT responsible
for their content.
HOW TO POST to comp.infosystems.www.authoring.cgi:
http://www.thinkspot.net/ciwac/howtopost.html
William Alexander Segraves
news:76gb8.83$Bw4....@news1.telusplanet.net...
>
> I'm attempting to submit some data, via a url, but it isn't being parsed
> properly by Perl.
>
> Example:
>
> The data: thequick;brownfox
>
> The URL : www.sample.com?keyword=thequick;brownfox
>
> Processing the input: $keyword=param('keyword');
>
> What perl Sees: $keyword="thequick"
>
>
> Can anyone explain this behavior?
Your problem is neither a Perl problem nor a CGI.pm problem; but rather, is
an incorrect use of HTML. You have simply URL-encoded the query string
incorrectly.
If, OTOH, you'll simply let your browser do the URL-encoding work it's
designed to do, regardless of whether you use GET or POST, you'll find that
Perl and CGI.pm will gladly do their parts of the job.
See http://segraves.tripod.com for examples of forms for which your
"thequick;brownfox" keyword gets correctly URL-encoded, after which Perl has
no problem handling it.
Bill Segraves
brian d foy
> The data: thequick;brownfox
>
> The URL : www.sample.com?keyword=thequick;brownfox
the standard says to use www.example.com for examples ;)
> Processing the input: $keyword=param('keyword');
>
> What perl Sees: $keyword="thequick"
> Can anyone explain this behavior?
it looks like CGI.pm thinks you are using the new CGI data
delimiter ";", which is the default in anything past version
2.64. see the CGI.pm documentation for "-newstyle_urls" and
its friend "-oldstyle_urls".
your fix may be as simple as
use CGI qw(:standard -oldstyle_urls);
--
brian d foy <com...@panix.com> - Perl services for hire
CGI Meta FAQ - http://www.perl.org/CGI_MetaFAQ.html
Troubleshooting Perl CGI scripts - http://www.perl.org/troubleshooting_CGI.html
brian d foy
> <ds...@memeticcandiru.com> wrote in message
> news:76gb8.83$Bw4....@news1.telusplanet.net...
> >
> > I'm attempting to submit some data, via a url, but it isn't being parsed
> > properly by Perl.
> >
> > Example:
> >
> > The data: thequick;brownfox
> >
> > The URL : www.sample.com?keyword=thequick;brownfox
> >
> > Processing the input: $keyword=param('keyword');
> >
> > What perl Sees: $keyword="thequick"
> >
> >
> > Can anyone explain this behavior?
>
> Your problem is neither a Perl problem nor a CGI.pm problem;
it's a CGI.pm problem. CGI.pm changes its default behaviour
frequently. now its default behaviour is to use the newstyle_url
option which separates pairs with ; rather than &.
> but rather, is
> an incorrect use of HTML. You have simply URL-encoded the query string
> incorrectly.
it's not HTML - it's a URL ;)
--
brian d foy <com...@panix.com> - Perl services for hire
CGI Meta FAQ - http://www.perl.org/CGI_MetaFAQ.html
Troubleshooting Perl CGI scripts - http://www.perl.org/troubleshooting_CGI.html
--
brian d foy
> On Feb 15, ds...@memeticcandiru.com inscribed on the eternal scroll:
> If you want & or ; as data in a hand-coded query string, and a whole
> lot of other characters that are listed in the spec, then you better
> %xx-encode them. CGI.pm (which I heartily recommend) implements what
> the specifications document. So reading the specs is always a useful
> move, IMHO.
reading the CGI.pm changes docs is essential. Lincoln changes the
default behaviour frequently. this problem probably was not a
problem with versions of CGI.pm earlier than 2.64. :)
--
brian d foy <com...@panix.com> - Perl services for hire
CGI Meta FAQ - http://www.perl.org/CGI_MetaFAQ.html
Troubleshooting Perl CGI scripts - http://www.perl.org/troubleshooting_CGI.html
--
William Alexander Segraves
news:150220022140526903%com...@panix.com...
> In article <a4k7a4$kg8$1...@slb1.atl.mindspring.net>, William Alexander
Segraves <wseg...@mindspring.com> wrote:
>
> > <ds...@memeticcandiru.com> wrote in message
> > news:76gb8.83$Bw4....@news1.telusplanet.net...
> > >
> > > I'm attempting to submit some data, via a url, but it isn't being
parsed
> > > properly by Perl.
> > >
> > > Example:
> > >
> > > The data: thequick;brownfox
> > >
> > > The URL : www.sample.com?keyword=thequick;brownfox
> > >
> > > Processing the input: $keyword=param('keyword');
> > >
> > > What perl Sees: $keyword="thequick"
> > >
> > >
> > > Can anyone explain this behavior?
> >
Hopefully, Brian and the OP will take the following as constructive, rather
than confrontational. What Brian states is useful and constructive; but,
IMO, misses the OP's question.
> > Your problem is neither a Perl problem nor a CGI.pm problem;
>
I hasten to add that CGI.pm does exactly what it's supposed to do with the
OP's string.
> it's a CGI.pm problem.
Nope. CGI.pm is not the problem. It works fine.
> CGI.pm changes its default behaviour
> frequently. now its default behaviour is to use the newstyle_url
> option which separates pairs with ; rather than &.
>
I concur in this. It doesn't alter my conclusions about the OP's problem,
though.
WADR to Brian, I stand by my original statements. The problem arises before
CGI.pm "sees" the URL-encoded string of name=value pairs because the
semi-colon has not been URL-encoded.
If the OP had simply coded his link as
www.sample.com?keyword=thequick%3bbrownfox
Perl and CGI.pm would not have "misinterpreted" the OP's submittal on
account of the incorrectly coded semi-colon. OTC, Perl and CGI.pm likely
used the semi-colon as a delimiter, as recent versions of CGI.pm do.
> > but rather, is
> > an incorrect use of HTML. You have simply URL-encoded the query string
> > incorrectly.
>
> it's not HTML - it's a URL ;)
>
I agree with Brian on this. It's a URL. But it appears in the posting as if
it's being used as a link in HTML. IN fact, it is indeed a URL embedded in
HTM in this M$OE mail program! But that's another issue. ;-)
Regardless of the interpretation, URL or URL in HTML, the previous
statements with which I still stand are IMO valid. IOW, if the link is coded
incorrectly, you can infer that the HTML/URLof which it is a part is also
coded incorrectly or will not perform as intended.
Even if the URL is entered directly as a URL in the browser, the OP's intent
is not accomplished because the semicolon was not URL encoded.
Finally, CGI.pm handles the properly coded URL correctly in both cases. That
is, it returns the string "thequick;brownfox" when the semicolon is
URL-encoded, while it returns "the quick" when the semi-colon is not
URL-encoded. FWIW, the semi-colon does indeed act as a delimiter when it's
not URL-encoded, as CGI.pm interprets the "brownfox" part of the original
string as the next, or rather another, keyword.
Examples (all entered as URLs):
http://localhost/cgi-bin/echo2txt_with_cgi.pl?keyword=thequick;brownfox
produces
keyword:thequick
brownfox:
as the result.
OTOH,
http://localhost/cgi-bin/echo2txt_with_cgi.pl?keyword=thequick%3bbrownfox
produces
keyword:thequick;brownfox
as the result.
Finally,
http://localhost/cgi-bin/echo2fdf.pl?keyword=thequick;brownfox=fluffycritter
produces the following tag-value pairs as part of the returned FDF
<< /T (keyword) /V (thequick) >>
<< /T (brownfox) /V (fluffycritter) >>
I rest my case.
Bill Segraves
> --
Alan J. Flavell
> it's a CGI.pm problem.
No, it isn't. I'm going to have to call you on this, brian, because
your reply is in error on at least two vital technical points.
> CGI.pm changes its default behaviour frequently.
It does change (the term "frequently" seems to be troll-fodder). But
it only changes within the scope of the relevant specifications!
> now its default behaviour is to use the newstyle_url
> option which separates pairs with ; rather than &.
That's its URL-writing default now, yes. But here we're discussing
its URL-decoding behaviour, and I'm confident that _that_ hasn't
changed in a relevant respect for as long as I can remember.
Recall that the specification of URL-encoding [was RFC1738, now
RFC2396] contains a list of ASCII characters which are designated
"reserved", and if these characters are intended to serve as data in a
URL, rather than performing some specified function, then they are
specified to be represented in %xx-encoding. The semicolon is one of
the reserved characters, as is the ampersand. See RFC2396 section
2.2.
Recall also that RFC1866/HTML2.0 recommends hand-coded query URLs
using an alternative parameter-separator character, and suggests
semicolon. This has been repeated in subsequent HTML specifications,
see e.g http://www.w3.org/TR/html401/appendix/notes.html#h-B.2.2
URL-decoding software is perfectly entitled to support both ampersand
and semi-colon as query-string parameter separators, at the same time;
and CGI.pm does so, and has done for - as I say - as long as I can
remember.
The thing that _changed_ (see the change log for version 2.64) was the
default format of URLs which were _generated_ by the program; but both
before and after that change, CGI.pm would _decode_ query strings
whose query parameters were separated by _either_ character. Indeed I
find no mention in the change log of such support having been
introduced, so I presume it's been there all along: certainly it's
been there as long as I have known and used CGI.pm.
> it's not HTML - it's a URL ;)
Indeed, and therefore needs to follow the rules for URL-encoding.
Which means that when reserved characters are to be included as data,
the specification calls for them to be %xx-encoded. CGI.pm handles
this correctly - of course (a failure to do so would be a very serious
bug!).
Even _in_ a context where HTML would be relevant, you don't start
HTML-ifying a URL reference until you'd got the URL properly
URL-encoded. That's where a lot of beginners go wrong - confusing
&entity; HTML references with %xx URL-encoding. (But you'd obviously
know better than that.)
cheers
William Alexander Segraves
news:Pine.LNX.4.40.02021...@lxplus023.cern.ch...
> On Feb 15, brian d foy inscribed on the eternal scroll:
>
> > it's a CGI.pm problem.
>
> No, it isn't. I'm going to have to call you on this, brian, because
> your reply is in error on at least two vital technical points.
Alan, you've stated much more eloquently the points I was trying to get
across to Brian, citing appropriate standards. Thanks for sharing your
expertise.
Bill Segraves
brian d foy
> On Feb 15, brian d foy inscribed on the eternal scroll:
>
> > it's a CGI.pm problem.
>
> No, it isn't. I'm going to have to call you on this, brian, because
> your reply is in error on at least two vital technical points.
i was probably wrong in more than that :)
--
brian d foy <com...@panix.com> - Perl services for hire
CGI Meta FAQ - http://www.perl.org/CGI_MetaFAQ.html
Troubleshooting Perl CGI scripts - http://www.perl.org/troubleshooting_CGI.html
--
brian d foy
> > it's a CGI.pm problem.
>
> Nope. CGI.pm is not the problem. It works fine.
i disagree, but not for the reasons we debate here, apparently.
CGI.pm treats either the ; or the & as a delimiter in the same
query string. however, i discover on further research that
RFC 2396 does not say you cannot do this. indeed, it says nothing.
> WADR to Brian, I stand by my original statements. The problem arises before
> CGI.pm "sees" the URL-encoded string of name=value pairs because the
> semi-colon has not been URL-encoded.
in the OP example, keyword=thequick;brownfox, there is only one
name-value pair, in my opinion, because there is only one =. the
RFC does not talk about this though, and the CGI specifications
defer to the RFC. perhaps i've missed something though.
i beleive the intent of any CGI data parser is to break the
query string into name-value pairs, and i think in this case
CGI.pm does not acheive this, because it does not try to recognize
name value pairs -- it recognizes characters that might separate
them.
in this query string
name1=value1;name2=value2&name3=value3
CGI.pm will see three name-value pairs. whether or not you think
that is correct behaviour (even though it is undefined as far as i
can tell), is the crux of our disagreement. :)
i think the various specifications fall short because they include
no hints to the CGI data parser about the data format. before ;
was a name-value separator, even though it was a reserved character
it did not need to be escaped. you only need to escape reserved
characters only need to be escaped if their unescaped presence causes
confusion. now either ; or & may be a separator, even at the same
time, which i think is silly, although the specifications don't
say you can't do that. i think the user-agent should have to choose
one, and then declare which one it chose.
but, no matter what i think, encoding the ; makes CGI.pm do the
expected thing, and i didn't mean to imply that we shouldn't
do that. theory and practice are different things. ;)
--
brian d foy <com...@panix.com> - Perl services for hire
CGI Meta FAQ - http://www.perl.org/CGI_MetaFAQ.html
Troubleshooting Perl CGI scripts - http://www.perl.org/troubleshooting_CGI.html
--
brian d foy
> "Alan J. Flavell" <fla...@mail.cern.ch> wrote in message
> news:Pine.LNX.4.40.02021...@lxplus023.cern.ch...
> > On Feb 15, brian d foy inscribed on the eternal scroll:
> > > it's a CGI.pm problem.
> > No, it isn't. I'm going to have to call you on this, brian, because
> > your reply is in error on at least two vital technical points.
> Alan, you've stated much more eloquently the points I was trying to get
> across to Brian, citing appropriate standards.
i'm not sure that any of the specifications say enough to define
correct behaviour in this case.
--
brian d foy <com...@panix.com> - Perl services for hire
CGI Meta FAQ - http://www.perl.org/CGI_MetaFAQ.html
Troubleshooting Perl CGI scripts - http://www.perl.org/troubleshooting_CGI.html
--
William Alexander Segraves
> In article <a4kpqd$n8n$1...@slb5.atl.mindspring.net>, William Alexander
Segraves <wseg...@mindspring.com> wrote:
>
> > > it's a CGI.pm problem.
> >
> > Nope. CGI.pm is not the problem. It works fine.
>
> i disagree, but not for the reasons we debate here, apparently.
> CGI.pm treats either the ; or the & as a delimiter in the same
> query string.
I disagree, too. Actually, I think we are in agreement on (recent) CGI.pm's
treatment of semi-colon as a delimiter. My examples demonstrated the
behaviour, both for URL-encode and unencoded semi-colons.
> > WADR to Brian, I stand by my original statements. The problem arises
before
> > CGI.pm "sees" the URL-encoded string of name=value pairs because the
> > semi-colon has not been URL-encoded.
>
> in the OP example, keyword=thequick;brownfox, there is only one
> name-value pair, in my opinion, because there is only one =. the
> RFC does not talk about this though, and the CGI specifications
> defer to the RFC. perhaps i've missed something though.
>
Yes. That's why I considered "thequick;brownfox" as a single value in the
URL, which in order to be treated as intended in a URL, must be coded as
www.example.com?keyword=thequick%3bbrownfox
Note I've changed the OP's URL as you had suggested. ;-)
> i beleive the intent of any CGI data parser is to break the
> query string into name-value pairs, and i think in this case
> CGI.pm does not acheive this, because it does not try to recognize
> name value pairs -- it recognizes characters that might separate
> them.
>
> in this query string
>
> name1=value1;name2=value2&name3=value3
>
> CGI.pm will see three name-value pairs. whether or not you think
> that is correct behaviour (even though it is undefined as far as i
> can tell), is the crux of our disagreement. :)
>
IMO, that is the correct behaviour. Wait one; I'll test it.
Firing up IndigoPerl's Apache server, as it's handy on this system ...
O.K. CGI.pm does what it is expected to do with the mixed coding of
separators.
http://localhost/cgi-bin/echo2txt_with_cgi.pl?name1=value1;name2=value2&name
3=value3
produces
name1:value1
name2:value2
name3:value3
as the result. Note: The Perl and CGI.pm script in the above simply returns
the name-value
pairs to the browser with colon as a separator.
<snip>
> but, no matter what i think, encoding the ; makes CGI.pm do the
> expected thing, and i didn't mean to imply that we shouldn't
> do that. theory and practice are different things. ;)
Indeed! For me, CGI.pm does what's expected, both when the semi-colon is
encoded, i.e., part of the value, and when it's not encoded, i.e., a
separator.
So it seems we've come to agreement. Good debate, IMO! Thanks for your
comments.
Bill Segraves
Alan J. Flavell
> i disagree, but not for the reasons we debate here, apparently.
> CGI.pm treats either the ; or the & as a delimiter in the same
> query string. however, i discover on further research that
> RFC 2396 does not say you cannot do this. indeed, it says nothing.
RFC2396 is about the generic syntax of URLs. It rates a number
of characters to be "reserved", and categorises them as potentially
having special functions in specific URL schemes. What those
functions are, you'd need to find elsewhere.
It is, perhaps, strange that the function of & and particularly of ;
in http-scheme URLs is found in the _HTML_ specifications. The HTTP
specification, RFC2616, gets as far as (in 3.2.2) the ?query string,
but seems to say no more about it in detail, as far as I can see.
RFC1867 also defers to the HTML specification (RFC1866) for the
definition of application/x-www-form-urlencoded. Of course there is,
as you know, no later definition of HTML in the form of an RFC, so
there the trail runs cold. (RFC2070 discusses the use of this
encoding in an i18n context, but I find nothing more.)
> in the OP example, keyword=thequick;brownfox, there is only one
> name-value pair, in my opinion, because there is only one =
I see your point, but this looks to me more like dealing with error
recovery than with the operation of a formal syntax.
> i beleive the intent of any CGI data parser is to break the
> query string into name-value pairs, and i think in this case
> CGI.pm does not acheive this, because it does not try to recognize
> name value pairs -- it recognizes characters that might separate
> them.
But the characters '=', '&' and ';' are all legal in value strings.
In fact, as I'll show in a moment, they also appear to be legal in
name strings.
My advice to authors would be to represent all three of these
characters in %xx form when they are intended to form part of the
data. Any other approach would seem to lead to hopeless ambiguities.
> in this query string
>
> name1=value1;name2=value2&name3=value3
>
> CGI.pm will see three name-value pairs.
I reckon so, yes. The relevant spec (which, as I say, rather oddly
seems to be the HTML spec) does not rule the use of ';' and '&' as
separators to be mutually exclusive, so I don't think CGI.pm is doing
anything wrong. I wouldn't advise authors to actually do this in
their hand-coded queries, but, as I say, CGI.pm's reaction could not
be rated incorrect IMO.
> i think the various specifications fall short because they include
> no hints to the CGI data parser about the data format.
But the URL RFCs _do_ tell the author to %xx-encode any reserved
character, _at least_ in contexts where any misunderstanding could
arise. My conclusion is that, since at least RFC1866/HTML2.0, this
_is_ such a context for the semicolon, and so a well-behaved author
WILL encode it. I think you'll find that all browsers will encode
this character in a "real" form submission, and when you're
hand-coding a query string you're basically aiming to imitate that
behaviour (modulo this little infelicity with the ampersand).
> before ;
> was a name-value separator, even though it was a reserved character
> it did not need to be escaped.
(although it's never wrong to do so. But the semicolon has been
documented as a potential separator since at least RFC1866, and I
think discussing what might have gone before that is a bit peripheral
to the main thrust of this discussion.)
> but, no matter what i think, encoding the ; makes CGI.pm do the
> expected thing, and i didn't mean to imply that we shouldn't
> do that.
Fine!
> theory and practice are different things. ;)
They could be, but in this case I reckon they're pretty close.
Now let's take a look at e.g
http://www.w3.org/TR/html401/interact/forms.html#control-name
A control's "control name" is given by its name attribute.
and look at a specific control to see what this "control name" can
legally be:
http://www.w3.org/TR/html401/interact/forms.html#adef-name-INPUT
name = cdata [CI]
it's defined to be case-insensitive CDATA.
So: formally, the "name" is CDATA, which means it can contain pretty
much anything - no different in this regard from the "value" (the
latter might be case-sensitive, but that's a side-issue here).
I don't find anything in the text of the spec which places any closer
limitations on what characters the "name" may contain, any more than
it places limitations on what the "value" may contain.
If I rustle up a form which contains the characters &, ; and = in
both the name and the value strings, and submit it, then I am not
surprised to find that all occurrences of these characters have been
put into %xx encoding.
My thesis is that authors need to do this when composing query strings
manually, and that anything else counts as error recovery - which
might in some cases produce the results intended by the author, but on
the other hand might not.
Hope that helps.
Jacqui caren
> In article
> <Pine.LNX.4.40.02021...@lxplus023.cern.ch>, Alan J.
> Flavell <fla...@mail.cern.ch> wrote:
>
>> On Feb 15, brian d foy inscribed on the eternal scroll:
>>
>> > it's a CGI.pm problem.
>>
>> No, it isn't. I'm going to have to call you on this, brian, because
>> your reply is in error on at least two vital technical points.
>
> i was probably wrong in more than that :)
>
We are all wrong at times - it shows real character to admit it
publically - pity clueless wanabbes cannot follow your example.
OTOH, it shows level of skill when it takes someone like Alan
to correct you.
I have to say I enjoyed this thread - no nastyness, lots of good
refs and well crafted responses by people I respect.
Just want to say thanks :-)
Jacqui
.