Hash.toQueryString changes

[Mislav Marohnić]

I'm combining Tobie's and mine Hash patches to see how they work out. These changes fix 3 bugs and remove some premature optimization from toQueryString method. Now, ticket #7494 by Skirch is about nested hash support - I like it and am thinking about rolling it in for the next release. There are a number of questions regarding all these changes, though. I'd like to sort out the inconsistencies.

1. undefined (or null) values are currently serialized as empty strings - should those key/value pairs be skipped instead?
2. undefined (or null) values are currently skipped if they're inside an array value - should it be consistent with #1?
3. should array values be flattened when serializing? (#7494 does this)
4. should we support nested hashes?
   

Same questions expressed in JS code:

1. { foo:null } => "foo="
2. { foo:['a', null, 'b']} => "foo=a&foo=b"; { foo:[null] } => "foo="
3. { foo:['a', ['b'], 'c']} => "foo=a&foo=[object]&foo=c"
4. { foo:{bar:'baz'}} => "foo[bar]=baz"
   

Remeber, the toQueryString is all about sending that data to the server. The result strings are also used inside Prototype to check if the form has been modified.

Please, state your opinions.

--
http://dev.rubyonrails.org/ticket/7494

[Tom Gregory]

My opinion:

1. I support skipping nulls, as (depending on server implementation) 'foo=' may result in an empty string on the server rather than null.  Serialize empty strings.

2. Yes, although you could pursue numerical indexes (not my preference), similar to how you handle hashes in #4, returning "foo[0]=a&foo[2]=b"

3. No, flatten as a object with numerical keys, then address as per #4. Provide the option for flatten as JSON instead; allow hooks for other serialization formats to be added (such as PHP's serialize).

4. Yes.

TAG

[Andrew Dupont]

#4 looks fine to me.

On #1 and #2: perhaps the answer is...

{ foo: undefined } => "foo="
{ foo: null } => ""

I don't know if this is useful or not, but it would allow you to
distinguish between these two cases, since null and undefined are
distinct values with different meanings.

I have no opinion on #3. That's an edge case that makes my brain cry.

Cheers,
Andrew

On Mar 2, 10:48 am, "Mislav Marohnić" <mislav.maroh...@gmail.com>
wrote:

> I'm combining Tobie's and mine Hash patches to see how they work out. These
> changes fix 3 bugs and remove some premature optimization from toQueryString
> method. Now, ticket #7494 by Skirch is about nested hash support - I like it
> and am thinking about rolling it in for the next release. There are a number
> of questions regarding all these changes, though. I'd like to sort out the
> inconsistencies.
>

>    1. undefined (or null) values are currently serialized as empty

>    strings - should those key/value pairs be skipped instead?

>    2. undefined (or null) values are currently skipped if they're inside

>    an array value - should it be consistent with #1?

>    3. should array values be flattened when serializing? (#7494 does
>    this)
>    4. should we support nested hashes?

>
> Same questions expressed in JS code:
>

>    1. { foo:null } => "foo="
>    2. { foo:['a', null, 'b']} => "foo=a&foo=b"; { foo:[null] } => "foo="
>    3. { foo:['a', ['b'], 'c']} => "foo=a&foo=[object]&foo=c"
>    4. { foo:{bar:'baz'}} => "foo[bar]=baz"

[Mislav Marohnić]

On 3/3/07, Andrew Dupont <goo...@andrewdupont.net> wrote:

#4 looks fine to me.

On #1 and #2: perhaps the answer is...

{ foo: undefined } => "foo="
{ foo: null } => ""

Interesting. My opinions are:

1. undefined should be skipped, but null should become an empty value (empty string on the server)
2. should be consistent with #1
3. no flattening, too much trouble
4. nested hashes are sweet!
   

[Colin Mollenhour]

Is 2 not incorrect? That should be "foo[]=a&foo[]=b" if I'm not
mistaken... On the server side you will get foo=b. Similarly for 3 you
will get foo=c.

I haven't looked at the code or thought about it much, but I would
like to see recursion of hashes and arrays. If you do 4, I would guess
that taking it all the way shouldn't be much harder.

{ foo:{bar:['a','b','c']}} => "foo[bar][]=a&foo[bar][]=b&foo[bar][]=c"

As for arrays, I think they should be given indexes because it would
more accurately reproduce the original data structure and with a good
recursion scheme probably be easier to code (but that's just
speculation).
{ foo:[['a','b'],['c','d']]} => "foo[0][0]=a&foo[0][1]=b&foo[1]
[0]=c&foo[1][1]=d"
{ foo:['a', ['b'], 'c']} => "foo[0]=a&foo[1][0]=b&foo[2]=c"
{ foo:['a',{bar:'baz',baz:'zab'},'c']} => "foo[0]=a&foo[1]
[bar]=baz&foo[1][baz]=zab&foo[2]=c"

I would probably find this stuff very useful in the future if it was
fully featured.
I've setup a simple test page to test such query strings here:
http://colin.mollenhour.com/posttest.php

I assume pretty much every server-side language will treat these the
same way, and performance probably isn't critical, so why not go the
whole nine? Just my .02

Colin

[Mislav Marohnić]

I'm back on the issue. In a recent talk we agreed that we should go by Rails conventions as much as applicable, so I went and ported some Rails tests to JavaScript. I also made minor changes to toQuery methods, and here is where my patch stands now:

1. { foo:'', bar:null } becomes "foo=&bar=". Sam, is this OK? "foo=&bar" really isn't a proper query :-/
   
2. { bar:undefined } is skipped
3. for now, "a=b&bar" becomes { a:'b', bar:undefined }. That should be null instead, right?
4. "foo=a=b=c" becomes { foo:'a=b=c' }
5. it handles "&a=b&&&&c=d" properly, too

Now, for the hard part - collections. Rails and PHP do some magic regarding the square brackets, but I don't think we are to follow that because we are at the client, not the backend. Magic around square brackets would break form serializing.

1. { foo:['a', 'b', 'c'] } becomes "foo=a&foo=b&foo=c". In Rails, square brackets are added to the keys
2. "foo=a&foo=b&foo=c" becomes { foo:['a', 'b', 'c'] }. Rails (PHP too, I think) would preserve only the first value. Ruby preserves all.
   
3. "foo[]=a&foo[]=b&foo[]=c" becomes { 'foo[]':['a', 'b', 'c'] }. Rails and PHP would now preserve all values, but remove the square brackets from the key.
4. { foo:[null] } becomes "foo="
5. { foo:[undefined] } or { foo:[] } becomes ""
6. "color=&color=blue" now becomes { color:['', 'blue'] }, not { color:'blue' } as before.

Objections?

[Christophe Porteneuve]

Hey Mislav,

According to the talk we had in Core's campfire, and to the way Rails
seems to behave, here's my take on your examples.

> 1. { foo:'', bar:null } becomes "foo=&bar=".
OK.

> 2. { bar:undefined } is skipped
OK.

> 3. for now, "a=b&bar" becomes { a:'b', bar:undefined }. That should
> be null instead, right?
Yes. Since we skip undefined on the way out, we should consider
presence on the way in to mean null.

> 4. "foo=a=b=c" becomes { foo:'a=b=c' }
Totally (is this even a valid URL component though?).

> 5. it handles "&a=b&&&&c=d" properly, too
That is, how? I figure { a: 'b', c: 'd' }...

> regarding the square brackets, but I don't think we are to follow that
> because we are at the client, not the backend. Magic around square

I sort of agree. I think we *may* use brackets in the QS as a hint that
it's an array, but we should not put brackets on the way out. J2EE, for
one, doesn't require any (and as you mentioned way back, URL-encoded
brackets in the parameter name look "like a car wreck" :-)).

Just a thought: optional argument to put brackets in? But this sounds
very much like edge case to me, since putting brackets in the form
element names will be preserved anyway, and we're just talking about
serializing a random object here... Plus, at any rate, later-coming
serialization hooks will let people do that anyhow.

> 1. { foo:['a', 'b', 'c'] } becomes "foo=a&foo=b&foo=c".
OK.

> 2. "foo=a&foo=b&foo=c" becomes { foo:['a', 'b', 'c'] }. Rails (PHP

> too, I think) would preserve only the first value. Ruby preserves all.

As do J2EE and, I believe, .NET. Backend-agnosticity mandates we deal
with this like HTTP and HTML intended us to, which is what you're
describing here.

> 3. "foo[]=a&foo[]=b&foo[]=c" becomes { 'foo[]':['a', 'b', 'c'] }.

> Rails and PHP would now preserve all values, but remove the square
> brackets from the key.

OK.

That's a backend concern, we don't need to be concerned with that. It
is paramount that we do not automagically alter any key name, otherwise
we'll break form serialization. As I mentioned, later serialization
hooks will deal with edge cases.

> 4. { foo:[null] } becomes "foo="
OK. Totally consistent.

> 5. { foo:[undefined] } or { foo:[] } becomes ""
OK? Consistent yet somehow feels weird... On the other hand, bijection
mandates this result.

> 6. "color=&color=blue" now becomes { color:['', 'blue'] }, not {
> color:'blue' } as before.
OK. Prime case of lack of bijection in the 1.5.0 implementation.

--
Christophe Porteneuve aka TDD
t...@tddsworld.com

[Mislav Marohnić]

On 3/13/07, Christophe Porteneuve <t...@tddsworld.com> wrote:

> 4. "foo=a=b=c" becomes { foo:'a=b=c' }
Totally (is this even a valid URL component though?).

Backends choose how they parse query strings. This is how Rails parses it, and here it preserves original data when serialized back to string.

> 5. it handles "&a=b&&&&c=d" properly, too
That is, how?  I figure { a: 'b', c: 'd' }...

Right.

Just a thought: optional argument to put brackets in?

My idea also. Ain't sure about its usefulness :-/

That's a backend concern, we don't need to be concerned with that.  It
is paramount that we do not automagically alter any key name, otherwise
we'll break form serialization.

Exactly.

> 5. { foo:[undefined] } or { foo:[] } becomes ""
OK?  Consistent yet somehow feels weird...  On the other hand, bijection
mandates this result.

The latter (empty array case) becomes "" in Rails, too (extracted from its tests)

Thanks for the input

[Richard Quadling]

On 13/03/07, Mislav Marohnić <mislav....@gmail.com> wrote:
> "foo=a&foo=b&foo=c" becomes { foo:['a', 'b', 'c'] }. Rails (PHP too, I
> think) would preserve only the first value. Ruby preserves all.

In PHP, this would end up as

array(1) { ["foo"]=> string(1) "c" }

$_GET is an array of 1 element with an index of 'foo' with a value of 'c'.

> "foo[]=a&foo[]=b&foo[]=c" becomes { 'foo[]':['a', 'b', 'c'] }. Rails and PHP
> would now preserve all values, but remove the square brackets from the key.

In PHP, this would end up as

array(1) { ["foo"]=> array(3) { [0]=> string(1) "a" [1]=> string(1)
"b" [2]=> string(1) "c" } }

$_GET is an array of 1 element with an index of 'foo' which is an
array of 3 elements, 'a', 'b' and 'c'.

If you have PHP, use this as showget.php

<?php var_dump($_GET); ?>

--
-----
Richard Quadling
Zend Certified Engineer : http://zend.com/zce.php?c=ZEND002498&r=213474731
"Standing on the shoulders of some very clever giants!"

[Colin Mollenhour]

Yes, as I stated in my previous post as well, this *definitely* need
to be changed to follow the array name with brackets, otherwise you
are simply redefining that variable over and over again and it will
simply equal the last value. Maybe rails handles this correctly but
probably nothing else will and it simply isn't correct. I thought we
all knew that to make an array in a query string you use []?

A simple test of the 1.5.1_rc1 code shows that
$H({blah:['foo','bar']}).toQueryString() => "blah=foo&blah=bar"

If you paste this into my test page [1] you will see that php gets:
[blah] => bar

The correct result of the above toQueryString should be:
"blah[]=foo&blah[]=bar"
Which will yield:
[blah] => Array (
[0] => foo
[1] => bar
)

Did anyone read my first post to this thread? I make some very good
points there but they seem to have been completely ignored thus far.

[1] http://colin.mollenhour.com/posttest.php

Colin

On Mar 14, 8:06 am, "Richard Quadling" <rquadl...@googlemail.com>
wrote:

[Colin Mollenhour]

Trac isn't responding at the moment so I can't explore and see what
other problems Hash.toQueryString is having, but I completely rewrote
it myself to support nested structures and the rewrite handles all of
the cases mentioned in this thread correctly (including for servers
out there that aren't running Rails). The string returned reproduces
the original data structure on the server-side flawlessly and is only
26 lines of code total!

For your consideration, I've included the code and a test page that
allows you to run your own tests and includes a good default test case
that also shows off it's abilities nicely. It can be tested alongside
the Prototype 1.5.1_rc1 version which obviously does not encode arrays
correctly and cannot encode the nested structures.

Code: http://pastie.caboo.se/47059
Test: http://colin.mollenhour.com/posttest.php

I'd appreciate your opinions and tests if you think you can break it.

Thanks,
Colin

[Tobie Langel]

Trac should be back up now... so you're welcome to submit that.

[Michael Peters]

Colin Mollenhour wrote:
> Maybe rails handles this correctly but
> probably nothing else will and it simply isn't correct. I thought we
> all knew that to make an array in a query string you use []?

That's a PHPism. I think Rails might do it to, but I believe the standard Rudy
libs don't. In fact, Perl, Python and most Java frameworks also don't need the
brackets. Makes me wonder why PHP needs them.

[Marius Feraru]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Colin Mollenhour wrote:
> Did anyone read my first post to this thread? I make some very good
> points there but they seem to have been completely ignored thus far.

IMHO, folks didn't ignore your previous mail, they merely assumed you
already reviewed the discussion in this thread and "fixed" your *wrong*
assumption that:

> Is 2 not incorrect? That should be "foo[]=a&foo[]=b" if I'm not
> mistaken... On the server side you will get foo=b.

Mislav said it multiple times right in this thread: *do not assume* a
*specific* backend.

> Yes, as I stated in my previous post as well, this *definitely* need to
> be changed to follow the array name with brackets, otherwise you are
> simply redefining that variable over and over again and it will simply
> equal the last value. Maybe rails handles this correctly but probably
> nothing else will and it simply isn't correct.

And now you stepped from "if I'm not mistaken" to "*definitely*"? Even more,
you assume "nothing else" *is* PHP. :(

> The correct result of the above toQueryString should be:
> "blah[]=foo&blah[]=bar" Which will yield: [blah] => Array ( [0] => foo
> [1] => bar )

Search the old spinoffs archive and the Trac for toQueryString and/or
toQueryParams. There were (way too) many discussions about these issues.
Everybody assumes whatever they wanna hear, biased by their favorite web
platform way of "understanding the universe" :(
e.g.: http://dev.rubyonrails.org/ticket/6645

I agree, all these discussions have their reason, even if we were to think
just about how scarcely/scattered is this application/x-www-form-urlencoded
based form submission procedure documented throughout the specs (RFC2396,
RFC1867, RFC2616, RFC3875, W3/HTML4.01Specs/Forms) :(

Ah, but no, we are NOT discussing about serializing HTML forms, we're just
trying to make up a wannabe abstract serialization engine which tries to
mimic application/x-www-form-urlencoded's encoding. :(

As a matter of fact, the only real reason I stepped into a discussion on
this issue is that toQuery(Params|String) are being used for XHR parameters
serialization and HTML forms serialization. Otherwise, I'd have *nothing* to
complain about *any* funky convention you may choose for this weird (YMMV)
serialization system. :(

Yes, *convention* IS the key point - e.g. remember the discussions about
"undefined" vs "" vs "false" vs "null" ? ;-/

My apologies again, in advance, if I may offended someone with my kinda
personal opinions on the matter.

yours, weeping,
- --
Marius Feraru
-----BEGIN PGP SIGNATURE-----

iD8DBQFF+VaUtZHp/AYZiNkRAqFiAKCORr1NRhIEv1FPDIEH2wbVPTSp7ACgsfF5
XN3W4yyGTZX6f8FFSEKBDQs=
=/WRP
-----END PGP SIGNATURE-----

[Colin Mollenhour]

WTF!?!?!? This is the fifth time I have posted a reply to a thread via
google groups and it says the post was successful and then it never
shows up!! Ok, I honestly don't care anymore as this is way more
trouble than it is worth. It is much easier to maintain a file of all
of my personal patches than to get anything changed in the core
because I have to argue my point endlessly and even *if* I'm right it
doesn't matter.. My first response was much more polite but I am in a
really bad mood now because I just waisted a few hours setting up a
rails server and a test page and writing responses, so I apologize for
for my tone.

Look, I installed and tested this on Rails (using
CGIMethods.parse_query_parameters) just for this thread and it NEEDS
brackets, ok? I went into details before but google groups said "No
sir! haha!" so I won't go into it again. If you think you're right,
run a test and prove it. I'd like to see tests on other platforms as
well to see how brackets and numeric indexes are handled (PHP as
arrays, Rails as hashes) but I'll leave that up to people on those
platforms who give a damn (probably nobody). Michael, care to copy and
paste my test page code into some tests for some of the other
platforms you mentioned? Just dump the post variable if it exists,
otherwise print out the page. Why not make this code work with as many
server-side languages as possible and THEN work on the toQueryParams
function?

[Mislav Marohnić]

On 3/15/07, Colin Mollenhour <cm5...@gmail.com> wrote:

Code: http://pastie.caboo.se/47059

I'd appreciate your opinions

Here's mine. First, it uses $H(value).findAll, so it isn't safe (try feeding it "each", for instance). Second, about arrays:
 
  Hash._toQueryString(key+'['+index+']',val);

I never understood why to put index in there. That complicates both serialization and parsing (think toQueryParams), plus we shouldn't be changing keys because it totally breaks form serialization.

I like how it supports nested hashes, though. We still haven't agreed on this internally, but I think nested hashes support may be in version 1.6.

I also think we need Array.prototype.toQueryString to move the logic there. Moreover, why not deprecating toQueryString/Params in favor of "toQuery"?

[Marius Feraru]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Colin Mollenhour wrote:
> WTF!?!?!? This is the fifth time I have posted a reply to a thread via
> google groups and it says the post was successful and then it never
> shows up!!

Sorry, web toys have this nasty habit of eating bytes :(
Anyway, desktop apps have it too, especially when using bleeding edge
"releases", they always crash right in the moment when you really do NOT
want them to crash :))

> Look, I installed and tested this on Rails (using
> CGIMethods.parse_query_parameters) just for this thread and it NEEDS
> brackets, ok?

Sorry x2, proving that this occult bracket convention was inherited by the
Web2.0 kings doesn't prove anything more than the fact that Rails folks
inherited it from Web1.0 kings (PHP) :))

(Which was already mentioned by Mislav, so you really wasted your time for
this - sorry x3) ;-)

> If you think you're right, run a test and prove it.

Try this "echo" service:

request:
http://www.neohub.com/ws/echo?foo=1;foo=2;foo=3;bar=4

answer:
{'bar':4,'foo':[1,2,3]}

> I'd like to see tests on other platforms as well to see how brackets and
> numeric indexes are handled (PHP as arrays, Rails as hashes) but I'll
> leave that up to people on those platforms who give a damn (probably
> nobody).

Hehe, now you reminded me that I wasted some time today too reading all the
crap^Wstuff at http://www.php.net/manual/en/language.variables.external.php
trying to understand how does your PHP bracket convention really work.
Look quite clear that if you want arrays, you have to apply some magic to
HTML "name" attributes (adding "[]").

So, if you are using this funky serialization engine for serializing your
magic HTML forms (as Prototype currently tries to do), I'd say yes, it
should do exactly what you expect.
OTOH, as I said in my previous message, folks here seem to be trying making
up some abstract serialization engine, so this discussion is bogus, as you
don't want a perfect Form.serialize(), but just to apply some PHPism into
this application/x-www-form-urlencoded serialization engine. Why don't you
keep on using the same convention you already use in your HTML with this
toy? It should work the same. For instance:

>>> $H({'foo[]':[1,2,3]}).toQueryString()
"foo%5B%5D=1&foo%5B%5D=2&foo%5B%5D=3"

Isn't it what you want?

- --
Marius Feraru
-----BEGIN PGP SIGNATURE-----

iD8DBQFF+c3ntZHp/AYZiNkRAuZWAJ9m5wHxfVbGBiAo3xQ9AOZIsqO1nwCgu95z
/m71y5jtI2XYb8Xs16gpi7E=
=JHTD
-----END PGP SIGNATURE-----

[Andrew Dupont]

You act as if the web isn't built upon conventions, even arbitrary
ones.

I think this is a POLS issue, only we disagree on which behavior is
least surprising. But since Prototype is designed to go hand-in-hand
with Rails, and since JavaScript methods can be redefined at runtime,
I think the empty brackets convention is the way to go here. If you
disagree (or if whatever server framework you use disagrees), you can
monkeypatch in your own serialization logic.

Cheers,
Andrew

> crap^Wstuff athttp://www.php.net/manual/en/language.variables.external.php

[Marius Feraru]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Andrew Dupont wrote:
> You act as if the web isn't built upon conventions, even arbitrary ones.

No idea how you got to this conclusion.

> I think this is a POLS issue, only we disagree on which behavior is least
> surprising. But since Prototype is designed to go hand-in-hand with
> Rails, and since JavaScript methods can be redefined at runtime, I think
> the empty brackets convention is the way to go here.

And this is really amazing, as it contradicts with what I heard about
Prototype (as not being tied with Rails). I liked that, and tried to defend
that. OFC, I may have heard it wrong, as I'm reading/using Prototype for
just about a couple of days, so you all have my apologies for this
misunderstanding.

> If you disagree (or if whatever server framework you use disagrees), you
> can monkeypatch in your own serialization logic.

It's not about one "server framework" or another, but merely about my yet
another mistake to presume your were discussing about creating an
application/x-www-form-urlencoded compatible serialization engine.

Anyway, hopefully no real harm done for you - besides Collin's headache -,
I'll keep quiet from now on, as I'm not currently interested in Rails.

Thank you all very much for supporting me and my deepest apologies for all
my mistakes. Prototype was a great experience thanks to all of you.

Best regards.

- --
Marius Feraru
-----BEGIN PGP SIGNATURE-----

iD8DBQFF+db6tZHp/AYZiNkRAmCyAJ9jQXTdADmCHNsygN+9kmTTb5g6jACgoMoW
fI+TE8VAX2r3ItKN/Cg2hzI=
=lEJO
-----END PGP SIGNATURE-----

[Colin Mollenhour]

> Try this "echo" service:
>
> request:http://www.neohub.com/ws/echo?foo=1;foo=2;foo=3;bar=4
>
> answer:
> {'bar':4,'foo':[1,2,3]}

Would you care to mention what platform and what method of parsing
into variables is being used in that test? The results are different
than my Rails test.

> Hehe, now you reminded me that I wasted some time today too reading all the

> crap^Wstuff athttp://www.php.net/manual/en/language.variables.external.php

> trying to understand how does your PHP bracket convention really work.
> Look quite clear that if you want arrays, you have to apply some magic to
> HTML "name" attributes (adding "[]").

That's why I setup my PHP test page for everyone to use...

> OTOH, as I said in my previous message, folks here seem to be trying making
> up some abstract serialization engine, so this discussion is bogus, as you
> don't want a perfect Form.serialize(), but just to apply some PHPism into
> this application/x-www-form-urlencoded serialization engine. Why don't you
> keep on using the same convention you already use in your HTML with this
> toy? It should work the same. For instance:
>
> >>> $H({'foo[]':[1,2,3]}).toQueryString()

For one, that is hackish and not always possible to do. Also, to
support nesting of arrays you *must* use brackets with indexes so if
you just want reliable serialization and deserialization the brackets
with indexes are the most reliable method. How else can you do this
correctly?
{foo:[[0,1],[2,3]]}

Colin

[Colin Mollenhour]

> Here's mine. First, it uses $H(value).findAll, so it isn't safe (try feeding
> it "each", for instance).

The new function in 1.5.1_rc1 has the same problem so I figured you
were comfortable with that fact already.. I don't see a good solution
to this as long as hash prototypes are extended.

> Second, about arrays:
>
> Hash._toQueryString(key+'['+index+']',val);
>
> I never understood why to put index in there. That complicates both
> serialization and parsing (think toQueryParams), plus we shouldn't be
> changing keys because it totally breaks form serialization.

That code will only get called when there is an actual JS array nested
inside a value. So, for HTML form serialization I don't see how this
code will ever be called anyway. It isn't changing the intended "name"
attributes in the form, if that is what you were alluding to. This
feature is really for when you build an array yourself in JS, not for
automatic form serialization.

> I like how it supports nested hashes, though. We still haven't agreed on
> this internally, but I think nested hashes support may be in version 1.6.

Sounds good, I can't think of a strong reason not to..

> I also think we need Array.prototype.toQueryString to move the logic there.

That doesn't make sense though... The encoding requires a key and
doing something like:
[5,6,7].toQueryString() doesn't specify a key anywhere. AFAIK there is
no correct way to serialize this.
The correct way for the programmer would be:
$H({mykey:[5,6,7]}).toQueryString()

Now the question seems to be to use brackets or not, and to use
indexes or not..
I am for using indexes (mykey[0]=5...). In Rails (in my tests) this
would give you a hash with indexes being "0", "1", "2". Is there any
reason that isn't doable? Treat the hash as if it was an array, order
is preserved, indexes are no different, probably more languages will
play well with this method than any other, everyone is happy, no?
Deserializing is at least well defined this way and shouldn't be too
hard, just test the keys to see if they're numeric and create either
an array or hash and fill it up.

I personally don't see a great need for deserializing.. If the
programmer is smart enough to keep everything as an object until the
very last step (sending it to the server) then there should be no
need. If you want to serialize and deserialize to watch for changes or
some other internal use, why not use the new JSON functions? JSON is a
much better encoding scheme than application/x-www-form-urlencoded.
For Ajax.Request, I think deserializing a passed string and then
reserializing is a bad idea. Why muck with the programmers data? If
they pass a string, just assume they know what they're doing and keep
it a string.

> Moreover, why not deprecating toQueryString/Params in favor of "toQuery"?

I'll let you guys make the API decisions, but the current API suits me
fine. :)

Thanks for your opinions and code review, Mislav.
Colin

[Andrew Dupont]

Marius, I don't want you to stop participating on this list because of
a misunderstanding. Let me clarify what I'm saying.

Prototype is not solely for use with Rails, obviously. I've spent the
last year using it completely outside of Rails. I'm saying that when
in doubt I'd do what Rails does, espcially since Rails didn't just
pull it out of thin air. We *are* trying to implement a serialization
method compatible with application/x-www-form-urlencoded, but you
yourself concede that this falls outside the scope of the
specification.

Can we get a few more opinions on this issue? I don't want this to be
decided only by the handful of people who've posted in this thread.

Cheers,
Andrew

[Mislav Marohnić]

On 3/16/07, Colin Mollenhour <cm5...@gmail.com> wrote:

> Here's mine. First, it uses $H(value).findAll, so it isn't safe (try feeding
> it "each", for instance).

The new function in 1.5.1_rc1 has the same problem so I figured you
were comfortable with that fact already..

No I'm not comfortable. Copy and paste this object in your test page:
{ _each:1, each:2, findAll:3 }

"eval with Prototype 1.5.1" serializes the object correctly.

[Christophe Porteneuve]

Hi guys,

JM2C:

- We use to fight for the idea of backend-agnosticity, which means we
CANNOT put brackets on a systematic basis, since at least J2EE doesn't
do anything special with them (and that's a sizeable market share). I
don't know about .NET, ColdFusion, and the rest.

- If we do provide bracketing, it should be as an option defaulting to
false on Hash.toQueryString (and higher-level methods such as
serialize/serializeElements). If such an option seems like too much to
set at every turn, maybe we can introduce Hash.defaultOptions that would
be used for defaults, and then you'd only have to change its bracketing
property once at script start or something.

[Michael Peters]

Andrew Dupont wrote:
> We *are* trying to implement a serialization
> method compatible with application/x-www-form-urlencoded

I think this makes the situation very simple then. We need Hash.toQueryString to
behave *exactly* like a form of type application/x-www-form-urlencoded. Forms
*do not* add brackets to inputs so neither should this method. If your backend
requires brackets, then you need to add them to your form input names as well as
your hash key names. It would be strange if you had to do one and not the other.
I think it would be even worse to force users of non-bracketed backends (J2EE,
.Net, Perl, Python, etc) to have to change their backend code to remove the
brackets just because the client side JS library thinks they should be there.

I also think this method is becoming way more complicated than it needs to be.
This is not a generic data serialization method. We don't need to deal with
nested hashes or arrays inside arrays. If you want to serialize arbitrary
structures of arbitrary depths use JSON. It's really good at that.

--
Michael Peters
Developer
Plus Three, LP

[Mislav Marohnić]

On 3/16/07, Michael Peters <michael...@gmail.com> wrote:

I also think this method is becoming way more complicated than it needs to be.
This is not a generic data serialization method. We don't need to deal with
nested hashes or arrays inside arrays.

Wisely said. I believe Sam thinks so, too.

[Tom Gregory]

Take Michael's argument a bit further (_exact_ encoding as a form
would do it), and perhaps it might be acceptable that any nested
arrays/hashes simply be JSON encoded and passed as a string. It's a
simple convention, with no edge cases. Keys are preserved just as
they are passed. Values are converted to strings immediately. The
programmer has full control over how something will be passed, as s/
he can change the key/val pair at any time. Then allow for an
onSerialize callback at the form level.

This problem didn't exist (at this volume level) until Prototype
started serializing Forms as Hashes instead of query strings, which
led to the problem of figuring out how to represent two inputs with
the same name, which led to this serialization discussion.

TAG

[Michael Peters]

Tom Gregory wrote:
> Take Michael's argument a bit further (_exact_ encoding as a form
> would do it), and perhaps it might be acceptable that any nested
> arrays/hashes simply be JSON encoded and passed as a string. It's a
> simple convention, with no edge cases.

I'm ok with this. It's the easiest to explain to someone and fits the POLS.

[Colin Mollenhour]

After seeing all of the server-side inconsistencies with url decoding,
I too agree that Hash.toQueryString is getting too complicated. In
fact, what was wrong with the old one? The one currently in 1.5.1_rc2
needs to go as it has several problems that the old one didn't have
and it is pretty convoluted for what it accomplishes. It'd be nice to
be able to encode any data structure but it doesn't look like it is
going to work.

Also, why can't Ajax.Request leave the parameters option a string if
it is passed a string? Converting to a hash and back to a string is
completely unnecessary and complicates things. If the user passes a
string, let's just assume he knows what he's doing and doesn't want
his string broken down and reconstructed. If an object is passed it
can still be converted to a string. That way problems like using
enumerable method names as keys can be circumvented by the user
building the parameters string manually.

Colin

[Mislav Marohnić]

On 3/16/07, Tom Gregory <to...@byu.net> wrote:

This problem didn't exist (at this volume level) until Prototype started serializing Forms as Hashes instead of query strings

What problem? I thought we were merely discussing about conventions and which to embrace/discard.

... which led to the problem of figuring out how to represent two inputs with the same name

When Prototype serialized forms to query string, those two inputs looked like this:

  "foo=1&foo=2"

.. and nobody complained. Now that we serialize using Hash.toQueryString , the same form inputs look like this:

  "foo=1&foo=2"

... and everybody complains! Suddenly you all want to replace the whole thing with JSON! :)

Remember, we've covered form serialization a long time ago. It works. Now we're just polishing and trying not to break stuff.

On 3/16/07, Colin Mollenhour <cm5...@gmail.com> wrote:

why can't Ajax.Request leave the parameters option a string if it is passed a string?

Then we would need to branch the logic around hacks into hacks for strings and hacks for hashes---obviously not a good way to code. String params are only for backwards compatibility, you really should handle it with hashes. Using hashes internally in the framework has slimmed down both Ajax params and Form serialization code, and it should reduce the number of string hacks in user code, also. It's also much nicer to look at.

The change had regressions, unfortunately. Version 1.5.1 is going to fix all the issues we know of, though.

[Tom Gregory]

On Mar 18, 2007, at 9:13 AM, Mislav Marohnić wrote:

On 3/16/07, Tom Gregory <to...@byu.net> wrote:

This problem didn't exist (at this volume level) until Prototype started serializing Forms as Hashes instead of query strings

What problem? I thought we were merely discussing about conventions and which to embrace/discard.

I think you're being creatively forgetful.  I'm referring to the topic that started this thread, which you posted:  How does one appropriately turn complex JS objects into query strings?

http://groups.google.com/group/prototype-core/msg/1127daf06a040a9b

A good rewrite of Hash.toQueryString would have brought the discussion about eventually, but the transition to Form serialization as hashes made the issue both more immediate and more pressing.

... which led to the problem of figuring out how to represent two inputs with the same name

When Prototype serialized forms to query string, those two inputs looked like this:

  "foo=1&foo=2"

Which is correct behavior.  We both know that.

.. and nobody complained. Now that we serialize using Hash.toQueryString , the same form inputs look like this:

  "foo=1&foo=2"

... and everybody complains! Suddenly you all want to replace the whole thing with JSON! :)

Remember, we've covered form serialization a long time ago. It works. Now we're just polishing and trying not to break stuff

It's not the Form serialization I'm worried about.  The question is Hash serialization, which involves more edge cases, as you've demonstrated.

I'm suggesting that it would have been very easy to serialize any complex objects beyond the first level as JSON.  However, serializing Forms as Hashes means we have to be able to make a round trip, so the solution can't be that easy. Tweak the idea to allow for arrays at the first level (but not nested), no nested hashes at all, anything that's not a primitive is converted to a string (Hashes and Arrays use toJSON(), other objects (e.g. Date) use implicit toString conversion.)

To go back to your original question (after thinking about it a good deal, I'm giving a different answer than I originally did):

   2. { foo:['a', null, 'b']} => "foo=a&foo=b"; { foo:[null] } => "foo="
   3. { foo:['a', ['b'], 'c']} => "foo=a&foo=[object]&foo=c"
   4. { foo:{bar:'baz'}} => "foo[bar]=baz" 

2.  "foo=a&foo=&foo=b"  // if We maintain that foo: null => "foo=", per #1

3.  "foo=a&foo=" +["b"].toJSON() + "&foo=c"

4.  "foo=" + {bar:'baz'}.toJSON()

Re #4, I'd suggest that if the programmer wished the result to be "foo[bar]=baz", then the original hash should be { "foo[bar]" : "baz" }.

This solution gives a mostly 1:1 relationship between input and output--the operation is reversible. I say "mostly", because null == "", and both would unserialize to "", however (un)serializing boolean values might not be completely reversible--I haven't tested.

TAG

[Tom Gregory]

On Mar 18, 2007, at 9:13 AM, Mislav Marohnić wrote:

On 3/16/07, Colin Mollenhour <cm5...@gmail.com> wrote:

why can't Ajax.Request leave the parameters option a string if it is passed a string?

Then we would need to branch the logic around hacks into hacks for strings and hacks for hashes---obviously not a good way to code. String params are only for backwards compatibility, you really should handle it with hashes. Using hashes internally in the framework has slimmed down both Ajax params and Form serialization code, and it should reduce the number of string hacks in user code, also. It's also much nicer to look at. 

I think Colin is right on this one.  There's no need to switch back and forth, or branch, as you suggest.

There is only a single instance that I found where Ajax modifies the params, and that's simulating other verbs over post. (ajax.js : 87-91) The hash is converted to a query string two lines later.  Simply don't make strings do the round trip, and move the Hash.toQueryString up a couple of lines.  The verb simulation can operate on a string.

Easy as pie, no hacks involved, and the code is smaller and faster.  The only functional change is that "this.parameters" would remain in the same format it was passed in as, instead of always being a hash.

TAG

[jdalton]

Michael Peters is correct.
Since forms of type application/x-www-form-urlencoded *do not* add

brackets to inputs so neither should this method.

PHP is my primary server side language and our form elements are named
'foo[]' if we want them to become an array on the serverside.
Its up to the dev to name their form elements properly not the JS
Framework to magically know.

Also, as to the other points.
I think that consistency is key.
If its null in then its null out (doesnt even appear in the render).
As to marking it null in the first place I am for it.
It makes a smaller query string if you have a bunch of nulls.
If you want them nulls to show up I, i set them to "" an empty string
instead of null.

Good discussion.

[Colin Mollenhour]

If we are talking about form serialization, then when is it even
possible to have null or undefined values? Form.serialize grabs all
form elements, so if it does it's job correctly then undefined is out
of the picture. On FF (haven't tested else where or looked up specs)
the .value property cannot be null.. create a new element and don't
assign it a value and it will evaluate to "" already. Even when I
manually assigned it to null it still evaluates to "". Is there a case
in which a form element's value can be null? Before you say "the user
can define null and undefined values directly in a hash", read on.

Creating arrays in an HTML form is really just a hack, defining
elements with the same "name" value repeatedly.. As we've seen,
interpreting this is not a well-defined operation. For example, a PHP
programmer (or others) may intend for one element's value to override
another if present by reusing the name without brackets while a Rails
programmer would expect an array. It seems like forms are better
represented as a collection of independent key->value pairs rather
than a hash, which are not the same things.
1) A hash is not guaranteed to maintain the original order in which
the elements were defined, and
2) a hash is composed of unique keys that have one value.
Serialization and deserialization of a collection of such key->value
pairs would be completely straightforward, and I suggest this is the
way it should be done. This is the only truly server-agnostic solution
IMO. I would then suggest providing a separate method for interpreting
that collection either as a strict hash, or as a hash with nested
arrays in the case of multiple definitions of the same key.

Example:
//PHP users would most *likely* use strict = true
//default result is same as current serializeElements([],true)
functionality
collectionToHash: function(collection, strict){
return collection.inject({}, function(result, pair) {
if (!strict && pair.key in result) {
if (result[pair.key].constructor != Array) result[pair.key] =
[result[pair.key]];
result[pair.key].push(pair.value);
}else result[pair.key] = value;
return result;
});
}

Doing the above would then allow for much better flexibility in
toQueryString and toQueryParams.. i.e. "real" arrays would be given
indexes and then any combination of arrays and hashes would be
possible with the caveat that some server-side languages will
interpret numerical indexes as hash keys rather than array indexes but
as I already said, that is a shortcoming of the server-side
interpretation.

Of course, code could be shuffled around a bit so that backwards
compatibility is maintained.. for example, serialize could get another
optional parameter which would simply be passed on to
collectionToHash, but for internal use, direct serialization of a form
to a collection to a string wouldn't care.

serialize(form,getHash,strict)

Thoughts? Basically I am separating Hash.toQueryString from
Form.serialize because forms are not most accurately represented as
hashes to begin with, putting the question of server-side or intended
interpretation out of the picture and therefore allowing toQueryString
to do it's thing independently of form serialization.

I'd be happy to write up a patch that fully implements what I've
proposed if the core team is interested.

Thanks,
Colin

[Colin Mollenhour]

I've written new code that I think will offer the most functionality
to the most people. The only functions modified are
Form.serializeElements, Form.Methods.serialize, Hash.toQueryString and
String.prototype.toQueryParams.

- Forms are serialized directly to strings so that there is no hash
middle-step to introduce inconsistencies favoring any server-side
technology
- When creating a hash from a string, an optional parameter
"strict" (use true) will produce:

foo=bar&foo=baz {foo:['baz']} rather than {foo:['bar','baz']} (the
default)
foo[]=bar&foo[]=baz is still {foo:['bar','baz']} in both cases
foo[0]=bar&foo[1]=baz is also supported to enable nesting

- Hash.toQueryString will add indexes to arrays which allows for
complete nesting of arrays and hashes. This shouldn't be a problem for
anyone when serializing forms because this is not used during form
serialization.
- String.prototype.toQueryParams can correctly deserialize any string
produced by toQueryString, even those with nested hashes and arrays
(only possible with explicit array indexes).

http://colin.mollenhour.com/toquery.php
http://pastie.caboo.se/48450

Does anyone have any problems with this proposal?

Thanks,
Colin