Why is v:none needed?

[tyru]

Dear Bram and Vimmers,

I have read the below thread.

[vim] jsonencode() does not work correctly with function references (#579)
https://groups.google.com/forum/#!msg/vim_dev/9rf5_YmPn28/qZKB3rKvCAAJ

But, I couldn't understand why Vim has had to add v:none variable.

> Although a missing item in an array is not according to the JSON
> standard, it is very useful in practice:
I don't think so.
If v:none variable has been added because *only* it is useful,
it should not be.
I don't think It should be added even breaking the JSON standard.

Because, v:none and v:null is very similar.
If a user mistake v:none for v:null,
an invalid JSON (for strict JSON parser) will be generated!

jsonencode([1,v:none,v:none,4])

"undefined" and "null" in JavaScript is totally a bad part.
Please don't follow that.

Best wishes,

tyru

[Nikolay Aleksandrovich Pavlov]

Since that thread is over I would reply here (did not reply previously
because I did not realize that json and channel features are
connected)

> The question is what to do with items that can't be converted to JSON.
> I have been in doubt, whether to silently drop them or give an error.

>
> Although a missing item in an array is not according to the JSON
> standard, it is very useful in practice:
>

> [1,,,,8]
>
> The missing items would be "undefined" in JavaScript. In Vim they are
> v:none. Proper JSON would be:
>
> [1,null,null,null,8]
>
> That's more than twice as long. Gets worse when there are more missing
> items, up to five times as long.

1. Arrays like `[1, null, null, null, 8]` are very rarely used when
communicating. So this is almost never “twice as long”. Especially
with the planned subfeatures of the channel feature.
2. `[1,,,,8]` with the current parser is `[1, v:none, v:none, v:none,
8]`. If people are testing for null values using `if {val} is v:null`
the fact that it can be written like this is absolutely useless. If
people are using `if {val}` this can be as well written as
`[1,0,0,0,8]`.
3. I have no problems in counting three null values, but commas are
harder to count and they are usually visually skipped because of
having very low importance.
4. Computer has no problems with either variants, performance impact
is negligible.
5. Handling v:none in VimL in case somebody cares code adds *far* more
ticks then parsing `null` in C code.
6. It is easy to miss v:none in cases like

[
"1"
, "2"
, "3",
, "4"
]
7. In msgpack the whole array is six bytes. JSON is more then three
times as long. Non-JSON you propose is still 1⅓ longer. For IPC it is
better. For user YAML is more readable (especially when one needs
multi-line strings), and almost always can be made less verbose then
JSON.
8. In javascript `[1,,,,8]` is `[1,undefined,undefined,undefined,8]`,
not `[1,null,null,null,8]`.

>
> I propose to allow this extension to JSON. However, it should not be
> created accidentally, only when intentionally using v:none as an item.

If documentation states that channel accepts JSON, it should accept
JSON and not something else. I am not much fond of idea of
jsondecode() extensions, but do not create *yet another* standard in
interprocess communications, this action is worse then creating yet
another non-strict parser.

>
> So, we should probably give an error when using a function, instead of
> producing JSON that's not according to the standard.

2016-02-03 16:28 GMT+03:00 tyru <tyru...@gmail.com>:
> Dear Bram and Vimmers,
>
> I have read the below thread.
>
> [vim] jsonencode() does not work correctly with function references (#579)
> https://groups.google.com/forum/#!msg/vim_dev/9rf5_YmPn28/qZKB3rKvCAAJ
>
> But, I couldn't understand why Vim has had to add v:none variable.
>
>> Although a missing item in an array is not according to the JSON
>> standard, it is very useful in practice:
> I don't think so.
> If v:none variable has been added because *only* it is useful,
> it should not be.
> I don't think It should be added even breaking the JSON standard.
>
> Because, v:none and v:null is very similar.
> If a user mistake v:none for v:null,
> an invalid JSON (for strict JSON parser) will be generated!

I can also confirm that I need to constantly remind myself what
exactly I need to use.

>
> jsonencode([1,v:none,v:none,4])
>
> "undefined" and "null" in JavaScript is totally a bad part.
> Please don't follow that.

It would be better if you have shown (pointed to an article) what is
so bad here.

>
> Best wishes,
>
> tyru
>
> --
> --
> You received this message from the "vim_dev" maillist.
> Do not top-post! Type your reply below the text you are replying to.
> For more information, visit http://www.vim.org/maillist.php
>
> ---
> You received this message because you are subscribed to the Google Groups "vim_dev" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to vim_dev+u...@googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.

[tyru]

Hi Nikolay,

Thanks for lots of follow-up examples :)

On Thu, Feb 4, 2016 at 8:33 AM, Nikolay Aleksandrovich Pavlov
<zyx...@gmail.com> wrote:
> Since that thread is over I would reply here (did not reply previously
> because I did not realize that json and channel features are
> connected)

BTW, I didn't understand why json and channel features are connected.
I only talked about JSON feature.

This is very likely probable case!
The following JSON string leaves v:none at the end
if it is parsed by current jsondecode().

[
"1",
"2",
"3",

"4",

"undefined" and "null" are totally different things.
So "undefined === null" returns true, of course.
But both values mean often an absense of a value.
They are very confusing.

[tyru]

oh sorry.
of course "undefined === null" returns false :)

[tyru]

> Bram

What do you think about this?

[Nikolay Aleksandrovich Pavlov]

Neovim developers agree with me that neither v:none, nor extensions to
JSON should be added to Neovim. So another point is that plugins
willing to be compatible with Neovim and use json*() functions will
not use v:none or proposed syntax extensions.

[Bram Moolenaar]

> What do you think about this?

I don't want to spend much time discussing this. v:null and v:none are
needed just like JavaScript has null and undefined.

I was thinking of taking this a step further to a more efficient
encoding that is similar to Javascript. Unfortunately I haven't been
able to find a specification. I thought it was used in combination with
protocol buffers. Besides empty entries in an array, it also removes
the quotes around object item names. It's more efficient and doesn't
drop any functionality. We could add protocol buffer support, but let's
leave that for some other time.

Also keep in mind that when you want to stick to the JSON standard
(well, one of them), you should not write the string yourself but use a
library to create it. Arguments that it's hard to type or spot a
mistake are hardly relevant.


--
hundred-and-one symptoms of being an internet addict:
154. You fondle your mouse.

/// Bram Moolenaar -- Br...@Moolenaar.net -- http://www.Moolenaar.net \\\
/// sponsor Vim, vote for features -- http://www.Vim.org/sponsor/ \\\
\\\ an exciting new programming language -- http://www.Zimbu.org ///
\\\ help me help AIDS victims -- http://ICCF-Holland.org ///

[Nikolay Aleksandrovich Pavlov]

2016-02-06 17:50 GMT+03:00 Bram Moolenaar <Br...@moolenaar.net>:
>
>> What do you think about this?
>
> I don't want to spend much time discussing this. v:null and v:none are
> needed just like JavaScript has null and undefined.
>
> I was thinking of taking this a step further to a more efficient
> encoding that is similar to Javascript. Unfortunately I haven't been
> able to find a specification. I thought it was used in combination with
> protocol buffers. Besides empty entries in an array, it also removes
> the quotes around object item names. It's more efficient and doesn't
> drop any functionality. We could add protocol buffer support, but let's
> leave that for some other time.
>
> Also keep in mind that when you want to stick to the JSON standard
> (well, one of them), you should not write the string yourself but use a
> library to create it. Arguments that it's hard to type or spot a
> mistake are hardly relevant.

JSON is used not only for communication. All arguments like “hard to
spot a mistake” apply to the configuration files that also can be
written in JSON.

Also when library is used argument like “this is twice as long” is
hardly relevant because it only matters when using Vim over very slow
channel or when sending a huge chunks of binary data, otherwise there
is no difference what data is sent exactly as it does not reach the
user, and even most likely fits in the same number of packets.

And as I said, more space-efficient encodings are binary ones like
msgpack (or protocol buffers). More human-readable is YAML. Also
binary encodings are almost always parsed faster then text ones.

>
>
> --
> hundred-and-one symptoms of being an internet addict:
> 154. You fondle your mouse.
>
> /// Bram Moolenaar -- Br...@Moolenaar.net -- http://www.Moolenaar.net \\\
> /// sponsor Vim, vote for features -- http://www.Vim.org/sponsor/ \\\
> \\\ an exciting new programming language -- http://www.Zimbu.org ///
> \\\ help me help AIDS victims -- http://ICCF-Holland.org ///
>

[tyru]

Sorry Bram, I forgot to send this mail to vim_dev.
Send again with +alpha about problems of current JSON features.

On Sat, Feb 6, 2016 at 11:50 PM, Bram Moolenaar <Br...@moolenaar.net> wrote:
>
>> What do you think about this?
>
> I don't want to spend much time discussing this. v:null and v:none are
> needed just like JavaScript has null and undefined.
>
> I was thinking of taking this a step further to a more efficient
> encoding that is similar to Javascript. Unfortunately I haven't been
> able to find a specification. I thought it was used in combination with
> protocol buffers. Besides empty entries in an array, it also removes
> the quotes around object item names. It's more efficient and doesn't
> drop any functionality. We could add protocol buffer support, but let's
> leave that for some other time.
>
> Also keep in mind that when you want to stick to the JSON standard
> (well, one of them), you should not write the string yourself but use a
> library to create it. Arguments that it's hard to type or spot a
> mistake are hardly relevant.

Okay.
Now I know you seem to follow JavaScript syntax rather than JSON standard.

But please remind JSON is not only for JavaScript.
It might be used for a communication with Vim and scripts, external
commands, and so on.

And more, currently, 'jsonencode({"key": v:none})' produces output '{"key":}'.
This is not even a correct JavaScript syntax.

And as ZyX said, please delegate the role of human readable format to
other formats, like YAML.
JSON should do just a communication work with something outside Vim.

> you should not write the string yourself but use a
> library to create it.

Hmm, why?

[Bram Moolenaar]

Tyru wrote:

> Sorry Bram, I forgot to send this mail to vim_dev.
> Send again with +alpha about problems of current JSON features.
>
> On Sat, Feb 6, 2016 at 11:50 PM, Bram Moolenaar <Br...@moolenaar.net> wrote:
> >
> >> What do you think about this?
> >
> > I don't want to spend much time discussing this. v:null and v:none are
> > needed just like JavaScript has null and undefined.
> >
> > I was thinking of taking this a step further to a more efficient
> > encoding that is similar to Javascript. Unfortunately I haven't been
> > able to find a specification. I thought it was used in combination with
> > protocol buffers. Besides empty entries in an array, it also removes
> > the quotes around object item names. It's more efficient and doesn't
> > drop any functionality. We could add protocol buffer support, but let's
> > leave that for some other time.
> >
> > Also keep in mind that when you want to stick to the JSON standard
> > (well, one of them), you should not write the string yourself but use a
> > library to create it. Arguments that it's hard to type or spot a
> > mistake are hardly relevant.
>
> Okay.
> Now I know you seem to follow JavaScript syntax rather than JSON standard.

True. Perhaps we should split this and add jsencode() / jsdecode().

> But please remind JSON is not only for JavaScript.
> It might be used for a communication with Vim and scripts, external
> commands, and so on.
>
> And more, currently, 'jsonencode({"key": v:none})' produces output '{"key":}'.
> This is not even a correct JavaScript syntax.

I'll fix that.

> And as ZyX said, please delegate the role of human readable format to
> other formats, like YAML.
> JSON should do just a communication work with something outside Vim.

YAML has its advantages and disadvantages, I don't like it for
inter-process communication.
JSON is a nice format in many ways, although the requirement for quotes
isn't that nice.

> > you should not write the string yourself but use a
> > library to create it.
>
> Hmm, why?

To avoid mistakes. But you can create the strings manually if you want
to, that's the advantage of using JSON over a binary format. Easier for
debugging too. And for writing tests.

--
hundred-and-one symptoms of being an internet addict:

159. You get excited whenever discussing your hard drive.

[tyru]

Hi Bram,

On Sun, Feb 7, 2016 at 2:42 AM, Bram Moolenaar <Br...@moolenaar.net> wrote:
>
> Tyru wrote:
>
>> Sorry Bram, I forgot to send this mail to vim_dev.
>> Send again with +alpha about problems of current JSON features.
>>
>> On Sat, Feb 6, 2016 at 11:50 PM, Bram Moolenaar <Br...@moolenaar.net> wrote:
>> >
>> >> What do you think about this?
>> >
>> > I don't want to spend much time discussing this. v:null and v:none are
>> > needed just like JavaScript has null and undefined.
>> >
>> > I was thinking of taking this a step further to a more efficient
>> > encoding that is similar to Javascript. Unfortunately I haven't been
>> > able to find a specification. I thought it was used in combination with
>> > protocol buffers. Besides empty entries in an array, it also removes
>> > the quotes around object item names. It's more efficient and doesn't
>> > drop any functionality. We could add protocol buffer support, but let's
>> > leave that for some other time.
>> >
>> > Also keep in mind that when you want to stick to the JSON standard
>> > (well, one of them), you should not write the string yourself but use a
>> > library to create it. Arguments that it's hard to type or spot a
>> > mistake are hardly relevant.
>>
>> Okay.
>> Now I know you seem to follow JavaScript syntax rather than JSON standard.
>
> True. Perhaps we should split this and add jsencode() / jsdecode().

Sounds great!
Do you have the plan to add jsencode() / jsdecode() ?
Or, will you merge the patch for those functions if I write?

>
>> But please remind JSON is not only for JavaScript.
>> It might be used for a communication with Vim and scripts, external
>> commands, and so on.
>>
>> And more, currently, 'jsonencode({"key": v:none})' produces output '{"key":}'.
>> This is not even a correct JavaScript syntax.
>
> I'll fix that.

Thanks!
Now I confirmed that 7.4.1269 raises an E474 error for the expression.
But yet it seems to return the string ('{"key":}').
Is it intentional?

[Nikolay Aleksandrovich Pavlov]

YAML is for things like configuration files, it was never meant for IPC.

JSON is one of the formats that tries to sit on two chairs: being
format for IPC and being human-readable, mostly because of its
background. And for IPC quotes do not matter and strict parser would
be preferred because “incoming” messages are also generated by a
computer and non-JSON stings indicate some bug.

Though JSON does not fit for IPC used by Vim: buffer lines can contain
strings which are not UTF-8, buffer names can contain such strings,
variable values are also not restricted; and all this even if
&encoding is UTF-8, so you can safely transfer basically no data*.
Format which allows binary strings would be preferred here.

* Python-3 bindings have exactly the same problem: you cannot use
vim.current.buffer.name, vim.eval, vim.current.buffer[N] without
possibly getting UnicodeEncodeError. Most plugins still ignore this
problem which they can because it is not present in Python-2 and
strings with characters that do not fit into &encoding are rare.

May be fixed by using 'surrogateescape' when encoding, which is also
one of the options for JSON, except that I cannot say how many
implementations can accept string which contains only low part of the
surrogate. Python JSON implementation accepts both strings
`’"\"\uDCFF\""` (U+DCFF it is the result of using 'surrogateescape' to
decode “UTF-8” string b'\xFF') and `"\"\\uDCFF\""`.

>
>> > you should not write the string yourself but use a
>> > library to create it.
>>
>> Hmm, why?
>
> To avoid mistakes. But you can create the strings manually if you want
> to, that's the advantage of using JSON over a binary format. Easier for
> debugging too. And for writing tests.
>
> --
> hundred-and-one symptoms of being an internet addict:
> 159. You get excited whenever discussing your hard drive.
>
> /// Bram Moolenaar -- Br...@Moolenaar.net -- http://www.Moolenaar.net \\\
> /// sponsor Vim, vote for features -- http://www.Vim.org/sponsor/ \\\
> \\\ an exciting new programming language -- http://www.Zimbu.org ///
> \\\ help me help AIDS victims -- http://ICCF-Holland.org ///
>

[Bram Moolenaar]

Tyru wrote:

> >> Sorry Bram, I forgot to send this mail to vim_dev.
> >> Send again with +alpha about problems of current JSON features.
> >>
> >> On Sat, Feb 6, 2016 at 11:50 PM, Bram Moolenaar <Br...@moolenaar.net> wrote:
> >> >
> >> >> What do you think about this?
> >> >
> >> > I don't want to spend much time discussing this. v:null and v:none are
> >> > needed just like JavaScript has null and undefined.
> >> >
> >> > I was thinking of taking this a step further to a more efficient
> >> > encoding that is similar to Javascript. Unfortunately I haven't been
> >> > able to find a specification. I thought it was used in combination with
> >> > protocol buffers. Besides empty entries in an array, it also removes
> >> > the quotes around object item names. It's more efficient and doesn't
> >> > drop any functionality. We could add protocol buffer support, but let's
> >> > leave that for some other time.
> >> >
> >> > Also keep in mind that when you want to stick to the JSON standard
> >> > (well, one of them), you should not write the string yourself but use a
> >> > library to create it. Arguments that it's hard to type or spot a
> >> > mistake are hardly relevant.
> >>
> >> Okay.
> >> Now I know you seem to follow JavaScript syntax rather than JSON standard.
> >
> > True. Perhaps we should split this and add jsencode() / jsdecode().
>
> Sounds great!
> Do you have the plan to add jsencode() / jsdecode() ?
> Or, will you merge the patch for those functions if I write?

Feel free to make a patch.

> >> But please remind JSON is not only for JavaScript.
> >> It might be used for a communication with Vim and scripts, external
> >> commands, and so on.
> >>
> >> And more, currently, 'jsonencode({"key": v:none})' produces output '{"key":}'.
> >> This is not even a correct JavaScript syntax.
> >
> > I'll fix that.
>
> Thanks!
> Now I confirmed that 7.4.1269 raises an E474 error for the expression.
> But yet it seems to return the string ('{"key":}').
> Is it intentional?

When there is an error it returns what it has. Would it be better to
return nothing?


--
You're as much use as a condom machine at the Vatican.
-- Rimmer to Holly in Red Dwarf 'Queeg'

[tyru]

Okay, thanks!

>
>> >> But please remind JSON is not only for JavaScript.
>> >> It might be used for a communication with Vim and scripts, external
>> >> commands, and so on.
>> >>
>> >> And more, currently, 'jsonencode({"key": v:none})' produces output '{"key":}'.
>> >> This is not even a correct JavaScript syntax.
>> >
>> > I'll fix that.
>>
>> Thanks!
>> Now I confirmed that 7.4.1269 raises an E474 error for the expression.
>> But yet it seems to return the string ('{"key":}').
>> Is it intentional?
>
> When there is an error it returns what it has. Would it be better to
> return nothing?

Hmm, patch 7.4.1270 seemed to fix this temporarily last night
and the result was totally what I was thinking;
which returns 'v:none' instead of what it has.

An explicit error return value is better I think
because immediately we can notice something went wrong.
and if only a user checks the return value (without surrounding :try ~ :catch),
a user can detect an error, like 'delete()' returns -1 with an error.

let json = jsonencode({"key": v:none})
if json is v:none
echoerr 'error!'
return ...
endif

And sorry, tyru is a nickname.
The real name is Takuya Fujiwara.
I think you prefer a real name :)
(changed my nickname on google groups, too)
(oh, ':helpgrep Tyru' also shows my nickname in some places...)

[Bram Moolenaar]

Well, let's just return an empty string. Returning a different type
often makes it more complicated.

> And sorry, tyru is a nickname.
> The real name is Takuya Fujiwara.
> I think you prefer a real name :)
> (changed my nickname on google groups, too)
> (oh, ':helpgrep Tyru' also shows my nickname in some places...)

Yes, I do prefer real names. Thanks.

--
hundred-and-one symptoms of being an internet addict:

166. You have been on your computer soo long that you didn't realize
you had grandchildren.

[Nikolay Aleksandrovich Pavlov]

Your code should not assume that execution continues after error. Wrap
the whole thing into :try/:catch and `echoerr` will never be run. And
it is not necessary you that will use :try/:catch, users of your
plugin may do the same thing and it will stop.

If you need to handle the error reliably use :try/:catch, there are no
other options. And *never* use :echoerr, it has just the same problem:
you do not know whether Vim will continue executing your code or
:echoerr error will be transformed into exception, jumping to :catch,
:finally or halting completely right after :echoerr.

>
>>
>>
>> --
>> You're as much use as a condom machine at the Vatican.
>> -- Rimmer to Holly in Red Dwarf 'Queeg'
>>
>> /// Bram Moolenaar -- Br...@Moolenaar.net -- http://www.Moolenaar.net \\\
>> /// sponsor Vim, vote for features -- http://www.Vim.org/sponsor/ \\\
>> \\\ an exciting new programming language -- http://www.Zimbu.org ///
>> \\\ help me help AIDS victims -- http://ICCF-Holland.org ///
>

[Nikolay Aleksandrovich Pavlov]

I do not think changing anything has sense, there is no way to
reliably use json variable in this example in any case. :try/:catch is
reliable, but json variable will not be defined in this case.

>
>> And sorry, tyru is a nickname.
>> The real name is Takuya Fujiwara.
>> I think you prefer a real name :)
>> (changed my nickname on google groups, too)
>> (oh, ':helpgrep Tyru' also shows my nickname in some places...)
>
> Yes, I do prefer real names. Thanks.
>
> --
> hundred-and-one symptoms of being an internet addict:
> 166. You have been on your computer soo long that you didn't realize
> you had grandchildren.
>
> /// Bram Moolenaar -- Br...@Moolenaar.net -- http://www.Moolenaar.net \\\
> /// sponsor Vim, vote for features -- http://www.Vim.org/sponsor/ \\\
> \\\ an exciting new programming language -- http://www.Zimbu.org ///
> \\\ help me help AIDS victims -- http://ICCF-Holland.org ///
>

[tyru]

I agree.
However, I think "E706: Variable type mismatch" should be removed
becase it is not so useful for me...

[tyru]

Yes.
You seem to know well about both Vim script and C implementation :)

I also think :try/:catch is a more practical way to handle an error in
Vim script.
But, what I wanted to say in that example is that,

1. I suppose that a Vim script function must return a value even when
an error raised
(like 'delete()' returns -1 on error) *not just throwing an exception*.
* (Please tell me if the implementation like *just throwing an
exception* function is possible.
Because I don't know much about Vim's implementation)

2. I also think it is simpler if jsonencode() just throws an exception.

BTW, in Vim script, an error and an exception are slightly different.
Also I don't use :echoerr for error message
and I add 'abort' to all functions (my old script doesn't have it,
though... ;) ).

The safest way to echo an error message, is to create a function like below.
But it is too complicated for an example :)

function s:error(msg) abort
try
echohl ErrorMsg
echomsg a:msg
finally
echohl None
endtry
endtry

[Nikolay Aleksandrovich Pavlov]

Functions cannot “just throw an exception”. It is possible that they
throw it (though I do not know built-ins which do this without using
:try), but return value will be present in any case.

>
> BTW, in Vim script, an error and an exception are slightly different.
> Also I don't use :echoerr for error message
> and I add 'abort' to all functions (my old script doesn't have it,
> though... ;) ).
>
> The safest way to echo an error message, is to create a function like below.
> But it is too complicated for an example :)

In the examples I usually use :echo, :echomsg (without highlighting)
or simply `" handle error`.

[Bram Moolenaar]

Takuya Fujiwara wrote:

[...]

> >> let json = jsonencode({"key": v:none})
> >> if json is v:none
> >> echoerr 'error!'
> >> return ...
> >> endif
> >
> > Well, let's just return an empty string. Returning a different type
> > often makes it more complicated.
>
> I agree.
> However, I think "E706: Variable type mismatch" should be removed
> becase it is not so useful for me...

Where do you get that? Or do you mean that the sticky type checking
isn't all that useful in general? I've been wondering whether we would
be better off without it.

--
In Joseph Heller's novel "Catch-22", the main character tries to get out of a
war by proving he is crazy. But the mere fact he wants to get out of the war
only shows he isn't crazy -- creating the original "Catch-22".

[tyru]

Hi Bram,

2016/02/08 3:22 "Bram Moolenaar" <Br...@moolenaar.net>:

>
>
> Takuya Fujiwara wrote:
>
> [...]
>
> > >>     let json = jsonencode({"key": v:none})
> > >>     if json is v:none
> > >>         echoerr 'error!'
> > >>         return ...
> > >>     endif
> > >
> > > Well, let's just return an empty string.  Returning a different type
> > > often makes it more complicated.
> >
> > I agree.
> > However, I think "E706: Variable type mismatch" should be removed
> > becase it is not so useful for me...
>
> Where do you get that?  Or do you mean that the sticky type checking
> isn't all that useful in general?  I've been wondering whether we would
> be better off without it.

I don't think it is useful in lightweight languages
(I don't think *only* a strict type checking is useful).
A strict type checking is useful for a language which has a strong type system.
Because we can detect errors by different types, and so on.

[tyru]

2016/02/08 10:53 "tyru" <tyru...@gmail.com>:

>
> Hi Bram,
>
> 2016/02/08 3:22 "Bram Moolenaar" <Br...@moolenaar.net>:
>
> >
> >
> > Takuya Fujiwara wrote:
> >
> > [...]
> >
> > > >>     let json = jsonencode({"key": v:none})
> > > >>     if json is v:none
> > > >>         echoerr 'error!'
> > > >>         return ...
> > > >>     endif
> > > >
> > > > Well, let's just return an empty string.  Returning a different type
> > > > often makes it more complicated.
> > >
> > > I agree.
> > > However, I think "E706: Variable type mismatch" should be removed
> > > becase it is not so useful for me...
> >
> > Where do you get that?  Or do you mean that the sticky type checking
> > isn't all that useful in general?  I've been wondering whether we would
> > be better off without it.
>
> I don't think it is useful in lightweight languages
> (I don't think *only* a strict type checking is useful).
> A strict type checking is useful for a language which has a strong type system.
> Because we can detect errors by different types, and so on.

Ah, sorry.
I mean, we can detect errors by different types, and so on *at compile-time*.

[Bram Moolenaar]

[adjusted the subject, it was "Why is v:none needed"]

Takuya Fujiwara wrote:

> > [...]
> >
> > > >> let json = jsonencode({"key": v:none})
> > > >> if json is v:none
> > > >> echoerr 'error!'
> > > >> return ...
> > > >> endif
> > > >
> > > > Well, let's just return an empty string. Returning a different type
> > > > often makes it more complicated.
> > >
> > > I agree.
> > > However, I think "E706: Variable type mismatch" should be removed
> > > becase it is not so useful for me...
> >
> > Where do you get that? Or do you mean that the sticky type checking
> > isn't all that useful in general? I've been wondering whether we would
> > be better off without it.
>
> I don't think it is useful in lightweight languages
> (I don't think *only* a strict type checking is useful).
> A strict type checking is useful for a language which has a strong type
> system.
> Because we can detect errors by different types, and so on.

At the time it seemed like a good idea, to have some type checking to
avoid mistakes. Over time I can't say this has uncovered a problem. I
did run into the error when I didn't want it. Currently I think we
might be better off without it.

So, let's ask: who finds the sticky type checking useful?

--
hundred-and-one symptoms of being an internet addict:

178. You look for an icon to double-click to open your bedroom window.

[tyru]

+1.

I'll show a example here, why we might be better off a sticky type checking.

for i in ['foo', 'bar', {'key': 'baz'}]
echo string(i)
endfor

Will output:

'foo'
'bar'
E706: Variable type mismatch for: i

We need 'unlet i'.
It is easy to miss it...

for i in ['foo', 'bar', {'key': 'baz'}]
echo string(i)

" If you :continue here, you need to :unlet in the condition, too...
if ...
unlet i
continue
endif

" We need this always!
unlet i
endfor

[Olaf Dabrunz]

+1 for these, they are somewhat a nuisance.

OTOH, sticky type checking has found problems for me, e.g. when I
changed the way a list was created, sticky type checking pointed to the
places where I forgot to update code that uses the list.


Generally, I believe type checking is more useful in low level
languages, where type mismatches are very likely to have real, even
catastrophic bugs as consequences.

In script languages I tend to prefer it when the language and script
functions do something "sensible" for each data type, including
automatic data type conversion. Errors can be found more easily through
testing.

Script languages typically avoid catastrophic errors such as
segfaults, and they are interpreted, so no setup time is required
for every test. So testing each change quickly is a much more
viable option than for low level languages.

That being said, I more or less often need a mechanism that finds all
uses of some data structure, including when used through aliases. This
helps to find all code that must be adapted after changing the
definition of a data structure. This could be done during tests by
injecting a wrong data type into a data structure and then looking for
errors, or by locking the changed data item in the structure (if that
works consistently).

So IMHO one alternative for sticky types is consistent locks.

--
Olaf Dabrunz (oda <at> fctrace.org)

[Alejandro Exojo]

El Saturday 06 February 2016, Bram Moolenaar escribió:
> I don't want to spend much time discussing this. v:null and v:none are
> needed just like JavaScript has null and undefined.

Having too many falsey types/values (null, undefined, NaN...) is considered
one of the flaws of JavaScript. Crockford, who is a JavaScript promoter, and
created JSON, did not include anything besides null.

> I was thinking of taking this a step further to a more efficient
> encoding that is similar to Javascript. Unfortunately I haven't been
> able to find a specification. I thought it was used in combination with
> protocol buffers. Besides empty entries in an array, it also removes
> the quotes around object item names. It's more efficient and doesn't
> drop any functionality. We could add protocol buffer support, but let's
> leave that for some other time.

You can consider CBOR as well, if you want something well standarized:

http://tools.ietf.org/html/rfc7049

There are some similarities with msgpack, but IIRC it has some additions. Here
is a library, in case you want to consider adding it or playing with it:

https://github.com/01org/tinycbor

> Also keep in mind that when you want to stick to the JSON standard
> (well, one of them),

Is there more than one JSON standard? The standard might have flaws, and even
more so the implementations, but there is supposed to be only one.

--
Alex (a.k.a. suy) | GPG ID 0x0B8B0BC2
http://barnacity.net/ | http://disperso.net

[Bram Moolenaar]

Alejandro Exojo wrote:

> El Saturday 06 February 2016, Bram Moolenaar escribió:
> > I don't want to spend much time discussing this. v:null and v:none are
> > needed just like JavaScript has null and undefined.
>
> Having too many falsey types/values (null, undefined, NaN...) is considered
> one of the flaws of JavaScript. Crockford, who is a JavaScript promoter, and
> created JSON, did not include anything besides null.

I know Crockford. He has some good ideas and some bad ideas. He isn't
really a JavaScript promotor, but he does make money from the book, so
you can't expect him to be too negative :-).

> > I was thinking of taking this a step further to a more efficient
> > encoding that is similar to Javascript. Unfortunately I haven't been
> > able to find a specification. I thought it was used in combination with
> > protocol buffers. Besides empty entries in an array, it also removes
> > the quotes around object item names. It's more efficient and doesn't
> > drop any functionality. We could add protocol buffer support, but let's
> > leave that for some other time.
>
> You can consider CBOR as well, if you want something well standarized:
>
> http://tools.ietf.org/html/rfc7049
>
> There are some similarities with msgpack, but IIRC it has some
> additions. Here is a library, in case you want to consider adding it
> or playing with it:
>
> https://github.com/01org/tinycbor

I don't like binary formats.

> > Also keep in mind that when you want to stick to the JSON standard
> > (well, one of them),
>
> Is there more than one JSON standard? The standard might have flaws,
> and even more so the implementations, but there is supposed to be only
> one.

https://tools.ietf.org/html/rfc4627
https://tools.ietf.org/html/rfc7159.html
http://www.ecma-international.org/publications/files/ECMA-ST/ECMA-404.pdf

There was also rfc7158 but it was discarded.

I already pointed out some flaws. Another one is that it should only
support utf-8 and nothing else to simplify encoders/decoders.

--
hundred-and-one symptoms of being an internet addict:

183. You move your coffeemaker next to your computer.

[Bram Moolenaar]

Sorry, I'm confused. Do you +1 sticky type checking or the removal?

> for i in ['foo', 'bar', {'key': 'baz'}]
> echo string(i)
> endfor
>
> Will output:
>
> 'foo'
> 'bar'
> E706: Variable type mismatch for: i
>
> We need 'unlet i'.
> It is easy to miss it...

Well, this is both an argument for and against. It's helpful if you
wanted to be warned about mixed types, it's annoying if you
intentionally have mixed types.

> for i in ['foo', 'bar', {'key': 'baz'}]
> echo string(i)
>
> " If you :continue here, you need to :unlet in the condition, too...
> if ...
> unlet i
> continue
> endif
>
> " We need this always!
> unlet i
> endfor

--
hundred-and-one symptoms of being an internet addict:

184. You no longer ask prospective dates what their sign is, instead
your line is "Hi, what's your URL?"

[Nikolay Aleksandrovich Pavlov]

I tend to have mixed types more often then fixed (mostly this is the
case in “generic” functions like the recursive map suggested by you in
thread about v:none), and usually sticky type checking is a *source*
of the bugs in this case, should I forget about it.

In cases when sticky type checking failure indicates a bug some type
error usually occurs later in expression like `let s = str_set_to_list
. "another str"` and this bug will be noticed immediately whether or
not sticky type checking errorred out. Most of time it is not hard to
trace where did that value come from.

On the other side in cases with iterating over containers I remember
cases when I added possibility for the container to contain a value of
another type and got an error some time later because I modified code
to work with new values in one place, but forgot `:unlet`. This means
that whenever I think container may contain values of different types
*now or in the future* I use `:unlet` and sticky type checking loses
any sense.

>
>> for i in ['foo', 'bar', {'key': 'baz'}]
>> echo string(i)
>>
>> " If you :continue here, you need to :unlet in the condition, too...
>> if ...
>> unlet i
>> continue
>> endif
>>
>> " We need this always!
>> unlet i
>> endfor
>
> --
> hundred-and-one symptoms of being an internet addict:
> 184. You no longer ask prospective dates what their sign is, instead
> your line is "Hi, what's your URL?"
>
> /// Bram Moolenaar -- Br...@Moolenaar.net -- http://www.Moolenaar.net \\\
> /// sponsor Vim, vote for features -- http://www.Vim.org/sponsor/ \\\
> \\\ an exciting new programming language -- http://www.Zimbu.org ///
> \\\ help me help AIDS victims -- http://ICCF-Holland.org ///
>

[tyru]

I +1 the removal, of cource! :)
I don't like it because I have so often struggled a type mismatch error...

[Alejandro Exojo]

El Monday 08 February 2016, Bram Moolenaar escribió:

> Alejandro Exojo wrote:
> > You can consider CBOR as well, if you want something well standarized:
> >
> > http://tools.ietf.org/html/rfc7049
> >
> > There are some similarities with msgpack, but IIRC it has some
> > additions. Here is a library, in case you want to consider adding it
> > or playing with it:
> >
> > https://github.com/01org/tinycbor
>
> I don't like binary formats.

Then you confuse me more. Isn't Protocol Buffers binary?

Also, given that the goal is RPC, and not just a format, note that both the
old and the new guys of Protocol Buffers are considering newer approaches:

https://capnproto.org/news/2014-06-17-capnproto-flatbuffers-sbe.html

> > > Also keep in mind that when you want to stick to the JSON standard
> > > (well, one of them),
> >
> > Is there more than one JSON standard? The standard might have flaws,
> > and even more so the implementations, but there is supposed to be only
> > one.
>
> https://tools.ietf.org/html/rfc4627
> https://tools.ietf.org/html/rfc7159.html
> http://www.ecma-international.org/publications/files/ECMA-ST/ECMA-404.pdf
>
> There was also rfc7158 but it was discarded.

One RFC obsoltes the other. The more up to date RFC is more recent than the
ECMA paper, but I don't see any significant difference, specially an
incompatible one. It just says:

RFC 4627 was written by Douglas Crockford. This document was
constructed by making a relatively small number of changes to that
document; thus, the vast majority of the text here is his.

I see some clarification with respect to encoding, actually.

> I already pointed out some flaws. Another one is that it should only
> support utf-8 and nothing else to simplify encoders/decoders.

The RFC doesn't say so. But yes, if you put an invalid text inside a text
type, you are going to get awkward results. A string type in some languages or
libraries just doesn't support broken Unicode.

If the definition of a text type is what is specified by Unicode (why would
you choose a different specification of something as complex as that?), you
need to be strict, and use a byte array if you need something more liberal.

[Bram Moolenaar]

Alejandro Exojo wrote:

> > > You can consider CBOR as well, if you want something well standarized:
> > >
> > > http://tools.ietf.org/html/rfc7049
> > >
> > > There are some similarities with msgpack, but IIRC it has some
> > > additions. Here is a library, in case you want to consider adding it
> > > or playing with it:
> > >
> > > https://github.com/01org/tinycbor
> >
> > I don't like binary formats.
>
> Then you confuse me more. Isn't Protocol Buffers binary?

Protocol buffers is a specification format. There is an implementation
with binary messages. Messages are quite efficient, but require bit
manipulation for encoding and decoding. Doesn't work so well in some
languages, e.g. JavaScript. And it's impossible to read the message
without a tool that knows the format being encoded. Bit of a problem if
you're single stepping through code.

The JS encoding is nearly as efficient and is (more or less) readable.
Encoding/decoding performance isn't much different in most languages,
it's much faster in JavaScript (so long as you trust eval()).

> Also, given that the goal is RPC, and not just a format, note that both the
> old and the new guys of Protocol Buffers are considering newer approaches:
>
> https://capnproto.org/news/2014-06-17-capnproto-flatbuffers-sbe.html

Don't know it. At least see some statements that aren't exactly true.
Seems very similar to protocol buffers.

> > > > Also keep in mind that when you want to stick to the JSON standard
> > > > (well, one of them),
> > >
> > > Is there more than one JSON standard? The standard might have flaws,
> > > and even more so the implementations, but there is supposed to be only
> > > one.
> >
> > https://tools.ietf.org/html/rfc4627
> > https://tools.ietf.org/html/rfc7159.html
> > http://www.ecma-international.org/publications/files/ECMA-ST/ECMA-404.pdf
> >
> > There was also rfc7158 but it was discarded.
>
> One RFC obsoltes the other. The more up to date RFC is more recent than the
> ECMA paper, but I don't see any significant difference, specially an
> incompatible one. It just says:
>
> RFC 4627 was written by Douglas Crockford. This document was
> constructed by making a relatively small number of changes to that
> document; thus, the vast majority of the text here is his.
>
> I see some clarification with respect to encoding, actually.

Perhaps I should write my own standard and state that this obsoletes the
others, since mine is of course going to be better. (hint: obsolete
doesn't mean unused).

> > I already pointed out some flaws. Another one is that it should only
> > support utf-8 and nothing else to simplify encoders/decoders.
>
> The RFC doesn't say so. But yes, if you put an invalid text inside a
> text type, you are going to get awkward results. A string type in some
> languages or libraries just doesn't support broken Unicode.
>
> If the definition of a text type is what is specified by Unicode (why would
> you choose a different specification of something as complex as that?), you
> need to be strict, and use a byte array if you need something more liberal.

Yep. And then you can also support NULs. But don't confuse syntax and
semantics, you can encode any binary byte sequence into a JSON string.

--
hundred-and-one symptoms of being an internet addict:

204. You're being audited because you mailed your tax return to the IRC.

[Nikolay Aleksandrovich Pavlov]

How exactly? Whatever you choose this is going to either make
resulting format less efficient or less portable. Example of less
portable: surrogateescape scheme where byte 0xFF is encoded as U+DCFF
(resulting in `"\uDCFF"` JSON string because literal U+DCFF is not a
valid UTF-8 string) without another character from the surrogate pair.
Works fine in Python, but it does not have to. “Less portable” variant
may turn into “less efficient” or “not working at all” with some
implementations.

Example of less efficient: quoted-printable encoding. Does not matter
what exactly you will choose, this thing will require all strings to
be additionally filtered on the other side, without custom JSON parser
this may make parsing+postprocessing your JSON even a magnitude or
more slower: in cases like Python where json module is written in C,
but postprocessor will be written in Python.

There is a middle option though: use channel types
“json-surrogateescape” and “json-quoted-printable”, it would be up to
server what exactly to choose. Though I would have chosen a binary
format in any case (but not msgpack like in Neovim unless you are
actually going to be compatible, protocol buffers binary messages look
better).

>
> --
> hundred-and-one symptoms of being an internet addict:
> 204. You're being audited because you mailed your tax return to the IRC.
>
> /// Bram Moolenaar -- Br...@Moolenaar.net -- http://www.Moolenaar.net \\\
> /// sponsor Vim, vote for features -- http://www.Vim.org/sponsor/ \\\
> \\\ an exciting new programming language -- http://www.Zimbu.org ///
> \\\ help me help AIDS victims -- http://ICCF-Holland.org ///
>

[Bram Moolenaar]

It seems base64 encoding is the most popular. There might be another
that's a bit more efficient. Anyway, to JSON it's just a text string,
you will have to encode your binary into that string, and on the other
side decode. There isn't really any other way, unless using raw format,
and then you have the problem that you need some way to separate
messages. You can use NL, but then you need to escape NL, and then
escape character...

--
Your mouse has moved. Windows must be restarted for the change
to take effect. Reboot now?

[tyru]

Is there any progress on this? >Removing sticky type checking

--
Takuya Fujiwara