Don't ban nil checking, discourage use of the # operator

[Dirk Laurie]

John Hind wrote:

> 8. When using the default length determination method (i.e. edge
> detection), the 'insert' function errors on parameters that would break
> 'proper sequence' i.e. it does not allow insertion of 'nil' values and
> it only allows the table to grow contiguously from the top.

> Dirk hates item 8 and despite a lengthy and occasionally acrimonious exchange
> of emails we've not reached a shared understanding of what the reason
> for the objection is, let alone agreement on whether it should be in
> or out!

We actually differ very little. It can be narrowed down to:

Dirk: There is no point in banning nils since the user can always insert
nils anyplace he likes in other ways.
John: That's not our responsibility. But to call a function "insert" and
allow it to make a list shorter is an affront to common sense.
Dirk: OK, put in an option to switch on the check, but let it be off by
default, otherwise it might break some programs.
John: If the test is not automatic, it is barely worth adding it because
it will rarely be used. If we do not add it, or not in a way that
actually gets used, it is not worth the effort of releasing and
supporting xtable.

John has suggested he would be willing to replace banning nils by instead
banning the default Lua `#` operator based on edge detection. That takes
care of my first objection, but it would break even more programs that
happily work with `table`.

But here is an idea I've not yet played past John.

Suppose that instead of making an existing program actually fail, it
merely turns it into nagware. The moment you call `insert` without
either an explicit `len` or a `__len` metamethod being in place,
a cautionary message appears right in your standard output, Cc to
standard error, and the program continues. You can switch that off
only by some not-too-easy method, e.g `debug.setupvalue(insert,1,false)`.

If John agrees to this, our differences would be reduced to the size
and wording of the message, and the number of times it appears before
the program accepts that you are ignoring it.

[steve donovan]

On Fri, Feb 15, 2013 at 8:56 AM, Dirk Laurie <dirk....@gmail.com> wrote:
> John has suggested he would be willing to replace banning nils by instead
> banning the default Lua `#` operator based on edge detection. That takes
> care of my first objection, but it would break even more programs that
> happily work with `table`.

Hm, well we _know_ # is not entirely reliable, which is why we try to
keep such tables in a state where it has a defined meaning; friends
don't let friends put holes in tables.

Apart from the inevitable newcomer confusion, is this really such a
terrible source of bugs?

These kind of drastic solutions aren't in the usual 'caveat emptor'
spirit of Lua.

steve d.

[Axel Kittenberger]

Not taking a side here, as the times I liked to discuss language
principles in depth are over :-)

A major strength of Lua has always been, it does not look back! Lua
never had a trouble with dropping backward compatibilty in favor of
proper language design. Thats where most languages usually fall pride
to in their evolution, "must not break old programs", so the shit
start piling up, until at one point the whole language is more or less
dropped for the next hyped lingua.

I remember back the days I think I used to suggest dropping any
default meaning for #. For tables as well as for strings (UTF
encoding?). Its about a user library to put something into the
specific global metatables if must be, but remove it from Lua core.
From the core provide a table.edge( ) function that calcs what # is
now for efficient use if a user environment wants to set # what it
currently is.

Peace out, Axel

[John Hind]

I'd be disappointed if I've just moved this debate online so we get 'n' irreconcilable views rather than just 2!

The question I'd like to answer with this post: Is anyone interested in using xtable?

a) Interested if it has default sequence protection that is easily switched off

b) Interested if it has sequence protection which is off by default, but easily switched on

c) Interested only if sequence protection is removed altogether

d) Not interested - the standard table library is fine

I do not think removing, or not supporting (we could simply raise an error in xtable if there is no __len metamethod and len is not supplied as a parameter) is really an option. I only raised that to clarify my argument that if we support it then we should support it as safely as possible.

I am far from being a newcomer, and I got into this after being affronted when I finally tracked a bug that mysteriously emptied a list altogether. It turned out the bug was a negative pos parameter to insert (anyone want to defend Lua not making that an error?). The negative pos was not right there as a constant - it was due to a complex chain of events in another part of the program. Looking into this deeper I found that insert would also allow nil as a value and did not check the upper boundary either. Either of these could make the list mysteriously shorter or cause it to be emptied.

I do not see this as a problem with edge detection - though the non-determinism of that makes it worse. Rather is is inherent in the definition: edge detection implicitly defines a list as being terminated by a nil value, rather like a string is in C. Even if Lua found the size with a deterministic linear search from 1, nil is not an ordinary value which one can legitimately insert - it is special by definition. If you want nil to be an ordinary value you have to redefine # using a metamethod. Therefore, insert should not allow nil as a value if it is relying on the nil-is-terminator definition - QED!

I do not think the compatibility argument is strong - we break compatibility with the implementation (as opposed to with the manual) in a number of ways in xtable, some of them much worse than this. Dirk struggles to provide a reasonable use case for inserting nils in a default list (mostly involving doing some immediate fix-up afterward). If someone has done this in the real world, they will get a nicely localised and explicit error message, check the manual and find they need to replace:

insert(tab,pos,nil) --> insert(#tab,tab,pos,nil)

Hardly the end of the world! 99% of the time the error message will be flagging a real and highly obscure bug they've been trying to find for weeks.

Neither do I find the "warning to stdout" a good solution - many Lua implementations do not show, or make it easy to see such output.

- John

[Axel Kittenberger]

If you take a look at my workarounds with "Array", "CountArray" and "Queue"

https://github.com/axkibe/lsyncd/blob/master/lsyncd.lua

I'd gladly take something that solves this in a more elegant, ready-made way

Kind regards,
Axel

> --
> You received this message because you are subscribed to the Google Groups
> "lua table semantics" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to lua-table-seman...@googlegroups.com.
> For more options, visit https://groups.google.com/groups/opt_out.
>
>