Why tcl raises a dict related error?

[MartinLemburg@Siemens-PLM]

Hi,

executing the following ...

set header [dict create length [dict create received 2956 calculated 9436 header 1] rgba {0.5960783958435059 0.6666666865348816 0.686274528503418 0.0} vertices 1080 strips 768]

dict with header {
dict with length {
puts $calculated
}
}

... raises the error:

9436
missing value to go with key
% set errorInfo
missing value to go with key
while executing
"dict with header {
dict with length {
puts $calculated
}
}"
% set errorCode
TCL VALUE DICTIONARY

The tcl version is:

% info patchlevel
8.6.4

Should I file a bug?

Cheers,

Martin

[MartinLemburg@Siemens-PLM]

BTW the following works:

set header [dict create length [dict create received 2956 calculated 9436 header 1] rgba {0.5960783958435059 0.6666666865348816 0.686274528503418 0.0} vertices 1080 strips 768]

set length [dict get $header length]

dict with length {
puts $calculated
}

Why suddenly "updating" the header dictionary, after leaving the "dict with header" scope fails?

[MartinLemburg@Siemens-PLM]

Am Donnerstag, 10. März 2016 11:35:34 UTC+1 schrieb MartinLemburg@Siemens-PLM:

And ...

set header [dict create length [dict create received 2956 calculated 9436 header 1] rgba {0.5960783958435059 0.6666666865348816 0.686274528503418 0.0} vertices 1080 strips 768]

dict update header length l {
dict with l {
puts $calculated
}
}

... fails like the sample in the first post.

[Rich]

MartinLemburg@Siemens-PLM <martin.lembur...@gmx.net> wrote:
> Hi,

> executing the following ...

Reformatted to make the structure a bit more visible on usenet:

> set header [dict create
length [dict create received 2956
calculated 9436
header 1]

^^^^^^
222222

rgba {0.5960783958435059 0.6666666865348816 0.686274528503418 0.0}
vertices 1080
strips 768]

> dict with header {

^^^^^^
111111

> dict with length {
> puts $calculated
> }
> }

> ... raises the error:

> 9436
> missing value to go with key
> % set errorInfo
> missing value to go with key
> while executing
> "dict with header {
> dict with length {
> puts $calculated
> }
> }"
> % set errorCode
> TCL VALUE DICTIONARY

> The tcl version is:

> % info patchlevel
> 8.6.4

> Should I file a bug?

No, you should fix your code.

Note that your outermost variable holding the nested dict is named
"header" (item labeled with "111111" above).

Note, also, that inside the nested dict, you also define a key named
"header" (item labeled with "222222" above).

Note the first sentence of the manpage documenation of "dict with":

dict with dictionaryVariable ?key ...? body
Execute the Tcl script in body with the value for each key in
dictionaryVariable mapped (in a manner similarly to dict update)
to a variable with the same name.

Each key in the dict becomes a new variable in the enclosing scope [1]
of the execution of the "dict with".

Now, what do you think happens when your original variable "header"
(originally containing a dict) is overwritten by a new variable
"header" (containing a scalar "1") due to the "dict with" operation on
the nested dict?


Making either of the following changes will fix your bug:

1) change "set header" to "set header2" and "dict with header" to
"dict with header2" (so that the nested dict with does not
overwrite "header" with a scalar).

2) change the key "header" inside the nested dict to header2 (or some
other, non conflicting, name) (again, so that the nested dict with
does not overwrite the outer "header" variable.



[1] Note that "enclosing scope" is that of the proc where this code
resides, or the global namespace if it runs outside of a proc. I.e.,
dict with does not change variables just inside its own code block, it
changes them for the entire proc.

[MartinLemburg@Siemens-PLM]

Am Donnerstag, 10. März 2016 11:35:34 UTC+1 schrieb MartinLemburg@Siemens-PLM:

Again - my own error.

The dict header contains the dict length, which contains a key header.
The nested "dict with" calls causes to overwrite the header dictioanry with the content of the header key, which itself is not a dictionary.
So with leaving the outer "dict with" scope the try to update the header dictionary fails.

Ok - my own fault.

Cheers,

Martin

[heinrichmartin]

On Thursday, March 10, 2016 at 12:25:43 PM UTC+1, MartinLemburg@Siemens-PLM wrote:
> The dict header contains the dict length, which contains a key header.
> The nested "dict with" calls causes to overwrite the header dictioanry with the content of the header key, which itself is not a dictionary.
> So with leaving the outer "dict with" scope the try to update the header dictionary fails.

That's why I don't use [dict with]: Adding a new key to an existing dictionary in your code base could break the code anywhere.

I once considered whether [dict with] should auto-detect such situations and either
* return an error or
* skip offending keys,
but none of these seems intuitive.

Other options are:
* [dict with] could keep the dict elsewhere and copy it back afterwards
* add an option to use dedicated keys only [dict with length {received calculated} {# body}]

[Rich]

heinrichmartin <martin....@frequentis.com> wrote:
> On Thursday, March 10, 2016 at 12:25:43 PM UTC+1, MartinLemburg@Siemens-PLM wrote:
> > The dict header contains the dict length, which contains a key
> > header. The nested "dict with" calls causes to overwrite the header
> > dictioanry with the content of the header key, which itself is not
> > a dictionary. So with leaving the outer "dict with" scope the try
> > to update the header dictionary fails.

> That's why I don't use [dict with]: Adding a new key to an existing
> dictionary in your code base could break the code anywhere.

Yes, it is a slightly dangerous tool, but danger is not always a good
reason to avoid it.

One workaround is if you always use lowercase variable names in your
scripts, is to use some form of camel case names for your dict keys.

I.e., in ML's case, "header" is his proc variable, so "Header" would be
his dict key.

No conflicts that way - but it does require disicipline on the
programmers part in order to "protect" the code.

> I once considered whether [dict with] should auto-detect such
> situations and either
> * return an error or
> * skip offending keys,
> but none of these seems intuitive.

Given that "skipping offending keys" can produce similar subtle bugs,
then returning an error "Error: dict with would overwrite existing
variable X" would seem more appropriate. That points directly at the
potential issue, rather than one staring at debug output wondering "why
isn't my dict key's value showing up here".

Given that the existing default is overwrite, this "safer" version
might need to be a switch that one turns on for backwards compatibility
sake:

dict with -safety dictvar { }

The switch, however, would still require some disipline.

> Other options are:
> * [dict with] could keep the dict elsewhere and copy it back
> afterwards

But, with this option, in the instance where you actually *did* want to
overwrite the local vars, you can't.

> * add an option to use dedicated keys only [dict with length
> {received calculated} {# body}]

Workable, but if you mistakenly forget about your existing 'header'
var, and add {received header calculated} to your key list, you get the
same silent overwrite all over again.

[Donal K. Fellows]

On 10/03/2016 11:35, heinrichmartin wrote:
> I once considered whether [dict with] should auto-detect such situations and either
> * return an error or
> * skip offending keys,
> but none of these seems intuitive.

How would it detect that a key is offending? That's a non-trivial
problem. Tell me that first. :-)

> Other options are:
> * [dict with] could keep the dict elsewhere and copy it back afterwards

That prevents updates from elsewhere, which is an explicitly-supported
usage scenario (useful particularly when dealing with nested dicts).
Determining whether a write to a variable is a problem or not is *very*
difficult. At the implementation level, without using traces all you've
got is examining the values at the point when the [dict with] body
finishes, and at that point you've just really got “equal”/“not-equal”,
“valid-dict”/“invalid-dict”, and whether variables are set at all.

With traces, you'd get to see a different timing of the problem, but
you'd still be faced with a non-trivial decision with only minimal
information. You'd also have the overhead of traces, which is
non-trivial for local variables (for various reasons) and isn't how the
[dict with] semantics are defined.

> * add an option to use dedicated keys only
> [dict with length {received calculated} {# body}]

That's what [dict update] is for. It doesn't handle the nested dict case
that [dict with] does, but it does handle key filtering (and binding
them to different names too).

Donal.
--
Donal Fellows — Tcl user, Tcl maintainer, TIP editor.

[heinrichmartin]

On Thursday, March 10, 2016 at 2:38:42 PM UTC+1, Donal K. Fellows wrote:
> > * skip offending keys,
> > but none of these seems intuitive.
>
> How would it detect that a key is offending? That's a non-trivial
> problem. Tell me that first. :-)

I thought of [info locals], but that was brainstorming only. While I feel that some of the dict commands have room for improvement, I know that changes are unlikely...

> > * add an option to use dedicated keys only
> > [dict with length {received calculated} {# body}]
>
> That's what [dict update] is for. It doesn't handle the nested dict case
> that [dict with] does, but it does handle key filtering (and binding
> them to different names too).

dict update header length l {

dict update l calculated c {
puts $c
}
}

I should start using this :-)

[neuro...@gmail.com]

I pointed out the danger of the dict interface 10 years ago, but unfortunately it went through as is anyway.
I shudder to think how much code will break only on certain rare data inputs where dict keys happen to match variable names.
Nobody really likes a 'told you so'.. so let me just say that overall I love the addition of 'dict' to Tcl, but yes it's a sharp tool.

Perhaps try a safer construct like the dictn wrapper over dict.
http://wiki.tcl.tk/17106

(but note that it's old code and could do with a review and update for changes made to dict in the meantime. Still - it gives an idea of some different ways to access dict structures)

[Donal K. Fellows]

On 11/08/2016 06:45, neuro...@gmail.com wrote:
> I pointed out the danger of the dict interface 10 years ago, but
> unfortunately it went through as is anyway.

It's never enough to just say “it might be a problem”. You've also got
to propose what to do instead, and you've got to balance usability in
the common cases against the rare cases. That's the difficult bit. The
fundamental difficulties with [dict] are that if we make paths
ubiquitous then we have problems with simple keys requiring packing with
[list] for safety, and that we can't really determine what the structure
“ought” to be just by looking, making things like a general tree
traversal impossible. It's just not possible to make all use cases work
well without introducing impossibly-many subcommands; we had to just
draw a line and stop worrying about it. (You *can* add more subcommands
to the [dict] ensemble FWIW.)

OTOH, I'm not convinced that doing lots of nesting is usually a good
idea anyway. The most complex “data structure” I've produced in Tcl has
been the output from the [tcl::unsupported::getbytecode] command, and
the structure there is designed to be exploded a part at a time...

[julz]

>... and you've got to balance usability in

> the common cases against the rare cases. That's the difficult bit. The
> fundamental difficulties with [dict] are that if we make paths
> ubiquitous then we have problems with simple keys requiring packing with
> [list] for safety,

Thanks for the explanation. That's a good point - and after all these years it gives me a better understanding of the tradeoffs you had to make.

>and that we can't really determine what the structure
> “ought” to be just by looking, making things like a general tree
> traversal impossible.

do you mean for example that with code like:

dictn set d $keylist myvalue

it's unclear how far this line of code intends to reach into the dict?
I guess there is a certain lack of self-documentation in that code which I hadn't considered.

I'm not understanding your comment about making 'a general tree traversal impossible' - isn't that just a general property of nested dicts rather than the access method? ( that you can't always tell from the data whether a particular value is intended to be a leaf or another dict)

[Donal K. Fellows]

On 12/08/2016 15:49, julz wrote:
> do you mean for example that with code like:
>
> dictn set d $keylist myvalue
>
> it's unclear how far this line of code intends to reach into the dict?

In this case, it's OK because you're describing what structure to
expect/build. (If you want a hand plugging this into [dict] itself, let
me know. It's not quite as easy as "put the code in the right namespace"
for various performance reasons; the bytecode compiler pokes around
inside [dict] and it works in a weird context.)

> I'm not understanding your comment about making 'a general tree
> traversal impossible' - isn't that just a general property of nested
> dicts rather than the access method? ( that you can't always tell
> from the data whether a particular value is intended to be a leaf or
> another dict)

It might not have been a comment that was directly applicable to
anything you're doing. However, here's the explanation. :-) If you've
got a structure with a bunch of nested dictionaries, there's no way to
safely say:

dict foreachLeaf {keyPathVar valueVar} $theDict {
some script...
}

The problem is that the dictionary code doesn't know which values at any
level it can treat as dictionaries and which it should not, since a list
with an even number of elements, or even a string with an even number of
words, might be a dictionary if asked to convert. Such a traversal
requires the *expected* structure to work right, which is approximately
the same as the type (except types are something else; Tcl's actual type
system is sufficiently complicated that we should pretend everything is
a string; strings are the true superclass type).

The structure descriptions just need to say what the non-leaf branches
are *or* what the leaf branches are (well, in so far as non-leaves are
the ones that will be recursed down). It's pretty simple except when
trying to figure out how to do it in general...

I *REALLY* want to continue to allow arbitrary values to be used as both
keys and values. That's important; it's the absolutely most core concept
that I started from with dictionaries, and is how they differ a bit from
how TclX's keyed lists work.

[Rich]

Donal K. Fellows <donal.k...@manchester.ac.uk> wrote:
> It might not have been a comment that was directly applicable to
> anything you're doing. However, here's the explanation. :-) If
> you've got a structure with a bunch of nested dictionaries, there's
> no way to safely say:
>
> dict foreachLeaf {keyPathVar valueVar} $theDict {
> some script...
> }
>
> The problem is that the dictionary code doesn't know which values at
> any level it can treat as dictionaries and which it should not, since
> a list with an even number of elements, or even a string with an even
> number of words, might be a dictionary if asked to convert. Such a
> traversal requires the *expected* structure to work right,

For a visual example, consider this dictionary:

% dict set data entry "the rain in spain"
entry {the rain in spain}

That is clearly a dict with one toplevel key (entry).

But to the general 'dict' code, which is sadly not omnipinent and does
not know what the programmer intended, is this value:

"the rain in spain"

meant to be a leaf, containing a string of 17 characters, or is it
meant to be a second level dict, with keys "the" and "in" mapping to
leaves "rain" and "spain".

Note:
% dict set subdict the rain
the rain
% dict set subdict in spain
the rain in spain
% dict set outerdict entry $subdict
entry {the rain in spain}

Identical net string representation from building the dict in two
different ways.