beta testers needed (proper subfunctions)

[Kevin Lawler]

All,

After a substantial amount of work, Tom has produced a fork with working subfunctions that's ready to be beta tested. This adds the last missing piece needed for full function support (see https://github.com/kevinlawler/kona/blob/master/kx.c#L328), which was to properly configure the namespace of variables visible to subfunctions. This was challenging work. The full function support has been in alpha for a while (https://github.com/kevinlawler/kona/issues/5). Function support is stable enough now that it is ready to be beta tested. 

Please help us by testing the new feature. You can do this by compiling the fork and hunting for subfunction snippets which return the wrong answer or crash Kona. Tom has added wide test coverage already (https://github.com/tavmem/kona/commit/b19f56bfb378ace505aba8c5d535ba3243fdc588#diff-c827ff5fc97c02d4b19c133788531961), so finding a bug may require some added creativity. The added tests give a good template for where to look. Some of the added functionality currently runs slower than optimal: we are testing for correctness first before optimizing for speed.

A good place to report any nits you find is in the issues thread. Starting an email chain on the kona-dev list might also be a good idea.

After beta testing is complete we will merge the fork into the main repo. If you would like to help, please clone the fork at tavmem/kona, play with the build, and report any issues you find.

Tom's beta fork with full function support:

https://github.com/tavmem/kona

test suite additions, for reference:

https://github.com/tavmem/kona/commit/b19f56bfb378ace505aba8c5d535ba3243fdc588#diff-c827ff5fc97c02d4b19c133788531961

subfunctions issue:

https://github.com/kevinlawler/kona/issues/5

Kevin

[Leon Baum]

Wow, great work guys! I use Q at work, but have been meaning to learn K
a bit better. It's quite possible my syntax is off, but using the
subfunction branch, I am getting type errors for what I thought should
be valid:

{x + {[a] a+a} y} . 1 2
5
{x + {[a] a+a} y} [1;2]
type error
{[a;b] a + {x+x} b} . 1 2
5
{[a;b] a + {x+x} b} [1;2]
type error


Leon

[Tom Szczesny]

Hi Leon -

Thanks for doing testing, and for reporting your findings.

Both of the problem cases appear to be syntax errors.

I tried each in k2.8 and got the following results:

K 2.8 2000-10-10 Copyright (C) 1993-2000 Kx Systems

  {x + {[a] a+a} y} [1;2]

syntax error

{x + {[a] a+a} y} [1;2]

                  ^

parse error

  

   {[a;b] a + {x+x} b} [1;2]

syntax error

{[a;b] a + {x+x} b} [1;2] 

                    ^

parse error

Tom

[Leon Baum]

Hi Tom,

Thanks for checking it out. I just tried the examples with KDB's K
prompt, and they did work. But I suppose that means KDB's K is just
non-standard.

Leon

[Bakul Shah]

Works in k-2.95 and k-3.2 once you take out the space between
} and [. But even with that it doesn't in Kona. Since the
first expression works, the second one should too.

IIRC adverbs and [ can't have any intervening space but no
such rule exists for } and [.

[Tom Szczesny]

Thanks Bakul !

Leon:  The k in Q is k4.  Kona is modelled on k3.2 which may be quite different. 

[Tom Szczesny]

A new commit was made to tavmem/kona with a fix for the 2 Leon Baum tests.

[Tom Szczesny]

The 2 new tests work in tavmem/kona with or without the space that can cause a syntax error in k2.8, k2.95 and k3.2.

[paul.a...@gmail.com]

Hello!

It seems like I have found a problem with subfunctions. It can be observed if 1) subfunction is modified with an adverb; 2) argument lists of subfunction and enclosing function differ; 3) there is a local variable.

  {{(7;x)}'[!3]}[]                /OK; no local vars
(7 0
 7 1
 7 2)
  {a:5;{(7;x)}'[!3]}[]          /not OK; there is a local var
((7;)
 (7;)
 (7;))
  {x;a:5;{(7;x)}'[!3]}[5]      /OK; the argument lists are the same
(7 0
 7 1
 7 2)
  {[n]a:5;{[n](7;n)}'[!3]}[5] /OK; the argument lists are the same
(7 0
 7 1
 7 2)
  {[n]n;a:5;{(7;x)}'[!3]}[5]  /not OK; different argument names
((7;)
 (7;)
 (7;))
  {x;a:5;{[n](7;n)}'[!3]}[5]  /not OK; different argument names
((7;)
 (7;)
 (7;))
  {x;{[n](7;n)}'[!3]}[]          /OK; local var matters
(7 0
 7 1
 7 2)

I'm using the latest HEAD on Linux 64.
Sorry if I'm mistaken as I'm pretty new to K, but it really looks like a bug to me.

[paul.a...@gmail.com]

In fact this problem affects even standalone functions. And it seems to break them somehow.

  f:{(7;x)}                   / so we have a very simple function here
{(7;x)}
  f'[!3]                         / it seems to work fine

(7 0
 7 1
 7 2)

  {a:x;f'[!a]}[3]            / argument lists are the same; everything is OK

(7 0
 7 1
 7 2)

  {[n]a:n;f'[!a]}[3]        / argument lists differ; oops

((7;)
 (7;)
 (7;))

  f'[!3]                          / f doesn't work anymore???

[Tom Szczesny]

Each of the problems identified by paul.a... (in the prior 2 postings) have been fixed.

[Michael Lazarev]

Were proper subfunctions merged into Kona?
I tried the following on the interpreter checked out from Github repo today.
Should there be pairs of numbers in the response, or am I doing something wrong?

  for:{[n;f] f'!n}
{[n;f] f'!n}
  for[5]{[i] i}
0 1 2 3 4
  for[5]{[i] for[i]{[j] i,j}}
(()
 ,(;0)
 ((;0)
  (;1))
 ((;0)
  (;1)
  (;2))
 ((;0)
  (;1)
  (;2)
  (;3)))

[Tom Szczesny]

Hi Michael,

Looks like a bug to me.  Thanks

k3.2 yields the following results:

   for:{[n;f] f'!n}

  for[5]{[i] i}

0 1 2 3 4

  for[5]{[i] for[i]{[j] i,j}}

(()

 ,1 0

 (2 0

  2 1)

 (3 0

  3 1

  3 2)

 (4 0

  4 1

  4 2

  4 3))

[Tom Szczesny]

This bug has been fixed.

[Michael Lazarev]

That's great, thanks!

I have tested the following expression:
  for[5]{[i] for[i]{[j] for[j]{[k] i,j,k}}}

What the current build of Kona prints out, seems unexpected to me:

(()
 ,()
 (()
  ,(1;;0))
 (()
  ,(1;;0)
  ((2;;0)
   (2;;1)))
 (()
  ,(1;;0)
  ((2;;0)
   (2;;1))
  ((3;;0)
   (3;;1)
   (3;;2))))

I believe the sequence should end on something like this:

4 1 0
4 2 0
4 2 1
4 3 0
4 3 1
4 3 2

[Tom Szczesny]

This is what I get using k3.2.

It does not agree with your expectations.

It also does not agree with Kona.

It's not immediately clear to me which is correct.

K 3.2 2005-06-25 Copyright (C) 1993-2004 Kx Systems

WIN32 2CPU 4000MB inspiron.home 0 EVAL

  for:{[n;f]f'!n}

  for[5]{[i] for[i]{[j] for[j]{[k] i,j,k}}}

(()

 ,()

 (()

  ,(;1;0))

 (()

  ,(;1;0)

  ((;2;0)

   (;2;1)))

 (()

  ,(;1;0)

  ((;2;0)

   (;2;1))

  ((;3;0)

   (;3;1)

   (;3;2))))

[Tom Szczesny]

k2.8 yields the same result as k3.2

[Tom Szczesny]

I could make Kona agree with k2.8 and k3.2,

but the k2.8 and k3.2 results may not be correct either.

[Michael Lazarev]

It's strange to find out that in k2.8 and k3.2 closures are limited to
two levels in being 'proper', i.e. keeping the information about the
enclosing state. Would be interesting to know what does Mr. Whitney
think about that, and if there are any justifications to such
behavior.

If the goal is to keep Kona as compatible as possible to k3, it might
be reasonable to reproduce the way k3 deals with closures. On the
other hand, the correctly implemented closures have their own
benefits, of which the most important is not burdening the programmer
with the necessity to reason how the variables get discarded on
different levels of nesting.

Regarding compatibility, it is possible that nobody encountered this
situation while using k3, it might be never reported and addressed, so
implementing the closures properly might not affect currently existing
programs.

I am only a beginner in K, learning by myself, not having enough
examples of code, so I may as well use atrociously wrong style that
nobody writes K in, so maybe these nested closures do not really
matter for anybody.

> --
> You received this message because you are subscribed to a topic in the
> Google Groups "Kona Developers" group.
> To unsubscribe from this topic, visit
> https://groups.google.com/d/topic/kona-dev/TayM2rh8k2w/unsubscribe.
> To unsubscribe from this group and all its topics, send an email to
> kona-dev+u...@googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.

[Ryan Gonzalez]

On November 9, 2015 6:08:55 PM CST, Michael Lazarev <lazarev...@gmail.com> wrote:
>It's strange to find out that in k2.8 and k3.2 closures are limited to
>two levels in being 'proper', i.e. keeping the information about the
>enclosing state. Would be interesting to know what does Mr. Whitney
>think about that, and if there are any justifications to such
>behavior.
>
>If the goal is to keep Kona as compatible as possible to k3, it might
>be reasonable to reproduce the way k3 deals with closures. On the
>other hand, the correctly implemented closures have their own
>benefits, of which the most important is not burdening the programmer
>with the necessity to reason how the variables get discarded on
>different levels of nesting.
>
>Regarding compatibility, it is possible that nobody encountered this
>situation while using k3, it might be never reported and addressed, so
>implementing the closures properly might not affect currently existing
>programs.
>
>I am only a beginner in K, learning by myself, not having enough
>examples of code, so I may as well use atrociously wrong style that
>nobody writes K in, so maybe these nested closures do not really
>matter for anybody.
>

FYI, https://github.com/JohnEarnest/ok/blob/gh-pages/docs/Programming.md covers the general K "style" well, even though it's for K5, and https://github.com/kevinlawler/kona/wiki is a good reference.

--
Sent from my Nexus 5 with K-9 Mail. Please excuse my brevity.

[Tom Szczesny]

Michael -

Thanks for the comments on closures.  My first reaction is that it is probably best to implement closures correctly in Kona.

And, by the way, the fix that I implemented for the second level is incorrect and needs further adjustment.

Kona yields 

  for:{[n;f]f'!n}

{[n;f]f'!n}

   for[3]{[i] for[i]{[j] i,j,k}}

(()

 ,(0;0;)

 ((0;0;)

  (0;1;)))

but should yield

  for:{[n;f]f'!n}

  for[3]{[i] for[i]{[j] i,j}}

(()

 ,1 0

 (2 0

  2 1))

--
You received this message because you are subscribed to the Google Groups "Kona Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to kona-dev+u...@googlegroups.com.

[Tom Szczesny]

Arthur Whitney stripped out much of the sub-function (i.e, lambda) features from k4.

Closures have even less support in k4 than they do in k3.2 or k2.8.

They do not even work at level 2.

In k4:

Welcome to kdb+ 32bit edition

For support please see http://groups.google.com/d/forum/personal-kdbplus

Tutorials can be found at http://code.kx.com/wiki/Tutorials

To exit, type \\

To remove this startup msg, edit q.q

q)\

  for:{[n;f]f'!n}

  for[5]{[i] i}

0 1 2 3 4

  for[5] {[i] for[i] {[j] i,j}}

k){[j] i,j}

'i

  )

[Tom Szczesny]

Not surprisingly, the same lack of support for closures also exists in q:

Welcome to kdb+ 32bit edition

For support please see http://groups.google.com/d/forum/personal-kdbplus

Tutorials can be found at http://code.kx.com/wiki/Tutorials

To exit, type \\

To remove this startup msg, edit q.q

q)for:{[n;f] f each til n}

q)for[5]{[i] i}

0 1 2 3 4

q)for[5] {[i] for[i] {[j] i,j}}

{[j] i,j}

'i

q))

[Tom Szczesny]

On a related note, here is a command that works in Kona, but not in k2.8 or k3.2

(Of course, it's possible that it should fail in Kona):

Kona:

  {[n;f] f'!n}[2]{n,x}

(2 0

 2 1)

k3.2:

  {[n;f] f'!n}[2]{n,x}

((;0)

 (;1))

[Tom Szczesny]

Level 2 Closure now works in Kona.

Re: k4 and q (reduced lambda support and improper level-2 closures relative to k3)

Both k4 and q are domain specific languages, rather than general purpose.

k4 is proprietary to kx Systems, and is a DSL for the creation of q.

q is a DSL for the creation and operation of the KDB+ time-series database system.

[Tom Szczesny]

On Sun, Nov 15, 2015 at 10:37 PM, Stevan Apter wrote:

tom, i think it's misleading to call k4 domain-specific.  true, kx changed its *business model* from language vendor to database software company, but the new language is no less general purpose than k2/3.

as for q, it's just a thin layer of syntactic sugar on k4, substituting keywords for monads.

neither language has real closures.  for example, you can't implement paul graham's "accumulator generator" in either k3 or k4.

i avoid the quasi-closures of k3 and don't miss them at all in k4.  instead, i pass everything into local lambdas as arguments.  a bit more verbose, but referentially transparent, which i like.

k3 has advantages over k4 (the k-tree for example), and k4 has advantages over k3 (non-atomic dictionaries, first-class tables and keytables.)

best,

stevan

[Stevan Apter]

i've implemented k4's 'select', 'update', and 'eval' here:

    http://nsl.com/k/select.k

i found them useful in backporting the treetable logic of the k4 hypertree:

    http://nsl.com/k/treetable/treetable_k3.html

you can get a sense of the relative strengths and weaknesses of k3 and k4 by comparing this with the hypertree implementation (which of course does more):

    https://github.com/stevanapter/hypertree

i neglected to mention a few other important differences between the languages.

- k3's attribute structure permits triggers and dependencies (and gui) on individual variables, whereas k4 has a single global trigger .z.t.  moreover, dependencies are only implemented for variables in the . directory.  by no means is this a handicap - just a difference.

- another major difference is this:  k3 dictionary members are limited to valid k names.  k4 dictionary keys can be ANYTHING.  for example, v:10 20 30!`x`y`z is a dictionary whose keys are 10 20 30 and whose values are `x `y and `z.  so e.g. `x = v 10.  since tables are lists of dictionaries, a keyed table is then just a dictionary whose domain and range are tables.  i.e. a map from dictionaries to dictionaries.  now isn't that just the coolest thing ... :-)

- one of the consequences of this decision is that we have nothing like the k tree in k4.  e.g. 10 is a valid dictionary key, but how could it be a variable on the k tree?

[theos]

hi,

when starting up a new kona session and just entering an undefined
variable name, does not result in an error. is this intended? k2.8
gives a value / parse error in this situation.

% rlwrap ./k
K Console - Enter \ for help

a
\\


% rlwrap k

K 2.8 2000-10-10 Copyright (C) 1993-2000 Kx Systems

Evaluation. Not for commercial use.
\ for help. \\ to exit.

a
value error
a
^
parse error
\\

thanks,
theo.

[Tom Szczesny]

Hi Theo -

The original design of Kona had this behavior as a purposeful deviation from k.

Since then, the general consensus of Kona users is that Kona should behave like k does.

This is the sole remaining open "feature" issue (#383) in the Kona repository.

Unfortunately, it is not that easy to change, and it will take some time.

[theos]

hi tom,
i see. thanks for explaining.
theo.