a way to add metadata to code?

[Mariano Guerra]

hi, I've been working on squim lately and a sister project called squide:

http://marianoguerra.github.com/squide/

that allows editing squim code in a structured way,

to play with it:

click on a value, edit and press enter
press esc to cancel
pres tab to cycle through fields
press del to remove the focus field
press the + button to add a value

this is just the beginning, it will improve with time since I'm using
it for a project as a way to configure minimal logic.

as you can see there is a type called "block" that has curly braces
and everything, under it it's just ($sequence ...), in that case I can
recover blocks by checking if it's a list and the first element is a
symbol called $sequence but there are other types that are not so easy
to deduct in both ways.

for example, I want to add metadata to a string saying that it's a
color or is a date to allow specialized input fields for those cases
and format it differently too and I need the metadata to survive in
both ways (since I store the config as a string containing squim code)

any ideas on how to achieve this without reinventing the wheel?

[Mariano Guerra]

one proposal (feel free to analyze it's unlispiness :)

a runtime set of functions:

(set-meta <thing> <key-value-pairs>)

sets the values in key-value-pairs as metadata for thing, thing can be
any type in squim

(get-meta <thing> key [default])

return metadata associated with key in thing, if not found return
default if provided, some default value if not*

but this still doesn't solve the problem of attaching metadata in the
code at parse time and have it serialized again, for this I propose an
extension to the syntax (maybe only for squim)

(<thing> #{key1 value1 key2 value2})

will attach the set of key values to thing (the element on the left of #{}

* nil, inert and ignore don't seem to fit the solution, is there a
"kernel" or "scheme" way to handle this? fail?

[Andres Navarro]

On Wed, Jul 25, 2012 at 9:36 AM, Mariano Guerra
<luismari...@gmail.com> wrote:
>
> one proposal (feel free to analyze it's unlispiness :)
>
> a runtime set of functions:
>
> (set-meta <thing> <key-value-pairs>)
>
> sets the values in key-value-pairs as metadata for thing, thing can be
> any type in squim
>
> (get-meta <thing> key [default])
>
> return metadata associated with key in thing, if not found return
> default if provided, some default value if not*
>

My javascript is pretty weak, but I assume you mean attaching this in
some kind of hidden slot in the object itself at runtime? There is
nothing unlispy about that, as a matter of fact in older lisps (and
also in CL) symbols had what was called a property list, which was
something like that.

Another implementation option is to use a hashtable This is actually
doable in klisp, thanks to Oto Havle and his work on exposing the
internal (eq? based) hashtables api. The api is still not final, but
it exposes hashtables using eq? as the equality predicate, and so you
can use a global table to store per object metadata (but please be
aware that some immutable objects can't be copied, e.g. All string=?
immutable strings are eq? to each other, so you may want to use
mutable strings (or encapsulated immutable strings) in that case). If
you follow this route, however, you should take into account that the
global table will not allow gc to reclaim objects in the table, even
if there are no other references. You can avoid this with the use of
weak tables, something internal klisp tables have, but still isn't
exposed in the current api.

> but this still doesn't solve the problem of attaching metadata in the
> code at parse time and have it serialized again, for this I propose an
> extension to the syntax (maybe only for squim)
>
> (<thing> #{key1 value1 key2 value2})
>
> will attach the set of key values to thing (the element on the left of #{}
>

Haha, This topic comes time and time again, it's like it's following
me. Since before I started klisp, the main hole of the Kernel Report
I wanted to address is the fact that there is no way to customize
read/write/eval/eq?/equal?, neither for new objects types
(encapsulations) nor for primitive types. I always leaned to generic
functions/multi-methods as a catch all solution, but as the Report
mentions, this is just one possible solution.

Of course read just needs a way to know about syntax extension and
what to do with them, and write can be solved with single-dispatch on
its argument.
As for external representations, I favour the following syntax for
extensions to Kernel:

#<thing>(obj1 obj2 ...)
and
#<thing>

The writer is trivially extended to support this. As for the reader,
when it sees this it can select a combiner based on <thing> and pass
it the (unevaluated) list as arguments to it. The combiner would
return the object just read or throw an error. I feel this is pretty
flexible and consistent with the existing syntax. For example it
could be used for vectors ("#vector(elem1 elem2 ...)", or like in
scheme just "#(elem1 elem2 ...)"), hashtables ("#hashtable(string=?
("key1" <value1>) ("key2" <value2>))"), new constants ("#pi",
"#null"), etc.
A more general solution of course are reader macros, but I think they
are overkill for just allowing external representations for new object
types.
Only two new applicatives are needed:
(add-output-representation <type-predicate?> <format-applicative>)
(add-input-representation <type-symbol> has-list? <constructor-applcative>)
where format-applicative could either return a string with the
external representation, or directly write to the output stream,
<type-symbol> is the symbol after "#" for this type, has-list? is a
boolean indicating if this type needs a list after <symbol> and
<constructor-applicative> takes the list or no arguments (depending on
has-list?) and returns the object to read.

> * nil, inert and ignore don't seem to fit the solution, is there a
> "kernel" or "scheme" way to handle this? fail?

If you mean that they are singleton constants and so can't have
different metadata for different appearances then you are correct.
The only solution is to not use singletons for them! The downside is
that you must allocate them on the heap as any other objects which may
be a very high price to pay, that's for you to decide. Depending on
your implementation you may be able to use the singleton constants by
default and just replace the ones who have metadata attached...


This is just from the top of my head, let me know if it was of any help.

Regards,
Andres Navarro

[Andres Navarro]

On Wed, Jul 25, 2012 at 8:27 AM, Mariano Guerra
<luismari...@gmail.com> wrote:
> as you can see there is a type called "block" that has curly braces
> and everything, under it it's just ($sequence ...), in that case I can
> recover blocks by checking if it's a list and the first element is a
> symbol called $sequence but there are other types that are not so easy
> to deduct in both ways.
>
> for example, I want to add metadata to a string saying that it's a
> color or is a date to allow specialized input fields for those cases
> and format it differently too and I need the metadata to survive in
> both ways (since I store the config as a string containing squim code)
>

While I answered about your implementation idea in a previous mail, I
want to mention that you need not represent the values as
s-expressions directly. You can use an intermediate representation
(actually an AST), which contains the metadata AND the kernel object,
and this could have a convenient external representation in kernel
without altering the external representation of regular types.
Whether this is lispy enough or not is a matter of debate, but it's
still a valid solution to your problem, I think.

> any ideas on how to achieve this without reinventing the wheel?

You may want to have a look at CLIM if you haven't already:
http://www.cliki.net/CLIM

Especially consider how the following features interact:
stream-oriented input and output extended with facilities such as
output recording, presentations, and context sensitive input; high
level "formatted output" facilities.

As you will see, a great job is done there on handling the case of
specialized input fields (even allowing selecting them with the mouse
from the output buffer!, this implies tracking the output buffer to
know which object generated each part of it). This has to be seen to
be completely understood, if you have the time watch the beginning of
this presentation about the lisp machine and see it in action (any of
the bottom links):
http://lists.common-lisp.net/pipermail/boston-lisp/2012-July/000292.html


Regards,
Andres Navarro