Question about :tag

[Nicola Mometto]

What exactly should be allowed on :tag metadata?

I've written tools.analyzer with the assumption that :tag metadata
attached to every IObj must have as its value either a Class, a Symbol
or a String that can be resolved to a Class, however Andy Fingerhut
recently pointed out that Stu Halloway's test.generative uses type hints
as "instructions for generating test input data", thus it's by design
that some symbols will have attatched as :tag metadata expression that
don't map to Classes.

Now, is test.generative wrongly exploiting an undefined behaviour of
clojure that allows wrong :tag values in some cases, or is the ":tag
should resolve to a class" rule limited only to symbols/expressions in
special positions? (e.g. symbols of def, let bindings etc)

http://clojure.org/java_interop says:
"Type hints are metadata tags placed on symbols or expressions that are
consumed by the compiler. They can be placed on function parameters,
let-bound names, var names (when defined), and expressions [..] Once a
type hint has been placed on an identifier or expression, the compiler
will try to resolve any calls to methods thereupon at compile time"

http://clojure.org/reader says:
"Such tags can be used to convey type information to the compiler."

Thanks,
Nicola

[Ambrose Bonnaire-Sergeant]

I remember this was briefly discussed by Rich at the Clojure Conj 2011 clojure-dev meeting. Rich announced

something along the lines of ":tag is getting more flexible, speak now if you want something changed!".

I got the impression that plain symbols are just fine. I think it was unclear whether other data structures like

vectors and sets might work, but definitely not lists IIRC.

Hope that helps.

Ambrose

--
You received this message because you are subscribed to the Google Groups "Clojure Dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to clojure-dev...@googlegroups.com.
To post to this group, send email to cloju...@googlegroups.com.
Visit this group at http://groups.google.com/group/clojure-dev.
For more options, visit https://groups.google.com/groups/opt_out.

[Colin Fleming]

I think putting things that are not resolvable somehow to a Class (i.e. using :tag for something other than type hinting) is going to complicate a lot of tooling. Why overload :tag rather than use some other keyword in the metadata to indicate the type of a generative test or whatever? It seems like that would be surprising to the vast majority of Clojure programmers (and tool writers, including myself).

[Ambrose Bonnaire-Sergeant]

It's the only way to use the ^foo syntax, which ends up in a predictable ^{:tag foo} entry. ^:foo is ^{:foo true}, which is arguably less consistent.

Seems like a legitimate use for code generation, which is exactly what test.generative does. :tag also triggers code generation in the compiler, but there's no reason why a macro might not intercept it and generate its own.

:tag might have special behaviour if it reaches the compiler, but I think the ^foo syntax is generally useful enough to reuse it for more specialised things.

Thanks,

Ambrose

[Colin Fleming]

I think that's a fairly large pandora's box to open for some minor syntactical convenience. But if that's the way we decide to go it should definitely be clearly documented. Anyone writing code analysis tools, especially static ones that work on non-macroexpanded code, are going to have to take that into account.

Cheers,
Colin

[Ambrose Bonnaire-Sergeant]

I don't think I fully understand the implications of using :tag. Could we show an example where it's problematic to use :tag for something other than type hints? I see how it might weaken tools.analyzer's ability to warn on bad :tags.

FWIW I interpreted the documentation as quoted by OP as implying :tag "can be" used for removing reflection, but not limited to. It also does not imply that :tag is reserved by Clojure.

But clearly such tools working on pure syntax have much bigger things to worry about eg. local scope. :) By their very nature, they are a bit of a stab in the dark in terms of consistency and correctness.

Thanks,

Ambrose

[David Nolen]

It's important think beyond the semantics of the JVM, the tags that are relevant for CLJS are quite different.

[Colin Fleming]

Fair point. Is it currently still used in the 'traditional' sense, i.e. for type hinting expressions/vars/params in CLJS? A quick Google didn't produce much info and I'm not very familiar with CLJS yet.

I think I'm more worried about opening up the semantics of :tag beyond "this thing is of this type". I must admit I don't have a particular case in mind that would be problematic, but I have an uneasy feeling about overloading these semantics and I'm not really seeing the benefit of doing so.

And yes, working on Clojure syntax is definitely tricky :-). Hence my resistance to making it trickier unless we can see a big win in doing so. I think it's much more important to consider what most developers will expect rather than exactly what the doc says right now - the doc is relatively easy to change.

[Ambrose Bonnaire-Sergeant]

IMO :tag can be thought of as a platform independent directive to the compiler, usually for speed.

It seems arbitrary to restrict it. New :tag metadata is often added (at least in CLJS).

Thanks,

Ambrose

[David Nolen]

On Fri, Dec 6, 2013 at 4:19 AM, Colin Fleming <colin.ma...@gmail.com> wrote:

Fair point. Is it currently still used in the 'traditional' sense, i.e. for type hinting expressions/vars/params in CLJS? A quick Google didn't produce much info and I'm not very familiar with CLJS yet.

ClojureScript does not need to jump through all the hoops the Clojure compiler does to generate fast code. So for a long time only ^boolean was used. However we now also use ^number to check non-higher order arithmetic. In the future I would like to use ^clj so that we can inline functions backed by protocols to eliminate the indirection overhead.

Different hosts have different needs and tying tags to JVM semantics seems needlessly restrictive.

David

[Alan Dipert]

From a syntactic analysis perspective, the rule seems straightforward: if there's a hat and the thing after it isn't a map, it's either a string or a symbol to add to the metadata of the symbol to follow under the :tag key.

It is the interpretation of that :tag key, an other meta we see, that is platform dependent.  So, I think the question boils down to: what all should this first analysis pass be doing?  If it should be trying to resolve classes, it may need to interpret the symbol or string as a Java class or Javascript type, which it may also choose to attempt to resolve - a very platform-specific process.

If we're interested in having analysis tools that are usable across platforms - the dream? - then it seems like a good approach might be to separate analysis into at least two steps, platform-agnostic and platform-specific.  In this way implementations of Clojure may share a platform-agnostic syntax analyzer that does universal stuff like meta aggregation and locals analysis.  Later steps might then apply platform-specific meaning to this structure.

My 0.2 as an excitable amateur peeking in.  Thank you all for your great work improving these tools.

Alan 

--

[Andy Fingerhut]

Nicola can comment more accurately, but I believe he already does try to separate the JVM specific portions of analysis into the lib tools.analyzer.jvm, separate from tools.analyzer.

I think his original question may have been: what should the interpretation of :tag metadata be for Clojure on the JVM?

Andy

Sent from my iPhone

[Nicola Mometto]

Thanks Andy, I was just writing a reply clarifying that point.

To David, Ambrose & all:
I mispoke on my first post, tools.analyzer and all the passes on that
library have no assumption whatsoever about :tag, in fact tools.analyzer
never deals with :tag at all.

I understand and agree that CLJS has different needs and :tag in CLJS
will obviously NEVER be a Class.

It goes further than that, c.t.a.passes.jvm/infer-tag itself
doesn't care at all whether :tag is a Class or something else, I'll
probably move some parts of that pass in a tools.analyzer pass as I'm
sure it will be useful for local tag inference on a CLJS port.

What's actually throwing an exception when :tag cannot be resolved to a
class is the c.t.a.passes.jvm/validate pass, this pass also throws on
method not found, on invokes with wrong arity etc.

So, my *strictly* clojure JVM related question is:
Given that it's stated in several pages on clojure.org that :tag should
be used as a hint to the compiler, is it wrong to assume that tag should
*always* have as a val something (either a Symbol, a String or a Class)
that can resolve to a Class and assume that the current Clojure
behaviour of allowing aribitrary :tag values in some cases should be
considered an implementation detail (just like (keyword " ") works)?

Some examples where arbitrary :tag throws:

user=> (def ^foo a)
CompilerException java.lang.RuntimeException: Unable to resolve symbol:
foo in this context, compiling:(NO_SOURCE_PATH:1:1)
user=> (let [^foo a []] (.hashCode a))
CompilerException java.lang.IllegalArgumentException: Unable to resolve
classname: foo, compiling:(NO_SOURCE_PATH:4:18)
user=> (let [a []] (.hashCode ^foo a))
CompilerException java.lang.IllegalArgumentException: Unable to resolve
classname: foo, compiling:(NO_SOURCE_PATH:7:13)
user=> (fn ^foo [])
CompilerException java.lang.IllegalArgumentException: Unable to resolve
classname: foo, compiling:(NO_SOURCE_PATH:11:1)
user=> (fn [^foo x])
CompilerException java.lang.IllegalArgumentException: Unable to resolve
classname: foo, compiling:(NO_SOURCE_PATH:12:1)
user=> (deftype x [] Object (equals [this ^foo that] nil))
CompilerException java.lang.IllegalArgumentException: Unable to resolve
classname: foo, compiling:(NO_SOURCE_PATH:3:1)
user=> (deftype x [^foo a])
CompilerException java.lang.IllegalArgumentException: Unable to resolve
classname: foo, compiling:(NO_SOURCE_PATH:4:1)

Thanks,
Nicola

[Nicola Mometto]

Alan,

This exactly what tools.analyzer is: an extendable
platform-agnostic analyzer with platform-agnostic passes, and
tools.analyzer.jvm is an extension of tools.analyzer with JVM-specific
passes/extension points.

[Nicola Mometto]

I agree, if arbitrary :tag never reaches the compiler it should be ok,
however that's not the case with test.generative as arbitrary :tag leaks
to the generated code:

user=> (require '[clojure.test.generative :refer (defspec)])
nil
user=> (require '[clojure.data.generators :as gen])
nil

user=> (binding [*print-meta* true] (pr (macroexpand-1 '(defspec foo
identity [^{:tag `gen/anything} o]))))
(clojure.core/defn ^{:clojure.test.generative/arg-fns
[#<generators$anything clojure.data.generators$anything@13197df2>]} foo
[o] (clojure.core/let [% (clojure.core/apply identity [^(quote
clojure.data.generators/anything) o])] %))nil

[Ambrose Bonnaire-Sergeant]

Hmm ok, that's surprising.

[Kevin Downey]

On 12/5/13, 6:59 PM, Ambrose Bonnaire-Sergeant wrote:
> It's the only way to use the ^foo syntax, which ends up in a predictable

clojure supports ^:foo (with a keyword) which is equivalent to ^{:foo true}

Using a keyword in this manner is much better than using :tag, because
the keywords can stack easily (with "^:foo ^:bar x" x will have {:foo
true :bar true} in the metadata) and keyword tags can also be namespaced.

--
And what is good, Phaedrus,
And what is not good—
Need we ask anyone to tell us these things?

[Ambrose Bonnaire-Sergeant]

My point was if you want some arbitrary value like a class name with the ^foo syntax, 

it's arguably more consistent to expect it as the value of a fixed key like :tag, rather 

than digging through the keys of your metadata map and guessing the original source.

Thanks,

Ambrose

[Kevin Downey]

On 12/9/13, 6:14 PM, Ambrose Bonnaire-Sergeant wrote:
> My point was if you want some arbitrary value like a class name with the
> ^foo syntax,
> it's arguably more consistent to expect it as the value of a fixed key like
> :tag, rather
> than digging through the keys of your metadata map and guessing the
> original source.

the annotation support for gen-class does exactly that (shifting the
contents of the metadata map looking for things that may be annotation
classes)

everyone using :tag is just not scalable, what if you want to use
multiple tools that each want a different thing in :tag?

with ^:foo you can at least namespace stuff, so you can stack and mix
and match

^:mything/skip-linter ^:clojurescript/boolean foo

[Ambrose Bonnaire-Sergeant]

Ah I see. Good point.

My narrow point of view was just considering specialised DSLs that fully control

any incoming metadata. test.generative probably falls into this category.

Thanks,

Ambrose

[Nicola Mometto]

In the meantime I've opened a ticket for tools.generative that fixes the
issue (http://dev.clojure.org/jira/browse/TGEN-5), I guess we'll have to
wait for Rich to express his opinion on this to know whether it's
reasonable to expect :tag to be usable with not "class-able" values or
not.

Thanks everybody,
Nicola

[Colin Fleming]

I think it's fair to say that we can't restrict it to just "class-able" values given CLJS (and CLJ-.NET and others). I'd still prefer to see it restricted to type information though, even if only by convention, and as Kevin pointed out overloading it a lot is going to be difficult because of interactions between different uses.

Nicola

[Nicola Mometto]

I think I made it clear that I'm not talking about CLJS here, it's clear
that :tag for CLJS cannot represent a Class.

The point I'm trying to make is that currently the Clojure compiler (and
I believe the CLR one too) will throw an exception if :tag doesn't
represent a Class in some positions, while allowing it to compile just
as fine in other positions.

Now, what I'd like to get clarification on is:

In a Clojure JVM/CLR implementation :tag should:
a- always represent a Class and the analyzer is allowed to throw an
exception if that's not the case -- in this case it's an implementation
detail that currently Compiler.java doesn't always throw.

b- always represent a Class when attached to forms in a special position
e.g. args symbols, args vector, method call args etc, and can be
whatever in forms that are not in this special positions -- in this case
the current implementation matches the expected behaviour and it should
be expected to be able to use whatever as :tag if attached to forms NOT
in a special position

c- possibly represent a Class, in that case the Compiler should accept
that as a type hint, otherwise just discard it -- in this case the
current implementation is wrong in throwing an exception expecting :tag
to represent a Class in some special positions.

[Chouser]

Maybe this is already covered in your definition of "class", but just in case you've overlooked it, the Clojure compiler also recognizes hints like ^floats and ^doubles, even those these are not really java class names

[Nicola Mometto]

Yes I know, that's why I'm writing "represents a Class", meaning that
the Compiler understands how to get a Class out of that representation.

Right now it allows for:
- a Class
- a Symbol/String representing a Class name (String) or a Class internal
representation ("[D") or some special names representing arrays
(objects)

[Chouser]

Very good. I wasn't sure and just wanted to make sure that detail hadn't slipped under your radar.

[Nicola Mometto]

Two examples where the current behaviour of Clojure to silently ignore
some mis-formed :tags result in bugs:
http://dev.clojure.org/jira/browse/CLJ-1307
http://dev.clojure.org/jira/browse/CLJ-1308

[Alex Miller]

I'm almost certain that 1307 is already in jira somewhere.

[Nicola Mometto]

You're right, I'm sorry I searched for "qualified" in JIRA and found
nothing.

I closed it as a duplicated of http://dev.clojure.org/jira/browse/CLJ-1232

[Alex Miller]

Thanks!