Variadic arguments to special forms

[Arto Bendiken]

Does there exist a Clojure convention for dealing with variadic
arguments to special forms? Specifically, I'm thinking of (new) and
(.)

To take a simple example, this is perhaps somewhat verbose:

(defn locale
([lang]
(new java.util.Locale (str lang)))
([lang country]
(new java.util.Locale (str lang) (str country)))
([lang country variant]
(new java.util.Locale (str lang) (str country) (str variant))))

The following is more concise, but avoiding (eval) would be
preferable:

(defn locale [& args]
(eval `(new java.util.Locale ~@(map str args))))

Now, having to do a bit of a dance with (new) isn't too bad, but how
do you invoke a variadic JVM function with the (.) operator? I was
trying to write a (printf) as a wrapper around java.util.Formatter,
but got stumped right at:

(defn printf [#^String str & args]
(let [formatter (new java.util.Formatter)]
(. formatter (format str)))) ;; TODO: args

Presumably this one would be doable using (eval), too - but surely
there's another way? Apologies if I'm missing anything obvious...

Incidentally, the above code fails even without trying to pass extra
arguments:

user=> (printf "hello, world!")
java.lang.IllegalArgumentException: No matching method found: format

I've tried a number of ways of invoking Formatter#format, but am
getting no further than a ClassCastException (if passing args) or
IllegalArgumentException.

As for (new) and (.), I'd be perfectly happy to (quote) the arguments
to them if procedural versions of them, callable with (apply), could
be provided. Alternatively, first class macros would be great ;-)

Any help much appreciated,
Arto

[MikeM]

On Mar 30, 11:42 am, Arto Bendiken <arto.bendi...@gmail.com> wrote:
> Does there exist a Clojure convention for dealing with variadic
> arguments to special forms? Specifically, I'm thinking of (new) and
> (.)
>
> To take a simple example, this is perhaps somewhat verbose:
>
>   (defn locale
>     ([lang]
>      (new java.util.Locale (str lang)))
>     ([lang country]
>      (new java.util.Locale (str lang) (str country)))
>     ([lang country variant]
>      (new java.util.Locale (str lang) (str country) (str variant))))
>
> The following is more concise, but avoiding (eval) would be
> preferable:
>
>   (defn locale [& args]
>     (eval `(new java.util.Locale ~@(map str args))))
>

Seems like a macro is what you want here:

(defmacro locale [& args]
(let [strargs# (map str args)]
`(new java.util.Locale ~@strargs#)))

> Now, having to do a bit of a dance with (new) isn't too bad, but how
> do you invoke a variadic JVM function with the (.) operator? I was
> trying to write a (printf) as a wrapper around java.util.Formatter,
> but got stumped right at:
>
>   (defn printf [#^String str & args]
>     (let [formatter (new java.util.Formatter)]
>       (. formatter (format str)))) ;; TODO: args
>
> Presumably this one would be doable using (eval), too - but surely
> there's another way? Apologies if I'm missing anything obvious...
>
> Incidentally, the above code fails even without trying to pass extra
> arguments:
>
>   user=> (printf "hello, world!")
>   java.lang.IllegalArgumentException: No matching method found: format
>
> I've tried a number of ways of invoking Formatter#format, but am
> getting no further than a ClassCastException (if passing args) or
> IllegalArgumentException.
>
> As for (new) and (.), I'd be perfectly happy to (quote) the arguments
> to them if procedural versions of them, callable with (apply), could
> be provided. Alternatively, first class macros would be great ;-)
>
> Any help much appreciated,
> Arto

I believe varargs in Java is syntactic sugar that actually ends up as
an array argument. Try passing (into-array ...) as your argument to
the java method.

[Arto Bendiken]

On Mar 30, 6:08 pm, MikeM <michael.messini...@invista.com> wrote:
>
> Seems like a macro is what you want here:
>
>    (defmacro locale [& args]
>      (let [strargs# (map str args)]
>      `(new java.util.Locale ~@strargs#)))

Well, no :-)

A macro for this purpose is, to me, as undesirable as a runtime (eval)
call. Both will get the job done, of course, but both would be
unnecessary if Clojure were to provide primitives that would allow
something like:

(apply %new 'java.util.Locale (map str args))

...where (new) could be redefined in terms of syntactic sugar for
calling the low-level (%new) without having to quote the class name -
which is really the only thing that the special form provides, as far
as I can see.

> I believe varargs in Java is syntactic sugar that actually ends up as
> an array argument. Try passing (into-array ...) as your argument to
> the java method.

OK, thanks for the pointer. This didn't yet work right away:

(. (new java.util.Formatter)
(format (into-array "hello, world!\n")))

...but I'll look some more into how varargs are implemented in Java
vis-a-vis the Clojure compiler.

Arto

[christop...@gmail.com]

I may lend you a hand: if you intend to invoke Formatter.format(String
format, Object... args), you have to pass two arguments: one String
and one array of Objects.

Note that (into-array "hello, world!\n") returns an array of
Characters. (into-array ["hello, world!\n"]) returns an array of
Strings which is not compatible with an array of Objects.
(to-array ["hello, world!\n"]) will return an array of Objects.

Christophe

[Arto Bendiken]

On Mar 30, 8:59 pm, "christo...@cgrand.net"

<christophe.gr...@gmail.com> wrote:
>
> I may lend you a hand: if you intend to invoke Formatter.format(String
> format, Object... args), you have to pass two arguments: one String
> and one array of Objects.
>
> Note that (into-array "hello, world!\n") returns an array of
> Characters. (into-array ["hello, world!\n"]) returns an array of
> Strings which is not compatible with an array of Objects.
> (to-array ["hello, world!\n"]) will return an array of Objects.

Ah, I see - I had misunderstood the vararg calling convention. Sweet,
this works perfectly now:

(defn printf [fmt & args]
(let [formatter (new java.util.Formatter)]
(. formatter (format (str fmt) (to-array args)))))

Thanks a bunch, Christophe :-)

Arto

[Arto Bendiken]

Anyone needing a working printf/sprintf function can grab them here:

http://svn.bendiken.net/clojure/util.clj

Arto

[Arto Bendiken]

So, I didn't figure out anything that useful in answer to my original
question regarding varargs for special forms, so I cheated and went
the ugly-but-effective route of using 'eval' to get "desugared" plain
procedural versions of '.' and 'new':

http://svn.bendiken.net/clojure/java.clj

This library defines the procedures 'java/new' and 'java/.', allowing
me to shorten the aforementioned 'locale' function down to:

(defn locale [& args]
(apply java/new 'java.util.Locale (map str args)))

While at it, I implemented some convenience stuff that became possible
with these forms. First, 'java/method' is a higher-order function that
is nifty for quickly making wrappers of standard library methods in a
fully declarative style:

(def uuid
(java/method java.util.UUID 'randomUUID))

(uuid)
=> 890c7d32-47b9-4210-a4fb-6d4eeb505e21

(def url-encode
(java/method java.net.URLEncoder 'encode))

(url-encode "hello, world!\n")
=> "hello%2C+world%21%0A"

Also, 'java/method*' is a variant of the previous that also handles
calling variadic Java functions based on what I learned today in this
thread:

(def format-xy
(java/method* (new java.util.Formatter) 'format "x=%d y=%d\n"))

(format-xy (int 123) (int 456))
=> x=123 y=456

There's also a 'java/field' and a 'java/call' in there, but they're
pretty obvious.

I'm sure I've managed to slow method execution down a couple of
factors with this stuff, but such is life... if there's some better
way to do all this in Clojure, I'd be happy to learn about it :-)

Arto

[Rich Hickey]

On Mar 30, 11:17 pm, Arto Bendiken <arto.bendi...@gmail.com> wrote:
> So, I didn't figure out anything that useful in answer to my original
> question regarding varargs for special forms,

Sorry - lots of traffic yesterday! I focused on bugfixes and missed
responding.

> so I cheated and went
> the ugly-but-effective route of using 'eval' to get "desugared" plain
> procedural versions of '.' and 'new':
>
> http://svn.bendiken.net/clojure/java.clj
>
> This library defines the procedures 'java/new' and 'java/.', allowing
> me to shorten the aforementioned 'locale' function down to:
>
> (defn locale [& args]
> (apply java/new 'java.util.Locale (map str args)))
>

I don't recommend this. None of your examples use import, but it will
get rid of the package prefixes. If it is still that important to say
(locale lang country) rather than (new Locale lang country), your
first (overloaded) formulation is far better. In both cases the result
of your function won't have a type known to the compiler unless you
add type hints, whereas it will be known if you call new, and all
subsequent use of the resulting object will be non-reflective.

More generally, I recommend to everyone to avoid the tendency to try
to hide Java. There are umpteen thousand Java methods - I'd hate to
see umpteen thousand Clojure functions written just to eliminate dot/
new or uppercase letters. There's no significant benefit and some
tangible costs - in the ability of dev environments to provide type-
based assistance/Javadoc help/completion, and in the ability of the
compiler to optimize.

Never say never, of course, but the emphasis should be on leveraging
Java libs, not hiding/wrapping them, and on making their direct use
succinct and easy. I'm not saying I've got that optimal yet, but it's
a better place to focus IMO than wrapper generation.

> While at it, I implemented some convenience stuff that became possible
> with these forms. First, 'java/method' is a higher-order function that
> is nifty for quickly making wrappers of standard library methods in a
> fully declarative style:
>
> (def uuid
> (java/method java.util.UUID 'randomUUID))
>
> (uuid)
> => 890c7d32-47b9-4210-a4fb-6d4eeb505e21
>
> (def url-encode
> (java/method java.net.URLEncoder 'encode))
>
> (url-encode "hello, world!\n")
> => "hello%2C+world%21%0A"
>

If you must have wrappers, it's better to use the existing facilities:

(def uuid #(. java.util.UUID (randomUUID)))

(def url-encode #(. java.net.URLEncoder (encode %)))

These versions will be much faster, and are as succinct. Although I
must say, while experimenting with your stuff I found the eval calls
to be surprisingly fast, but I can't promise they will always be so.

Again though, if you were to say:

(let [id (uuid)]

the compiler knows nothing about the type of id, vs:

(let [id (. UUID (randomUUID))]

where it knows that id is a UUID.

> Also, 'java/method*' is a variant of the previous that also handles
> calling variadic Java functions based on what I learned today in this
> thread:
>
> (def format-xy
> (java/method* (new java.util.Formatter) 'format "x=%d y=%d\n"))
>
> (format-xy (int 123) (int 456))
> => x=123 y=456
>

Support for auto-creating the array for Java 'variadic' calls is on my
todo list. apply ./new are interesting concepts I'll have to think
about.

Hope this helps,

Rich

[Arto Bendiken]

On Mar 31, 3:49 pm, Rich Hickey <richhic...@gmail.com> wrote:
> On Mar 30, 11:17 pm, Arto Bendiken <arto.bendi...@gmail.com> wrote:
>
> > This library defines the procedures 'java/new' and 'java/.', allowing
> > me to shorten the aforementioned 'locale' function down to:
>
> > (defn locale [& args]
> > (apply java/new 'java.util.Locale (map str args)))
>
> I don't recommend this. None of your examples use import, but it will
> get rid of the package prefixes. If it is still that important to say
> (locale lang country) rather than (new Locale lang country), your
> first (overloaded) formulation is far better. In both cases the result
> of your function won't have a type known to the compiler unless you
> add type hints, whereas it will be known if you call new, and all
> subsequent use of the resulting object will be non-reflective.

OK, good point.

> More generally, I recommend to everyone to avoid the tendency to try
> to hide Java. There are umpteen thousand Java methods - I'd hate to
> see umpteen thousand Clojure functions written just to eliminate dot/
> new or uppercase letters. There's no significant benefit and some
> tangible costs - in the ability of dev environments to provide type-
> based assistance/Javadoc help/completion, and in the ability of the
> compiler to optimize.
>
> Never say never, of course, but the emphasis should be on leveraging
> Java libs, not hiding/wrapping them, and on making their direct use
> succinct and easy. I'm not saying I've got that optimal yet, but it's
> a better place to focus IMO than wrapper generation.

Hmm :-) The UUIDs and Locales happened to be very straightforward
examples, which is why I asked about them. Arguably they don't indeed
further benefit from wrapping, especially using 'import' and the nifty
lambda syntax you demonstrated.

However, for my other printf/sprintf example, there are several lines
of code involved in getting this functionality going - I certainly
don't want to be writing those cumbersome lines by hand every time I
need formatted output, hence the wrappers.

In general, there are sane APIs and then there are not-so-sane APIs. I
spent most of yesterday figuring out and playing with the Java
libraries that I'd need in order to utilize Clojure for my work. From
the looks of it, all the libraries I need are available (no surprise),
but their APIs tend to fall into the latter category (not so sane,
from a Lisper's point of view).

For instance, I work with RDF a lot, so let's take the Jena [1]
library as an example. Jena is, I'm told, the most widely-used and
comprehensive RDF/SPARQL library for Java. But being used to elegant,
sane APIs from the likes of Ruby, Scheme or Python, I really can't
describe Jena's API with any term more charitable than a clusterfuck.

There is no way that I want to use Jena directly on a daily basis; it
would take the fun out of Clojure, plus simply make for a whole lot of
verbose coding. Yet the library provides all the _functionality_, per
se, that I need. Hence, if I will use it, I will first want to wrap
it.

While Clojure could perhaps still improve with e.g. handling varargs
and autoboxing number types, I don't think there's anything further
Clojure-the-language can provide to make up for the design
deficiencies of third-party libraries; it's just a matter of manually
figuring out sane high-level Lispy/Clojuresque wrappers to make them
usable.

To take another example perhaps closer to your heart, I was reading up
on the ASM bytecode generation library [2] you use. Again, while
there's no doubt all the functionality one could desire is in there, I
have to say (with the caveat that this is based on reading just the
asm-guide.pdf) that it looks like a somewhat painful API to use from a
Lispy point of view, what with the explicit visitor pattern and all.

To sum it up, I'm interested in using Lisp, not Java. If I'm to be
using Clojure, I want to eventually reach a point where I don't need
to keep the JDK javadoc open in order to be able to do stuff, and the
fewer times I need to see any 'import java...' in my programs, the
better. Hence my first order of business now being creating interfaces
and wrappers in order to assemble something resembling a Lispy
standard library catering to my requirements.

> > While at it, I implemented some convenience stuff that became possible
> > with these forms. First, 'java/method' is a higher-order function that
> > is nifty for quickly making wrappers of standard library methods in a
> > fully declarative style:
>

> > ...

>
> If you must have wrappers, it's better to use the existing facilities:
>
> (def uuid #(. java.util.UUID (randomUUID)))
>
> (def url-encode #(. java.net.URLEncoder (encode %)))
>
> These versions will be much faster, and are as succinct. Although I
> must say, while experimenting with your stuff I found the eval calls
> to be surprisingly fast, but I can't promise they will always be so.

Excellent, thanks - this is certainly succinct enough, at least for
the non-variadic function cases.

> Again though, if you were to say:
>
> (let [id (uuid)]
>
> the compiler knows nothing about the type of id, vs:
>
> (let [id (. UUID (randomUUID))]
>
> where it knows that id is a UUID.

OK - but just to make sure: presumably the type information _is_
preserved if I wrap it using your succinct 'uuid' example from above?

> > Also, 'java/method*' is a variant of the previous that also handles
> > calling variadic Java functions based on what I learned today in this
> > thread:
>
> > (def format-xy
> > (java/method* (new java.util.Formatter) 'format "x=%d y=%d\n"))
>
> > (format-xy (int 123) (int 456))
> > => x=123 y=456
>
> Support for auto-creating the array for Java 'variadic' calls is on my
> todo list. apply ./new are interesting concepts I'll have to think
> about.

Sounds good!

I actually ran into quite a bit further trouble yesterday with
invoking variadic functions, even after I got 'printf' working; most
often, I'm getting ClassCastExceptions and 'IllegalArgumentException:
No matching method found' for (seemingly) no discernible reason.

These problems were with non-JDK libraries, so I'll try and reproduce
them with some variadic function in the JDK for easier
troubleshooting, and then post back to the group.

I'm sure it's just my ineptitude, but certainly, some built-in varargs
support would be superb; or perhaps even more importantly, a script to
take a Java class and produce the Clojure code which would
successfully invoke its methods. Could be useful for unit testing
purposes, too - hmm, guess I better look into the Reflection API.

Arto

[1] http://jena.sourceforge.net/
[2] http://asm.objectweb.org/

[Rich Hickey]

>...

> There is no way that I want to use Jena directly on a daily basis; it
> would take the fun out of Clojure, plus simply make for a whole lot of
> verbose coding. Yet the library provides all the _functionality_, per
> se, that I need. Hence, if I will use it, I will first want to wrap
> it.
>
> While Clojure could perhaps still improve with e.g. handling varargs
> and autoboxing number types, I don't think there's anything further
> Clojure-the-language can provide to make up for the design
> deficiencies of third-party libraries; it's just a matter of manually
> figuring out sane high-level Lispy/Clojuresque wrappers to make them
> usable.
>

> To sum it up, I'm interested in using Lisp, not Java. If I'm to be
> using Clojure, I want to eventually reach a point where I don't need
> to keep the JDK javadoc open in order to be able to do stuff, and the
> fewer times I need to see any 'import java...' in my programs, the
> better. Hence my first order of business now being creating interfaces
> and wrappers in order to assemble something resembling a Lispy
> standard library catering to my requirements.
>

Fair enough. My comments were mostly directed towards wrapping-in-the-
small, if you will, and the Java calling syntax of Clojure. Wrapping
in the large is an interesting area. In some cases it's just bad/
awkward names, in others it's tedious call sequences, class
hierarchies, need to derive etc. This kind of wrapping is more
difficult to generalize, and may ultimately be personal, but to the
extent there are common needs or idioms I'd like to look at them as
potential Clojure features.

>
> > > While at it, I implemented some convenience stuff that became possible
> > > with these forms. First, 'java/method' is a higher-order function that
> > > is nifty for quickly making wrappers of standard library methods in a
> > > fully declarative style:
>
> > > ...
>
> > If you must have wrappers, it's better to use the existing facilities:
>
> > (def uuid #(. java.util.UUID (randomUUID)))

> > (let [id (uuid)]
>
> > the compiler knows nothing about the type of id, vs:
>
> > (let [id (. UUID (randomUUID))]
>
> > where it knows that id is a UUID.
>
> OK - but just to make sure: presumably the type information _is_
> preserved if I wrap it using your succinct 'uuid' example from above?
>

No. Once you wrap it in a fn, its return type is Object. fns can have
multiple heterogeneous returns, and right now there is no return-type
analysis done. Even if there were, the potential exists for the
function to be redefined later with different returns. If you intend
that a fn will always return a single type, you need to declare that
with a type hint.

Similarly, vars that are def'ed do not adopt the type of their
initializers as a type hint, because, over their lifetime, they may
hold objects of different types, so if you intend that a var will only
ever hold values of a particular type, you need to declare that with a
type hint (if it matters).

> I actually ran into quite a bit further trouble yesterday with
> invoking variadic functions, even after I got 'printf' working; most
> often, I'm getting ClassCastExceptions and 'IllegalArgumentException:
> No matching method found' for (seemingly) no discernible reason.
>

Sometimes you can get no matching or more than one matching errors due
to Java functions overloaded (on types) with the same arity. Clojure
may need type hints on the arguments in order to distinguish the
options.

> I'm sure it's just my ineptitude, but certainly, some built-in varargs
> support would be superb; or perhaps even more importantly, a script to
> take a Java class and produce the Clojure code which would
> successfully invoke its methods. Could be useful for unit testing
> purposes, too - hmm, guess I better look into the Reflection API.
>

There are some useful facilities in the Reflector.java code, although
not yet packaged for external consumption.

Rich