Enhanced Primitive Support Syntax

88 views
Skip to first unread message

RJ Nowling

unread,
Jan 13, 2011, 4:31:39 PM1/13/11
to Clojure
Hi all,

I've been reading about the changes to Clojure that are occurring with
regards to improving performance of arithmetic operations. For
example, + no longer auto promotes and +' has been introduced as an
auto-promoting addition operator.

The idea of introducing additional syntax is a bit odd to me, so I had
thought that it could be solved by setting an environmental variable
for the namespace and the use of macros. For example, maybe set
*enhanced-primitive-support* true or false will determine the behavior
of the operators (+, -, etc.) for an entire namespace. If a user
wants a different set of behaviors for a block of code in that
namespaces, they could wrap the code in a macro called something like
enhanced-primitive-support or old-primitive-support:

(enhanced-primitive-support (+ a b) )

I was wondering why an approach like this was not taken?

Thanks,
RJ

trptcolin

unread,
Jan 13, 2011, 6:46:02 PM1/13/11
to Clojure
I can't speak for Rich, but your suggestion would mean an additional
if statement being executed every time you call +. Not sure if
Hotspot (or other JVMs) would compile that away or not...

Colin

Stuart Sierra

unread,
Jan 13, 2011, 7:28:53 PM1/13/11
to clo...@googlegroups.com
The goal of primitive math is better performance in the common case. The implementation makes the assumptions that Java long is big enough for nearly all cases, and that auto-promotion to BigInteger (and the resulting performance hit) is rarely desirable.

The + function still promotes, it just promotes everything to long or double. If you want BigInts or BigDecimals, just insert one of them into the calculation, and everything else will get promoted to BigInt or BigDecimal.

The only difference between + and +' is that +' will promote to BigInt if you overflow a long, whereas + will thrown an exception.

Ken Wesson

unread,
Jan 13, 2011, 7:45:22 PM1/13/11
to clo...@googlegroups.com
On Thu, Jan 13, 2011 at 7:28 PM, Stuart Sierra
<the.stua...@gmail.com> wrote:
> The goal of primitive math is better performance in the common case.

Of course, this better performance is not needed "in the common case",
IMO, but only in hotspots that do number crunching, where people
already optimize using primitive locals, coercion, and unchecked-foo.

> The implementation makes the assumptions that Java long is big enough for
> nearly all cases

It also makes the assumption that the Java long is as fast as native
arithmetic. Which, on a lot of 32-bit hardware, it won't be.

> and that auto-promotion to BigInteger (and the resulting performance hit) is
> rarely desirable.

Debatable. I, for one, prefer to have unadorned arithmetic be correct
at the expense of a little speed, while still having a way to get the
speed in performance-critical parts of my code.

Stuart Sierra

unread,
Jan 14, 2011, 5:40:50 PM1/14/11
to clo...@googlegroups.com
Debatable it is, endlessly. ;)  So Clojure committers made a choice.  Hopefully, they have made a choice that has:

* a small positive effect (better performance with less effort) on a majority of users

* a small negative effect (worse performance, extra effort) on a minority

-S

Armando Blancas

unread,
Jan 14, 2011, 8:40:51 PM1/14/11
to Clojure
They used to give you compile switches for that kind of stuff, not
hope and wholesome wishes. Seems like every performance improvements
makes the language more complex, uglier or both.

Saul Hazledine

unread,
Jan 15, 2011, 8:29:06 AM1/15/11
to Clojure
On Jan 15, 2:40 am, Armando Blancas <armando_blan...@yahoo.com> wrote:
> They used to give you compile switches for that kind of stuff, not
> hope and wholesome wishes. Seems like every performance improvements
> makes the language more complex, uglier or both.
>

I don't feel strongly about integer limits at all and am always
surprised when this comes up. I did scientific programming on a 32 bit
platform for several years and never met anyone who hit big problems
with fixed size integers. For illustration, Long.max is:

9 223 372 036 854 775 807

which is so much bigger than I was used to. I know encryption requires
BigInteger but I have yet to see a native clojure encryption library.
It would help people like me understand the debate if some mainstream
examples of applications requiring (seamless) BigInteger support could
be identified.

Saul

Bob Hutchison

unread,
Jan 15, 2011, 11:17:06 AM1/15/11
to clo...@googlegroups.com
I'm going to re-organise this a bit…

> On Jan 14, 2:40 pm, Stuart Sierra <the.stuart.sie...@gmail.com> wrote:
>> Debatable it is, endlessly. ;) So Clojure committers made a choice.
>> Hopefully, they have made a choice that has:

I agree that they've made a choice, and I really don't want to be too critical here. However, since Clojure 1.3 is still in an alpha stage, maybe this discussion can still contribute something. Maybe what I'm saying is that I want to stay constructive and that there's maybe still time to be constructive.

>>
>> * a small positive effect (better performance with less effort) on a
>> majority of users
>>
>> * a small negative effect (worse performance, extra effort) on a minority

These goals are good, but I don't know that the approach taken achieves them.

In my experience, errors are the problem and we should be avoiding them, almost at all costs. Numbers are confusing to people. Numbers approximated on a computer are far more confusing. How many times do you see threads discussing how a compiler is buggy because it can't divide two numbers and get the right answer? I've been doing this stuff for years and I can come up with an awful lot of amusing and/or horribly nasty examples. But I don't think this needs to be re-established.

Given my experience I *strongly* lean towards not making a 'mistake' due to compiler optimisations. In other words, I'd be very annoyed, and I'd expect others to be annoyed too, if a numerical error was introduced to one of my programs because of an unexpected, silent, compiler optimisation.

Secondly, Clojure has already established that we will use type annotations to signal to the compiler what's what. When we annotate, we are relaxing our requirements on the compiler to not make a mistake by assuming that responsibility ourselves.

I would suggest the following:

1) if there's type annotation on both values of, say, an addition, then the optimised version can be used. If there isn't, or the compiler isn't sure, then use safe operations.

2) if the compiler isn't cooperating (because it isn't sure what's going on) we should be helping it by again assuming the responsibility of being right and marking the operator, say with a tick.

And yes, this likely has problems too. I'm not saying that this is an issue with easy solutions.

We're heading for a hodgepodge of annotation purposes, some for optimisation, some for correctness (and one of these days I'll mention what I think of the @/deref thing :-) And now we're pretty much guaranteed ugly code no matter what. Though I'd prefer no ugliness, I'd trade ugly code for speed, but I'd rather not for correctness.

And there's a practical problem with mixed annotation purposes. If you want to track down a bug you can't just remove all annotations temporarily. You'd have to remove some and add others. Not looking forward to that. Maybe a macro: make-this-safe could be written. Hmm. Maybe a 'defn-safe' would be something to think about???

This is also the kind of thing that you just can't fix later. Imagine how we'll feel in ten or twenty years about this decision.


>>
>> -S


>
On 2011-01-14, at 8:40 PM, Armando Blancas wrote:

> They used to give you compile switches for that kind of stuff, not
> hope and wholesome wishes. Seems like every performance improvements
> makes the language more complex, uglier or both.
>


Compiler switches were/are problematic too, but at least they are explicit and have to be *added*.

Cheers,
Bob

----
Bob Hutchison
Recursive Design Inc.
http://www.recursive.ca/
weblog: http://xampl.com/so


Stefan Kamphausen

unread,
Jan 15, 2011, 11:30:53 AM1/15/11
to clo...@googlegroups.com
Hi,

just for the record: from what I have done in the past I wholeheartedly agree with Bob.  Of course the developers of Clojure have the final say on this.  It's just my 2ct.

Kind regards,
Stefan

David Powell

unread,
Jan 15, 2011, 11:43:46 AM1/15/11
to clo...@googlegroups.com

Bob Hutchison said:

> In other words, I'd be very
> annoyed, and I'd expect others to be annoyed too, if a numerical
> error was introduced to one of my programs because of an unexpected, silent, compiler optimisation.

Just to be clear, Clojure 1.3-alpha does not introduce numerical
errors, unless you explicitly ask for them; it throws a
RuntimeException - which I guess is analogous to it being a
dynamically-typed language and throwing RuntimeExceptions to signal
type errors.

user=> (* 100000000000 100000000000)
ArithmeticException integer overflow clojure.lang.Numbers.throwIntOverflow (Numbers.java:1583)

user=> (*' 100000000000 100000000000)
10000000000000000000000N

user=> (* 100000000000 100000000000N)
10000000000000000000000N

user=> (unchecked-multiply 100000000000 100000000000)
1864712049423024128

--
Dave

Lee Spector

unread,
Jan 15, 2011, 11:48:34 AM1/15/11
to clo...@googlegroups.com

On Jan 15, 2011, at 8:29 AM, Saul Hazledine wrote:

> It would help people like me understand the debate if some mainstream
> examples of applications requiring (seamless) BigInteger support could
> be identified.


I doubt that many will consider this "mainstream," but I evolve programs using genetic programming techniques and I've found that in this context BigIntegers can arise in all sorts of unexpected and weird and wonderful and sometimes adaptive ways.

I still haven't figured out exactly what the 1.3 changes will mean for this work -- maybe it'll be fine or even better -- but I've liked not having to think about integer sizes much at all previously (as in Common Lisp, where the handling of complex numbers is also nice).

-Lee

Armando Blancas

unread,
Jan 15, 2011, 1:29:27 PM1/15/11
to Clojure
I think this is looking at the situation backwards. I don't want
BigInts, why should I want *them*? Nor longs or whatever for that
matter. What I want is a numerical tower and a language that can
handle it correctly, without blowing up, and if possible with clean
code. I can type 9223372036854775807 or something bigger in a
spreadsheet and get 9.22337203685478E+018, and take it from there.
Now, I don't know if spreadsheet programmers *want* doubles, but
something correct (if less precise) must be done depending on what *I,
the user* wants to do.

Now we face the choice of putting limits on what our users can do, use
a different set of operators, or decide that we "want" BigInts. Again,
this isn't about wanting bigints, that's a red herring. Part of the
problem is the complex, confusing, and sometimes quite mysterious Type
System that's been creeping into Clojure for the sake of performance.
Another is the conflicted attitude of being an untyped language: a
kind of guilty pleasure with the remorse it brings of all those
reflective calls and boxing/unboxing, whose negative effects on
performance supposedly makes the language lose credibility.

If a Type System is what it takes to establish Clojure in the server/
enterprise market, maybe this is be start of a family of languages,
with one simple and elegant and another pragmatic and subordinated to
the demands of business, a la Microsoft C++ with COM or something that
ugly.

Stuart Halloway

unread,
Jan 15, 2011, 4:06:57 PM1/15/11
to clo...@googlegroups.com
> In my experience, errors are the problem and we should be avoiding them, almost at all costs.

This debate always starts by conflating three things into two, and then goes downhill from there. :-( It isn't

(a) safe/slow vs.
(b) unsafe/fast.

It is

(a) unsafe/incorrect value on overflow/fastest/unifiable* vs.
(b) safe/error on overflow/fast/unifiable vs.
(c) safe/promoting on overflow/slow/not-unifiable

*unifiable: able to deliver same semantics for primitives and objects

We have thought about this quite a bit, and an argument from one axis only (e.g safe/unsafe) that doesn't even mention some of the other axes is not likely to be persuasive. Would be more interesting to see a new axis we haven't thought of...

Stu

Stuart Halloway
Clojure/core
http://clojure.com


Stuart Halloway

unread,
Jan 15, 2011, 5:04:10 PM1/15/11
to clo...@googlegroups.com
> I think this is looking at the situation backwards. I don't want
> BigInts, why should I want *them*? Nor longs or whatever for that
> matter. What I want is a numerical tower and a language that can
> handle it correctly, without blowing up, and if possible with clean
> code. I can type 9223372036854775807 or something bigger in a
> spreadsheet and get 9.22337203685478E+018, and take it from there.
> Now, I don't know if spreadsheet programmers *want* doubles, but
> something correct (if less precise) must be done depending on what *I,
> the user* wants to do.

The spreadsheet example may be a useful one. Clojure is a language you could write a spreadsheet in. Would you want to use a spreadsheet written entirely in a JVM language that did not provide access to the primitives?

> Now we face the choice of putting limits on what our users can do, use
> a different set of operators, or decide that we "want" BigInts. Again,
> this isn't about wanting bigints, that's a red herring. Part of the
> problem is the complex, confusing, and sometimes quite mysterious Type
> System that's been creeping into Clojure for the sake of performance.

Clojure is not getting a type system. nor is the behavior in 1.3 complex. It can be confusing, because it is addressing a multifaceted problem, and it certainly is mysterious, because (1) we haven't spent enough effort documenting it, and (2) lots of people have misdocumented it.

I'll make a documentation update higher priority; hopefully that will help.

> Another is the conflicted attitude of being an untyped language: a
> kind of guilty pleasure with the remorse it brings of all those
> reflective calls and boxing/unboxing, whose negative effects on
> performance supposedly makes the language lose credibility.

The Clojure design process is not about achieving credibility, it is about solving problems. Credibility has followed, and will continue to follow, to the extent that Clojure solves problems well.

Mark Engelberg

unread,
Jan 15, 2011, 9:40:33 PM1/15/11
to clo...@googlegroups.com
On Sat, Jan 15, 2011 at 1:06 PM, Stuart Halloway
<stuart....@gmail.com> wrote:
> We have thought about this quite a bit, and an argument from one axis only (e.g safe/unsafe) that doesn't even mention some of  the other axes is not likely to be persuasive. Would be more interesting to see a new axis we haven't thought of...

Here's an axis that hasn't gotten much discussion: How evident is the
behavior of Clojure code, and what features help and hinder this
clarity?

Clojure is a dynamically typed language, which means that generally
speaking, it is not obvious what the type of a given variable is,
since there aren't annotations immediately prior to the variable
telling you what it must be. Similarly, a Clojure IDE does not offer
any way to to "hover" over a variable and see what the type is. There
is a lot of freedom that comes with this, but the cost is that a
dynamic programmer must be careful to document in some way what kinds
of things are acceptable inputs, and what kinds of promises are made
of the outputs. The compiler can't check this, so it's up to the
programmer. As Clojure programmers, we take on the responsibility of
tracking a certain amount of "unseen information" that isn't readily
evident from the code itself, but there's a limit to how much
responsibility programmers can take on before programs become brittle,
so new features should take this "axis" into account.

Primitives are especially problematic because there is no good way to
determine whether something is a primitive or not. Consider the
following interactions in the 1.1 REPL:
user> (type 1)
java.lang.Integer
user> (type (int 1))
java.lang.Integer
Any features involving primitives should be assessed from the
standpoint that it is extremely difficult to know from looking at code
whether something is a primitive or not. Many of the new features
(e.g., static functions can now return primitives, literals are
primitives, but numbers that get stored in collections or cross
certain kinds of function boundaries are not), means that you'll
frequently end up with a mixture of primitives and non-primitives, and
it won't always be obvious which is which.

When designing math operators that behave one way for longs and
another for bigints, one question that needs to be asked is: "How
apparent will it be whether a variable represents a long or a bigint?
If it's not apparent, how will the programmer know which behavior to
expect? Is there any tooling that can help make this more
discoverable?" One possibility is that Clojure programmers will need
to evolve ways to track this information, perhaps by explicitly
commenting in code whether a function can gracefully handle both longs
and bigints. On the other hand, there's already a history in Clojure
and similar languages of just documenting certain vars as "numbers"
without needing to get more precise than that, so this could be a
painful transition for many programmers who are not used to thinking
about specifying their numeric types in greater detail than that.

Because it's difficult to do "typeflow analysis" within a
dynamically-typed language as Clojure, this clarity axis also comes
into play when thinking about what sorts of burdens are going to be
placed on library developers. As a case in point, I developed the
expt function in clojure.contrib.math because I was surprised when I
first came to Clojure that no generic exponentiation operator existed
in the language. The expt in contrib handles all of Clojure's numeric
types seamlessly. But what am I supposed to do with expt in Clojure
1.3? New expectations are being created with the new model -- some
people will expect expt with primitives to return primitives; some
will expect computation with longs to return bigints when necessary,
since exponentiation frequently overflows. Do I need to provide an
expt and expt' function to make both camps happy? (For that matter,
is there even a way to overload expt for both primitive longs and
primitive doubles, or do I need to make separate expt-long and
expt-double functions?) Are we going to see a proliferation of
variations for all mathematical functions once we start going down
this road?

This is an axis I think about a lot, and I hope this is something that
the Clojure dev team is carefully considering as well.

Ken Wesson

unread,
Jan 15, 2011, 9:59:33 PM1/15/11
to clo...@googlegroups.com
On Sat, Jan 15, 2011 at 9:40 PM, Mark Engelberg
<mark.en...@gmail.com> wrote:
> On Sat, Jan 15, 2011 at 1:06 PM, Stuart Halloway
> <stuart....@gmail.com> wrote:
>> We have thought about this quite a bit, and an argument from one axis only (e.g safe/unsafe) that doesn't even mention some of  the other axes is not likely to be persuasive. Would be more interesting to see a new axis we haven't thought of...
>
> Here's an axis that hasn't gotten much discussion:  How evident is the
> behavior of Clojure code, and what features help and hinder this
> clarity?

[massive snip]

> This is an axis I think about a lot, and I hope this is something that
> the Clojure dev team is carefully considering as well.

+1 to all of that.

Michael Gardner

unread,
Jan 16, 2011, 12:26:38 AM1/16/11
to clo...@googlegroups.com
On Jan 15, 2011, at 4:04 PM, Stuart Halloway wrote:

> I'll make a documentation update higher priority; hopefully that will help.

This should help. I feel like the discussion is going in circles because there's no single, official source that summarizes exactly what is happening with numerics in 1.3. (I know about http://www.assembla.com/wiki/show/clojure/Enhanced_Primitive_Support, but it's terse and a bit confusing.)

Jason Wolfe

unread,
Jan 16, 2011, 12:35:28 AM1/16/11
to Clojure
> (a) unsafe/incorrect value on overflow/fastest/unifiable* vs.
> (b) safe/error on overflow/fast/unifiable vs.
> (c) safe/promoting on overflow/slow/not-unifiable

If I understand correctly, the issue with auto-promotion is that we
have to box the output of an operation even if it turns out to fit in
a long, since the compiler must specify whether the result will be
long or Object.

This is probably a stupid question, but since Clojure is already
testing for overflow in (b) to throw an exception, would it be
possible to jump into an alternative (c)-type compilation of the
function just-in-time to promote on overflow instead? It seems like
this could achieve the performance of (b) while still allowing for
auto-promotion (albeit perhaps with a performance hit in that case,
code bloat, compromises about how many versions to compile, etc.).

For what it's worth, I'm personally happy with the approach of the
current alpha.

Sam Roberton

unread,
Jan 16, 2011, 7:56:54 AM1/16/11
to clo...@googlegroups.com

When Clojure compiles your function, it emits JVM bytecode for a new
class which is then loaded by the classloader. That JVM bytecode
defines a function (well, a method as far as the JVM is concerned)
which returns either a primitive type or Object. Your suggestion
would involve redifining the class while it is executing. That's not
possible on the JVM. Even if it were possible -- your function now
returns Object instead of long. But the variable that the result of
your function is about to be assigned to is a long, because that's
what your function used to be defined to return, and the next bytecode
operation in the calling code is the one that subtracts a primitive
long from a primitive long. Now what?

Fundamentally, Clojure has to contend with the fact that the JVM as a
platform distinguishes between primitives and Objects. The bytecode
operations which the Clojure compiler emits have to differ based on
that distinction. Essentially, the distinction cannot be magicked
away, because JVM bytecode is going to statically enforce that we're
either working with a (long or double) or a (Long or BigInt or
BigDecimal or whatever), and never the twain shall meet. So if we
ever want to be able to access the speed of the primitive bytecode
operations, then the primitive/Object distinction has to leak into
Clojure. (Or we have to redefine or move away from the JVM.
Obviously not really an option, but I mention it to point out that the
reason Clojure has to make this decision is that it's a hosted
language. That has a lot of benefits; this is one of the trade-offs
we have to contend with in return.)

I think everyone agrees that it's important to make the speed of the
primitive bytecode operations available in Clojure (whether or not
it's the default), so that rules out the option of always doing
auto-promotion. I think it's probably also agreed that allowing
unchecked overflow is not good (at least, no one seems to be arguing
for it).

So we're left with option (b) and a choice about the default behaviour
of the core library functions. If we default to boxing and treating
everything as an Object then we get the nice comfy numeric tower that
we never have to worry about, but the default case suffers in
performance. Otherwise, we default to primitive, and accept that if
we're dealing with numbers which might get bigger than Long.MAX_VALUE,
then we might need to explicitly use a BigInt to get contagion, or use
an operator like +' which will always deal with Objects.

By choosing to make speed the default preference in the core library
functions, I suppose there's more for Clojure programmers to think
about, because whenever you're dealing with numbers, you need to have
in the back of your mind the question of whether this might ever need
to be bigger than Long.MAX_VALUE, and so whether you might need +'
instead of +. Then again, how often do you write code that might be
doing maths with numbers that big and not realise it? For that
matter, how often do you write code that might be doing maths with
numbers that big and not spend time thinking carefully about its
performance anyway?

Armando Blancas

unread,
Jan 16, 2011, 5:52:11 PM1/16/11
to Clojure
> Then again, how often do you write code that might be
> doing maths with numbers that big and not realise it?  For that
> matter, how often do you write code that might be doing maths with
> numbers that big and not spend time thinking carefully about its
> performance anyway?

This reminds me of a little joke:

Cooper: Honey, will you please - what are the odds of the Russians
attacking on a Thursday night?
--The Man with One Red Shoe (1985)

Having heard from various posters how reasonable it all is, and from
Rich now necessary [1], I still have a feeling that the implications
and consequences of the new semantics aren't well understood, and if
anything, any points of real concerned are downplayed. I'll second
Mark's concerns, above. Like others, I don't like to annotate and
postquote my way to the semantics I want, but that may be the easy
part. But I hope I'm wrong; I guess we'll see.

[1] (in this thread
http://groups.google.com/group/clojure/browse_thread/thread/c8c850595c91cc11/8a4eee5ac4eab3f9?lnk=gst&q=autopromotion#8a4eee5ac4eab3f9)

Jason Wolfe

unread,
Jan 16, 2011, 6:50:04 PM1/16/11
to Clojure

> >> (a) unsafe/incorrect value on overflow/fastest/unifiable* vs.
> >> (b) safe/error on overflow/fast/unifiable vs.
> >> (c) safe/promoting on overflow/slow/not-unifiable
>
> > If I understand correctly, the issue with auto-promotion is that we
> > have to box the output of an operation even if it turns out to fit in
> > a long, since the compiler must specify whether the result will be
> > long or Object.
>
> > This is probably a stupid question, but since Clojure is already
> > testing for overflow in (b) to throw an exception, would it be
> > possible to jump into an alternative (c)-type compilation of the
> > function just-in-time to promote on overflow instead?  It seems like
> > this could achieve the performance of (b) while still allowing for
> > auto-promotion (albeit perhaps with a performance hit in that case,
> > code bloat, compromises about how many versions to compile, etc.).
>
> When Clojure compiles your function, it emits JVM bytecode for a new
> class which is then loaded by the classloader.  That JVM bytecode
> defines a function (well, a method as far as the JVM is concerned)
> which returns either a primitive type or Object.  Your suggestion
> would involve redifining the class while it is executing.  That's not
> possible on the JVM.  Even if it were possible -- your function now
> returns Object instead of long.  

If I understand correctly, the return type of your function is not up
for interpretation. It is Object if undeclared, and primitive only if
declared -- so that is not a problem (if you ask for long, that's what
you'll get). I am only talking about behavior within a single
function call -- no calling code needs to change.

Moreover, we do not need to redefine the class at run-time. A simple
way to do this: when you compile a function with arithmetic
operations, concatenate bytecode for two versions: essentially, one
with the unprimed (exception-throwing) ops and one with the primed
(overflowing) ones. Now, replace the exceptions in the first path
with local jumps into the second path. There are probably ways to get
around generating two copies of the bytecode in advance, and making
such a unilateral shift to inefficient semantics, but I don't know
enough about JVM internals to say for sure.

Sean Corfield

unread,
Jan 16, 2011, 9:18:37 PM1/16/11
to clo...@googlegroups.com
On Sun, Jan 16, 2011 at 3:50 PM, Jason Wolfe <ja...@w01fe.com> wrote:
> Moreover, we do not need to redefine the class at run-time.  A simple
> way to do this: when you compile a function with arithmetic
> operations, concatenate bytecode for two versions: essentially, one
> with the unprimed (exception-throwing) ops and one with the primed
> (overflowing) ones.  Now, replace the exceptions in the first path
> with local jumps into the second path.  There are probably ways to get
> around generating two copies of the bytecode in advance, and making
> such a unilateral shift to inefficient semantics, but I don't know
> enough about JVM internals to say for sure.

The problem is that if you have an arbitrary form that can operate
entirely in primitives (some loop/recur perhaps) and you allow
primitives to magically convert to Objects in that code, then the
entire piece of code has to handle both primitives AND Objects and
every single sub-form must be capable of handling primitives as input
AND Objects as input and returning primitives if possible...

You can't have automatic promotion to Object from primitive and expect
any reasonable code to be generated that can maintain primitive
performance across arbitrary expressions. Either everything can work
with Objects - and you lose performance - or everything must be able
to work within primitives (and at most throw exceptions) and remain
performant.
--
Sean A Corfield -- (904) 302-SEAN
Railo Technologies, Inc. -- http://getrailo.com/
An Architect's View -- http://corfield.org/

"If you're not annoying somebody, you're not really alive."
-- Margaret Atwood

chris

unread,
Jan 16, 2011, 10:10:23 PM1/16/11
to Clojure
Is it insane to suggest that perhaps clojure should work with scala
such that we can write both languages in the same file?

Use scala to do you strongly typed work and things where you are
really concerned that auto-promotion. Let the language made for
helping a programmer lots of information about his/her code (and
perhaps overspecify the solution a gross amount) be used where that is
necessary.

Look, annotating shit in a clojure file to ensure you don't have a
crapload of boxing and inference going in is a large PITA. So in
areas where you know you are going to need to do lots of things like
that, why wouldn't you use scala?

Specifying the type of everything going on is obviously, for a lot of
cases, grossly overspecifying the problem. But what if we could just
write scala code in a clojure file or vice versa? It seems that you
would bypass a lot of the odder 'improvements' to clojure for expert
users and just be better off.

Obviously one of the answers is 'if you think it is a good idea then
you do it' and I don't have the time. But I know this:

1. Meta programming is an insanely powerful tool for compressing
code.
2. Type inference is insanely powerful for producing programs that do
'exactly' what the programming said they should.
3. Clojure-in-java is fast and thus clojure-in-scala would be fast,
no language additions required. And probably 30-50% shorter but
perhaps not more than that.
4. Type annotations are as bad as C programming in terms of verbosity
and to write fast code you need them. A little inference would make a
large difference.

Types in a lisp-like language suck. There isn't a way around it.
Lets use a powerful type-inference tool where appropriate and ditch
them completely where it is inappropriate.

You want a hover-over or tooltip to tell you what type something is
producing or taking (or to guarantee details about what you are
doing)? Use scala.

You want to write extremely compressed code in a way that is very easy
to change and specifies as little of the answer to the problem as
possible (thus giving a lot of leeway for implementation and
extension)? Use clojure.

Chris

nicola...@gmail.com

unread,
Jan 17, 2011, 7:55:59 AM1/17/11
to clo...@googlegroups.com
On Mon, Jan 17, 2011 at 3:10 AM, chris <cnue...@gmail.com> wrote:
> Is it insane to suggest that perhaps clojure should work with scala
> such that we can write both languages in the same file?
>

A lot of reasons for which it is not possible:
- it would mean coordinating two implementations/implementers.
- it would prevent to go to platform for which there is no support in
the other language.
- A type checker would not be really happy to deal with a lot of
Object -> Object functions...
- it would be ugly

Having a bit of (optional) type inference for performance and
compile-time safety in Clojure could be interesting though.

chris

unread,
Jan 17, 2011, 1:22:39 PM1/17/11
to Clojure

Bob Hutchison

unread,
Jan 17, 2011, 4:05:47 PM1/17/11
to clo...@googlegroups.com
Hi Stuart,

On 2011-01-15, at 4:06 PM, Stuart Halloway wrote:

>> In my experience, errors are the problem and we should be avoiding them, almost at all costs.
>
> This debate always starts by conflating three things into two, and then goes downhill from there. :-( It isn't
>
> (a) safe/slow vs.
> (b) unsafe/fast.

That's how us outsiders are left to look at it.

>
> It is
>
> (a) unsafe/incorrect value on overflow/fastest/unifiable* vs.
> (b) safe/error on overflow/fast/unifiable vs.
> (c) safe/promoting on overflow/slow/not-unifiable
>
> *unifiable: able to deliver same semantics for primitives and objects

This doesn't really help me understand your argument.

It looks to me as though Clojure is trying to steer itself through the middle of something. The trouble is that I don't know where the edges of the middle are.

Maybe it is just a documentation problem. But I'd also suggest that there's a bit of a sales job necessary here.

>
> We have thought about this quite a bit,

Nobody doubts that, certainly I don't. And I'm not trying to minimise or dismiss what you've done. And I'm not claiming that I've thought about it better or more or deeper. But I do have concerns and I don't see them being addressed, and I'd like it if they weren't minimised either. Maybe my concerns are completely addressed. Maybe not. I don't know, and I'd like to be convinced.

> and an argument from one axis only (e.g safe/unsafe) that doesn't even mention some of the other axes is not likely to be persuasive. Would be more interesting to see a new axis we haven't thought of...

Numerical correctness, for some of us, is an overwhelming issue. This is purely from experience... bad experience... 30+ years of bad experience in my case :-) From my point of view, the approach Clojure is taking isn't persuasive, not to say it couldn't be made persuasive.

I think I did add what might be considered an additional axis. Syntax. Specifically what annotations are needed and for what purpose. I don't think this should be dismissed out of hand.

Cheers,
Bob

>
> Stu
>
> Stuart Halloway
> Clojure/core
> http://clojure.com
>
>

> --
> You received this message because you are subscribed to the Google
> Groups "Clojure" group.
> To post to this group, send email to clo...@googlegroups.com
> Note that posts from new members are moderated - please be patient with your first post.
> To unsubscribe from this group, send email to
> clojure+u...@googlegroups.com
> For more options, visit this group at
> http://groups.google.com/group/clojure?hl=en

Jason Wolfe

unread,
Jan 17, 2011, 4:17:01 PM1/17/11
to Clojure


On Jan 16, 6:18 pm, Sean Corfield <seancorfi...@gmail.com> wrote:
> On Sun, Jan 16, 2011 at 3:50 PM, Jason Wolfe <ja...@w01fe.com> wrote:
> > Moreover, we do not need to redefine the class at run-time.  A simple
> > way to do this: when you compile a function with arithmetic
> > operations, concatenate bytecode for two versions: essentially, one
> > with the unprimed (exception-throwing) ops and one with the primed
> > (overflowing) ones.  Now, replace the exceptions in the first path
> > with local jumps into the second path.  There are probably ways to get
> > around generating two copies of the bytecode in advance, and making
> > such a unilateral shift to inefficient semantics, but I don't know
> > enough about JVM internals to say for sure.
>
> The problem is that if you have an arbitrary form that can operate
> entirely in primitives (some loop/recur perhaps) and you allow
> primitives to magically convert to Objects in that code, then the
> entire piece of code has to handle both primitives AND Objects and
> every single sub-form must be capable of handling primitives as input
> AND Objects as input and returning primitives if possible...
>
> You can't have automatic promotion to Object from primitive and expect
> any reasonable code to be generated that can maintain primitive
> performance across arbitrary expressions. Either everything can work
> with Objects - and you lose performance - or everything must be able
> to work within primitives (and at most throw exceptions) and remain
> performant.

I think you can. Let me elaborate on my simplistic example. Compile
the code for a function twice -- once where everything works within
primitives, and once where everything works with Objects -- and
concatenate the bytecode together. Start by running the primitive
version. On overflow, jump into the corresponding spot in the object
version (after some patching up / boxing) -- rather than throwing an
exception. If no overflow happens, you run *exactly* the same
bytecode as the current unprimed ops. (I'm not sure how this would
interact with JIT compared to exceptions, though). As soon as
overflow happens, you run code as if all of the operations were
primed, until function call exit.

I believe this is related to the approach used by TraceMonkey,
Firefox's Javascript compiler:
http://hacks.mozilla.org/2009/07/tracemonkey-overview/

Anyway, I'll leave this be now; I just wanted to mention the idea
since I hadn't seen it discussed.

Mark Engelberg

unread,
Jan 17, 2011, 4:17:33 PM1/17/11
to clo...@googlegroups.com
On Mon, Jan 17, 2011 at 1:05 PM, Bob Hutchison <hutch...@recursive.ca> wrote:
> Numerical correctness, for some of us, is an overwhelming issue. This is purely from experience... bad experience... 30+ years of bad experience in my case :-) From my point of view, the approach Clojure is taking isn't persuasive, not to say it couldn't be made persuasive.

Under the current proposal, you should never get an incorrect answer.
You might get an error, though. It's a subtle difference, but this is
the main reason why the developers don't see it as a "correctness
issue". If your program runs, and gives you back an answer, it will
be correct. If it crashes, you convert to the overflow version of
arithmetic, or typecast some of your numbers to bigints, and you'll
get the right answer. I think a lot of the argument from both sides
boils down to how much you fear runtime crashes.

Brian Goslinga

unread,
Jan 17, 2011, 6:24:55 PM1/17/11
to Clojure
On Jan 17, 3:17 pm, Jason Wolfe <ja...@w01fe.com> wrote:
> I think you can.  Let me elaborate on my simplistic example.  Compile
> the code for a function twice -- once where everything works within
> primitives, and once where everything works with Objects -- and
> concatenate the bytecode together.  Start by running the primitive
> version.  On overflow, jump into the corresponding spot in the object
> version (after some patching up / boxing) -- rather than throwing an
> exception.  If no overflow happens, you run *exactly* the same
> bytecode as the current unprimed ops.  (I'm not sure how this would
> interact with JIT compared to exceptions, though).  As soon as
> overflow happens, you run code as if all of the operations were
> primed, until function call exit.
I'm not sure the JVM allows methods to do that; it might be possible
though.

This doesn't address the issue that a method has to decide up front
whether it is going to return a primitive or an object. I suppose you
could compile two versions of each function, but brings up two issues:
First, there would be massive code bloat, which might hurt the ability
of the code to be JITed well. Second, if a function is hinted to
return a primitive, it would almost certainly be an error to return
something that isn't a primitive.

Jason Wolfe

unread,
Jan 17, 2011, 6:41:45 PM1/17/11
to Clojure


On Jan 17, 3:24 pm, Brian Goslinga <quickbasicg...@gmail.com> wrote:
> On Jan 17, 3:17 pm, Jason Wolfe <ja...@w01fe.com> wrote:> I think you can.  Let me elaborate on my simplistic example.  Compile
> > the code for a function twice -- once where everything works within
> > primitives, and once where everything works with Objects -- and
> > concatenate the bytecode together.  Start by running the primitive
> > version.  On overflow, jump into the corresponding spot in the object
> > version (after some patching up / boxing) -- rather than throwing an
> > exception.  If no overflow happens, you run *exactly* the same
> > bytecode as the current unprimed ops.  (I'm not sure how this would
> > interact with JIT compared to exceptions, though).  As soon as
> > overflow happens, you run code as if all of the operations were
> > primed, until function call exit.
>
> I'm not sure the JVM allows methods to do that; it might be possible
> though.

I believe it's possible, as long as the bytecodes are combined into a
single method body (due to the local jump restriction).

> This doesn't address the issue that a method has to decide up front
> whether it is going to return a primitive or an object. I suppose you
> could compile two versions of each function, but brings up two issues:
> First, there would be massive code bloat, which might hurt the ability
> of the code to be JITed well.

This is an issue. I believe it might be avoidable with some tricks,
but I don't know enough about JVM internals to say for sure. (I can
elaborate on ideas if requested).

> Second, if a function is hinted to
> return a primitive, it would almost certainly be an error to return
> something that isn't a primitive.

Correct. As I mentioned earlier, if you hint the return as long, we
would always return a long (truncating/throwing if necessary). This
is only talking about behavior *within* a function; the external
interface of the function is not up for interpretation. And, taking a
step back, what people seem to be concerned with is "long" semantics
being inferred when that's not what they want. If the user explicitly
declares the return value to be a long, this concern would no longer
apply.


Brian Goslinga

unread,
Jan 17, 2011, 6:46:48 PM1/17/11
to Clojure
I'll also add that type inference wouldn't solve the problem, it would
just move the pain the the design of the type system and details
relating to it. The type system would probably be at least as complex
as Java generics to be something worthwhile if you do the type
inferencing for perf primarily and type checking incidentally; if you
want type checking primarily it would most likely be even more
complicated.

The changes in 1.3 don't really make the language more complicated; in
contrast, type inferencing would.

Stuart Sierra

unread,
Jan 18, 2011, 1:02:53 PM1/18/11
to clo...@googlegroups.com
Yes, very important difference that. Clojure will *not* return incorrect results on overflow (Java will) but it might throw an exception.
Reply all
Reply to author
Forward
0 new messages