Fwd: [scalatest-users] Binary incompatibilities with 2.9.1?

[Bill Venners]

Hi,

A ScalaTest user got a stack trace trying to use ScalaTest 1.6.1 with
2.9.1 that looks like it could be a binary compatibility between 2.9.1
and 2.9.0, but I'm not certain. The way I deployed that version was by
redeploying the binaries I built against 2.9.0 under the artifact tag
scalatest_2.9.1, because all 2.9.x are supposed to be binary
compatible. Does that stack trace look familiar to anyone?

Bill

---------- Forwarded message ----------
From: vvilhonen <vesa.v...@gmail.com>
Date: Tue, Nov 1, 2011 at 5:04 AM
Subject: [scalatest-users] Binary incompatibilities with 2.9.1?
To: scalatest-users <scalate...@googlegroups.com>

Hi,

I'm using scalatest 1.6.1 with sbt 0.11.0 and scala 2.9.1. When trying
to execute a simple true should equal (true) spec I get a
ClassCastException right after stepping to scalatest from sbt. Stack
trace:

java.lang.ClassCastException: scala.collection.immutable.Set$EmptySet$
cannot be cast to scala.collection.generic.Addable
       at org.scalatest.tools.Runner$.parseCompoundArgIntoSet(Runner.scala:
1037)
       at org.scalatest.tools.ScalaTestFramework
$ScalaTestRunner.run(ScalaTestFramework.scala:87)
       at sbt.TestRunner.delegateRun(TestFramework.scala:61)
       at sbt.TestRunner.run(TestFramework.scala:55)
       at sbt.TestRunner.runTest$1(TestFramework.scala:75)
       at sbt.TestRunner.run(TestFramework.scala:84)
       at sbt.TestFramework$$anonfun$6$$anonfun$apply$8$$anonfun$7$$anonfun
$apply$9.apply(TestFramework.scala:183)
       at sbt.TestFramework$$anonfun$6$$anonfun$apply$8$$anonfun$7$$anonfun
$apply$9.apply(TestFramework.scala:183)
       at sbt.TestFramework$.sbt$TestFramework$
$withContextLoader(TestFramework.scala:195)
       at sbt.TestFramework$$anonfun$6$$anonfun$apply$8$$anonfun
$7.apply(TestFramework.scala:183)
       at sbt.TestFramework$$anonfun$6$$anonfun$apply$8$$anonfun
$7.apply(TestFramework.scala:183)
       at sbt.Tests$$anonfun$makeParallel$1$$anonfun$apply
$7.apply(Tests.scala:113)
       at sbt.Tests$$anonfun$makeParallel$1$$anonfun$apply
$7.apply(Tests.scala:113)
       at sbt.std.Transform$$anon$3$$anonfun$apply$2.apply(System.scala:47)
       at sbt.std.Transform$$anon$3$$anonfun$apply$2.apply(System.scala:47)
       at sbt.std.Transform$$anon$5.work(System.scala:67)
       at sbt.Execute$$anonfun$submit$1$$anonfun$apply$1.apply(Execute.scala:
221)
       at sbt.Execute$$anonfun$submit$1$$anonfun$apply$1.apply(Execute.scala:
221)
       at sbt.ErrorHandling$.wideConvert(ErrorHandling.scala:13)
       at sbt.Execute.work(Execute.scala:227)
       at sbt.Execute$$anonfun$submit$1.apply(Execute.scala:221)
       at sbt.Execute$$anonfun$submit$1.apply(Execute.scala:221)
       at sbt.CompletionService$$anon$1$$anon$2.call(CompletionService.scala:
26)
       at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
       at java.util.concurrent.FutureTask.run(FutureTask.java:138)
       at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:
441)
       at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
       at java.util.concurrent.FutureTask.run(FutureTask.java:138)
       at java.util.concurrent.ThreadPoolExecutor
$Worker.runTask(ThreadPoolExecutor.java:886)
       at java.util.concurrent.ThreadPoolExecutor
$Worker.run(ThreadPoolExecutor.java:908)
       at java.lang.Thread.run(Thread.java:662)

## the spec

package com.test

import org.scalatest._
import org.scalatest.matchers.ShouldMatchers

class ServerSpec extends FeatureSpec with ShouldMatchers {
 feature("test") {
   scenario("test") {
     true should equal(true)
   }
 }
}

## sbt build.sbt

name := "test"

version := "0.1"

scalaVersion := "2.9.1"

libraryDependencies ++= {
 val ufVersion = "0.5.1"
 Seq(
   "net.databinder" %% "unfiltered-filter" % ufVersion,
   "net.databinder" %% "unfiltered-jetty" % ufVersion,
   "net.databinder" %% "unfiltered-json" % ufVersion,
   "net.databinder" %% "unfiltered-scalatest" % ufVersion,
   "org.scalatest" %% "scalatest" % "1.6.1" % "test"
 )
}

--
You received this message because you are subscribed to the Google
Groups "scalatest-users" group.
To post to this group, send email to scalate...@googlegroups.com
To unsubscribe from this group, send email to
scalatest-use...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/scalatest-users?hl=en
ScalaTest itself, and documentation, is available here:
http://www.artima.com/scalatest

--
Bill Venners
Artima, Inc.
http://www.artima.com

[Mirco Dotta]

Hi Bill, 

Scala library only ensures *forward* binary compatibility, not backward. 

From the use case you described, it looks like your user is on 2.9.1 and 

he's trying to run his code against ScalaTest compiled with 2.9.0, which 

is not ensured to be binary compatible (and it is likely not). 

Shameless plug, you could use the Scala Migration Manager [1] to verify that.

Cheers, 

 Mirco

[1] http://typesafe.com/technology/migration-manager

---------------

Mirco Dotta

Typesafe - Enterprise-Grade Scala from the Experts 

PSE-D, 1015 Lausanne, Switzerland

Work: +41 (0)21 691 49 65

Twitter: @mircodotta

[Michael Bayne]

On Tue, Nov 1, 2011 at 8:21 AM, Mirco Dotta <mirco...@typesafe.com> wrote:
> Scala library only ensures *forward* binary compatibility, not backward.

I don't mean to start a long debate on the subject, but isn't backward
compatibility far more useful than forward?

As a Scala user, a problem I'm going to frequently face when upgrading
from, say, 2.9.0 to 2.9.1 is that the dozens of third-party libraries
I make use of will likely not be updated for 2.9.1, or perhaps not as
quickly as I like. So I'm forced to remain on 2.9.0 until every single
one of the third party libraries that I use has been recompiled and
reissued by its extremely diligent maintainer.

The only value I can imagine from forward compatibility is that if I
were using 2.9.0, and one of my third party library providers moved on
to 2.9.1, I could make use of their new library without having to
upgrade to 2.9.1 myself. But if I had backward compatibility, I could
just upgrade to 2.9.1 in order to use that new library, and benefit
from all the bug fixes in 2.9.1 to boot.

Am I missing some great benefit of forward compatibility? Because it
doesn't seem worth the trouble. Might as well have no inter-release
compatibility and spend the extra time on new features.

-- m...@samskivert.com

[Bill Venners]

Hi Mirco,

I agree with Michael, and it sure wasn't obvious to me that this was
the binary compatibility promise. The drag on Scala adoption has
nothing to do with a lack forward compatibility and everything to do
with a lack of backward compatibility.

Bill

On Nov 1, 8:57 am, Michael Bayne <m...@samskivert.com> wrote:

[Mirco Dotta]

Hi Bill, Michael,

Actually, in my first answer I was wrong, so let me clear things up

(and please forgive me for the confusion I created).

2.9.x releases are forward binary compatible. It means that if I'm on 

2.9.1 and I use a third party library that was compiled against 2.9.0, that 

is ensured to work (i.e., it *is* binary compatible).

Your user is on 2.9.1and he is using ScalaTest 1.6.1 which, as far as I 

understand, it is compiled against 2.9.0. That is exactly the setup I just 

described above, so that should just work.

I checked the binaries of scala-test 1.6.1 and I couldn't find the `checkcast` 

to `scala.collection.generic.Addable` that is reported in the stacktrace you 

provided. I'm really puzzled, I wonder if your user is really using 2.9.1 (I

fear he could be using 2.8.x, or maybe having two scala libraries in his 

classpath).

To track this down maybe it would be useful to have a look at the build file, 

if he can make it public. I'd be glad to have a look at it (as of now I'm really 

curious to understand what it is actually going on :)

Last point about backward binary compatibility (which is indeed not very useful, 

and it would basically prevent any evolution of both the compiler and the library).

Let's say you are on 2.9.0 and you use a third party library that is compiled 

against 2.9.1, that is *not* ensured to work. I think you will agree that this is 

trade-off that is worth paying.

Cheers,

  Mirco

[Bill Venners]

Hi Mirco,

Ah, OK. That's how I thought it was supposed to work. Sorry I've
always gotten confused by the difference between forward and backward
compatibility. I agree that a guarantee that you could use a library
compiled with Scala 2.9.1 when using Scala 2.9.0 is not needed. That's
what I thought (and I think Michael thought) you were saying was the
guarantee. But a library compiled with Scala 2.9.0 should work with
Scala 2.9.1. That should be, an I think you're saying is, the
"guarantee" or at least goal for typesafe/EPFL.

I'll dig into what this user's problem is. One thing I'll try is to
deploy a snapshot built with 2.9.1 and see if that has the same
problem.

By the way, the reason I haven't done a new build for each binary
compatible Scala point release is that I want to reduce risk that a
build error will happen. Builds need to be reproducible, but build
mistakes can and do happen. I also want to be able to know when a user
reports a problem, exactly what bits they are using by the version
number of the software (in this case ScalaTest), not by version number
plus a little Scala version number hanging off the artifact ID. Some
users don't use Maven or sbt, and download a zip file. They wouldn't
even have an artifact ID. But sbt's %% operator is convenient, and I
don't mind redeploying the same version in multiple ways. So I deploy
a zip file, I deploy it to scala-tools under each artifact ID, and
that gets mirrored to Maven central. So the same 1.6.1 release, which
works for Scala 2.9.x, gets released in a lot of different places. But
it is always the same exact bits.

Bill

[Heiko Seeberger]

On Nov 1, 2011, at 3:30 PM, Bill Venners wrote:

So I deploy
a zip file, I deploy it to scala-tools under each artifact ID, and
that gets mirrored to Maven central. So the same 1.6.1 release, which
works for Scala 2.9.x, gets released in a lot of different places. But
it is always the same exact bits.

Interesting approach. Good test case for binary compatibility!

Heiko

[vvilhonen]

Hi,

Here you can find the sbt project that shows the exception when trying
to run sbt test.

https://github.com/vvilhonen/binarybackward

Also requires sbt:

https://github.com/harrah/xsbt/wiki

- Vesa

On Nov 1, 9:57 pm, Heiko Seeberger <heiko.seeber...@googlemail.com>
wrote:

[Grzegorz Kossakowski]

On 1 November 2011 23:43, vvilhonen <vesa.v...@gmail.com> wrote:

Hi,

Here you can find the sbt project that shows the exception when trying
to run sbt test.

https://github.com/vvilhonen/binarybackward

Also requires sbt:

https://github.com/harrah/xsbt/wiki

I reproduced the problem and even dig into a bit but I have no clue how this could happen. Mirco, i think you need to dig more. Report back your findings as this looks really interesting.

--
Grzegorz Kossakowski

[Todd Vierling]

On Tuesday, November 1, 2011 3:30:19 PM UTC-4, Bill Venners wrote:

I'll dig into what this user's problem is. One thing I'll try is to
deploy a snapshot built with 2.9.1 and see if that has the same
problem. 

The line number mentioned in the backtrace is a hint: Runner.scala:1037. This line number isn't in the method parseCompoundArgIntoSet until you roll all the way back to Runner.scala r2478.

Digging further, building the example project (https://github.com/vvilhonen/binarybackward) from a clean ~/.sbt and ~/.ivy2 reveals that the process to bootstrap and build pulls in:

~/.ivy2/cache $ grep -r 'org/scalatest/tools/Runner\$' .

Binary file ./org.scalatest/scalatest_2.9.1/jars/scalatest_2.9.1-1.6.1.jar matches

Binary file ./org.scalatest/scalatest/jars/scalatest-1.3.jar matches

Lo and behold, scalatest-1.3.jar does use a org/scalatest/tools/Runner$.class that says, per javap -c:

===

public scala.collection.immutable.Set parseCompoundArgIntoSet(scala.collection.immutable.List, java.lang.String);

  Code:

[...]

   22:  checkcast       #780; //class scala/collection/generic/Addable

[...]

  LineNumberTable: 

   line 1037: 0

===

Methinks a pom somewhere is doing the wrong thing, or worse, sbt itself is doing the wrong thing.

[Todd Vierling]

On Tuesday, November 1, 2011 11:57:24 PM UTC-4, Todd Vierling wrote:

Methinks a pom somewhere is doing the wrong thing

Yup, someone was a bad pom writer:

http://scala-tools.org/repo-releases/net/databinder/unfiltered-scalatest_2.9.1/0.5.1/unfiltered-scalatest_2.9.1-0.5.1.pom

(search the xml text for "scalatest")

[Mirco Dotta]

Well done Todd! 

I'm glad you could figure out the bit that was messing up the build.

-- Mirco

[Grzegorz Kossakowski]

On 2 November 2011 07:10, Mirco Dotta <mirco...@typesafe.com> wrote:

Well done Todd! 

I'm glad you could figure out the bit that was messing up the build.

Oh well! I should have looked into classpath first instead of trying to chase problems in scala library. Anyway, it looks like Scala is not to be blamed :)

--
Grzegorz Kossakowski

[Josh Suereth]

Check out how to exclude dependencies in SBT:

https://github.com/harrah/xsbt/wiki/Library-Management

This should remove the incompatible scalatest versions.