moving partest, scala-xml out, sbt interface for partest: progress report

[Adriaan Moors]

Hi,

I'm almost done moving partest out, so that we can move scala-xml to its own repository.

https://github.com/adriaanm/scala-partest-test-interface/commits

https://github.com/adriaanm/scala-partest/commits

https://github.com/adriaanm/scala-xml/commits

https://github.com/adriaanm/scala/tree/modularize-partest

I could use some help with the remaining TODOs:

  - in scala-partest-test-interface: get dependencyClasspath in Test from SBT and use that to start partest

  - rework the test/partest script to resolve the current release of partest and launch it

cheers

adriaan

[Mark Harrah]

On Tue, 23 Jul 2013 17:56:00 -0700
Adriaan Moors <adriaa...@typesafe.com> wrote:

> Hi,
>
> I'm almost done moving partest out, so that we can move scala-xml to its
> own repository.
>
> https://github.com/adriaanm/scala-partest-test-interface/commits
> https://github.com/adriaanm/scala-partest/commits
> https://github.com/adriaanm/scala-xml/commits
> https://github.com/adriaanm/scala/tree/modularize-partest
>
> I could use some help with the remaining TODOs:
>
> - in scala-partest-test-interface: get dependencyClasspath in Test from
> SBT and use that to start partest

If the class loader that the Runner gets isn't enough, you can pass the classpath in an argument. If that doesn't sound quite right, can you explain the requirements? (If it is about right, I can provide details tomorrow.)

> - rework the test/partest script to resolve the current release of
> partest and launch it

If I understand correctly, this is partly about how to determine what the current release's version is. If so, you probably want the version to be explicit in the build so that the build is reproducible. (Sorry if that was obvious!) For resolving, I assume you mean via a Maven repository, in which case Josh probably knows how to better integrate that into the Ant build.

-Mark

> cheers
> adriaan
>
> --
> You received this message because you are subscribed to the Google Groups "scala-internals" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to scala-interna...@googlegroups.com.
> For more options, visit https://groups.google.com/groups/opt_out.
>
>

[Adriaan Moors]

Hi Mark

On Tuesday, July 23, 2013, Mark Harrah wrote:

On Tue, 23 Jul 2013 17:56:00 -0700
Adriaan Moors <adriaa...@typesafe.com> wrote:

> Hi,
>
> I'm almost done moving partest out, so that we can move scala-xml to its
> own repository.
>
> https://github.com/adriaanm/scala-partest-test-interface/commits
> https://github.com/adriaanm/scala-partest/commits
> https://github.com/adriaanm/scala-xml/commits
> https://github.com/adriaanm/scala/tree/modularize-partest
>
> I could use some help with the remaining TODOs:
>
>   - in scala-partest-test-interface: get dependencyClasspath in Test from
> SBT and use that to start partest

If the class loader that the Runner gets isn't enough, you can pass the classpath in an argument.  If that doesn't sound quite right, can you explain the requirements?  (If it is about right, I can provide details tomorrow.)

Ah right, the class path for the loader passed to the framework is the dependency classpath configured for test in SBT?

That's perfect. 

>   - rework the test/partest script to resolve the current release of
> partest and launch it

If I understand correctly, this is partly about how to determine what the current release's version is.  If so, you probably want the version to be explicit in the build so that the build is reproducible.  (Sorry if that was obvious!)  For resolving, I assume you mean via a Maven repository, in which case Josh probably knows how to better integrate that into the Ant build.

Thanks, the ant build is done -- it's the bash script version of partest that I wanted to change to use a launcher to download partest from maven. I can also just call into ant and get the jars that way I guess. 

cheers

adriaan

[Som Snytt]

> it's the bash script version of partest that I wanted to change to use a launcher to download partest from maven.

That will be nice.  By coincidence, I had a minor stumble today (unsent email follows) where it suddenly couldn't find diffutils.jar because I accidentally started sbt in `test`.  I fixed the line that prefers test/target over ../build, but that would be blessedly moot.

apm@mara:~/projects/snytt/test$ ./partest --update-check files/neg/t7622-cyclic-dependency

Exception in thread "main" java.lang.RuntimeException: No diffutils.jar found in '/home/apm/projects/snytt/test/target/pack/lib'.
    at scala.sys.package$.error(package.scala:27)
    at scala.tools.partest.nest.PathSettings$$anonfun$diffUtils$1.apply(PathSettings.scala:79)
    at scala.tools.partest.nest.PathSettings$$anonfun$diffUtils$1.apply(PathSettings.scala:79)
    at scala.Option.getOrElse(Option.scala:120)
    at scala.tools.partest.nest.PathSettings$.diffUtils$lzycompute(PathSettings.scala:79)
    at scala.tools.partest.nest.PathSettings$.diffUtils(PathSettings.scala:78)
    at scala.tools.partest.nest.ReflectiveRunner.main(ReflectiveRunner.scala:58)
    at scala.tools.partest.nest.NestRunner$.main(NestRunner.scala:13)
    at scala.tools.partest.nest.NestRunner.main(NestRunner.scala)

But I just ran this command two shell prompts ago!  Did I delete something?

No, I ran sbt! just to try something out.

It creates target in my test dir, and the rest is history, or search order.  Caveat computer!

[Adriaan Moors]

On Tue, Jul 23, 2013 at 11:36 PM, Adriaan Moors <adriaa...@typesafe.com> wrote:

Ah right, the class path for the loader passed to the framework is the dependency classpath configured for test in SBT?

That's perfect. 

Actually, I need to reverse engineer the classpath from the classloader to pass it to scalac as a string... 

Can I safely assume it's going to be a URLClassLoader? (DUN DUN DUN)

[Simon Ochsenreither]

I'm almost done moving partest out, so that we can move scala-xml to its own repository.

What's the benefit of it? I see only a slight drawback of making things a bit more annoying to test and build.

[Mark Harrah]

No, it isn't one in 0.13, for example. I think there are two options. The first is to pass the classpath you need via a test argument. The second is to grab the classpath by slurping the content of the app.class.path resource. This is sbt-specific, but safe. I don't think it is available when forking, though.

-Mark

[Adriaan Moors]

... but its parent is, right? This works (on SBT 0.13) for now: https://github.com/adriaanm/scala-partest-test-interface/blob/master/src/main/scala/com/typesafe/testing/partest/TestingInterface.scala#L76

Is this likely to change/break?

[Adriaan Moors]

The benefit of moving partest out? The benefit of moving anything out?

The main benefit is clear separation of concerns. Everyone agrees this is a good idea for software itself,

so I think it should also be applied to the practice of developing software.

Is a contributor who's interested to work on scaladoc more likely to clone scala/scala,

and find scaladoc in there (along with how to test it, build it, release it,...),

or would we perhaps get (even) more contributions if they could simply clone scala/scaladoc instead?

Keep in mind that that project will have its own PR queue, issue tracker, releases,...

Sure, it's a bit harder for us to integrate these separate pieces, but the net win is a nimbler process overall,

and more openness for new external modules (if we have a story for dealing with our own modules, it's a small step to accept yours).

I want the core Scala team to focus on the standard library and compiler in scala/scala,

and make it as easy as possible to contribute to e.g., Scala's xml library (or forks thereof!) in (via) scala/scala-xml,

and so on for scala-partest, scaladoc, scala-repl, scala-interactive, scala-parser-combinators,...

The first repo will be scala-xml, which needs partest for testing.

Make sense?

cheers

adriaan

On Wed, Jul 24, 2013 at 10:17 AM, Simon Ochsenreither <simon.och...@gmail.com> wrote:

I'm almost done moving partest out, so that we can move scala-xml to its own repository.

What's the benefit of it? I see only a slight drawback of making things a bit more annoying to test and build.

--

[Mark Harrah]

On Wed, 24 Jul 2013 12:22:32 -0700
Adriaan Moors <adriaa...@typesafe.com> wrote:

> ... but its parent is, right? This works (on SBT 0.13) for now:
> https://github.com/adriaanm/scala-partest-test-interface/blob/master/src/main/scala/com/typesafe/testing/partest/TestingInterface.scala#L76
> Is this likely to change/break?

The parent is currently one, but it isn't guaranteed. If you fork, however, it is guaranteed (in 0.13+) that everything is in the application class loader. So, you can use java.class.path in that case. Earlier versions of sbt forked via Scala, so that wasn't true.

-Mark

[Erik Bruchez]

Adriaan,

The split of scala-xml is great news. If I understand well, there are
two aspects to this:

1. The XML library itself.
2. The compiler generating code which uses that library when parsing
XML literals.

Now in order to plug an alternate implementation, one would need to
implement the right API. Where is the best place to look right now to
figure this out? Maybe this?

https://github.com/scala/scala/blob/master/src/compiler/scala/tools/nsc/ast/parser/SymbolicXMLBuilder.scala

-Erik

[Adriaan Moors]

Hi Erik,

Yes, that's right! The compiler is not linked to any particular XML library, as long as the method calls emitted by the symbolic xml builder are implemented appropriately.

I'm sorry we don't have docs on this right now, but happy to help out here where we can!

cheers

adriaan

[Erik Bruchez]

Great, thanks!

-Erik

On Wed, Jul 24, 2013 at 3:04 PM, Adriaan Moors

[Fernando Racca]

Makes a lot of sense!

Thanks

Fernando

[Jason Zaugg]

On Thu, Jul 25, 2013 at 5:30 AM, Adriaan Moors <adriaa...@typesafe.com> wrote:

The benefit of moving partest out? The benefit of moving anything out?

 

The first repo will be scala-xml, which needs partest for testing.

Is this a leap of logic? scala-xml requires scala-library, but we aren't moving that out.

I don't particularly mind the decision to split out scala-partest, but I don't yet see it as an essential step. 

If we do split it, we might consider keeping some scalac specific parts under scala/scala in src/partest-extensions. Some test infrastructure (ie, base classes like DirectTest / CompilerTest) is pretty tightly coupled to internal compiler APIs, and I'm a bit worried about the friction of separate repos.

-jason

[Simon Ochsenreither]

I'm a bit worried about the friction of separate repos.

That was my concern as well.

Adriaan mentioned ...

Keep in mind that that project will have its own PR queue, issue tracker, releases,...

... which makes me a bit skeptical whether having more PR queues solves more issues than it creates (we seem to have trouble handling the one we already have).

But if splitting the libraries would make it possible to move those libraries toward a more standard build/testing motel, e. g. SBT + some testing framework, that would be an improvement imho.

Anyway, if we go ahead, I'd like to see the same happen for scala-swing. (I don't think it is worthwhile to do it for scala-actors if we get rid of them for 2.12.)

[Jason Zaugg]

On Thu, Jul 25, 2013 at 7:39 PM, Simon Ochsenreither <simon.och...@gmail.com> wrote:

I'm a bit worried about the friction of separate repos.

That was my concern as well.

I should add that my concern is mostly specific to partest.

-jason

[Adriaan Moors]

scala-xml's tests need partest

I just published scala-partest 1.0-RC1, and as soon as scala-partest-interface is out, scala-xml can be tested with `sbt test`

friction requires surfaces to meet -- I find that technical solutions often follow the organizational structure

if we want our code to be modular, we should arrange our "work environment" correspondingly

yes, we should avoid fragmentation, but I think the advantage is that we, Scala compiler developers,

experience living in a modular world and the problems that go with it (e.g., enforcing binary compatibility),

so that we can solve them better for everyone

--

[Grzegorz Kossakowski]

On 25 July 2013 09:35, Adriaan Moors <adriaa...@typesafe.com> wrote:

scala-xml's tests need partest

I just published scala-partest 1.0-RC1, and as soon as scala-partest-interface is out, scala-xml can be tested with `sbt test`

friction requires surfaces to meet -- I find that technical solutions often follow the organizational structure

if we want our code to be modular, we should arrange our "work environment" correspondingly

yes, we should avoid fragmentation, but I think the advantage is that we, Scala compiler developers,

experience living in a modular world and the problems that go with it (e.g., enforcing binary compatibility),

so that we can solve them better for everyone

Personally, I find that argument very appealing.

--
Grzegorz Kossakowski

Scalac hacker at Typesafe

twitter: @gkossakowski

github: @gkossakowski

[Paul Phillips]

On Thu, Jul 25, 2013 at 9:49 AM, Grzegorz Kossakowski <grzegorz.k...@gmail.com> wrote:

Personally, I find that argument very appealing.

It's the class of the field, argumentwise.

Have to coat the dog food modules with flame retardant so there can be friction without conflagration.