Is it possible to implement a "pendingUntilCompiles" without untyped macros?

[Jean-Rémi Desjardins]

Hi,

It would be absolutely awesome, if I could use a sort of "pendingUntilCompiles" on some of my specifications.

I find many use cases for this:

• Any project using implicits to extend the Scala system (like the Scala typeclass pattern among others)
• A project that needs to test/specify macros
• Any project that wants to follow TDD by the letter and still be able to execute tests even though some of the other tests don't even compile

Hopefully I have convinced you that this would be useful (if not, please let's continue the discussion). So now my question is, would it be possible to implement this "pendingUntilCompiles" without untyped macros? I think the answer is no.

If so, then I will proceed to lobbying the appropriate people to get untyped macros into Scala in some distant future. If it is possible, please let me know where I missed something.

Cheers!

Jean-Rémi

[etorreborre]

Do you mean that you want to integrate something like that:

https://github.com/milessabin/shapeless/blob/master/core/src/main/scala/shapeless/test/typechecking.scala

So that it returns Pending if a string doesn't compile and Failure("you should remove pendingUntilCompiles") otherwise?

E.

[Jean-Rémi Desjardins]

Sort off. But I was thinking not a string. So you can do something like:

"my test" in {

    val t = new MyClass()

    t.doSomething() ==== expected

}.pendingUntilCompiles("Define and implement MyClass")

Obviously this example does not seem all that useful, but I think it is useful for the use cases outlined in my original message.

Cheers

--
You received this message because you are subscribed to the Google Groups "specs2-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to specs2-users...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[etorreborre]

I had a quick look around and it looks like we really need untyped macros to do that. Eugene Burmako was looking for a "killer" use-case for untyped macros, I don't know about "killer" but that looks like the essential use case.

I'm going to ping him.

Eric.

To unsubscribe from this group and stop receiving emails from it, send an email to specs2-users+unsubscribe@googlegroups.com.

[etorreborre]

Until we get untyped macros I have added some facility for typechecking code:

https://etorreborre.github.io/specs2/guide/SPECS2-3.4/org.specs2.guide.Matchers.html#optional

See the "Typecheck" tab and add "specs2-matcher-extra" to your build to get it.

Eric.

[Jean-Rémi Desjardins]

That's awesome Eric! Thx and I hope we can get untyped macros in so that I can both test that my macro compiles and works as expected without duplicating the test.

The more I think of it. I don't think that pendingUntilCompiles is exactly the right form for this. I think a marker to indicate that a typechecking failure should be a runtime failure as opposed to a compiler error would be more appropriate and then that can integrate with the current pendingUntilFixed. Of course, this marker might shadow some "real" typechecking errors that are a function of an error in the test code as opposed to a failure of a macro failure (or implicit missing). Personally, that is a trade off I am more than willing to make in order to able to run my whole test suite even though I may have broken a specific use case.

What do you think?

Cheers!

To unsubscribe from this group and stop receiving emails from it, send an email to specs2-users...@googlegroups.com.

[Eugene Burmako]

Hi guys,

Here's a recent discussion between Li Haoyi and Adriaan Moors about untyped macros: https://twitter.com/li_haoyi/status/570408843231907840. So it might be safe to assume that the prospects of pushing the original approach to untyped macros into Scala are not very good.

There are alternatives, however. Macro annotations are already untyped, and they aren't ruled out completely. Therefore, we might be able to come up with an alternative syntax for untyped macros, and that might look more acceptable. Denys has been proposing this for a while. Some ideas: `arg: @untypedMacro` or `@untypedMacro arg`. Maybe you would have any ideas in this area?

Cheers,

Eugene

Eric.

To unsubscribe from this group and stop receiving emails from it, send an email to specs2-users...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[etorreborre]

Are you thinking we could write something like this?

def typecheck(arg: @untypedMacro): Typechecked = arg

And arg in the implementation of typecheck would be replaced by whatever the macro annotation does on the "arg" parameter? I am not sure I understand how to interpret "arg: @untypeMacro".

E.

[Jean-Rémi Desjardins]

@Eugene. Thx for the link. It's a very interesting conversation and I still need to think about it some more to come up with my own opinion.

@Erik In the mean time, I have been thinking some more about `Typechecked`. There are a few drawbacks to the current implementation.

The current implementation is expressed as a macro which means that the tests are effectively run at compile time. This implies a few things, one of which is that if a user has a big test suite and wants to reduce execution time by only running tests that match a tag or using `testOnly`, they might be surprised to find out that all their Typechecked tests are still "being run". It also means that tests are not being run in parallel*. 

Another serious drawback with the current version is that it is completely untyped. In an ideal world, any typing failure that's fixed by modifying the test code should yield a compile error whereas a typing failure that's fixed by modifying a macro or some typelevel code should yield a runtime test failure. In practice, I don't think that is possible, but quasiquotes would get us part of the way there.

I propose a version of Typechecked that accepts a Tree (which would be build using quasiquotes) and evals it using the Toolbox. It then checks the output for a MatchResult. This implementation would not suffer from the problems mentioned above. Completely invalid syntax would wield a compile error, tests would be run in parallel*, it would play well with filtering and testOnly and finally one could verify both compilation and behavior in the same test conveniently.

Sorry for the long email. Thoughts?

Cheers!

JR

* to be fair, in my experience, the Toolbox blows up about every 10 times I run tests that make use of it (because they are running in parallel), it's not quite totally Thread safe yet

[etorreborre]

Those are good points Jean-Remi.

I am away for 4 days but I would like if you could post some code showing how you build your quasiquotes and how you use the Toolbox to compile the trees.

Even better would be a PR :-) but for now a few code snippets would really help me.

Thanks.

To unsubscribe from this group and stop receiving emails from it, send an email to specs2-users+unsubscribe@googlegroups.com.

[Jean-Rémi Desjardins]

Ya, sounds good. I will have a look at it.

Cheers,

[Eugene Burmako]

What I had in mind is that much like we have `@ann` in `@ann class C` take `class C` as an untyped argument, we can have `@ann` in `x: @ann` take `x` as an untyped argument. What do you think of this syntax?

--
You received this message because you are subscribed to a topic in the Google Groups "specs2-users" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/specs2-users/zBK5W557qrk/unsubscribe.
To unsubscribe from this group and all its topics, send an email to specs2-users...@googlegroups.com.

[Jean-Rémi Desjardins]

I am not sure that I understand the syntax, would it be possible to do:

({

    invalidReference

}: @dynamic) + validReference

or is that syntax reserved for the parameter list of a function definition?

[Eugene Burmako]

Yes, it would be possible.

[Jean-Rémi Desjardins]

Cool! :-)

[Jean-Rémi Desjardins]

@Eric

So quasiquotes ended up being a bit of a pain because they required the code to be modified to use absolute references to things since in their current form they do not capture the local context for references.

So since quasiquotes were probably overkill for this problem anyway, I rolled out my own macro that does what I previously described.

It has nothing specs2 specific about it: https://github.com/jedesah/scala-dynamic-typing

It's so trivial I don't think it should live on it's own, so we could just copy into specs2.

This does not solve the first issue I expressed which is the lesser one, but solves the second one.

It would be interesting I think when integrating into specs2 to explore having it only fail at runtime on macro expansion and implicit missing typechecking failures. I think very few other typechecking failures at runtime are likely to be useful in the context of testing with specs2.

[Jean-Rémi Desjardins]

Btw, you may notice that I did not use the new typecheck matcher in order to test this macro. I could not get them to work (compile). Is there a trait I need to mix in?

[etorreborre]

That's a good use of string interpolation. I'll integrate it to specs2 and will attribute it to you, thanks.

> Is there a trait I need to mix in?

Yes that trait is in the "optional matchers" section, it's the TypecheckMatchers trait.

E.

PS: I have a one more urgent thing to fix in specs2 (the executor service used for Scalaz Future is not the one I thought I was using) before being able to work on that so stay tuned.

Cool! :-)

To unsubscribe from this group and all its topics, send an email to specs2-users+unsubscribe@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "specs2-users" group.

To unsubscribe from this group and stop receiving emails from it, send an email to specs2-users+unsubscribe@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to a topic in the Google Groups "specs2-users" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/specs2-users/zBK5W557qrk/unsubscribe.

To unsubscribe from this group and all its topics, send an email to specs2-users+unsubscribe@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "specs2-users" group.

To unsubscribe from this group and stop receiving emails from it, send an email to specs2-users+unsubscribe@googlegroups.com.

[etorreborre]

Hi Jean-Remi,

I'm not getting one point with the code you posted.

It looks to me that the actual typechecking still occurs at compile-time. I thought it would be working at first glance but if I put println in the code I can see (and it makes sense) that the dyn"" string are first being replaced, and all the code typechecked, before the specification runs.

I am not sure it is really possible to achieve what you want here but I might be misunderstanding something.

E.

On Sunday, April 12, 2015 at 4:27:27 PM UTC+10, Jean-Rémi Desjardins wrote:

Cool! :-)

To unsubscribe from this group and all its topics, send an email to specs2-users+unsubscribe@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "specs2-users" group.

To unsubscribe from this group and stop receiving emails from it, send an email to specs2-users+unsubscribe@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to a topic in the Google Groups "specs2-users" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/specs2-users/zBK5W557qrk/unsubscribe.

To unsubscribe from this group and all its topics, send an email to specs2-users+unsubscribe@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "specs2-users" group.

To unsubscribe from this group and stop receiving emails from it, send an email to specs2-users+unsubscribe@googlegroups.com.

[Jean-Rémi Desjardins]

You are absolutely correct Eric. That is what I attempted to express in one of my previous emails but did a bad job of it: 

"This does not solve the first issue I expressed which is the lesser one, but solves the second one."

Basically, my line of thought was flawed. The two issues I brought up were fundamentally at odds. How can you attempt to detect a type-checking error that is due to test code or main code without type-checking the string at compile time (I feel a little silly now).

Either of those objectives are attainable separately, but I much prefer the version that tries to be smart about what should fail at compile time and what should be pushed to runtime. And besides, ignored tests are still compiled with vanilla Scala so it's quite reasonable to do the same with these strings.

So to recap, the advantage of the alternative I presented is that it will fail at compile time on invalid Scala syntax, but at runtime if it's a type-checking error as opposed to always at runtime with the current version. The second advantage is that it returns the value of the expression (albeit typed as Any) which can then be used to conduct further testing, which is essential in my macro project to avoid code duplication if I am going to use this kind of thing.

I have been exploring with the possibility of making a much more fine-grained version that only fails at runtime on macro-expansion type-checking failures. And thx to Eugene's response here, I believe it is possible to achieve this, at least in the case of blackbox macros.

[etorreborre]

I have now integrated your suggestions (in https://github.com/etorreborre/specs2/commit/eb5f30d7b945158ca644205ba29df857bd0208b6) and made a distinction between throwing parse errors at compile-time or runtime.

Having another set of methods to control macro-expansion is also interesting. I guess we can add new methods/interpolators to cover that case. Please report on your experiments!

To unsubscribe from this group and stop receiving emails from it, send an email to specs2-users+unsubscribe@googlegroups.com.

[Jean-Rémi Desjardins]

So here is what I have come up with: https://github.com/jedesah/scala-dynamic-typing/blob/master/src/test/scala/MainSpec.scala

This macro fails at compile time on anything but type-checking exceptions that are due to a macro expansion or a missing implicit value. There are three versions of it. One that fails at runtime for both, the other two fail at runtime for macro expansion errors and missing implicit values respectively.

I am having a lot of trouble coming up with adequate names for these macros!! :P

[etorreborre]

Oh my! Ideally we want different combinations of "fail at":

failure\when                      Compile-time     Runtime 

parse

missing implicit

macro-expansion

other typecheck errors

At the moment (3.5) we have:

                                           Compile-time     Runtime 

parse                                        tc                      ptc

missing implicit                                                  tc, ptc

macro-expansion                                               tc, ptc

other typecheck errors                                       tc, ptc

And ideally we want

                                           Compile-time     Runtime 

parse                                        tc                      ptc

missing implicit                         itc,imtc             tc, ptc, mtc

macro-expansion                      mtc,imtc          tc, ptc, itc

other typecheck errors                                      tc, ptc, itc, mtc

where itc is "implicit type check" and mtc is "macro-expansion" typecheck (I'm not sure the whole table is clear, I hope it is).

Do you think it is possible to have a more generic form which would be:

def eval(parameters: TypecheckParameters)(code: String) = macro evalImpl

and case class TypecheckParameters(parse: Boolean, missingImplicits: Boolean, macroExpansion: Boolean) which would decide when to fail at compile-time?

My fear is that we multiply names when we come-up with another situation where we might want to fail at compile-time and not runtime.

What do you think?

Eric.

To unsubscribe from this group and stop receiving emails from it, send an email to specs2-users+unsubscribe@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

[Jean-Rémi Desjardins]

I think the table is overkill, but maybe it's because I am focusing solely on my own use case.

This is how I see it:

I am having a very hard time coming up with a use case for failing to parse at runtime. I think there are only two ways to influence how code is type-checked from within "user space" and that is either with macros or typelevel programming through implicits.

So I see three use cases:

• I want to test my macro
• I want to test my type-level programming through implicits
• I want to test my crazy awesome project that uses both macros and implicits to do some crazy awesome stuff

So I think there should be 3 functions with the following 3 semantics:

• Push type-ckecking errors related to macro expansion to runtime
• Push type-checking errors related to missing implicits to runtime
• Push both to runtime

Do you have any use cases in mind that I am missing?