[SIP 19] New SIP: Implicit Source Locations

392 views
Skip to first unread message

Philipp Haller

unread,
Mar 30, 2012, 5:18:36 AM3/30/12
to scala...@googlegroups.com
Hi all,

Here is a new SIP which describes Implicit Source Locations [1], a feature that has been in Scala-Virtualized for a while (with extensions), and which has been used in frameworks and DSLs built on top of it, such as Delite and OptiML [2].

It is now a SIP so that it can be discussed and voted on formally for inclusion in Scala 2.10.

Cheers,
Philipp

Eugene Burmako

unread,
Mar 30, 2012, 5:22:57 AM3/30/12
to <scala-sips@googlegroups.com>
Why not expose a Position, which can also be a range position?

Eugene Burmako

unread,
Mar 30, 2012, 5:43:03 AM3/30/12
to scala...@googlegroups.com
Another suggestion would be writing an auxiliary macro, which gives you access to the position and does not require changing the original method at all.

Example:

def foo(implicit pos: SourcePosition) = {
  val bar = f(pos)
  baz
}

Would become:

def foo = macro impl

def impl(c: Context) = {
  reify { foo(c.pos.eval) }
}

def foo(pos: SourcePosition) = <original definition of foo>

Benefit of this approach is reducing magic and unifying metaprogramming facilities under macros infrastructure.

Also, could you please elaborate on the differences between SourceContext and SourceInfo, so that we can discuss them w.r.t macros.

Eugene Burmako

unread,
Mar 30, 2012, 6:02:40 AM3/30/12
to scala...@googlegroups.com
Okay, the boilerplate can be reduced to just one method:

@inline final def foo = withPosition(foo)

def foo(pos: Position) = <original definition of foo>

Where withPosition is a macro that goes from T to T and injects the position into the invocation it wraps.

Miles Sabin

unread,
Mar 30, 2012, 7:10:47 AM3/30/12
to scala...@googlegroups.com
On Fri, Mar 30, 2012 at 11:02 AM, Eugene Burmako <eugene....@epfl.ch> wrote:
> Okay, the boilerplate can be reduced to just one method:
>
> @inline final def foo = withPosition(foo)
>
> def foo(pos: Position) = <original definition of foo>
>
> Where withPosition is a macro that goes from T to T and injects the position
> into the invocation it wraps.

Maybe I'm missing something, but isn't there only one invocation of
foo(pos: Position) being wrapped, namely the one in,

@inline final def foo = withPosition(foo)

^^^

So wouldn't we only ever see that one source position?

That said, I'm very much in favour or trying to unify these mechanisms
if it's at all possible.

Cheers,


Miles

--
Miles Sabin
tel: +44 7813 944 528
gtalk: mi...@milessabin.com
skype: milessabin
g+: http://www.milessabin.com
http://twitter.com/milessabin
http://underscoreconsulting.com
http://www.chuusai.com

Eugene Burmako

unread,
Mar 30, 2012, 7:40:14 AM3/30/12
to scala...@googlegroups.com
Yeah, true, my bad.

However, we've come up with an implicit macro idea that seems to be working. We'll post the update when we're sure.

Eugene Burmako

unread,
Apr 1, 2012, 1:14:47 PM4/1/12
to scala-sips
Okay, here we go:
[1] Location-generating macro:
https://github.com/scalamacros/kepler/blob/0cabc96861c4d7b0e9dcdb8ea1394f401ceba5d5/test/files/run/macro-sip19/Impls_Macros_1.scala
[2] Location-aware function:
https://github.com/scalamacros/kepler/blob/0cabc96861c4d7b0e9dcdb8ea1394f401ceba5d5/test/files/run/macro-sip19/Test_2.scala

Good news:
+ It's totally possible to implement SIP-19 without any changes to the
compiler.

Bad news:
- Chaining of source locations becomes ugly (see [2] for the example),
because implicit parameter of type SourceLocation and implicit macro
of type => SourceLocation conflict. Any ideas how to fix that?

On Mar 30, 1:10 pm, Miles Sabin <mi...@milessabin.com> wrote:

Jason Zaugg

unread,
Apr 1, 2012, 2:29:36 PM4/1/12
to scala...@googlegroups.com
On Sun, Apr 1, 2012 at 7:14 PM, Eugene Burmako <eugene....@epfl.ch> wrote:
> Okay, here we go:
> [1] Location-generating macro:
> https://github.com/scalamacros/kepler/blob/0cabc96861c4d7b0e9dcdb8ea1394f401ceba5d5/test/files/run/macro-sip19/Impls_Macros_1.scala
> [2] Location-aware function:
> https://github.com/scalamacros/kepler/blob/0cabc96861c4d7b0e9dcdb8ea1394f401ceba5d5/test/files/run/macro-sip19/Test_2.scala
>
> Good news:
> + It's totally possible to implement SIP-19 without any changes to the
> compiler.
>
> Bad news:
> - Chaining of source locations becomes ugly (see [2] for the example),
> because implicit parameter of type SourceLocation and implicit macro
> of type => SourceLocation conflict. Any ideas how to fix that?

I would love a means to issue a casting vote in case of implicit
ambiguity. Scalaz has a ton of boilerplate (example [1]) that would
melt away with something more refined than LowPriorityImplicits.

This would work for me, although I understand that using annotations
to influence type checking isn't to everyone's taste.

@weight(+1) implicit def sourceLocation = ...

-jason

[1] https://github.com/scalaz/scalaz/blob/scalaz-seven/core/src/main/scala/scalaz/OptionT.scala

martin odersky

unread,
Apr 1, 2012, 5:09:48 PM4/1/12
to scala...@googlegroups.com
On Sun, Apr 1, 2012 at 7:14 PM, Eugene Burmako <eugene....@epfl.ch> wrote:
Okay, here we go:
[1] Location-generating macro:
https://github.com/scalamacros/kepler/blob/0cabc96861c4d7b0e9dcdb8ea1394f401ceba5d5/test/files/run/macro-sip19/Impls_Macros_1.scala
[2] Location-aware function:
https://github.com/scalamacros/kepler/blob/0cabc96861c4d7b0e9dcdb8ea1394f401ceba5d5/test/files/run/macro-sip19/Test_2.scala

Good news:
+ It's totally possible to implement SIP-19 without any changes to the
compiler.

Bad news:
- Chaining of source locations becomes ugly (see [2] for the example),
because implicit parameter of type SourceLocation and implicit macro
of type => SourceLocation conflict. Any ideas how to fix that?

But as far as I can see SIP 19 does not specify chaining?

Cheers

 - Martin


 
On Mar 30, 1:10 pm, Miles Sabin <mi...@milessabin.com> wrote:
> On Fri, Mar 30, 2012 at 11:02 AM, Eugene Burmako <eugene.burm...@epfl.ch> wrote:
> > Okay, the boilerplate can be reduced to just one method:
>
> > @inline final def foo = withPosition(foo)
>
> > def foo(pos: Position) = <original definition of foo>
>
> > Where withPosition is a macro that goes from T to T and injects the position
> > into the invocation it wraps.
>
> Maybe I'm missing something, but isn't there only one invocation of
> foo(pos: Position) being wrapped, namely the one in,
>
>   @inline final def foo = withPosition(foo)
>                                        ^^^
>
> So wouldn't we only ever see that one source position?
>
> That said, I'm very much in favour or trying to unify these mechanisms
> if it's at all possible.
>
> Cheers,
>
> Miles
>
> --
> Miles Sabin
> tel: +44 7813 944 528
> gtalk: mi...@milessabin.com
> skype: milessabin
> g+:http://www.milessabin.comhttp://twitter.com/milessabinhttp://underscoreconsulting.comhttp://www.chuusai.com



--
Martin Odersky
Prof., EPFL and Chairman, Typesafe
PSED, 1015 Lausanne, Switzerland
Tel. EPFL: +41 21 693 6863
Tel. Typesafe: +41 21 691 4967

Eugene Burmako

unread,
Apr 1, 2012, 5:47:05 PM4/1/12
to scala...@googlegroups.com, Philipp Haller
Good question.

Here's what the SIP says: "First, if there is already an implicit argument of type SourceLocation, this argument is selected. Otherwise, an instance of SourceLocation is generated by invoking the apply method of the object scala.reflect.SourceLocation, passing the components of the source location as arguments".

If you follow the specification to the letter, it says that fresh SourceLocations must be generated every time when the implicit argument is not provided explicitly. That is, even if there's a compatible implicit in scope. Let's ask Philipp (/cc'd) for clarifications.

Philipp Haller

unread,
Apr 1, 2012, 9:36:29 PM4/1/12
to <scala-sips@googlegroups.com>
On Apr 1, 2012, at 11:09 PM, martin odersky wrote:



On Sun, Apr 1, 2012 at 7:14 PM, Eugene Burmako <eugene....@epfl.ch> wrote:
Okay, here we go:
[1] Location-generating macro:
https://github.com/scalamacros/kepler/blob/0cabc96861c4d7b0e9dcdb8ea1394f401ceba5d5/test/files/run/macro-sip19/Impls_Macros_1.scala
[2] Location-aware function:
https://github.com/scalamacros/kepler/blob/0cabc96861c4d7b0e9dcdb8ea1394f401ceba5d5/test/files/run/macro-sip19/Test_2.scala

Good news:
+ It's totally possible to implement SIP-19 without any changes to the
compiler.

Bad news:
- Chaining of source locations becomes ugly (see [2] for the example),
because implicit parameter of type SourceLocation and implicit macro
of type => SourceLocation conflict. Any ideas how to fix that?

But as far as I can see SIP 19 does not specify chaining?

That's true, it doesn't specify chaining. So far, chaining is only implemented in Virtualized Scala.

I think what Eugene wanted to point out is that it's a bit cumbersome to get access to the implicit SourceLocation parameter of a method, and at the same time create a new SourceLocation for the next callee (preventing the existing implicit value to simply be passed as an argument also to the callee).

To do that we need to save the implicit parameter, and subsequently shadow it, so that the implicit macro is invoked again. If we do that then we could implement chaining explicitly, which is of course not as nice as in Virtualized Scala. So, I think we still need to clarify whether SIP 19 should specify chaining.

Cheers,
Philipp

Eugene Burmako

unread,
Apr 2, 2012, 4:20:58 AM4/2/12
to scala-sips
Allrighty, with a help of a little hack (not on the compiler side!)
implicit location macro now has bigger priority than implicit location
values in scope. I'm not very into implicit prioritization rules, so
maybe there exists a more elegant way to solve the ambiguity problem.

This not only gets rid of all the ugliness, but also enables
transparent chaining as shown in [1]. Updated macro is here: [2].

[1] https://github.com/scalamacros/kepler/blob/e388ddc559549573e3b3357f619c1bf6a422ebaa/test/files/run/macro-sip19-revised.check
[2]
https://github.com/scalamacros/kepler/blob/e388ddc559549573e3b3357f619c1bf6a422ebaa/test/files/run/macro-sip19-revised/Impls_Macros_1.scala

On Apr 2, 3:36 am, Philipp Haller <philipp.hal...@epfl.ch> wrote:
> On Apr 1, 2012, at 11:09 PM, martin odersky wrote:
>
> On Sun, Apr 1, 2012 at 7:14 PM, Eugene Burmako <eugene.burm...@epfl.ch<mailto:eugene.burm...@epfl.ch>> wrote:
> Okay, here we go:
> [1] Location-generating macro:https://github.com/scalamacros/kepler/blob/0cabc96861c4d7b0e9dcdb8ea1...
> [2] Location-aware function:https://github.com/scalamacros/kepler/blob/0cabc96861c4d7b0e9dcdb8ea1...
>
> Good news:
> + It's totally possible to implement SIP-19 without any changes to the
> compiler.
>
> Bad news:
> - Chaining of source locations becomes ugly (see [2] for the example),
> because implicit parameter of type SourceLocation and implicit macro
> of type => SourceLocation conflict. Any ideas how to fix that?
>
> But as far as I can see SIP 19 does not specify chaining?
>
> That's true, it doesn't specify chaining. So far, chaining is only implemented in Virtualized Scala.
>
> I think what Eugene wanted to point out is that it's a bit cumbersome to get access to the implicit SourceLocation parameter of a method, and at the same time create a new SourceLocation for the next callee (preventing the existing implicit value to simply be passed as an argument also to the callee).
>
> To do that we need to save the implicit parameter, and subsequently shadow it, so that the implicit macro is invoked again. If we do that then we could implement chaining explicitly, which is of course not as nice as in Virtualized Scala. So, I think we still need to clarify whether SIP 19 should specify chaining.
>
> Cheers,
> Philipp
>
> On Mar 30, 1:10 pm, Miles Sabin <mi...@milessabin.com<mailto:mi...@milessabin.com>> wrote:
>
>
>
>
>
> > On Fri, Mar 30, 2012 at 11:02 AM, Eugene Burmako <eugene.burm...@epfl.ch<mailto:eugene.burm...@epfl.ch>> wrote:
> > > Okay, the boilerplate can be reduced to just one method:
>
> > > @inline final def foo = withPosition(foo)
>
> > > def foo(pos: Position) = <original definition of foo>
>
> > > Where withPosition is a macro that goes from T to T and injects the position
> > > into the invocation it wraps.
>
> > Maybe I'm missing something, but isn't there only one invocation of
> > foo(pos: Position) being wrapped, namely the one in,
>
> >   @inline final def foo = withPosition(foo)
> >                                        ^^^
>
> > So wouldn't we only ever see that one source position?
>
> > That said, I'm very much in favour or trying to unify these mechanisms
> > if it's at all possible.
>
> > Cheers,
>
> > Miles
>
> > --
> > Miles Sabin
> > tel: +44 7813 944 528<tel:%2B44%207813%20944%20528>
> > gtalk: mi...@milessabin.com<mailto:mi...@milessabin.com>
> > skype: milessabin
> > g+:http://www.milessabin.comhttp://twitter.com/milessabinhttp://undersco...<http://twitter.com/milessabinhttp://underscoreconsulting.comhttp://ww...>
>
> --
> Martin Odersky
> Prof., EPFL<http://www.epfl.ch/> and Chairman, Typesafe<http://www.typesafe.com/>

Heather Miller

unread,
Apr 2, 2012, 10:28:28 AM4/2/12
to scala...@googlegroups.com
So, while I understand the motivation at the moment to demonstrate that as many compiler-changes as possible can be expressed with macros, I'm not so sure it makes sense to replace the current implementation of SourceLocations outright with macros.

This is mostly due to the pragmas proposed in SIP-18…

Assuming (a) the proposed language imports would come to be, and (b) these implicit macro defs designed to implement SIP-19 also come to be, then wouldn't that mean that DSL authors would have to import macros in order to use SourceLocations?

Bottom-line: it's probably best if DSL authors, didn't have to know that macros even exist. 

-- 
Heather Miller
Doctoral Assistant
EPFL, IC, LAMP

Eugene Burmako

unread,
Apr 2, 2012, 10:43:57 AM4/2/12
to scala...@googlegroups.com
SIP-18 restricts macro definitions, not macro usages, so SourceLocation macros will still work without a flag.

Heather Miller

unread,
Apr 2, 2012, 11:38:28 AM4/2/12
to scala...@googlegroups.com
In my previous email, I was talking about the language import statements, I didn't mention compiler flags. The email was about how this new proposition looked to a user. It's not clear how one should use it, or where it should go. A clearer answer would've been that it could go in the SourceLocation object (whose source would of course import macros as proposed in SIP-18), which could be imported (normally) by the user. Is that what you intend to do with it?

====
The rest isn't written to Eugene, but to the SIP committee in general...

It's not really clear from the chain of emails where or really in depth *how* SourceLocations would be implemented (yes, there's a sketch linked to, but it's preliminary and it's not clear where it'd go in the library) and thus how they'd appear (and how they could misbehave) before users. 

Before settling on macros for this one, maybe it's a good idea to have it fully implemented and tested first? Or at least, an organized plan would be nice before this one's voted on… 

So, the one big thing going for the current approach is that SourceLocations have been in Delite/scala-virtualized for a while, they're understood, used, and tested, they're fully-implemented and sitting as a pull-request on scala/master: https://github.com/scala/scala/pull/343

The new macro-based approach would have to be better-baked, integrated, and more thoroughly tested pretty soon…

-- 
Heather Miller
Doctoral Assistant
EPFL, IC, LAMP

Eugene Burmako

unread,
Apr 2, 2012, 12:04:24 PM4/2/12
to scala...@googlegroups.com
Yes, this is right, proposed macros would need to be imported, or they could go into the SourceLocation object to be located without imports.

Regarding the rest, please, let me know if you need me to elaborate on the details of the provided sketch.

Also, I would like to note that in comparison with the original implementation, macro-based implementation transparently supports chaining, something that cannot be easily achieved without macros. I hope, this shows flexibility of macro-based approaches to metaprogramming and will be an argument towards giving macros a chance.

martin odersky

unread,
Apr 2, 2012, 4:52:58 PM4/2/12
to scala...@googlegroups.com
On Mon, Apr 2, 2012 at 8:38 AM, Heather Miller <heather...@epfl.ch> wrote:
In my previous email, I was talking about the language import statements, I didn't mention compiler flags. The email was about how this new proposition looked to a user. It's not clear how one should use it, or where it should go. A clearer answer would've been that it could go in the SourceLocation object (whose source would of course import macros as proposed in SIP-18), which could be imported (normally) by the user. Is that what you intend to do with it?

Yes, I think that's where it should go.
 
====

It's not really clear from the chain of emails where or really in depth *how* SourceLocations would be implemented (yes, there's a sketch linked to, but it's preliminary and it's not clear where it'd go in the library) and thus how they'd appear (and how they could misbehave) before users. 

Before settling on macros for this one, maybe it's a good idea to have it fully implemented and tested first? Or at least, an organized plan would be nice before this one's voted on… 

So, the one big thing going for the current approach is that SourceLocations have been in Delite/scala-virtualized for a while, they're understood, used, and tested, they're fully-implemented and sitting as a pull-request on scala/master: https://github.com/scala/scala/pull/343

The new macro-based approach would have to be better-baked, integrated, and more thoroughly tested pretty soon…

I agree. But I also think it would be really attractive to do it with macros. The motivation for macros was that they can simplify the language and the compiler. If we do not use them for that, we have failed. 

And, btw, I believe that the total effort to implement SIP 19 as a macro should be not higher than to implement it as a language extension. In particular, it should be simpler to write ScalaDoc for source locations than to put them in the spec.

Cheers

 -- Martin

 

Paul Phillips

unread,
Apr 2, 2012, 10:18:06 PM4/2/12
to scala...@googlegroups.com
I find SourceLocation less interesting than I'd hoped. You can
implement it coarsely without any compiler support at all; I did so
long ago. Dredged up old version appended.

With respect to chaining, why isn't it a linked list? As long as it's
getting special support it may as well support that much.

trait SourceLocation extends Serializable {
def fileName: String = ""
def line: Int
def charOffset: Int = 0
// the missing piece
def previous: SourceLocation = NoSourceLocation
}


// From extempore's giant collection of abandoned git branches


package scala.tools
package reflect

import scala.tools.nsc.util.{ JavaStackFrame, FrameContext, StackSlice }
import scala.collection.{ mutable, immutable, SeqLike }
import mutable.{ WrappedArray, ArrayBuilder }
import Caller._

class CallerOps(override val repr: Caller) extends
SeqLike[StackTraceElement, Caller] {
override protected[this] def thisCollection:
WrappedArray[StackTraceElement] = repr.stack
override protected[this] def toCollection(repr: Caller):
WrappedArray[StackTraceElement] = repr.stack
override protected[this] def newBuilder = new
ArrayBuilder.ofRef[StackTraceElement] mapResult { xs => Caller(xs, 0)
}

def apply(idx: Int) = repr.stack(idx)
def length = repr.stack.length
def iterator = repr.stack.iterator
def seq = repr.stack.seq
}

/** A class for recording from whence a certain call originated.
* This trick is accomplished by having an implicit value of type Caller
* available, and when said implicit is invoked, a stack trace is recorded
* and trimmed to start with the original caller.
*/
class Caller private (val complete: StackSlice, val index: Int) {
def stack = complete drop index
def top = JavaStackFrame(stack(index))
def context() = stack map (x => FrameContext(x))
def frames() = stack map (x => JavaStackFrame(x))

def dropConstructors = dropWhile(_.isConstructor)
def dropWhile(pred: StackTraceElement => Boolean): Caller =
dropUntil(x => !pred(x))
def dropUntil(pred: StackTraceElement => Boolean): Caller = {
stack indexWhere pred match {
case -1 => Caller.Empty
case idx => drop(idx)
}
}
def drop(numFrames: Int): Caller = Caller(complete, index + numFrames)

def intersect(other: Caller): Caller = {
if (stack.length == 0 || other.stack.length == 0)
Caller.Empty
else {
val idx1 = stack indexWhere (_ == other.stack.head)
lazy val idx2 = other.stack indexWhere (_ == stack.head)

if (idx1 > 0)
this drop idx1
else if (idx2 > 0)
other drop idx2
else
(this drop 1) intersect (other drop 1)
}
}
def combine(implicit other: Caller) = intersect(other)

def show() { show(complete.length) }
def show(num: Int) { context() take num foreach println }

private def methodString = dropConstructors match {
case Empty => ""
case frame => " (" + frame.top.tag + ")"
}
override def toString =
if (top.isConstructor) "constructor for class " + top.className +
methodString
else if (top.isStaticConstructor) "static initializer for " + top.className
else top.tag
}
trait LowPriorityCaller {
self: Caller.type =>

private val ThreadName = classOf[Thread].getName
private val CallerName = classOf[Caller].getName
private val CallerFile = nonThreadFrames.head.getFileName

private def nonThreadFrames = Thread.currentThread.getStackTrace
dropWhile (_.getClassName == ThreadName)
private def skipFrame(ste: StackTraceElement) = ste.getFileName == CallerFile

def trace() = apply(nonThreadFrames, 0) dropWhile skipFrame
implicit def reifyCallerStack: Caller = apply(nonThreadFrames, 0)
dropWhile skipFrame
}

object Caller extends LowPriorityCaller {
object Empty extends Caller(Array(), -1) {
override def top = null
override def intersect(other: Caller) = this
override def toString = "NoCaller"
}

def apply(complete: StackSlice, index: Int): Caller = {
if (index < 0 || index >= complete.length) Caller.Empty
else new Caller(complete, index)
}
def caller[T](implicit c: Caller) = c
}

jul...@twitter.com

unread,
Aug 10, 2015, 2:02:35 PM8/10/15
to scala-sips
These links are dead.
Any chance to find those somewhere else?
Reply all
Reply to author
Forward
0 new messages