string interpolation restrictions are too severe

[Paul Phillips]

Am I missing it, or is there no way to divorce a string interpolation template from the interpolation itself? This seems like a big problem to me.  This is what I mean:

object Test {

  implicit class FooOps(val sc: StringContext) {

    def op(x: Int): String = ???

    def ints(foo: Int, bar: Int): String = ???

    def doubles(foo: Int, bar: Int): String = ???

  }

  // The compiler will inline this string constant, but I can't take advantage of it

  final val template = """This is string $interpolation."""

  

  // This does not compile of course: "not found: value interpolation"

  val s = op"""This is string $interpolation."""

  

  // Oh, I'm sure you thought this compiled

  val s1 = optemplate

  

  // I can't do any kind of composition, code organization, or abstraction

  // across the types of the interpolation variables.

  final val s1 = "I am $foo"

  final val s2 = "I am $bar"

  

  // I think these should work - not necessarily with this syntax

  def twoInts(x1: Int, x2: Int) = ints(s1 + ", and" + s2)(x1, x2)

  def twoDoubles(x1: Double, x2: Double) = doubles(s1 + ", and" + s2)(x1, x2)

}

Let me emphasize that we are not talking about having any less information about the interpolation at compile time.  The literal strings are literally inlined and from the standpoint of the compiler, it is - literally - the same thing.  Here is a peek at the constant pool given the above s1, s2, and the expression from twoInts.

// #17 = Utf8               I am $foo

// #18 = String             #17            //  I am $foo

// #19 = Utf8               LineNumberTable

// #20 = Utf8               I am $bar

// #21 = String             #20            //  I am $bar

// #22 = Utf8               twoInts

// #23 = Utf8               (II)Ljava/lang/String;

// #24 = Utf8               I am $foo, and I am $bar

// #25 = String             #24            //  I am $foo, and I am $bar

[Paul Phillips]

On Fri, Jun 29, 2012 at 8:53 AM, Paul Phillips <pa...@improving.org> wrote:

def doubles(foo: Int, bar: Int): String = ???

It it's not abundantly obvious, those are supposed to be Doubles.

[Daniel Sobral]

On Fri, Jun 29, 2012 at 12:53 PM, Paul Phillips <pa...@improving.org> wrote:
> Am I missing it, or is there no way to divorce a string interpolation
> template from the interpolation itself? This seems like a big problem to me.

AFAIK, not outside an untyped macro, which will not be in 2.10 either.
That was clear to me from the SIP, but I never considered it a problem
because I don't know of any string interpolation that does that -- or
even allows user-customization of the interpolation, for that matter.

However, at that point in time untyped macros were still in the radar,
as they'd have to be if we were to use string interpolation for quasi
quotations.

--
Daniel C. Sobral

I travel to the future all the time.

[Miles Sabin]

On Fri, Jun 29, 2012 at 4:53 PM, Paul Phillips <pa...@improving.org> wrote:
> Am I missing it, or is there no way to divorce a string interpolation
> template from the interpolation itself? This seems like a big problem to me.
>  This is what I mean:
>
> object Test {
>   implicit class FooOps(val sc: StringContext) {
>     def op(x: Int): String = ???
>     def ints(foo: Int, bar: Int): String = ???
>     def doubles(foo: Int, bar: Int): String = ???
>   }
>   // The compiler will inline this string constant, but I can't take
> advantage of it
>   final val template = """This is string $interpolation."""
>
>   // This does not compile of course: "not found: value interpolation"
>   val s = op"""This is string $interpolation."""
>
>   // Oh, I'm sure you thought this compiled
>   val s1 = optemplate

A big part of the problem is the embedded free name. At one point we
were able to use function literal wildcard syntax with string
templates,

https://gist.github.com/a69d8ffbfe9f42e65fbf

It'd be nice if we could get that back and write something like,

// Inferred type should be Int => String
final val template = """This is string $(_:Int)."""

The all of your examples boil down to function application and/or composition.

Cheers,


Miles

--
Miles Sabin
tel: +44 7813 944 528
gtalk: mi...@milessabin.com
skype: milessabin
g+: http://www.milessabin.com
http://twitter.com/milessabin

[Eugene Burmako]

Could you submit an enhancement request for that please?

[Paul Phillips]

On Fri, Jun 29, 2012 at 11:15 AM, Eugene Burmako <eugene....@epfl.ch> wrote:

Could you submit an enhancement request for that please?

https://issues.scala-lang.org/browse/SI-6010 

[Paul Phillips]

On Fri, Jun 29, 2012 at 11:12 AM, Miles Sabin <mi...@milessabin.com> wrote:

 // Inferred type should be Int => String
final val template = """This is string $(_:Int)."""

The all of your examples boil down to function application and/or composition.

And then all we need is for it to map %d to (_: Int) and we have the presently missing typesafe format, something I thought there's no way we'd miss given the addition of string-interpolation specific syntax.  Maybe it's possible with macros, I don't know, but I don't want it to be possible, I want to be unavoidable.

// Inferred type should also be Int => String

final val template = """This is string %d."

[Rex Kerr]

Hang on, I want triply quoted strings to _be strings no matter what_.  What if I need to produce output with %d in it--as a code generator for another language (or Scala for that matter), or for comments in Matlab, or anything else that might require this (some role-playing games, perhaps?).

We need _some_ sort of identifier that indicates that this is to be interpolated, especially since doing what you suggest will break all sorts of existing code.  I have lots of printing code that I will switch over to using string interpolation eventually, but I don't want it to break catastrophically with dozens of "format is not a member of Int" or "format is not a member of Int => String" messages.

  --Rex

[Paul Phillips]

On Fri, Jun 29, 2012 at 3:37 PM, Rex Kerr <ich...@gmail.com> wrote:

Hang on, I want triply quoted strings to _be strings no matter what_.  What if I need to produce output with %d in it--as a code generator for another language (or Scala for that matter), or for comments in Matlab, or anything else that might require this (some role-playing games, perhaps?).

We need _some_ sort of identifier that indicates that this is to be interpolated, especially since doing what you suggest will break all sorts of existing code.  I have lots of printing code that I will switch over to using string interpolation eventually, but I don't want it to break catastrophically with dozens of "format is not a member of Int" or "format is not a member of Int => String" messages.

The absence of a letter was a typo.  The point was to be able to have type-safe format at all, not to avoid typing 's'.

[Paul Phillips]

To clarify my interest, in case anyone is unaware, here is the signature of the "f" method on StringContext.

    def f(args: Any*): String

What a tragic signature when the common case looks like this:

  "Times were %.3f , %.3f, %.3f, and %s did %s"(ms1, ms2, ms3, who, what)

This is probably my most frequently incurred runtime error, %s where I needed %d, one too many arguments, etc. etc. And aaagh, it's a fixed compile time constant with a fixed, directly derivable method signature.

[Daniel Sobral]

Here's a barebones typesafe printf. I tested it just today, reading up
on macros:

http://scalamacros.org/documentation/gettingstarted.html

The f string context can use the same trick. So can StringOps.format.

[Eugene Burmako]

Iirc, Dominik planned to write (or is writing) a macro-based f-interpolator. Dominik, could you, please, provide a status update?

[Eugene Burmako]

Hmm? Quasiquotes work fine even without untyped macros: https://github.com/scalamacros/kepler/tree/topic/interpolation.

[Paul Phillips]

On Fri, Jun 29, 2012 at 4:04 PM, Daniel Sobral <dcso...@gmail.com> wrote:

Here's a barebones typesafe printf. I tested it just today, reading up
on macros:

That was good to hear. I wrote my first macro.  Okay, I guess eugene wrote it.  Let's say I adapted it.

  https://gist.github.com/3022109

Next stop, generality.

[Jason Zaugg]

On Sat, Jun 30, 2012 at 5:47 AM, Paul Phillips <pa...@improving.org> wrote:
> That was good to hear. I wrote my first macro.  Okay, I guess eugene wrote
> it.  Let's say I adapted it.
>
>   https://gist.github.com/3022109
>
> Next stop, generality.

Would also be nice to avoid reinterpreting the format string at
runtime and boxing the arguments. (Although, as with embellished
asserts, this would mean more bytecode.)

-jason

[Miles Sabin]

On Fri, Jun 29, 2012 at 11:37 PM, Rex Kerr <ich...@gmail.com> wrote:
> Hang on, I want triply quoted strings to _be strings no matter what_.

Yes, that was my typo ... I intended s"""..."""

Cheers,


Miles

--
Miles Sabin
tel: +44 7813 944 528
gtalk: mi...@milessabin.com
skype: milessabin
g+: http://www.milessabin.com
http://twitter.com/milessabin

http://underscoreconsulting.com
http://www.chuusai.com

[Paul Phillips]

On Fri, Jun 29, 2012 at 11:32 PM, Jason Zaugg <jza...@gmail.com> wrote:
> Would also be nice to avoid reinterpreting the format string at
> runtime and boxing the arguments. (Although, as with embellished
> asserts, this would mean more bytecode.)

Short of reimplementing the entire String#format logic -- something
which I think should be avoided, because I'd really rather live with
whatever bugs are in java's version than enjoy discovering a few dozen
in a reimplementation - I don't know how it's possible to avoid boxing
the arguments for the call to format.

[Miles Sabin]

Thanks ... I was about to do that :-)

Cheers,


Miles

--
Miles Sabin
tel: +44 7813 944 528
gtalk: mi...@milessabin.com
skype: milessabin
g+: http://www.milessabin.com
http://twitter.com/milessabin

http://underscoreconsulting.com
http://www.chuusai.com

[Daniel Sobral]

I've tried a small variation on it, using Typed arguments instead of
declaring vals. Sadly, it only reports the first argument with
incorrect type. Anyway, for the curious, here it is:

https://gist.github.com/3026809

[Paul Phillips]

On Sat, Jun 30, 2012 at 9:27 PM, Daniel Sobral <dcso...@gmail.com> wrote:

I've tried a small variation on it, using Typed arguments instead of
declaring vals. Sadly, it only reports the first argument with
incorrect type. Anyway, for the curious, here it is:

https://gist.github.com/3026809

Here's a more direct route, no new trees needed:

   (args, tpes).zipped foreach (c.typeCheck(_, _))

Same issue though, only reports the first error.  I believe it is the error code trying to be helpful, but I find it impenetrable.  Can anyone tell me whether I should expect more than one error to be issued? For comparison, in regular compiler land, if I give it this:

      val x: (Double, String, Int, Float, Double) = (true, true, true, true, true)  

It tells me about all five of them.

[Eugene Burmako]

Relevant pieces of code are:

* post-expand typecheck (Daniel's use case) [1]

* in-flight typecheck (Paul's use case) [2]

In the first case, I'm just doing typer.typed with some additional error reporting after the macro returns. Hubert, can this cause the described problems?

In the second case, there's an issue with silent typechecking as it returns only the head of the error buffer. Hubert, would it be okay to retain all the errors from the buffer? Maybe there were difficulties with that, so you didn't implement this functionality?

[1] https://github.com/scalamacros/kepler/blob/0ce5c06c85a956f620b77f05803de29d39417409/src/compiler/scala/tools/nsc/typechecker/Macros.scala#L977

[2] https://github.com/scalamacros/kepler/blob/0ce5c06c85a956f620b77f05803de29d39417409/src/compiler/scala/tools/nsc/typechecker/Typers.scala#L742

[Paul Phillips]

On Sat, Jun 30, 2012 at 11:17 PM, Eugene Burmako <eugene....@epfl.ch> wrote:

Maybe there were difficulties with that, so you didn't implement this functionality?

I imagine it's because we've been plagued with spurious errors the last half year or more.  But dropping all the errors after the first on the floor is a less than ideal remedy.

Also, you said "there's an issue with silent typechecking as it returns only the head of the error buffer." but I wasn't typechecking in silent mode, at least assuming that's what the "silent" parameter to typecheck communicates.  If it is actually silent despite silent=false, how might I discover that?

[Eugene Burmako]

c.typecheck always uses typer.silent in order not to contaminate the original buffer. Silent version is different from a non-silent one only in returning EmptyTree instead of crashing.

[Paul Phillips]

On Sun, Jul 1, 2012 at 12:00 AM, Eugene Burmako <eugene....@epfl.ch> wrote:

c.typecheck always uses typer.silent in order not to contaminate the original buffer. Silent version is different from a non-silent one only in returning EmptyTree instead of crashing.

When should one prefer the crashing variation? Maybe the parameter which allows for it should be called "crash" instead of "silent".

[Eugene Burmako]

Crashing variation returns information about the errors if they occur. Silent version silences all the errors and simply returns an EmptyTree. 

About naming. You present a valid point - indeed, non-silent typecheck in buffer mode and non-silent typecheck in macros behave differently. Let's discuss this on a Scala meeting.

[Gruntz Dominik]

I have a first version available, not for printf but for the f-interpolator. The reason we worked on the latter is, that here we have the guarantee, that the string which is passed is always a literal. For printf, the format string could in principle also be an expression. We also do not support the position modifier, as they make no sense for string interpolation.

 

The interpolation string is checked and used to define the expected types (one could also implement it the other way round and flag errors on the format string). At the end a "string format (args)" expression is returned, i.e. the format string is reinterpreted at runtime.

 

The implementation follows the example of Eugene/Paul. However, I have tried to support several types for some arguments, e.g. for date/time arguments.

I am currently working on moving this stuff into the library.

 

Examples:

 

scala> f"Times were ${ms1}%.3f, $ms2%s, $ms3%d, and %%%% and $who%s did $what%H"

res3: String = Times were 0.333, false, 54, and %%%% and bippy did B09FD41C

// note, that according to SIP11, a % is converted into %% before invocation of the f-method.

 

import java.util.Calendar

import java.util.Locale

val c = Calendar.getInstance(Locale.US)

c.set(2012, Calendar.JULY, 3)

 

scala> f"${c}%TD"

res7: String = 07/03/12

 

scala> f"${c.getTime}%TD"

res8: String = 07/03/12

 

scala> f"${c.getTime.getTime}%TD"

res9: String = 07/03/12

 

scala> f"${new java.util.Date {} }%TD"

res15: String = 07/03/12

 

Dominik

[Jason Zaugg]

On Tue, Jul 3, 2012 at 10:44 AM, Gruntz Dominik <dominik...@fhnw.ch> wrote:
> The implementation follows the example of Eugene/Paul. However, I have tried
> to support several types for some arguments, e.g. for date/time arguments.

It occurs to me that you could probably even support, for example,
JodaTime in that macro without introducing a hard dependency on that
library. Of course it would be nicer to provide an extensible approach
(type class based?), but that's a bigger project.

-jason

[Daniel Sobral]

I think the SIP-11 comment you refer to is an artifact of a previous
revision of it. The original way % formats on f-interpolator were
handled was rather convoluted (imho), and was changed afterwards.
There's a simple question to ask here: is there any point in doubling
%? I don't have time to go over it right now, but my vague memories
are that there was such a reason in the original spec, but not in the
revised spec.

[Dominik Gruntz]

% signs in the interpolated string are simply taken verbatim, so there is no need to double them.
I simply copied the example of Paul from his printf-macro implementation. With f-string-interpolation, this example could have been written as

scala> f"Times were ${ms1}%.3f, $ms2%s, $ms3%d, and %% and $who%s did $what%H"

res3: String = Times were 0.333, false, 54, and %% and bippy did B09FD41C

The spec was not changed on how % are handled, only the mapping to the StringContext was changed (originally, the %-sign was passed to a synthetic %s-part, now they are inserted as %% in the string  which is passed to format).

Dominik

[Dominik Gruntz]

It occurs to me that you could probably even support, for example,
JodaTime in that macro without introducing a hard dependency on that
library. Of course it would be nicer to provide an extensible approach
(type class based?), but that's a bigger project.

-jason

The problem is that the Java Formatter (which is used in the implementation) only supports a finite set of types, for date/time parameters these are Long, Calendar and Date. The macro has to convert the actual argument to one of these types. What the macro already supports are implicits, so you can write:

scala> import org.joda.time._
import org.joda.time._

scala> implicit val toCal = (dt: DateTime) => dt.toGregorianCalendar
toCal: org.joda.time.DateTime => java.util.GregorianCalendar = <function1>

scala> f"${new DateTime}%TD"
res0: String = 07/04/12

- Dominik

 

[Jason Zaugg]

That's pretty useful by itself.

My observation was that you could probably move the call to
toGregorianCalendar into the formatting macro itself, rather than
requiring the user to define it, and do so in a way that would be
invisible for those not using Joda Time.

-jason

[Dominik Gruntz]

I could hard-code it (and invoke toGregorianCalendar using reflection), but this way only a finite set of hard-coded types could be supported.
A type class based approach would need to extend the signature of the string-interpolation StringContext(....).xxx methods, right?
Dominik

[Jason Zaugg]

Yeah, I'm not sure it's the right thing to do, it would just be very convenient.

If you can infer an implicit view provided by the macro body, you
could also use a type class based conversion without changing the
signature. Just run the implicit search for Datish[T] rather than T =>
Date. It would be a touch safer, as the caller might not want to have
the implicit view in scope.

-jason

[Daniel Sobral]

There was a change in the spec regarding what it said about % -- it's
even in the history. At any rate, yes, all of the above is true.