Why doesn't @sprintf evaluate its format string?

[Ronald L. Rivest]

I'm new to Julia, and got burned (aka wasted a fair amount of time)

trying to sort out why @sprintf didn't work as I expected.  

julia> @sprintf("%2d",29)

"29"

julia> fmt = "%2d"

"%2d"

julia> @sprintf(fmt,29)

ERROR: @sprintf: first argument must be a format string

julia> @sprintf("%"*"2d",29)

ERROR: @sprintf: first argument must be a format string

I would expect that @sprintf would allow an arbitrary string expression as its

format string.  It obviously doesn't...

There are many good reasons why one might want a format string expression instead

of just a constant format string.  For example, the same format may be needed in

several places in the code, or you may want to compute the format string based on

certain item widths or other alignment needs.

At a minimum, this should (please!) be noted in the documentation.  

Better would be to have the extended functionality...

(Or maybe it exists already -- have I missed something?)

Thanks!

Cheers,

Ron Rivest

[Ronald L. Rivest]

Another reason for (perhaps) wanting an expression as a format

string:

Julia doesn't have a line continuation mechanism.

How does one split a long format string into parts without having

a string expression?  The following doesn't work:

    @sprintf("This is the first part of a rather long format string, which I" *

                   "would like to extend onto %d (or maybe more) lines.", 5)

Thanks.

Cheers,

Ron

[John Myles White]

I think both of the problems you're hitting are the same core problem: @sprintf is a macro, so its behavior is surprising when you think in terms of the run-time behaviors that  control function evaluation. In particular, your line continuation problem wouldn't be fixed by having line continuations. No matter what you're going to end up passing the wrong thing (a call to *) to a macro that expects a literal string.

I believe Dahua tried to work on writing a more function oriented package for doing formatting: https://github.com/lindahua/Formatting.jl

There might be some other solutions floating around as well.

 -- John

[Mike Innes]

To also clarify on why @sprintf behaves this way, the reason is that it compiles down to specialised code for printing e.g. a float, followed by a double, etc., whatever you specify. If you use a variable like `fmt` the actual value is only available long after the code has already been compiled.

This could certainly be supported this by falling back to interpreting the string at runtime, if necessary – the code just hasn't been written yet.

Another workaround is to call eval(:@sprintf($fmt, 29)). This will probably be fairly slow, though. I have another solution which will help with repeated format strings, but will need to send a PR to base to get that working.

[Tony Fong]

Just to add to John's comment on Formatting.jl: yes use it. There's this NumFormat.jl (of yours truly) but I've since proposed to merge its entire code into Formatting.j, subject to @lindahua's blessing. So we should have a wealth of options to format numbers by runtime arguments.

Tony

[Stefan Karpinski]

I gave this answer on StackOverflow, which I think explains it decently. However, this has caused enough people to be confused or frustrated that it should probably be revisited. I'm not sure what the right solution is. Note that we can't just call C's printf function because it's not type safe; Julia's printf is safe – it does type conversion for you.

[Jeff Waller]

I think this could be done by instead of expanding into specific (optimized) code dedicated to each argument, it instead would expand into a tree of if statements connected to a bunch of Expr(:call,:typeof,<more args>) and ? : (whatever the parse tree is for that).  Essentially a function call turned inline.  Is there support for that (inlining functions) already?

[Stefan Karpinski]

If Julia didn't do inlining, all of your code would take years to run.

[Stefan Karpinski]

I don't see how inlining solves this problem though – if format strings are run-time, you can't inline anything since you don't know what to inline until it runs, at which point it's too late.

[Jeff Waller]

You could not inline any one thing, but you could inline everything.  It would be slower, but instead of the type specific code expanded for each %<x> in the static string, you'd expand to all possible types -- one for each of the different possible x,  a cascade of if else that compare the % and wrap that inside a while statement.  

[Stefan Karpinski]

Unfortunately the number of types or arguments one can encounter is unbounded. Are you talking about the format specifiers or the arguments?

[Jeff Waller]

Unfortunately the number of types or arguments one can encounter is unbounded. Are you talking about the format specifiers or the arguments?

Yea,  the format specifiers, poor choice of words, my bad.  

[Mike Innes]

#9423 should help with the repetition of format strings issue. It occurs to me now that you can always just write a function wrapper for `@sprintf` to solve that issue but this might still be useful.

[ele...@gmail.com]

As Stefan said above, the problem with traditional (s)printf functions is type safety.

On Thursday, December 4, 2014 4:53:17 AM UTC+10, Mike Innes wrote:

#9423 should help with the repetition of format strings issue. It occurs to me now that you can always just write a function wrapper for `@sprintf` to solve that issue but this might still be useful.

[...]

[Mike Innes]

I'm not sure I understand – C's sprintf certainly has problems with type safety, but a function wrapping Julia's @sprintf can't not be strongly typed. No?

[Ivar Nesje]

A function wrapping Julia @sprintf will be typesafe, but will have terrible performance. One could cache the generated functions though, so that the format string will only be used as a lookup in a hash table that points to the specialized (typesafe) function for that format string.

[Mike Innes]

To clarify, I meant something like

function printnums(x,y,z)

  @sprintf("%.2f %.2f %.2f", x, y, z)

end

Which will both be type safe and have good performance. It doesn't solve all the issues, but as I said, it does solve the specific issue of repetition of format strings.

[Ivar Nesje]

But that has a constant format string, and is a great example on how to use @sprintf. The question here is about non-constant format strings (as far as I can tell).

[Tony Fong]

That's exactly the approach in the c-style portion of Formatting.jl.

[Stefan Karpinski]

One way of doing this would be to generate named functions that do a particular formatting, e.g. when someone does printf("%5.2d %-s %73f\n", ...) do

@eval function $(symbol("__printf__%5.2d %-s %73f\n"))(arg1, arg2, arg3)

    # generate code here

end

Then call that function. That uses the module as a lookup table. Unfortunately, there's still overhead to lookup the function – using a hash table might be more efficient anyway. Honestly, the whole printf way of doing formatting is kind of a horrid hack and should probably be replaced with something saner.

[ele...@gmail.com]

On Friday, December 5, 2014 12:39:13 AM UTC+10, Stefan Karpinski wrote:

One way of doing this would be to generate named functions that do a particular formatting, e.g. when someone does printf("%5.2d %-s %73f\n", ...) do

@eval function $(symbol("__printf__%5.2d %-s %73f\n"))(arg1, arg2, arg3)

    # generate code here

end

Then call that function. That uses the module as a lookup table. Unfortunately, there's still overhead to lookup the function – using a hash table might be more efficient anyway. Honestly, the whole printf way of doing formatting is kind of a horrid hack and should probably be replaced with something saner.

And extensible to user types.  But probably not C++'s << system.

Cheers

Lex

[Stefan Karpinski]

On Thu, Dec 4, 2014 at 9:45 AM, <ele...@gmail.com> wrote:

But probably not C++'s << system.

Haha, yeah, no, we're not going to overload the shift operators for I/O, and choose an API that guarantees that formatted I/O is super slow.