@sprintf with a format string

[Ferran Mazzanti]

Dear all,

I could use some help here, because I can't believe I'm not able to easily print formatted numbers under Julia in a easy way. What I try to do is to write a function that, given a vector, prints all its components with a user-defined format. I was trying something of the form

function Print_Vec(aux_VEC,form_VEC)
    form_VEC :: ASCIIString
    str_VEC  = "%16.8f"
    for elem_VEC in aux_VEC
        str_VEC += @sprintf(form_VEC,elem_VEC)
    end
    return str_VEC
end

However, that doesn't work because it looks like the first argument in @sprintf must be a explicit string, and not a variable.
Is there anything I can do with that?

Thanks a lot for your help.

[Michael Hatherly]

https://github.com/JuliaLang/Formatting.jl might help.

— Mike

​

[Ferran Mazzanti]

Thanks, that seems to work.
Still it amazes me how Julia, being a language made for numerical calculations, does not natively support a simple mechanism to print/write
large bunches of numbers. I've been in the numerical world for 20+ years and I know printing lots of numbers is something you get on a daily
basis. I know now the formatting package can help on that (thanks :), what I do not like is the idea of having to install every time a new package
to get added functionality. I understand there are things that have to go to external packages because of its limited or specialized use, but
come on... printing number os definitely not one of those.
Just my 2cents :)

[Tomas Lycken]

Julia is a young language.

This comes with a lot of benefits - for example, it's possible to do things *right* from the start. We're currently in a phase where there is a multitude of possible solutions to every problem that arises, and we honestly don't know which solution proves to be the best one. Testing various solutions out in packages, outside of the base distribution, where they can compete for popularity, and keep building on top of each-other until we reach something which we believe not only is adequate, but *really* hits the sweet spot. In other words: sure, formatting numeric output is fundamental, but it's a problem with a large solution space, and we don't want to lock in on one path yet.

But Julia's young age also means that there are lots of problems that aren't really solved yet, or that have solutions proposed but not implemented, etc. For the future, there are plans to include some packages (which ones have not been decided, or AFAIK really even discussed) in a sort-of "base distribution", so they will be installed upon installation of Julia, and available with a single `using` statement. But the infrastructure for this is not in place yet, and even if it were, there would still be lots of reasons not to make a final decision on which packages make the cut until we approach a 1.0 release of the language.

So yeah, it might seem silly to have to install a package just to get fine-grained control of numeric output formatting, but, at least for now, that's part of what one could call Julia's "Package Deal" ;)

// T

[Tomas Lycken]

By the way, you might also note that the Formatting.jl package is written and maintained by people who are all active contributors to the main language repo, and very active members of the community. Even if it's "external" in terms of how you install it, it's by no means external "in spirit".

// T

[Daniel Carrera]

I might be wrong, but to me Formatting.jl looks next to useless. The "sprintf" functions it provides only accept one parameter. The main function provided is `sprintf1()`, but even the very clumsy `generate_formatter()` function fails at the most basic tasks:

julia> fmtrfunc = generate_formatter( "%10.3f   %6d  %3d" )

ERROR: Only one AND undecorated format string is allowed

 in generate_formatter at /home/daniel/.julia/v0.3/Formatting/src/cformat.jl:23

I really don't understand why sprintf() is such a big deal. If C can have one, why can't Julia? I understand the argument that you might want to rewrite the implementation later. Fine. Just call the function "__temp_sprintf()" and put it in a package called "FunctionThatWillGoAwayLater". I don't care. I just want to be able to print formatted strings without a ton of needless hassle.

[Tomas Lycken]

> If C can have one, why can't Julia?

The hard truth, and nothing but it: If Julia is missing a feature, or has one that works in a way that's not the way you want, it's because no-one has wanted what you want enough to actually implement it.

That's it. There's no "it can't be done" - Julia is Turing complete, so "everything" can be done. There's no "it's not allowed" - no-one is stopping you from writing your own package, with your own implementation that does what you want the way you want it done, putting it in the package repository and seeing it take off. Chances are that if you write something that it turns out many others want as well, it will be included in the default package distribution in the future (or even in base Julia). Or it might not take off, but at least be there for your own enjoyment.

But the entire language, with all tooling, packages, etc - everything that is Julia - is very much a community-built product. Everything that is there, is there because someone at some point said hey, this is so important to me that I'm going to put it at the very top of my priority list. Creating a sprintf function that fulfills your needs might not have made it there for anyone yet. If it's that important to you, maybe that's your next contribution?

// T

[Daniel Carrera]

Coding sprintf() is beyond my skill, I tried. My contributions to Julia have to lie elsewhere (recently I've been putting a lot of time making mockups of a Julia IDE, and helping a new user port his Matlab code). I think it is rude to say "if you don't like it, fix it yourself"; especially after you wrote a post claiming that the non-implementation of sprintf() was an intentional omission because developers weren't sure how to do it right. I showed, correctly, that that argument is nonsense; so you spinned around and changed it to "if you don't like it, fix it yourself". That answer is only a way to reject valid complaints, and it feels like a thinly veiled "fuck you". Imagine if that's how people responded in bug reports. This is not the type of answer that Julia developers normally give, and it is not the type of answer that I expect in Julia; especially after giving a very different answer to the same question. I do try to contribute to Julia, and it is rude and unreasonable to hold me personally responsible for fixing everything that does not work in Julia. A healthy community needs room for receiving legitimate complaints.

Daniel.

[Tom Breloff]

Chill guys.

Anyways... I started to tackle this problem, but didn't get any (much needed) comments on whether I was on the right track.  Here's some sample preliminary usage of what I was working on.  Please let me know if you think it's worth continuing, and what you'd ideally like to see in a formatter.

https://github.com/tbreloff/Formatting.jl/blob/tom-fmt/test/fmt.jl

[Kristoffer Carlsson]

http://stackoverflow.com/questions/19783030/in-julia-why-is-printf-a-macro-instead-of-a-function

[Tomas Lycken]

If anything I wrote came off as a "thinly veiled 'fuck you'", I am sincerely sorry; that's nowhere near what I meant to say.

My two answers were directed at very different pieces of critique: first, I reacted to the statement "what I do not like is the idea of having to install every time a new package to get added functionality", explaining why this is actually a Good Thing for Julia at the moment, and was really quite unrelated to formatting.

Then, you replied that "Formatting.jl looks next to useless" and "I really don't understand why sprintf is such a big deal. If C can have one, why can't Julia?" I interpreted that, apparently wrongfully, as implying not only that you disliked the currently available implementations, but also that you thought it couldn't be so hard to do better, so my second answer had a different solution to the problem. (Note also that the two posts aren't directed at the same person.)

Re-reading the thread I think both our tones are harsher than necessary, and I apologize for my part in that.

// T

On Tuesday, September 22, 2015 at 4:20:41 PM UTC+2, Tom Breloff wrote:

[Michael Hatherly]

Anyways… I started to tackle this problem

As an interesting/entertaining hack you can “lift” the formatting string out of @printf with a staged function:

immutable Formatter{format} end

fmt(str) = Formatter{symbol(str)}()

@generated function printf{format}(io :: IO, :: Formatter{format}, args...)
    xs = [:(args[$i]::$(args[i])) for i = 1:length(args)]
    quote
        $(Expr(:meta, :inline))
        @inbounds $(Expr(:macrocall, symbol("@printf"), :io, string(format), xs...))
        nothing
    end
end

julia> printf(STDOUT, fmt("%f"), pi)

Though, of course, the normal warnings with relying heavily on @generated should be noted.
Performance and resulting code seems to be roughly the same as using @printf with
hard-coded formatting strings, and faster that what Formatting.jl is able to get.

and what you’d ideally like to see in a formatter.

http://www.gigamonkeys.com/book/a-few-format-recipes.html… but maybe that’s just me :)

— Mike

​

[Stefan Karpinski]

Responding with a solicitation for help in an open source project is not a rhetorical ploy – it's a legitimate request for contribution. I wrote the printf code in Julia some time ago and it clearly could use revisiting – the API is not working out for people. However, that is a large amount of design work and a tricky, performance-sensitive problem in generic programming (Julia's has to support printing an open-ended set of types, not just the limited set C has). Since I currently have a few other things to worry about, one of the most effective ways to see that work done is to do it yourself. Barring that, you just have to wait until someone does this work.

A lecture on the importance of printing numbers is also not helpful and pretty irritating. Yes, printing things is important, especially numbers. Everyone is well aware, myself included – that's why I spent a month implementing our generic printf in the first place. It will get fixed.

In the mean time, several viable solutions have been provided (repeatedly), including...

1. Use eval. This lets you generate formatter functions for specific runtime format strings:

julia> const fmt = "%05d %-5s %5.2f\n"

"%05d %-5s %5.2f\n"

julia> @eval formatter(i, s, x) = @printf($fmt, i, s, x)

formatter (generic function with 1 method)

julia> formatter(123, "flux", pi)

00123 flux   3.14

If you don't need to change the format in response to data, this is a fine way to do things.

2. Use Formatting.jl. If you don't care for the interface, help improve it. Help could take the form of pull requests, of just polite issues opened to suggest how the interface could be enhanced. Calling it "next to useless" and dismissing out of hand is rude and not particularly constructive.

3. Use C's printf. Since this seems to be what you want and you can easily call C from Julia, what's stopping you?

julia> ccall(:printf, Cint,

             (Cstring, Cint, Cstring, Cdouble),

             "%05d %-5s %5.2f\n", 123, "flux", pi)

00123 flux   3.14

18

It's not the most convenient interface in the world, but a little wrapping would solve that. (It may also be incorrect since printf is a varargs function – I was a little surprised that it worked – but using vprintf would address that.)

[Daniel Carrera]

On 22 September 2015 at 16:38, Tomas Lycken <tomas....@gmail.com> wrote:

If anything I wrote came off as a "thinly veiled 'fuck you'", I am sincerely sorry; that's nowhere near what I meant to say.

Thank you. I guess I misinterpreted your comments and reacted more harshly than I should.

Then, you replied that "Formatting.jl looks next to useless" and "I really don't understand why sprintf is such a big deal. If C can have one, why can't Julia?" I interpreted that, apparently wrongfully, as implying not only that you disliked the currently available implementations, but also that you thought it couldn't be so hard to do better, so my second answer had a different solution to the problem. (Note also that the two posts aren't directed at the same person.)

Well, I apologize if I made it sound that printf() was easy to write. You have to admit that most languages provide printf() out of the box, so it does seem odd that Julia does not.

 

Re-reading the thread I think both our tones are harsher than necessary, and I apologize for my part in that.

I apologize as well. Both of my emails were harsher than they should have been.

Daniel.

[LarryD]

Something I miss from Fortran is the very convenient default "print *, ..... "  It handled almost 100% of my needs while working on a program and was easily replaced by real formatting when the time came. Is there any chance that Julia could get something like this?

Thanks

[Tom Breloff]

Larry: can you provide details on exactly what you like about Fortran's print statement?  Did it provide good defaults?  Was it easy to customize?

[Daniel Carrera]

On 22 September 2015 at 17:42, Stefan Karpinski <ste...@karpinski.org> wrote:

Responding with a solicitation for help in an open source project is not a rhetorical ploy – it's a legitimate request for contribution.

It can be either. It depends on context and intent. I have seen that comment used both ways. I once saw a seminar that suggested that as a technique to get rid of people you don't want to deal with (e.g. because the person is disruptive in some way).

A lecture on the importance of printing numbers is also not helpful and pretty irritating. Yes, printing things is important, especially numbers. Everyone is well aware, myself included – that's why I spent a month implementing our generic printf in the first place. It will get fixed.

I don't know if this is a response to me or someone else. I never gave a lecture on the importance of printing numbers. At least, I never intended to and I don't know what I said that would come across like that.

Anyway, I appreciate that you are busy with probably more important things, and I thank you for the work you have already done. My email was harsher than it should have been, but I was replying comments from Tomas. I was not trying to file a complaint with the Julia devs. I know that you know about the @sprintf() issue, and I was not trying to remind you of it. I took issue with Tomas saying that printf() was not implemented because the devs were trying to decide what the right implementation should be. I don't think that this is, or should be, the right reason to not implement printf(). I suspected that the real reason was, as you said, that there is more work on your planet than just printf().

 

In the mean time, several viable solutions have been provided (repeatedly), including...

I was not aware of the eval() option, but I was aware of the others. In particular, I spent a while experimenting with C's printf() function, but I was not able to make a generic wrapper around it. This means that, for me, a direct call to C's printf is no more useful than the existing @sprintf macro. I also compared Formatter.jl with @sprintf and I decided that @sprintf served me better.

 

2. Use Formatting.jl. If you don't care for the interface, help improve it. Help could take the form of pull requests, of just polite issues opened to suggest how the interface could be enhanced. Calling it "next to useless" and dismissing out of hand is rude and not particularly constructive.

I was rude and I apologize. I don't think that Formatting.jl can be fixed. I think that there is probably a good reason it only provides sprintf1(). I don't think it would be news to the developers that a sprintf() that takes more than one parameter is desirable. If they haven't done it, it is because they can't.

 

3. Use C's printf. Since this seems to be what you want and you can easily call C from Julia, what's stopping you?

julia> ccall(:printf, Cint,

             (Cstring, Cint, Cstring, Cdouble),

             "%05d %-5s %5.2f\n", 123, "flux", pi)

00123 flux   3.14

18

It's not the most convenient interface in the world, but a little wrapping would solve that. (It may also be incorrect since printf is a varargs function – I was a little surprised that it worked – but using vprintf would address that.)

Yes, this is the approach I tried to work on. I was trying to make a wrapper. But I was unable to make a generic one. Similar to the @eval option, you end up having to make a wrapper for the specific format you want to print. This means that, in the end, I am better off just writing very long lines with @sprintf.

Anyway, I again apologize for my harsh tone, I hope I have clarified some of my thoughts, I hope I have re-assured you that I did look at the alternatives, and I thank you for your work in Julia.

Daniel.

[Stefan Karpinski]

I think that before any further discussion takes place of how easy or hard implementing a high-performance printf is, anyone who'd like to comment should spend some time perusing GNU libc's vfprintf implementation. This code is neither easy nor trivial – it's batsh*t crazy. And we want to match its performance yet be much more flexible and generic. The current printf implementation does just that, while being somewhat less insane GNU's printf code. If someone has bright ideas for how to also allow runtime format specification without sacrificing performance or generality, I'm all ears. I have some thoughts, but they're just that – thoughts. One option is to change the design and avoid printf-style formatting altogether. But then I'm sure I'll never hear the end of it with people kvetching about how we don't have printf.

[lawrence dworsky]

Hi Tom

What I like about it is that you can just use print *, dumbly and it always provides useful, albeit not beautiful, results. When I'm writing a program, I use print statements very liberally to observe what's going on - I find this more convenient than an in-line debugger. 

As the last line in my program below shows, it's easy to switch to formatted output when you want to. The formatting capability is pretty thorough, I'm just showing a simple example.

This Fortran program doesn't do anything, it just illustrates what the print statement produces:

real x, y

integer i, j

complex z

character*6  name

x = 2.6

y = -4.

i = 36

j = -40

z = cmplx(17., 19.)

name = 'Larry'

print *, x, y, i, j, z

print *, 'x = ', x, ' and j = ', j

print *, 'Hello, ', name, j

print '(2f8.3, i5)', x, y, j

stop

end

The output is:

        2.60000             -4.00000                   36             -40  (17.0000, 19.0000)

x =         2.60000       and j =                -40

Hello, Larry                 -40

  2.600   -4.000  -40

Is this what you are looking for?

Larry

[Tom Breloff]

Thanks Larry, that's helpful.  Just for discussions sake, here's a quick macro that calls my proposed `fmt` method under the hood, and does something similar to what you showed.  What do you think about this style (and what would you do differently)?

 

using Formatting

macro fmt(args...)
 expr = Expr(:block)
 expr.args = [:(print(fmt($(esc(arg))), "\t\t")) for arg in args]
 push!(expr.args, :(println()))
 expr
end

And then an example usage:

In:

x = 1010101
y = 555555.555555555
fmt_default!(width=15)

@fmt x y

fmt_default!(Int, :commas)
fmt_default!(Float64, prec=2)

@fmt x y

Out:

        1010101  555555.555556

      1,010,101      555555.56

[Daniel Carrera]

On 22 September 2015 at 20:40, Stefan Karpinski <ste...@karpinski.org> wrote:

I think that before any further discussion takes place of how easy or hard implementing a high-performance printf is, anyone who'd like to comment should spend some time perusing GNU libc's vfprintf implementation. This code is neither easy nor trivial – it's batsh*t crazy.

That is insane... 2388 lines, half of it macros, and I have no idea how it works.

 

And we want to match its performance yet be much more flexible and generic. The current printf implementation does just that, while being somewhat less insane GNU's printf code. If someone has bright ideas for how to also allow runtime format specification without sacrificing performance or generality, I'm all ears.

This might be a stupid question, but what's the harm in sacrificing performance as long as we keep the current @sprintf for scenarios that call for performance? I don't always need printf() to be fast.

 

I have some thoughts, but they're just that – thoughts. One option is to change the design and avoid printf-style formatting altogether. But then I'm sure I'll never hear the end of it with people kvetching about how we don't have printf.

Probably. Everyone is used to printf and they are comfortable with it.

Daniel.

[Stefan Karpinski]

Possible, but I don't relish the thought of forever explaining to people that they need to use printf with or without the @ depending on if they want it to be fast or flexible. If you really don't care about speed, you can just do this right now:

printf(fmt::AbstractString, args...) = @eval @printf($(bytestring(fmt)), $(args...))

But actually don't do that because it's so horrifically slow and inefficient I just can't.

[Daniel Carrera]

Hmm... I know it's horrible, but I just added that to my juliarc file :-)

This function is 100x slower than the macro, at about 100 lines in 0.5s. I know that's horribly slow for traditional printf() but it's fast enough for terminal output.

[Stefan Karpinski]

It also allocates 1MB just to print three numbers. Yikes.

[lawrence dworsky]

I'm afraid my beginner status with Julia is showing:

I ran Pkg.add("Formatting"), and then   using Formatting   came back with a whole bunch of warnings, most about  Union(args...) being depricated, use Union(args....) instead. 

When all is said and done,   fmt_default!  gives me a  UndefVarError.

Help!

[Tom Breloff]

Sorry I wasn't expecting you to run it... just comment.  You'll have to do:

Pkg.rm("Formatting")

Pkg.clone("https://github.com/tbreloff/Formatting.jl.git")

Pkg.checkout("Formatting", "tom-fmt")

Let me know if that works.

[ele...@gmail.com]

Stefan, don't beat yourself up so much :)

Sometimes the choice of fast or flexible just has to be made.  Maybe call them @fast_printf() and printf() (after appropriate deprecation time of course).  So long as the flexible printf is "adequate" performance (as Rolls Royce used to say).

Then if your ideas for flexible printf work out they can improve it over time, as convenient.

And of course other non-printf APIs can be added (but please don't (re)invent C++ << :).

Cheers

Lex

[Daniel Carrera]

Really? What does it *DO* that needs 1MB? I admit I don't really know how it works. I have tried to learn macros, but I still think they are basically black magic.

[Luke Stagner]

Would it be possible to rewrite @printf as a generated function instead of a macro. That way the calling syntax would be more familiar.

[Tom Breloff]

I think not Luke.  Generated functions work on the types of the arguments.  If someone wants to format based on an input string, then you either need a hardcoded value which can go into a macro, or a dynamic value that would go in a normal function.

If all you want to do is print out some arbitrary list of objects with pre-defined format, then a normal function should be plenty fast (and if it's a fixed format string, the current macro is the way to go).

[Bob Nnamtrop]

But Michael Hatherly's showed code above that uses a generated function to solve this. It seems to work pretty well in the short while I tried it in the REPL. Granted I didn't time it or do anything complicated. It works on Stefan's example above with no problem. The only difference is that one must type fmt("format") instead of "format". Possibly that could be shortened to fmt"format" using a str_macro (although that had some effects when I tried it).

What am I missing?

Bob

[Tim Holy]

On Tuesday, September 22, 2015 05:21:10 PM Luke Stagner wrote:
> Would it be possible to rewrite @printf as a generated function instead of
> a macro. That way the calling syntax would be more familiar.

That's a good suggestion.

At the risk of encouraging emacs users to "fix" the syntax with ctrl-T, I'd
propose the following (apparently complete?) solution:


immutable FormatString{S} end

FormatString(str::AbstractString) = FormatString{symbol(str)}

macro f_str(arg)
:(FormatString{symbol($arg)})
end

@generated function Base.print{format}(::Type{FormatString{format}}, args...)
meta = Expr(:meta, :inline)
fmt = string(format)
allargs = [:(args[$d]) for d = 1:length(args)]
quote
@printf($fmt, $(allargs...))
end
end



Demo:
julia> print(f"%.3f", pi)
3.142
julia> function foo(strs)
for str in strs
print(FormatString(str), pi)
end
end
foo (generic function with 1 method)

julia> strs = ("%.3f\n", "%.5f\n")
("%.3f\n","%.5f\n")

julia> foo(strs)
3.142
3.14159

julia> @time 1 # just to warm up @time
0.000004 seconds (148 allocations: 10.151 KB)
1

julia> @time foo(strs)
3.142
3.14159
0.000106 seconds (18 allocations: 704 bytes)


Nice that we get to re-use the macro that Stefan worked so hard on!

Best,
--Tim

>
> On Tuesday, September 22, 2015 at 1:07:23 PM UTC-7, Stefan Karpinski wrote:
> > Possible, but I don't relish the thought of forever explaining to people
> > that they need to use printf with or without the @ depending on if they
> > want it to be fast or flexible. If you really don't care about speed, you
> > can just do this right now:
> >
> > printf(fmt::AbstractString, args...) = @eval @printf($(bytestring(fmt)),
> > $(args...))
> >
> >
> > But actually don't do that because it's so horrifically slow and
> > inefficient I just can't.
> >
> > On Tue, Sep 22, 2015 at 3:57 PM, Daniel Carrera <dcar...@gmail.com
> >
> > <javascript:>> wrote:
> >> On 22 September 2015 at 20:40, Stefan Karpinski <ste...@karpinski.org
> >>
> >> <javascript:>> wrote:
> >>> I think that before any further discussion takes place of how easy or
> >>> hard implementing a high-performance printf is, anyone who'd like to
> >>> comment should spend some time perusing GNU libc's vfprintf
> >>> implementation

> >>> <http://repo.or.cz/w/glibc.git/blob/ec999b8e5ede67f42759657beb8c5fef87c8
> >>> cc63:/stdio-common/vfprintf.c>. This code is neither easy nor trivial –

> >>> it's batsh*t crazy.
> >>
> >> That is insane... 2388 lines, half of it macros, and I have no idea how
> >> it works.
> >>
> >>> And we want to match its performance yet be much more flexible and
> >>> generic. The current printf implementation does just that, while being
> >>> somewhat less insane GNU's printf code. If someone has bright ideas for
> >>> how

> >>> to *also* allow runtime format specification without sacrificing

[Eric Forgy]

If this were Facebook, I would "Like" this :)

[ele...@gmail.com]

Tim,

How well does it work if, instead of pi, the thing being printed is some abstract type so its concrete type is only known at runtime?

Cheers

Lex

[Tim Holy]

It Just Works.

--Tim

[Daniel Carrera]

This implementation requires Julia 0.4. I use Julia in production work, and 0.4 is in RC2. Is it safe to migrate or should I just wait?

Cheers,

Daniel.

[Tim Holy]

If you're willing to make updates to your own code---and possibly contribute
fixes to packages you depend on---I highly recommend migrating. In my hands,
it's more stable & predictable than julia 0.3 despite 0.3's long history.

--Tim

[Tomas Lycken]

That depends on what you do in your “production work” ;)

0.4 RC2 is good enough that it would become 0.4 if no serious bugs or regressions are found. There’s currently one issue tagged with milestone 0.4.0, and another one tagged backport pending 0.4, and I don’t know if there are other places you should look to find things that need to be fixed before the RC cycle is considered finished, but I guess there’s at least going to be an RC3 too.

It’s very possible that, for the areas of 0.4 that you would use, nothing will change between now and the final release. It’s also very possible that 0.4 is unusable to you in its current state - it depends entirely on your specific needs. But I would encourage you to try it out - there’s nothing that says you have to uninstall 0.3.11 yet :)

// T

PS:

I have tried to learn macros, but I still think they are basically black magic.

No matter what Julia concept we speak of, I could re-phrase that and apply it to myself, as

I think I have a pretty good understanding of [Concept X], but in the hands of Tim Holy it is still black magic…

[Tom Breloff]

Tim:  I love that you just whipped up a generated function for this (one great thing about Julia is how many different ways you can tackle a problem), however do you see this type of solution as being better than Formatting's sprintf1?  I would be worried about the sheer number of compiled versions (one for each format string, plus the macro expansion), but maybe that's not as much overhead as I think it is?

Also I still think there's room for something like my `fmt` proposal, which would allow for global defaults for formatting using the type hierarchy. The proposal is not meant to replace printf, but to supplement it.  Sometimes you want to control how individual types are printed globally, without using a format string repeatedly.  In my proposal, you could set global fixed widths and precision (and other stuff) to both concrete and abstract types, and would allow for bonuses like nested calls within array/dataframe printing (potentially solving this type of problem:  https://groups.google.com/forum/?hl=en#!topic/julia-users/S1aBoT9mQ_Y) 

If anyone thinks this is useful, please tell me.  If you think it sucks, please tell me why.

[Michael Hatherly]

But Michael Hatherly’s showed code above that uses a generated function to solve this.

What am I missing?

It’s probably because the thread was getting quite long and what I wrote simply got missed.

— Mike

​

[Daniel Carrera]

On 23 September 2015 at 13:51, Tomas Lycken <tomas....@gmail.com> wrote:

That depends on what you do in your “production work” ;)

What I mean by that is that there is basically no tolerance for errors or bugs. What I do with Julia is not overly complex, but it is a critical part of my research, and I really really really need to get this paper out. Also, even a brief downtime might be a problem if it happens at the wrong time. I have a personal laptop. I think I'll install 0.4 RC2 there. Unless it really dazzles me, my workstation will stay on 0.3.11 until the final release.

Cheers,

Daniel.

[Tim Holy]

Actually, I looked back for what you wrote and didn't find it---even search
(probably badly) julia-users on Google Groups. May this lazy-good-for-nothing
request a link?

--Tim

> >>>>>> implementation
> >>>>>> <http://repo.or.cz/w/glibc.git/blob/ec999b8e5ede67f42759657beb8c5fef8
> >>>>>> 7c8cc63:/stdio-common/vfprintf.c>. This code is neither easy nor

> >>>>>> trivial – it's batsh*t crazy.
> >>>>>
> >>>>> That is insane... 2388 lines, half of it macros, and I have no idea
> >>>>> how it works.
> >>>>>
> >>>>>> And we want to match its performance yet be much more flexible and
> >>>>>> generic. The current printf implementation does just that, while
> >>>>>> being
> >>>>>> somewhat less insane GNU's printf code. If someone has bright ideas
> >>>>>> for how

> >>>>>> to *also* allow runtime format specification without sacrificing

[Tim Holy]

The size of the method cache is definitely something to consider. An advantage
of the current @(s)printf, when the format string is known explicitly, is that
it works at the call site and doesn't add to the cache.

And yes, I like your proposal quite a lot. I don't have any personal
experience with Formatting.jl, but together with your macro it looks quite
powerful.

Best,
--Tim

On Wednesday, September 23, 2015 10:28:21 AM Tom Breloff wrote:
> Tim: I love that you just whipped up a generated function for this (one
> great thing about Julia is how many different ways you can tackle a
> problem), however do you see this type of solution as being better than
> Formatting's sprintf1? I would be worried about the sheer number of
> compiled versions (one for each format string, plus the macro expansion),
> but maybe that's not as much overhead as I think it is?
>
> Also I still think there's room for something like my `fmt` proposal, which
> would allow for global defaults for formatting using the type hierarchy.
> The proposal is not meant to replace printf, but to supplement it.
> Sometimes you want to control how individual types are printed globally,
> without using a format string repeatedly. In my proposal, you could set
> global fixed widths and precision (and other stuff) to both concrete and
> abstract types, and would allow for bonuses like nested calls within
> array/dataframe printing (potentially solving this type of problem:
> https://groups.google.com/forum/?hl=en#!topic/julia-users/S1aBoT9mQ_Y)
>
> If anyone thinks this is useful, please tell me. If you think it sucks,
> please tell me why.
>
> On Wed, Sep 23, 2015 at 7:51 AM, Tomas Lycken <tomas....@gmail.com>
>
> wrote:
> > That depends on what you do in your “production work” ;)
> >
> > 0.4 RC2 is good enough that it would become 0.4 if no serious bugs or
> > regressions are found. There’s currently one issue tagged with milestone

> > 0.4.0 <https://github.com/JuliaLang/julia/milestones/0.4.0>, and another

> > one tagged backport pending 0.4

> > <https://github.com/julialang/julia/issues?q=is%3Aopen+is%3Aissue+label%3A
> > %22backport+pending+0.4%22>, and I don’t know if there are other places

[Tom Breloff]

The original PR: https://github.com/JuliaLang/Formatting.jl/pull/10

The tests (which show some basic usage): https://github.com/tbreloff/Formatting.jl/blob/tom-fmt/test/fmt.jl

In addition to this, we could have a `@fmt` macro which is analogous to `@show`, but uses fmt instead.

Also, I think some helper methods would be good.  An example:

@withfmt width=20 align=right begin

  <do lots of printing>

end  # resets format spec

The naming and API can be changed, but I want people to weigh in on the concept of per-type global formatting defaults.  This will likely not be as performant as the @printf macro, but it should be fast enough for most uses.

[Stefan Karpinski]

At this point it's safe to switch to 0.4-rc2 with one important condition: when 0.4 comes out, switch to it as soon as you can. There *can* be incompatibilities between release candidates and releases, so if you linger too long on a candidate, you can find yourself a bit stuck later on when you want to upgrade to get a bug fix.

On Wed, Sep 23, 2015 at 7:51 AM, Tomas Lycken <tomas....@gmail.com> wrote:

[Michael Hatherly]

May this lazy-good-for-nothing request a link?

Of course, https://groups.google.com/forum/#!msg/julia-users/hBbEGEopi0A/fKQcqDEVBgAJ. Took me a while to actually find it again myself…

Implementation-wise they were pretty similar. I was dispatching of an instance Formatter{...}() rather than the FormatString{...} type itself, though the resulting generated code is identical as far as I can tell.

— Mike

​

[Stefan Karpinski]

That 1MB is allocation done by code generation. And this generates the code every time it runs. That's why it's bad.

[Stefan Karpinski]

This is very  but it doesn't entirely solve the problem. The f"..." macro still entails code generation, which can be done at compile time, but to support runtime formats the code generation has to be deferred until run time – and the exact same thing could be done with the @printf macro. The main thing that the generated function approach gives you is a way of caching generated code associated with a particular format. But we could also do code caching with @printf by just maintaining a dict mapping format strings to generated code (this is essentially what generated functions do) and it would avoid polluting the generated function cache, so it's arguably better. The concern with this is that someone is going to write a format string that depends on variables and they'll end up generating a huge amount of formatting code. To see why this could be an exponential explosion of code, consider the format "%$(w1)d %$(w2)d %$(w3)d %$(w4)d\n", and suppose that each of w1 through w4 ranges from 1 to 32. That would generate over a million formatting functions. Ruh roh.

I don't think there's a clever way around this without fundamentally changing the design – you have make some of the values that are currently being specialized on a compile time into runtime parameters so that you can reuse the same code for more formats. The good news is that I'm pretty sure that this is entirely doable, but it's a chunk of work.

[Stefan Karpinski]

On Wed, Sep 23, 2015 at 12:21 PM, Stefan Karpinski <ste...@karpinski.org> wrote:

This is very

cool

[Tom Breloff]

But we could also do code caching with @printf by just maintaining a dict mapping format strings to generated code

AFAIK, this is exactly what Formatting's sprintf1 does  (except it calls @sprintf instead of @printf, so that change would be easy).

[Stefan Karpinski]

sprintf1 avoids the combinatorial explosion issue since it only supports a single format specification.

[Tim Holy]

This does work at runtime, i.e., with a format string that is not known at
compile time. A new format string (one you've never seen before) will indeed
cause codegen, but only once; after that, every usage of the same format
string will be fast. It's basically a cached-method-table version of
@eval @printf($fmt, ...)
but in practice, the caching is a big deal.

We could, of course, do it the old-fashioned way, using a let block and
caching the compiled versions in a Dict{String,Function}.

That's not to say there isn't room for a purely-runtime solution, but this
occupies a nice niche of runtime-vs-compiletime performance.

Best,
--Tim

On Wednesday, September 23, 2015 12:21:20 PM Stefan Karpinski wrote:
> This is very but it doesn't entirely solve the problem. The f"..." macro

> still entails code generation, which *can* be done at compile time, but to

[lawrence dworsky]

Hi Tom

Sorry to take so long to get back to you, I had to go away for a couple of days. Thanks for the installation information, @fmt is working fine now.  It's still not as useful as the Fortran print * formatting however because it ​requires the user to know what's coming. For example, the Fortran code

x = -2.34e-12
do i = 1, 5
  x = -x*5000.
  print *, i, x
end do

produces

1     1.170000E-08

2    -5.850000E-05

3     0.292500

4     -1462.5

5     7.312501e+06

As you can see, print * figured out when exponential notation is necessary and automatically used it.

I'm retired now, but when I was working I spent a lot of time writing numerical analysis programs for various engineering issues (elastic material deformation, electron trajectories, etc.) While a  program was being developed I didn't care about the aesthetics of my printout, I just needed useful information - and early on, numerical or algebraic or programming errors could easily produce results off by 10 order of magnitude!

I think a capability such as this in Julia would be heavily used. I wish I had the expertise to write it.

Larry

On Tue, Sep 22, 2015 at 4:59 PM, Tom Breloff <t...@breloff.com> wrote:

Sorry I wasn't expecting you to run it... just comment.  You'll have to do:

Pkg.rm("Formatting")

Pkg.clone("https://github.com/tbreloff/Formatting.jl.git")

Pkg.checkout("Formatting", "tom-fmt")

Let me know if that works.

On Tue, Sep 22, 2015 at 5:52 PM, lawrence dworsky <ma...@lawrencedworsky.com> wrote:

I'm afraid my beginner status with Julia is showing:

I ran Pkg.add("Formatting"), and then   using Formatting   came back with a whole bunch of warnings, most about  Union(args...) being depricated, use Union(args....) instead. 

When all is said and done,   fmt_default!  gives me a  UndefVarError.

Help!

On Tue, Sep 22, 2015 at 2:45 PM, Tom Breloff <t...@breloff.com> wrote:

Thanks Larry, that's helpful.  Just for discussions sake, here's a quick macro that calls my proposed `fmt` method under the hood, and does something similar to what you showed.  What do you think about this style (and what would you do differently)?

 

using Formatting

macro fmt(args...)
 expr = Expr(:block)
 expr.args = [:(print(fmt($(esc(arg))), "\t\t")) for arg in args]
 push!(expr.args, :(println()))
 expr
end

And then an example usage:

In:

x = 1010101
y = 555555.555555555
fmt_default!(width=15)

@fmt x y

fmt_default!(Int, :commas)
fmt_default!(Float64, prec=2)

@fmt x y

Out:

        1010101  555555.555556

      1,010,101      555555.56

On Tuesday, September 22, 2015 at 3:08:35 PM UTC-4, lawrence dworsky wrote:

Hi Tom

What I like about it is that you can just use print *, dumbly and it always provides useful, albeit not beautiful, results. When I'm writing a program, I use print statements very liberally to observe what's going on - I find this more convenient than an in-line debugger. 

As the last line in my program below shows, it's easy to switch to formatted output when you want to. The formatting capability is pretty thorough, I'm just showing a simple example.

This Fortran program doesn't do anything, it just illustrates what the print statement produces:

real x, y

integer i, j

complex z

character*6  name

x = 2.6

y = -4.

i = 36

j = -40

z = cmplx(17., 19.)

name = 'Larry'

print *, x, y, i, j, z

print *, 'x = ', x, ' and j = ', j

print *, 'Hello, ', name, j

print '(2f8.3, i5)', x, y, j

stop

end

The output is:

        2.60000             -4.00000                   36             -40  (17.0000, 19.0000)

x =         2.60000       and j =                -40

Hello, Larry                 -40

  2.600   -4.000  -40

Is this what you are looking for?

Larry

On Tue, Sep 22, 2015 at 11:57 AM, Tom Breloff <t...@breloff.com> wrote:

Larry: can you provide details on exactly what you like about Fortran's print statement?  Did it provide good defaults?  Was it easy to customize?

On Tue, Sep 22, 2015 at 12:55 PM, LarryD <larryd...@gmail.com> wrote:

Something I miss from Fortran is the very convenient default "print *, ..... "  It handled almost 100% of my needs while working on a program and was easily replaced by real formatting when the time came. Is there any chance that Julia could get something like this?

Thanks

On Monday, September 21, 2015 at 3:46:31 AM UTC-5, Ferran Mazzanti wrote:

Dear all,

I could use some help here, because I can't believe I'm not able to easily print formatted numbers under Julia in a easy way. What I try to do is to write a function that, given a vector, prints all its components with a user-defined format. I was trying something of the form

function Print_Vec(aux_VEC,form_VEC)
    form_VEC :: ASCIIString
    str_VEC  = "%16.8f"
    for elem_VEC in aux_VEC
        str_VEC += @sprintf(form_VEC,elem_VEC)
    end
    return str_VEC
end

However, that doesn't work because it looks like the first argument in @sprintf must be a explicit string, and not a variable.
Is there anything I can do with that?

Thanks a lot for your help.

[Tom Breloff]

So if I understand your post correctly, one item on your wishlist is for the output to change formatting dynamically based on the value of the floating point number?  There's nothing inherently hard about that... it could just be a setting for `fmt` which you can switch on or off (and that could possibly be the default approach, since it's probably a better default than always printing 3 decimal places).

Don't worry so much about whether you could implement it yourself... now's the time to put your wishlist out there.  What is your best-case-scenario for formatting?

[Miguel Bazdresch]

With this Julia code:

x = -2.34e-12;

for i in 1:5

  x=-x*5000.

  println("$i $x")

end

I get this output:

1 1.17e-8

2 -5.8500000000000006e-5

3 0.29250000000000004

4 -1462.5000000000002

5 7.312500000000001e6

I don't think this is too bad. True, the output is a bit longer, but I actually prefer it, because you can copy/paste the numbers back into Julia and be sure you're getting the exact number that was used in the code.

-- mb

[Scott Jones]

I'm sorry - I meant to test it out right after you submitted the PR, I *do* think you are on the right track,

but I've been caught up in the fun of writing code for my happy Belgian company (luckily, it's in Julia),

and we hadn't gotten to the point where we needed nice formatting yet.

I've since been thinking about having a string macro, that would have a simple syntax for calling your

fmt function (and probably some other goodies):

Something like:

v1 = 10.23

v2 = 1234567

fmt_default!(Int, :commas, width = 12)

println(f"This is a test of formatting <\{v1,3,7}> <\{v2}>")

would print:

This is a test of formatting < 10.230> < 1,234,567>

So that essentially, \{...} is simply a shortcut that acts like $(fmt(...)).

On Tuesday, September 22, 2015 at 10:20:41 AM UTC-4, Tom Breloff wrote:

Chill guys.

Anyways... I started to tackle this problem, but didn't get any (much needed) comments on whether I was on the right track.  Here's some sample preliminary usage of what I was working on.  Please let me know if you think it's worth continuing, and what you'd ideally like to see in a formatter.

https://github.com/tbreloff/Formatting.jl/blob/tom-fmt/test/fmt.jl

On Tue, Sep 22, 2015 at 10:05 AM, Daniel Carrera <dcar...@gmail.com> wrote:

Coding sprintf() is beyond my skill, I tried. My contributions to Julia have to lie elsewhere (recently I've been putting a lot of time making mockups of a Julia IDE, and helping a new user port his Matlab code). I think it is rude to say "if you don't like it, fix it yourself"; especially after you wrote a post claiming that the non-implementation of sprintf() was an intentional omission because developers weren't sure how to do it right. I showed, correctly, that that argument is nonsense; so you spinned around and changed it to "if you don't like it, fix it yourself". That answer is only a way to reject valid complaints, and it feels like a thinly veiled "fuck you". Imagine if that's how people responded in bug reports. This is not the type of answer that Julia developers normally give, and it is not the type of answer that I expect in Julia; especially after giving a very different answer to the same question. I do try to contribute to Julia, and it is rude and unreasonable to hold me personally responsible for fixing everything that does not work in Julia. A healthy community needs room for receiving legitimate complaints.

Daniel.

On 22 September 2015 at 15:49, Tomas Lycken <tomas....@gmail.com> wrote:

> If C can have one, why can't Julia?

The hard truth, and nothing but it: If Julia is missing a feature, or has one that works in a way that's not the way you want, it's because no-one has wanted what you want enough to actually implement it.

That's it. There's no "it can't be done" - Julia is Turing complete, so "everything" can be done. There's no "it's not allowed" - no-one is stopping you from writing your own package, with your own implementation that does what you want the way you want it done, putting it in the package repository and seeing it take off. Chances are that if you write something that it turns out many others want as well, it will be included in the default package distribution in the future (or even in base Julia). Or it might not take off, but at least be there for your own enjoyment.

But the entire language, with all tooling, packages, etc - everything that is Julia - is very much a community-built product. Everything that is there, is there because someone at some point said hey, this is so important to me that I'm going to put it at the very top of my priority list. Creating a sprintf function that fulfills your needs might not have made it there for anyone yet. If it's that important to you, maybe that's your next contribution?

// T

On Tuesday, September 22, 2015 at 3:17:55 PM UTC+2, Daniel Carrera wrote:

I might be wrong, but to me Formatting.jl looks next to useless. The "sprintf" functions it provides only accept one parameter. The main function provided is `sprintf1()`, but even the very clumsy `generate_formatter()` function fails at the most basic tasks:

julia> fmtrfunc = generate_formatter( "%10.3f   %6d  %3d" )

ERROR: Only one AND undecorated format string is allowed

 in generate_formatter at /home/daniel/.julia/v0.3/Formatting/src/cformat.jl:23

I really don't understand why sprintf() is such a big deal. If C can have one, why can't Julia? I understand the argument that you might want to rewrite the implementation later. Fine. Just call the function "__temp_sprintf()" and put it in a package called "FunctionThatWillGoAwayLater". I don't care. I just want to be able to print formatted strings without a ton of needless hassle.

On Tuesday, 22 September 2015 10:03:52 UTC+2, Tomas Lycken wrote:

By the way, you might also note that the Formatting.jl package is written and maintained by people who are all active contributors to the main language repo, and very active members of the community. Even if it's "external" in terms of how you install it, it's by no means external "in spirit".

// T

On Tuesday, September 22, 2015 at 10:01:24 AM UTC+2, Tomas Lycken wrote:

Julia is a young language.

This comes with a lot of benefits - for example, it's possible to do things *right* from the start. We're currently in a phase where there is a multitude of possible solutions to every problem that arises, and we honestly don't know which solution proves to be the best one. Testing various solutions out in packages, outside of the base distribution, where they can compete for popularity, and keep building on top of each-other until we reach something which we believe not only is adequate, but *really* hits the sweet spot. In other words: sure, formatting numeric output is fundamental, but it's a problem with a large solution space, and we don't want to lock in on one path yet.

But Julia's young age also means that there are lots of problems that aren't really solved yet, or that have solutions proposed but not implemented, etc. For the future, there are plans to include some packages (which ones have not been decided, or AFAIK really even discussed) in a sort-of "base distribution", so they will be installed upon installation of Julia, and available with a single `using` statement. But the infrastructure for this is not in place yet, and even if it were, there would still be lots of reasons not to make a final decision on which packages make the cut until we approach a 1.0 release of the language.

So yeah, it might seem silly to have to install a package just to get fine-grained control of numeric output formatting, but, at least for now, that's part of what one could call Julia's "Package Deal" ;)

// T

On Tuesday, September 22, 2015 at 9:43:05 AM UTC+2, Ferran Mazzanti wrote:

Thanks, that seems to work.
Still it amazes me how Julia, being a language made for numerical calculations, does not natively support a simple mechanism to print/write
large bunches of numbers. I've been in the numerical world for 20+ years and I know printing lots of numbers is something you get on a daily
basis. I know now the formatting package can help on that (thanks :), what I do not like is the idea of having to install every time a new package
to get added functionality. I understand there are things that have to go to external packages because of its limited or specialized use, but
come on... printing number os definitely not one of those.
Just my 2cents :)

On Monday, September 21, 2015 at 10:52:52 AM UTC+2, Michael Hatherly wrote:

https://github.com/JuliaLang/Formatting.jl might help.

— Mike

​

[Tony Fong]

did you see the format function in the last part of Formatting.jl's readme?

[Scott Jones]

On Sunday, September 27, 2015 at 8:33:15 PM UTC-4, Tony Fong wrote:

did you see the format function in the last part of Formatting.jl's readme?

Yes - and there were some ambiguity problems with that, that limited its extensibility,

because you could have:

1) format(formatstring, args...)

2) format(formatexpr, args...)

3) format(number, args...)

Also, the formatstring or formatexpr could contain a lot of different format "directives", which means if you compile them, you get into that same problem with generating lots of different code I believe.

Tom's approach allows you to have a fmt methods for strings, for integers, for binary floats, decimal floats, whatever you want to add, and set up default format characteristics as well.

If combined with a simple string macro, that can interpolate values and format information in-line in the string,

I think that could get much better performance than is possible with C/C++s *printf functions, as well as being a lot easier to read, since the values (or expressions) being formatting are right next to the formatting,

not passed in as arguments sometimes far separated from where they are formatted.

Scott