Fancy progress and stats messages?

[Joachim Breitner]

Hi,

I’m quite happy with shake, right now using it to implement a tool to
build all Haskell packages in Debian, making sure that all reverse
dependencies are also built.

But I’m a bit disappointed by shake’s default output. Depending on the
target, some of the programs to run are rather boring, other very
interesting targets have no external command that are being run...

What I would like to see would be a status like this at the bottom of
the output:

target4: ✓ done
target3: ✓ done
target6: ☠ failed
target1: … running gccc
target2: waiting for target3
target3: … buiding

So, what this means is that shake is runnning with "-j2", currently
working on the (independent) targets target1 and target2. It already
has built target4, target3 and target6, where the latter has failed
(presumably due to shakeStaunch=True). target2 has also been started,
but is right now waiting for target3. target1 is actually running a
command, while target3 is doing “something”. Ideally, there would be a
"Action a -> Action a" combinator that I can use to tell the system
what I’m doing right now.

It would update the lines for the “active” targets (target1, target2
and target3) as they change (using curses or whatever is required to do
that), while finished targets scroll up as usual.

Ideally, any output from running targetn would also appear above the
“active” targets, prefixed with "targetn: ". But that’s maybe asking
too much.


Is it such a view possible, or maybe even implemented somewhere? Can I
query shake for its current view somehow? And does this even fit
Shake’s data model, as I maybe naively assume?


Thanks,
Joachim

--
Joachim “nomeata” Breitner
ma...@joachim-breitner.de • http://www.joachim-breitner.de/
Jabber: nom...@joachim-breitner.de • GPG-Key: 0xF0FBF51F
Debian Developer: nom...@debian.org

[Neil Mitchell]

Hi Joachim,

> But I’m a bit disappointed by shake’s default output.

Fortunately, tweaking up some UI output is much easier than writing a
build system, so hopefully we can get somewhere!

> Depending on the
> target, some of the programs to run are rather boring, other very
> interesting targets have no external command that are being run...

There are two ways to change what gets output:

* Write it yourself with putNormal, e.g. putNormal "This command is important"
* Run a command with quietly to suppress its normal output.

There's a bit more on this topic at
http://stackoverflow.com/questions/31852826/how-to-change-the-information-printed-by-shake-during-a-build

> What I would like to see would be a status like this at the bottom of
> the output:
>
> target4: ✓ done
> target3: ✓ done
> target6: ☠ failed
> target1: … running gccc
> target2: waiting for target3
> target3: … buiding
>
> So, what this means is that shake is runnning with "-j2", currently
> working on the (independent) targets target1 and target2. It already
> has built target4, target3 and target6, where the latter has failed
> (presumably due to shakeStaunch=True). target2 has also been started,
> but is right now waiting for target3. target1 is actually running a
> command, while target3 is doing “something”. Ideally, there would be a
> "Action a -> Action a" combinator that I can use to tell the system
> what I’m doing right now.

This seems to capture a lot of interesting information that I'm not
convinced that build system can figure out. Taking some examples of
programs that produce output similar to what I think Shake could
perhaps produce:

* Stack prints out "configuring foo", "building foo", "installed foo"
as lines that scroll upwards, and a single status line at the bottom
that says how many things it has left to do.

* Ninja prints out a single line of the most recent thing it started.

* Buck prints out N lines, one for each processor (e.g. 2 with -j2)
saying the current command it is working on.

How did you pick the 6 specific targets to be the ones that are displayed?

> It would update the lines for the “active” targets (target1, target2
> and target3) as they change (using curses or whatever is required to do
> that), while finished targets scroll up as usual.

Yes, requiring curses starts to make things a lot harder.

> Ideally, any output from running targetn would also appear above the
> “active” targets, prefixed with "targetn: ". But that’s maybe asking
> too much.

That's possible, if you're willing to take a performance hit as it
can't just connect up stdout with the console stdout as it does now.

> Is it such a view possible, or maybe even implemented somewhere? Can I
> query shake for its current view somehow? And does this even fit
> Shake’s data model, as I maybe naively assume?

There's nothing you can do on the outside to get exactly what you
want, but let's figure out what heuristics you are thinking of
applying, and we can figure out if Shake should provide something.
There's already https://github.com/ndmitchell/shake/issues/18 which
tracks the desire for something shorter.

Thanks, Neil

[Joachim Breitner]

Hi,

Am Dienstag, den 11.08.2015, 21:10 +0100 schrieb Neil Mitchell:
> Fortunately, tweaking up some UI output is much easier than writing a
> build system, so hopefully we can get somewhere!

do you think so? I generally find that UI is the most tedious part of
many programs. At least as soon as it exceeds the basic command-line
-flag-and-messages-to-stdout scheme...

> > What I would like to see would be a status like this at the bottom
> > of the output:
> >
> > target4: ✓ done
> > target3: ✓ done
> > target6: ☠ failed
> > target1: … running gccc
> > target2: waiting for target3

> > target3: … building

> >
> > So, what this means is that shake is runnning with "-j2", currently
> > working on the (independent) targets target1 and target2. It already
> > has built target4, target3 and target6, where the latter has failed
> > (presumably due to shakeStaunch=True). target2 has also been started,
> > but is right now waiting for target3. target1 is actually running a
> > command, while target3 is doing “something”. Ideally, there would be a
> > "Action a -> Action a" combinator that I can use to tell the system
> > what I’m doing right now.
>
> This seems to capture a lot of interesting information that I'm not
> convinced that build system can figure out.

Really? Given the moment of the build that I am describing, which bits
of information are not available to the build system?

> How did you pick the 6 specific targets to be the ones that are
> displayed?

In my design, you don’t have to pick: The 6 lines above are exactly
what I want to see!

I can create a mockup, i.e. a sequences of outputs as a hypothetical
build proceeds, but maybe not in this mail, as it would become a large
file. Let me know if it would help.

> > Is it such a view possible, or maybe even implemented somewhere?
> > Can I
> > query shake for its current view somehow? And does this even fit
> > Shake’s data model, as I maybe naively assume?
>
> There's nothing you can do on the outside to get exactly what you
> want, but let's figure out what heuristics you are thinking of
> applying, and we can figure out if Shake should provide something.
> There's already https://github.com/ndmitchell/shake/issues/18 which
> tracks the desire for something shorter.

Isn’t maybe trace list something close to what I imagine?


I guess I could create a hack to demonstrate my idea by wrapping the
various rule-generating functions (e.g. %>) to insert custom status
commands at the beginning and the end, and similar for "cmd", "traced"
and "need". These commands could signal something to a separate thread
that keeps some state and does the pretty display that I want. If I
then make sure everything else is quiet, it might work well enough to
see if the idea is viable and as pretty as I hope.

Greetings from DebConf,

[Neil Mitchell]

Hi Joachim,

> do you think so? I generally find that UI is the most tedious part of
> many programs. At least as soon as it exceeds the basic command-line
> -flag-and-messages-to-stdout scheme...

It's probably the bit that takes most time - but often bad UI is a
reflection of inconsistent underlying concepts. Yes, there's certainly
a lot of work to get a good UI.

> In my design, you don’t have to pick: The 6 lines above are exactly
> what I want to see!

I don't see how Shake decided on them. I guess Shake has built 100's
of rules, so why is it showing you information about those six. Are
they the last six to have completed? Are the the top-level want's you
did? Are they specially tagged some way?

> I can create a mockup, i.e. a sequences of outputs as a hypothetical
> build proceeds, but maybe not in this mail, as it would become a large
> file. Let me know if it would help.

I still don't think I know exactly what you are after, so that might
be useful, although a small example might capture it too.

>> There's nothing you can do on the outside to get exactly what you
>> want, but let's figure out what heuristics you are thinking of
>> applying, and we can figure out if Shake should provide something.
>> There's already https://github.com/ndmitchell/shake/issues/18 which
>> tracks the desire for something shorter.
>
> Isn’t maybe trace list something close to what I imagine?

The trace is something I assemble after the fact. It's certainly
possible that Shake could dump information about the steps its taking,
and that could be used to synthesise some kind of view.

> I guess I could create a hack to demonstrate my idea by wrapping the
> various rule-generating functions (e.g. %>) to insert custom status
> commands at the beginning and the end, and similar for "cmd", "traced"
> and "need". These commands could signal something to a separate thread
> that keeps some state and does the pretty display that I want. If I
> then make sure everything else is quiet, it might work well enough to
> see if the idea is viable and as pretty as I hope.

To keep everything quiet you probably want to shakeVerbosity=Quiet
then throw away all Stdout from processes (EchoStdout False). A
prototype would be quite cool. I'm sure there is something in the
compressed output that would be useful. For my large Shake system, it
basically dumps a wall of text at me, and I just rely on the titlebar
progress messages to see what is going on.

Have you already tried --progress? That doesn't give the depth of
information you are after, but it gives something.

Enjoy DebConf!

Thanks, Neil

[Joachim Breitner]

Hi,

Am Mittwoch, den 12.08.2015, 21:38 +0100 schrieb Neil Mitchell:
> > In my design, you don’t have to pick: The 6 lines above are exactly
> > what I want to see!
>
> I don't see how Shake decided on them. I guess Shake has built 100's
> of rules, so why is it showing you information about those six. Are
> they the last six to have completed? Are the the top-level want's you
> did? Are they specially tagged some way?

ah. this is just a small example, with indeed only 6 targets that have
been started.

> > I can create a mockup, i.e. a sequences of outputs as a
> > hypothetical
> > build proceeds, but maybe not in this mail, as it would become a
> > large
> > file. Let me know if it would help.
>
> I still don't think I know exactly what you are after, so that might
> be useful, although a small example might capture it too.

Here is an attempt to write a self-contained ad-hoc-terminal
-presentation-tool in bash. Just run it in bash, and click yourself
through the example:
https://gist.github.com/nomeata/a146d35fd9f6eeee6568

The rules are roughly (in
all:
need [target1, target2, target3, target4]

target1:
threadDelay 1000

target2:
threadDelay 300
cmd gcc
threadDelay 300

target3:
threadDelay 300
cmd gcc
need [target5]

target4:
threadDelay 1000

target5:
cmd gcc
threadDelay 300


The numbers are of course bogus. And the
all: ✓ finished
in the last frame is also not quite correct, but should still give an
idea.

> Have you already tried --progress? That doesn't give the depth of
> information you are after, but it gives something.

In another project of mine (which is smaller) it just says
"Starting..."


Greetings,

Joachim
--
Joachim "nomeata" Breitner

Debian Developer
nom...@debian.org | ICQ# 74513189 | GPG-Keyid: F0FBF51F
JID: nom...@joachim-breitner.de | http://people.debian.org/~nomeata

[Neil Mitchell]

>> I still don't think I know exactly what you are after, so that might
>> be useful, although a small example might capture it too.
>
> Here is an attempt to write a self-contained ad-hoc-terminal
> -presentation-tool in bash. Just run it in bash, and click yourself
> through the example:
> https://gist.github.com/nomeata/a146d35fd9f6eeee6568

Thanks for that, it does clarify what is going on.

My concern there is that for large build systems there may be 1000's
of active rules at any time, and then the presentation probably breaks
down. My large build system has about 600 end want's.

As to whether it's feasible, there are certainly pointers in the data
structure giving the graph you describe, but at the moment they only
store pointers to IORefs etc. Adding names should be possible, the
only tricky bit is to add them without slowing anything down.

>> Have you already tried --progress? That doesn't give the depth of
>> information you are after, but it gives something.
>
> In another project of mine (which is smaller) it just says
> "Starting..."

It should say that for the first 5 seconds, and then start displaying
information, once it has taken two samples and has a rough idea what
your system might be doing. You can pass --progress=1 to make it start
reporting after 1s, and update every 1s, but if you entire system
takes < 5s then you probably don't need progress.

Thanks, Neil

[Joachim Breitner]

Hi,

Am Mittwoch, den 12.08.2015, 22:09 +0100 schrieb Neil Mitchell:
> > > I still don't think I know exactly what you are after, so that
> > > might
> > > be useful, although a small example might capture it too.
> >
> > Here is an attempt to write a self-contained ad-hoc-terminal
> > -presentation-tool in bash. Just run it in bash, and click yourself
> > through the example:
> > https://gist.github.com/nomeata/a146d35fd9f6eeee6568
>
> Thanks for that, it does clarify what is going on.
>
> My concern there is that for large build systems there may be 1000's
> of active rules at any time, and then the presentation probably
> breaks down. My large build system has about 600 end want's.

sure, it won’t suit everyone. Maybe with a bit variation, i.e. not list
targets that are active, but waiting for another target, or only have
one summary line like

target all, target 3 and 12342132 other targets are waiting for dependencies

Endless possibilities – at least once the data is accessible :-)

> > > Have you already tried --progress? That doesn't give the depth of
> > > information you are after, but it gives something.
> >
> > In another project of mine (which is smaller) it just says
> > "Starting..."
>
> It should say that for the first 5 seconds, and then start displaying
> information, once it has taken two samples and has a rough idea what
> your system might be doing. You can pass --progress=1 to make it
> start
> reporting after 1s, and update every 1s, but if you entire system
> takes < 5s then you probably don't need progress.
>

Right, I only tested it with my “other” shake project, which is much
smaller. The one where I want this feature take half a day to
complete... but there, I already slammed my own command line parser in
front, supporting only a few shake options to be modified, so it
becomes non-trivial to test that with --progress :-)

Greetings,
Joachim


--
Joachim “nomeata” Breitner

[Luke]

I think this would be a really cool addition to Shake. Perhaps just being able to list the top level rules you want to see would be enough to solve the big dependency tree problem with thousands of rules.

[Joachim Breitner]

Hi,

oh, and let me add these days, there is a fancy library to render the
dynamic status information to the bottom of the screen:
http://hackage.haskell.org/package/concurrent-output

Greetings,
Joachim

[Joachim Breitner]

Hi,

Am Mittwoch, den 12.08.2015, 22:09 +0100 schrieb Neil Mitchell:

> > > I still don't think I know exactly what you are after, so that
> > > might
> > > be useful, although a small example might capture it too.
> > Here is an attempt to write a self-contained ad-hoc-terminal
> > -presentation-tool in bash. Just run it in bash, and click yourself
> > through the example:
> > https://gist.github.com/nomeata/a146d35fd9f6eeee6568
> Thanks for that, it does clarify what is going on.

I wanted to give it a shot, building on the very nice
http://hackage.haskell.org/package/concurrent-output
in particular the regions support therein.

This would be my approach:
 * shake, shakeArgs etc. wrap themselves in displayConsoleRegions
 * Every action-running rule such as "%>" wraps its action in
   withConsoleRegion. Initially, the content is a generic
   # build/artifact.c: … building
 * Blocking commands such as "need", "cmd" etc. write to the region of 
   the current action using setConsoleRegion, indicating both the name
   of the current target and what it is doing. E.g.
   # build/artifact.so:  waiting for build/artifact.c
   or
   # build/artifact.c: … running gcc
   After the command finishes, the region is changed back to the 
   generic
   # build/artifact.c: … building
 * At the end of an action, finishConsoleRegion is used to write
   # build/aritfact.c: ✓ done
   above the list of processed actions, and now gets scrolled out of 
   the picture.

I starting writing a wrapper module around Development.Shake that adds
that functionality, but I could not find a nice way of handing down
some information (in particular the name of the current target, and the
ConsoleRegion) from %> to the commands like need and cmd. Any idea on
how to implement that nicely, short of creating a custom Action type
(that wraps Shake’s Action in a reader monad) as well?

For my purposes it might suffice to have a 
  targetId :: Action something
where something represents the current target in some opaque way, and
which I can I use as a key in a (global) map.


But I think I’ll see how far the Action-wrapping approach gets me, at
least for a proof of concept. I hope that the result is so amazing that
you’ll rush to add the necessary hooks to shake so that it is more
easily usable.

Greetings,
Joachim

--
Joachim “nomeata” Breitner

  ma...@joachim-breitner.de • https://www.joachim-breitner.de/
  XMPP: nom...@joachim-breitner.de • OpenPGP-Key: 0xF0FBF51F
  Debian Developer: nom...@debian.org

[Joachim Breitner]

Hi,

Am Sonntag, den 24.04.2016, 11:30 +0200 schrieb Joachim Breitner:
> But I think I’ll see how far the Action-wrapping approach gets me, at
> least for a proof of concept. I hope that the result is so amazing that
> you’ll rush to add the necessary hooks to shake so that it is more
> easily usable.

here we go:
https://asciinema.org/a/c7f3jybf7qwwt0i2lo6r02ifs
Isn’t it great?

The wrapper is here:
https://github.com/nomeata/gipeda/blob/master/src/Development/Shake/Fancy.hs
and here are the changes necessary to the code:
https://github.com/nomeata/gipeda/commit/17b3adb20d54490099aa112e67b7b6f1c30b6f53

Major wart 1: I was not able to define an instance of CmdArguments for
my Action type, as that class’s methods are not exposed. Have you
considered adding a set of ".Internal" modules to shake to allow people
to use cmd with their own monads?

Major wart 2: Mixing this wrapping module with other modules providing
Action’s (such as my git-repo-access-module¹) was annoying, and it
would be really great if this logging could simply be added by way of
hooks.

Or maybe you are so amazed by the possibilities of concurrent-output
that you’ll simply provide this functionality within shake proper :-)

Greetings,
Joachim


¹ https://github.com/nomeata/gipeda/blob/master/src/Development/Shake/Gitlib.hs

[Neil Mitchell]

Hi,

First off, that's incredibly cool, and yes, I definitely want
something like that easily available for Shake. On most terminals, I
think I'd probably like it to be the default.

As to whether it goes into Shake, that really depends on the
dependencies Shake has. At the moment, the dependencies of Shake are
deliberately light, for two reasons: 1) when I started Shake we only
had Cabal, which makes dependencies more fraught; 2) if Shake becomes
part of the GHC bootstrap process the dependencies might matter a lot.
Point 1 is somewhat addressed by Stack and maturity over semantic
versioning. I'm going to be talking to the GHC team about point 2
later this week. I definitely won't go wild with dependencies, but
things like ansi-terminal seem quite plausible.

If it doesn't go into Shake itself, having the necessary hooks will
have to happen to get that working because its just cool not to.
Having targetId isn't infeasible - the closed-source version of Shake
actually had that and I removed it. Doing the wrapping is certainly
feasible for your project, but I suspect the wrapping is just too high
a cost to enable it for most projects, so it really needs to be more
integrated.

If we did go with the hooks approach, one way would be to hijack the
Shake Progress data type, and append the "current progress" messages
and then something could poll that and update the screen. Only
downside is it would be slightly slower to react.

So in summary, very cool, definitely the way to go, and I'll email
back by the end of the week as to whether this should be outside Shake
or inside Shake, and then we can get it going.

Thanks, Neil

PS. Asciinema is incredibly cool too!

[Joachim Breitner]

Hi,

I agree that it is desirable to have this functionality via sensible
and powerful hooks, not only because of the avoided dependency, but
also because of other things it might provide. Imagine someone building
a GUI application that is driven by shake – they also would want more
semantic status hooks.

Here is a first rough draft for a data type that can be used as the
type of an output hook field. Note that "env" is deliberately
existentially quantified, as shake does not have to care about this.

data ShakeOutput = forall env. ShakeOutput
  { initialize    :: IO a -> IO a
  , processTarget :: String -> (env -> IO a) -> IO a
  , updateTarget  :: String -> env -> IO a
  , failTarget    :: String -> env -> IO a
  , succeedTarget :: env -> IO a
  }

Some details can be refined. For example, the updateTarget (which
describes what the target is currently doing) could have a more
semantic argument instead of String, do distinguish between "waiting",
"running a command", "other stuff" etc.

But maybe you have something entirely different in mind; as long as I
get my fancy output, that’s fine with me :-)

Greetings,
Joachim

[Neil Mitchell]

I've attached a link to your stuff to
https://github.com/ndmitchell/shake/issues/18 which I'll use as the
issue.

I agree that your record type might make more sense - I'm not really
certain yet - but I agree that the fancy output is definitely
essential however it gets done. Currently I'm investigating how to
change the Rule type in Shake, so I'll probably leave this until after
that, so I'm not building on top of something that changes straight
away.

After talking to the GHC team it's been agreed Shake dependencies will
never be installed as bootstrap dependencies by GHC, which means I
have a little more freedom to add additional dependencies to Shake -
e.g. ansi-terminal.

Thanks, Neil


On Sun, Apr 24, 2016 at 10:35 PM, Joachim Breitner