Proposal: GN "query server" mode for fast query commands over the build graph

[David Turner]

I'd like to propose a new feature that helps tremendously when trying to inspect a given build graph. At the moment, all commands that act as queries over the build graph (e.g. `desc`, `refs`, `outputs`, `path` and `meta`) must first parse the whole build input files before displaying anything, which can take several seconds. This is annoying when one is trying to inspect various parts of the same build graph (e.g. 12s to 20s for the Fuchsia build).

The idea is to add a new `gn query` meta-command that could be used as follows:

- `gn query start-server` would load the build input files and construct the build graph in memory, then bind to a local socket, waiting for client connections.

- `gn query desc`, `gn query refs`, and similar sub-commands would be used in a different terminal to connect to the server, and perform read-only queries on the in-memory graph, retrieving the same output as their parent command, only dramatically faster. The output would be exactly the same as their parent command.

Implementation wise, this requires minimal changes to GN: Functions that implement the base query commands (e.g. RunDesc() implements `gn desc`) all create a Setup object in the same way. They could be changed very little to just take an instance from the caller instance.

The actual socket binding/connection code, and stdout/stderr redirection can be scoped to limited classes, and would actually only be used by `gn query start-server` itself.

This would only support read-only queries that need a simple build graph construction. In practical terms, this means:

  Setup* setup = new Setup();
  if (!setup->DoSetup(args[0], false))
    return 1;
  if (!setup->Run())
    return 1;

Note that other commands like "gn gen" do construct the Setup object differently, so would not be supported by this mode. Which is not a problem since the scope of that feature is to inspect the build graph rapidly.

There is a very simple proof-of-concept written by jayz...@google.com for this feature, available at https://gn-review.googlesource.com/c/gn/+/12780. Warning: this only works on Linux, but it demonstrates that this works very well.

Before going further and submitting a completed implementation, I'd like to know if this new feature seems in line with GN goals (i.e. do you consider this would add too much complexity?)

The most difficult and time-consuming parts of this would probably be proper and expansive testing: the test framework would need to spawn new processes for example, and IPCs are always a bit tricky. Also Windows stdout/stderr redirection with proper handling of ANSI color escape sequences.

Finally, there is no plan to implement a "daemon mode" where the server would move automatically in the background. That's a lot of complexity that isn't needed here.

- Digit

[Andrew Grieve]

I can definitely see the use case.

Since this is limited to queries, and not a generic "daemon mode", would it be simpler to have "gn query" drop you into a REPL rather than deal with IPCs (assuming there's a windows equivalent to readline)?

[K. Moon]

I was also thinking a REPL would be a more straightforward solution to this than spawning a separate server. This could even be used by automation if the REPL supported some sort of text-based protocol, like an LSP does.

[Dirk Pranke]

I am generally supportive of the idea, as I have had similar frustrations with how GN does this over the years. In particular, one concern I've long had is that we don't require loading the GN build graph to be a read-only operation, which means that running `gn <x>` may have side effects, particularly ones that make it unsuitable for invoking GN from inside the compile itself (whether or not doing so is a good idea for other reasons is probably better discussed separately, of course).

That said, it's unclear to me what complexity you're hoping to avoid here by this not being a "daemon mode" (and I don't know what you mean by "server would move automatically in the background". Can you expand on this more?

I assume much of it has to do with not having to track the filesystem for updates ...

-- Dirk

 

- Digit

--
To unsubscribe from this group and stop receiving emails from it, send an email to gn-dev+un...@chromium.org.

[Dirk Pranke]

Having a repl would limit you to being able to only call the server from a single client (or a single process), and I can definitely imagine cases where you'd want to be able to query the graph from different processes.

I.e., I suspect the repl would only be a partial solution. On the other hand, once you have a daemon mode, you could trivially implement a repl on top of it via separate tools, you wouldn't need the REPL logic to be in GN itself.

-- Dirk

[Brett Wilson]

I think a way to run multiple queries without re-running the whole build would be useful. But I agree with Andrew that a "repl mode" would be a better approach than a client/server design.

One of my other projects is the Fuchsia debugger which has a client/server architecture. We spent a great deal of effort getting the communication working well across the different platforms and use-cases, and we only had Linux & Mac. Windows would be a whole other thing, and GN supports many more platforms and I suspect a client/server mode would never be implemented on many of them.

Aside from implementation complexity, there is also the user complexity. By far the biggest hurdle we have in getting people to use the Fuchsia debugger is the difficulty in starting it. in GDB you can type "gdb" and then "run foo". In the Fuchsia debugger you need to start the remote server and then connect to it. We created wrapper commands that people normally use to do a one-command execution, but it's still a hurdle. There are all kinds of problems that happen, like the socket wasn't closed correctly last time because the server was killed and now everything is in a weird state, or the server is already running in a different window that you forgot about and are getting confusing errors. Symbol configuration issues and connection problems account for 90% of the debugger support requests that we get.

In the proposed design, it's actually more complicated for a user than the debugger design because it seems like you're requiring two terminal windows. The Fuchsia infra team already has a "gn desc" caching tool that tries to solve a similar problem. But AFAIK it's seldom used, I suspect mostly because it's a separate thing you have to run and think about. A client/server mode is a similar hurdle, while saying "type one command and then give it "desc" commands just like you would on the command-line" eliminates that.

For the debugger I wrote a reasonably-fully-featured line editor with no dependencies. It would need porting to Windows but I think would be a good starting point. I would not want to bring in one of the typical big console/readline libraries.

Brett

[Nico Weber]

Another idea, simpler than a repl, could be to just allow multiple queries per gn invocation (e.g. `gn query -c 'refs' -c desc ...`) for starters.

[Dirk Pranke]

Another option would be to flesh out the generated `--ide=json` output to be comprehensive and standardized, and just generate that at `gn gen` time and have tools parse the output to figure out what you needed. 

It would be slower than querying an in-memory graph, of course, but presumably much faster than actually parsing the BUILD.gn files and building the graph from that.

-- Dirk

[Dirk Pranke]

On Thu, Jan 20, 2022 at 12:44 PM Dirk Pranke <dpr...@google.com> wrote:

Another option would be to flesh out the generated `--ide=json` output to be comprehensive and standardized, and just generate that at `gn gen` time and have tools parse the output to figure out what you needed. 

It would be slower than querying an in-memory graph, of course, but presumably much faster than actually parsing the BUILD.gn files and building the graph from that.

I suppose it would mean clients would have to duplicate the logic GN uses for `path`, `refs`, etc, and that wouldn't be great, on reflection ...

-- Dirk

[Roland McGrath]

In the same vein I was going to suggest a "JSON REPL" rather than doing anything directly human-usable.  That is, a mode that reads queries in JSON on stdin and writes their responses on stdout, so it could be used interactively as a server, but without sockets and all that.  If you wanted to build a client-server interface for it, you can easily do that using gn in json-repl mode as a subprocess of the server.  But you can also just build a user-friendly REPL that talks the JSON protocol to the subprocess.

I think Dirk's idea of exporting complete semantic information in JSON such that even the kind of queries to support doesn't need to be decided in the gn tool itself is probably the best idea yet.  The existing diagnostic tools could be replaced by a library in Python or Go or whatever floats your boat to decipher the JSON.  I suspect that the volume of JSON data required would be enough that you'd want this kind of export to be a separate mode and not necessarily one you usually run on `gn gen`.  It would certainly be nice if GN gave you a way to define an action target for running gn that automatically got all the input deps that the re-gen rules get, so you could say `ninja gn-metadata.json` easily to do the even-slower-than-gen step to update the giant JSON file.

On Thu, Jan 20, 2022 at 10:47 AM Nico Weber <tha...@chromium.org> wrote:

[Brett Wilson]

I think this brings up the question that was not clearly addressed in the initial proposal: who is the target audience for this feature?

If it's build tooling, a better JSON output might be best. If it's individual users doing random exploration, a REPL would be best.

Brett

[David Turner]

Thanks for your replies.

First, the target audience for this feature is anybody who needs to inspect a complex build graph by doing repeated queries like `gn desc` / `gn refs` / `gn path`.
Which includes myself, since I have to routinely perform these when debugging issues in the Fuchsia build system, where each query can have 12 to 25 seconds of delay, depending on build configuration).
All I can say is that having the query results immediately is quite addictive compared to the current situation :-)

What I call a "daemon" is a server process that moves itself to the background. In this case, this would mean that the first `gn query <command>` call would start a background server process automatically, which would be reused by later `gn query` invocations.
However, managing the lifetime of this daemon requires much more complexity that what is being proposed here (the client needs to reliably detect whether a daemon is already running for the current directory, daemons can get stuck and must be killed manually, or the daemon becomes stale as soon as one input file changes, plus a few other things). I am perfectly happy with having to start the server process in a different terminal explicitly. Note that a daemonized mode can be implemented with a small shell script on Unix anyway (though this is more difficult on Windows).

A REPL is a nice feature to have too, but would require more code to handle parsing inputs, which is not always trivial, especially to support history and cursor management. And by the way, I don't know any non-GPL readline alternative that works on Windows.
In a way, the proposal just uses your existing shell as the REPL. And if you really want one, it is possible to implement one as a wrapper script or program on top of `qn query ...` commands.

Finally, JSON output is an interesting idea, but it seems a completely different feature.

In other words, this proposal is really to add a very valuable feature with the least amount of complexity to GN. Nothing precludes a daemon mode, a REPL, or JSON output to be implemented on top, or alongside this, though.

[Brett Wilson]

On Thu, Jan 20, 2022 at 3:13 PM David Turner <di...@google.com> wrote:

Thanks for your replies.

First, the target audience for this feature is anybody who needs to inspect a complex build graph by doing repeated queries like `gn desc` / `gn refs` / `gn path`.
Which includes myself, since I have to routinely perform these when debugging issues in the Fuchsia build system, where each query can have 12 to 25 seconds of delay, depending on build configuration).
All I can say is that having the query results immediately is quite addictive compared to the current situation :-)

What I call a "daemon" is a server process that moves itself to the background. In this case, this would mean that the first `gn query <command>` call would start a background server process automatically, which would be reused by later `gn query` invocations.
However, managing the lifetime of this daemon requires much more complexity that what is being proposed here (the client needs to reliably detect whether a daemon is already running for the current directory, daemons can get stuck and must be killed manually, or the daemon becomes stale as soon as one input file changes, plus a few other things). I am perfectly happy with having to start the server process in a different terminal explicitly. Note that a daemonized mode can be implemented with a small shell script on Unix anyway (though this is more difficult on Windows).

A REPL is a nice feature to have too, but would require more code to handle parsing inputs, which is not always trivial, especially to support history and cursor management. And by the way, I don't know any non-GPL readline alternative that works on Windows.

The Fuchsia readline library we wrote for the debugger has all of the features you might need and I think it should be easy enough to port to Windows.

For command parsing, I would factor all commands into two halves: one that takes a Setup object and all of the parameters other than the directory, and one that has the existing code that makes the Setup object. The latter would be implemented in terms of the former, and the REPL would call the forner. I think you would probably need something similar for a remote thing anyway.

 

In a way, the proposal just uses your existing shell as the REPL. And if you really want one, it is possible to implement one as a wrapper script or program on top of `qn query ...` commands. 

As I mentioned previously, I think implementing the client/server in a reliable way across platforms is actually more work and less generally useful to users.

[David Turner]

Le ven. 21 janv. 2022 à 00:36, Brett Wilson <bre...@chromium.org> a écrit :

On Thu, Jan 20, 2022 at 3:13 PM David Turner <di...@google.com> wrote:

Thanks for your replies.

First, the target audience for this feature is anybody who needs to inspect a complex build graph by doing repeated queries like `gn desc` / `gn refs` / `gn path`.
Which includes myself, since I have to routinely perform these when debugging issues in the Fuchsia build system, where each query can have 12 to 25 seconds of delay, depending on build configuration).
All I can say is that having the query results immediately is quite addictive compared to the current situation :-)

What I call a "daemon" is a server process that moves itself to the background. In this case, this would mean that the first `gn query <command>` call would start a background server process automatically, which would be reused by later `gn query` invocations.
However, managing the lifetime of this daemon requires much more complexity that what is being proposed here (the client needs to reliably detect whether a daemon is already running for the current directory, daemons can get stuck and must be killed manually, or the daemon becomes stale as soon as one input file changes, plus a few other things). I am perfectly happy with having to start the server process in a different terminal explicitly. Note that a daemonized mode can be implemented with a small shell script on Unix anyway (though this is more difficult on Windows).

A REPL is a nice feature to have too, but would require more code to handle parsing inputs, which is not always trivial, especially to support history and cursor management. And by the way, I don't know any non-GPL readline alternative that works on Windows.

The Fuchsia readline library we wrote for the debugger has all of the features you might need and I think it should be easy enough to port to Windows.

Can you provide a link, I cannot seem to find it under //zircon.

For command parsing, I would factor all commands into two halves: one that takes a Setup object and all of the parameters other than the directory, and one that has the existing code that makes the Setup object. The latter would be implemented in terms of the former, and the REPL would call the forner. I think you would probably need something similar for a remote thing anyway.

 

Yes, that's how it's implemented in the prototype as well.
 

[Brett Wilson]

On Thu, Jan 20, 2022 at 4:43 PM David Turner <di...@google.com> wrote:

Le ven. 21 janv. 2022 à 00:36, Brett Wilson <bre...@chromium.org> a écrit :

On Thu, Jan 20, 2022 at 3:13 PM David Turner <di...@google.com> wrote:

Thanks for your replies.

First, the target audience for this feature is anybody who needs to inspect a complex build graph by doing repeated queries like `gn desc` / `gn refs` / `gn path`.
Which includes myself, since I have to routinely perform these when debugging issues in the Fuchsia build system, where each query can have 12 to 25 seconds of delay, depending on build configuration).
All I can say is that having the query results immediately is quite addictive compared to the current situation :-)

What I call a "daemon" is a server process that moves itself to the background. In this case, this would mean that the first `gn query <command>` call would start a background server process automatically, which would be reused by later `gn query` invocations.
However, managing the lifetime of this daemon requires much more complexity that what is being proposed here (the client needs to reliably detect whether a daemon is already running for the current directory, daemons can get stuck and must be killed manually, or the daemon becomes stale as soon as one input file changes, plus a few other things). I am perfectly happy with having to start the server process in a different terminal explicitly. Note that a daemonized mode can be implemented with a small shell script on Unix anyway (though this is more difficult on Windows).

A REPL is a nice feature to have too, but would require more code to handle parsing inputs, which is not always trivial, especially to support history and cursor management. And by the way, I don't know any non-GPL readline alternative that works on Windows.

The Fuchsia readline library we wrote for the debugger has all of the features you might need and I think it should be easy enough to port to Windows.

Can you provide a link, I cannot seem to find it under //zircon.

Sure, it's actually in //src/lib:

https://fuchsia.googlesource.com/fuchsia/+/refs/heads/main/src/lib/line_input/ 

[David Turner]

Le ven. 21 janv. 2022 à 05:07, Brett Wilson <bre...@chromium.org> a écrit :

On Thu, Jan 20, 2022 at 4:43 PM David Turner <di...@google.com> wrote:

Le ven. 21 janv. 2022 à 00:36, Brett Wilson <bre...@chromium.org> a écrit :

On Thu, Jan 20, 2022 at 3:13 PM David Turner <di...@google.com> wrote:

Thanks for your replies.

First, the target audience for this feature is anybody who needs to inspect a complex build graph by doing repeated queries like `gn desc` / `gn refs` / `gn path`.
Which includes myself, since I have to routinely perform these when debugging issues in the Fuchsia build system, where each query can have 12 to 25 seconds of delay, depending on build configuration).
All I can say is that having the query results immediately is quite addictive compared to the current situation :-)

What I call a "daemon" is a server process that moves itself to the background. In this case, this would mean that the first `gn query <command>` call would start a background server process automatically, which would be reused by later `gn query` invocations.
However, managing the lifetime of this daemon requires much more complexity that what is being proposed here (the client needs to reliably detect whether a daemon is already running for the current directory, daemons can get stuck and must be killed manually, or the daemon becomes stale as soon as one input file changes, plus a few other things). I am perfectly happy with having to start the server process in a different terminal explicitly. Note that a daemonized mode can be implemented with a small shell script on Unix anyway (though this is more difficult on Windows).

A REPL is a nice feature to have too, but would require more code to handle parsing inputs, which is not always trivial, especially to support history and cursor management. And by the way, I don't know any non-GPL readline alternative that works on Windows.

The Fuchsia readline library we wrote for the debugger has all of the features you might need and I think it should be easy enough to port to Windows.

Can you provide a link, I cannot seem to find it under //zircon.

Sure, it's actually in //src/lib:

https://fuchsia.googlesource.com/fuchsia/+/refs/heads/main/src/lib/line_input/ 

Thank you. I had a cursory look, and it seems to have a few dependencies on fxl, fit and syslog libraries, but these could probably be replaced by something else.
I'm not too sure about the Win32 port though, I don't have a Windows machine to develop on, maybe with Wine on Linux? Do you know if it emulates the Win32 Console API and behaviour accurately?

Apart from that, I've been thinking about my use cases, and I don't think a simple REPL is going to be useful in practice. For example the output of "desc" if often very long, which means that having a good pager is necessary to navigate and search into it.
I also frequently write query outputs to files to compare them later with diffing tools. All of this can be done easily with the initial proposal, and becomes difficult with a REPL, or it means requiring a REPL with a much more advanced interface that what line_input would provide :-/

So I still prefer the client/server mode after all.
 

[K. Moon]

I'm very leery of the complexity of spawning a server on different platforms (for example, I think Bazel has a limitation where it locks the tree to one instance), but I think the language server model (like clangd's) would be a pretty good one to follow. (I think this corresponds to the "daemon mode" proposal; I/O is over pipes to stdin/stdout, which gets rid of some of the complexity of standalone servers.)

For extra credit, you could even imagine having formal LSP support at some point for GN files (although I'm not sure how useful that would be in practice, and probably wouldn't support the kind of queries we're talking about).

As an experiment, I think you could probably build a separate query server on top of GN's current JSON output, and see how well that works out.

[Aaron Wood]

I took a similar route, and used the `gx desc --format json` output in a separate tool (written in Rust).  It doesn't handle multiple queries, but it loads a couple-hundred MB json file (from the entire Fuchsia build graph) in ~4 seconds.  But I plan on adding a repl-like interface, where you load the file, and then run what are currently CLI commands against it and see the results.

[Aaron Wood]

David, here's the tool:
https://fuchsia-review.googlesource.com/c/fuchsia/+/482000

I wrote this while trying to make heads/tails of the image assembly steps in the build graph, because of both the complexity of the steps, and the number of templates.  The CLs are in a bit of a sorry state, but I want to dust them off and get them landed, soonish.

-Aaron

[Aaron Wood]

On the subject of LSPs, I'd love for the GN LSP that's currently available for VSCode to have more semantic information.  It has syntax, but it can't do symbol lookup, especially across files, and it would be so, so, nice to have the ability to inspect the values of GN args (based on the configured outdir), and be able to find where variables are declared, when dealing with templates (and the ability to jump to a template declaration from it's point of use).

[Brett Wilson]

This tool sounds useful but I think it would have a much greater impact if this feature was built-in. This would be another way to solve David's use-case without the client/server thing that sounds like it would be more useful and could be even easier to implement.

I'm a little uncomfortable duplicating the GN introspection commands in a separate tool because it sounds like a lot of work and there will be different kinds of mismatches and version skew. Serializing the full build graph isn't trivial but it's not *that* much work either. And then you get all of the existing commands available instantly.

As a first pass, we could have a way to dump this (a flag to "desc"?) and then a flag to use the serialized version to use with the different introspection commands. As a second phase we could hook up dirty management and automatically use/regenerate the file as-needed. In v2 where it's automatic, I would not have "gn gen" generate this serialization automatically because it will slow down the build. Instead, the introspection commands could dump this out so you pay the cost for the first one you use but subsequent runs are fast.

Brett

On Fri, Jan 21, 2022 at 4:56 PM Aaron Wood <aaro...@google.com> wrote:

[David Turner]

For the record, I'm still slowly exploring the query/REPL ideas. I found that GN commands will peek directly into base::CommandLine to see if switches like --print or --as=TYPE are being used.
This makes them difficult to use with a REPL, where each invocation would need a different set of switches, so I've uploaded to CLs to work-around this with minimal impact on the source code.
https://gn-review.googlesource.com/c/gn/+/12862
https://gn-review.googlesource.com/c/gn/+/12863

I also have a a setup where I can use clang-cl and lld-link to generate Windows binaries on my Linux workstation, and run them with Wine. Will try to see how easy porting the line_input library will be.

[Brett Wilson]

Thanks, I think my preferred solution would actually be loading/saving the serialized build graph in GN itself. It could be a similar amount of work (or maybe a little more) and might be more useful long-term depending on future plans.

Brett