Proposal: A ninja tool to list out all inputs for a given target

277 views
Skip to first unread message

Nicolas Arciniega

unread,
Jan 9, 2020, 7:34:23 AM1/9/20
to ninja-build
Hello,

I'm working on a Chromium change that would benefit from being able to list out all inputs for a given target. I've included full details for the change if that's interesting.

Today, there exists the "commands" tool that spits out all commands required to build a given target(s). I'd like to do basically the same thing, but only printing out inputs. I've got this working locally and I think it'll be a fairly simple change. Would this be a tool that would be interesting to others? I'm happy to put together a pull request for it.

More context on the Chromium Change:

I'm working on an update to the Auditor tool for Network Traffic Annotation. In short, these annotations are meant to clearly describe the intent of each network call going through the Chromium network stack. The auditor tool ensures each annotation is correctly formatted and documents existing annotations. Today, the auditor tool does not audit test files. We'd like to be able to audit test files and, more generally, all files that don't ship in the product so that we can we can ensure that these only use the "TEST" annotation. This will help make sure that the documentation compiled for annotations is completely accurate.

In order to accomplish this, we need to figure out a way to identify which files don't ship with the product. One way of achieving this would be to create a ninja tool that tells us each input in the "shipped_binaries" target. Each source file that is not in the output of this new ninja tool would be identified as a file that does not ship in the product.

Thanks,
Nicolas

Charles Nicholson

unread,
Jan 9, 2020, 10:18:57 AM1/9/20
to Nicolas Arciniega, ninja-build
We have an appetite for this functionality as well, though it wasn't immediately obvious to us if it should be a ninja or GN feature (I don't think we have a strong preference either way, so pushing it lower seems more generally useful). It's very helpful for medical device regulations and compliance to have automation answer the question of "which modules are covered by which tests?". Of course that alone doesn't answer the question but it's a very helpful starting point, and good supporting evidence.

Charles

--
You received this message because you are subscribed to the Google Groups "ninja-build" group.
To unsubscribe from this group and stop receiving emails from it, send an email to ninja-build...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/ninja-build/1033bedc-2102-4af5-b539-7c5829e868b2%40googlegroups.com.

Evan Martin

unread,
Jan 9, 2020, 7:06:17 PM1/9/20
to nicho...@verily.com, Nicolas Arciniega, ninja-build
In the limit, Ninja doesn't know all the files a given command uses -- e.g. the C compiler/linker will search all over your disk looking for header files and libraries.  At the other extreme, I suspect your build system knows more than Ninja about which files a given command uses.  Those facts together suggest to me that Ninja might not be the right layer for this feature, but I haven't thought about it too hard.

Dirk Pranke

unread,
Jan 13, 2020, 3:23:08 PM1/13/20
to Nicolas Arciniega, Charles Nicholson, ninja-build, Evan Martin
In addition to the limitation that Evan mentions (about the unmentioned header files and libraries), Ninja may not know about files that are "runtime dependencies" in GN's sense (files that are needed at runtime but not necessarily compiled).

This suggests to me that you'd be better off building the functionality into GN rather than Ninja, at least at first.

You might have an issue with GN in that it may miss some header files and other dependencies that should've been mentioned (and that will get picked up by the compiler), but, at least as far as Chromium's build is concerned, I would address that issue by adding the header files and adding checks to fail when they are missed, rather than trying to work around the tool by implementing the check in Ninja instead.

-- Dirk

Mathias Stearn

unread,
Jan 14, 2020, 8:04:11 AM1/14/20
to Dirk Pranke, Nicolas Arciniega, Charles Nicholson, ninja-build, Evan Martin
I think ninja should be the source of truth about what was actually included by a build since it is the only thing that knows about discovered dependencies. This will be a bigger problem in C++20 with modules where there will be much more leaning on dependencies discovered during the build phase. The metabuild system doesn't really have any way of knowing about dynamic dependencies without either querying ninja or intercepting build phases to extract data. Given that ninja already has the info, it seems best to expose it directly.

Dirk Pranke

unread,
Jan 14, 2020, 11:38:01 AM1/14/20
to Mathias Stearn, Nicolas Arciniega, Charles Nicholson, ninja-build, Evan Martin
You are correct about this, and I don't have any objection to adding the feature to ninja; it seems like it would be useful.

However, my point was that it may not really accomplish what Nicolas is trying to do in that specific case. I don't know offhand whether any runtime dependencies will meet the criteria, but if there are some, ninja will miss them.

-- Dirk

Nicolas Arciniega

unread,
Jan 16, 2020, 10:14:27 AM1/16/20
to ninja-build
I think that in my case, I don't have to worry too much about unmentioned header files, libraries, or runtime dependencies. The only thing this feature must know about are files that specifically get compiled in the Chromium project. So, I suspect Ninja can provide me all the information I need. It's good to know that these limitations exist, though, and I think it makes sense to add a disclaimer to the tool explaining them.

I'm glad such a tool would be acceptable! I did see an existing pull request for an inputs tool:

I had prototyped a different version of the tool where, similar to the "commands" tool, I simply traverse each Edge and GetBinding("in_newline") on each Edge to get the inputs. I think I prefer this approach since the underlying code can correctly populate what it defines as inputs instead of trying to do this myself, but I'm not entirely certain if that's overkill. What are your thoughts on how best to implement this tool?
To unsubscribe from this group and stop receiving emails from it, send an email to ninja...@googlegroups.com.

--
You received this message because you are subscribed to the Google Groups "ninja-build" group.
To unsubscribe from this group and stop receiving emails from it, send an email to ninja...@googlegroups.com.

--
You received this message because you are subscribed to the Google Groups "ninja-build" group.
To unsubscribe from this group and stop receiving emails from it, send an email to ninja...@googlegroups.com.

--
You received this message because you are subscribed to the Google Groups "ninja-build" group.
To unsubscribe from this group and stop receiving emails from it, send an email to ninja...@googlegroups.com.

Nicolas Arciniega

unread,
Jan 16, 2020, 10:14:27 AM1/16/20
to ninja-build
For the purposes of the feature I'm working on, I believe that what Ninja provides should be sufficient. I don't need to strictly know every single external library that is used to compile Chromium. I just need to know which original files in the Chromium project are used to build a given target or targets. Is Ninja able to provide this or have I missed something?

It's true that the build system (GN) knows more about which files are used for what targets. That is an option I've explored, but the limitation I've hit there has been performance. GN is slow at figuring out what files are used in which targets whereas Ninja is very fast. This is the main motivation for me to pursue a new tool in Ninja.


On Thursday, January 9, 2020 at 4:06:17 PM UTC-8, Evan Martin wrote:
In the limit, Ninja doesn't know all the files a given command uses -- e.g. the C compiler/linker will search all over your disk looking for header files and libraries.  At the other extreme, I suspect your build system knows more than Ninja about which files a given command uses.  Those facts together suggest to me that Ninja might not be the right layer for this feature, but I haven't thought about it too hard.

On Thu, Jan 9, 2020 at 7:18 AM Charles Nicholson <nicho...@verily.com> wrote:
We have an appetite for this functionality as well, though it wasn't immediately obvious to us if it should be a ninja or GN feature (I don't think we have a strong preference either way, so pushing it lower seems more generally useful). It's very helpful for medical device regulations and compliance to have automation answer the question of "which modules are covered by which tests?". Of course that alone doesn't answer the question but it's a very helpful starting point, and good supporting evidence.

Charles

On Thu, Jan 9, 2020 at 7:34 AM 'Nicolas Arciniega' via ninja-build <ninja...@googlegroups.com> wrote:
Hello,

I'm working on a Chromium change that would benefit from being able to list out all inputs for a given target. I've included full details for the change if that's interesting.

Today, there exists the "commands" tool that spits out all commands required to build a given target(s). I'd like to do basically the same thing, but only printing out inputs. I've got this working locally and I think it'll be a fairly simple change. Would this be a tool that would be interesting to others? I'm happy to put together a pull request for it.

More context on the Chromium Change:

I'm working on an update to the Auditor tool for Network Traffic Annotation. In short, these annotations are meant to clearly describe the intent of each network call going through the Chromium network stack. The auditor tool ensures each annotation is correctly formatted and documents existing annotations. Today, the auditor tool does not audit test files. We'd like to be able to audit test files and, more generally, all files that don't ship in the product so that we can we can ensure that these only use the "TEST" annotation. This will help make sure that the documentation compiled for annotations is completely accurate.

In order to accomplish this, we need to figure out a way to identify which files don't ship with the product. One way of achieving this would be to create a ninja tool that tells us each input in the "shipped_binaries" target. Each source file that is not in the output of this new ninja tool would be identified as a file that does not ship in the product.

Thanks,
Nicolas

--
You received this message because you are subscribed to the Google Groups "ninja-build" group.
To unsubscribe from this group and stop receiving emails from it, send an email to ninja...@googlegroups.com.

--
You received this message because you are subscribed to the Google Groups "ninja-build" group.
To unsubscribe from this group and stop receiving emails from it, send an email to ninja...@googlegroups.com.

Nicolas Arciniega

unread,
Jan 16, 2020, 10:14:27 AM1/16/20
to ninja-build
It appears there's an existing pull request for this feature:

Is this something that can be looked at in more detail? I left some code review comments.

Nicolas Arciniega

unread,
Feb 4, 2020, 11:34:56 AM2/4/20
to ninja-build
It seems like people are generally receptive to adding this tool to ninja, so I've created a pull request for your consideration:
Reply all
Reply to author
Forward
0 new messages