Compile commands dumper

[maK]

Hi,

I am trying to use rtags (https://github.com/Andersbakken/rtags) and YouCompleteMe (https://github.com/Valloric/YouCompleteMe calles as YCM) with tup.

Basically what I need it to have simple access to commands used to compile my sources.

There was a similar thread a year ago (https://groups.google.com/d/msg/tup-users/f6N0da57tn0/bGKby5hcJkEJ) but unfortunately it is not helpful.

I was able to workarount tup by querying database file to have commands for single compilation unit to pass "-I" flags to YCM, but rtags needs a little more from build system. And this is nasty solution and I was praying that database format won't change ;-)

What I need is to have all the commands used to compile project from scratch.

I am aware that tup have command 'generate', but this requires clean source tree, so it is inadequate.

What is the most simple way to have the following:

1. Get compile command for specific file in project - so I can pass this to YCM (as YCM requires compilation flags per file, on demand).
2. Get all compilation commands for whole project - so rtags could index all my sources.

(If I wrote something unclear - please forgive me, I will be happy to explain)

[Ben Boeckel]

On Fri, Dec 18, 2015 at 14:49:16 -0800, maK wrote:
> 1. Get compile command for specific file in project - so I can pass this

> to YCM (as YCM requires compilation flags per file, on demand).

> 2. Get all compilation commands for whole project - so rtags could index
> all my sources.

It'd be nice if Tup would write something conforming to:

http://clang.llvm.org/docs/JSONCompilationDatabase.html

--Ben

[maK]

Moreover

Even the most simple way suggested by rtags to index source files is not working with tup.

The suggestion is to precede compilation command with "rc -c", for example: rc -c gcc -I... -fsomeflag -c foobar.c

I've did so by adding following lines to tup:

export CXX

CXX = `[ "$CXX" ] && echo "$CXX" || echo 'g++'`

This should cause to recompile whole project if executed like this: CXX='rc -c g++'; tup

and everywhere compiler is called via $(CXX), but then rtags indexer (rc) is not generating output files (*.o and others) so tup complain about this.

So I am out of ideas how to workaround this.

[Mike Shal]

Attached is a patch that might provide a good start for this - it uses a trimmed down version of the 'tup graph' logic. However, it dumps every command, not just compilation commands. I'm not sure if that would mess up the compilation database. Also, I'm not sure how to generate the "file" line - is that required? Maybe it could just pull it out of the command string.

-Mike

[Ben Boeckel]

On Mon, Jan 04, 2016 at 16:47:38 -0500, Mike Shal wrote:
> Attached is a patch that might provide a good start for this - it uses a
> trimmed down version of the 'tup graph' logic. However, it dumps every
> command, not just compilation commands. I'm not sure if that would mess up
> the compilation database. Also, I'm not sure how to generate the "file"
> line - is that required? Maybe it could just pull it out of the command
> string.

It is described as "the translation unit’s main file", so I'd say it
should be the %f argument of the command. It also says that it is used
by tools for searching the database, so I'd say it's fairly important.
For any command without a %f (whether used literally in the command or
not), it might be best to just skip those (i.e., any tool using this
file would probably not care about Python script commands).

--Ben

[Mike Shal]

Hmm, in that case it might be best instead to generate this at the parser level rather than trying to do it after the fact from the database. Unfortunately the context of which file was listed from "%f" may be lost - they look the same as order-only inputs in the database. Additionally, parser level dependencies ("sticky links" in the tup source & the database) are ignored on normal files, so there may not even be a link from foo.c to the compilation command until the command actually executes.

So we'd probably want something that calls tup_db_reparse_all() to force it to re-parse all Tupfiles, and then have an alternate parsing mode that spits out things into the compilation database instead of the tup database (this would probably just be a separate implementation of parser.c:do_rule()).

-Mike

[maK]

Hi Mike,

As a start it looks good on my project.

Today I don't have much time, but I'll try to integrate this solution with rtags and YCM soon and share my results.

Thanks for the patch.

Regarding "file" line - generally I agree with Ben.

[maK]

Hi Mike,

Today I tried command dumper again, but it is not working.

It prints: 

tup: Unable to find tupid for 'dump'

in couple of my projects. Where should I look to find the cause?

[maK]

Hi again,

Looks like I've did some dumb thing and used not patched version of tup last time.

With current format I've found some issues:

• whole dump is not enclosed with '[' and ']' braces, so this is not a valid json file - this is a minor
• all the commands that contains double quotation chars should be escaped by \" as those are messing up json format This also forces to to specially treat all the escaped quotations \" as escaping already escaped quotation results in \\" and this also mess the json format

And regarding the missing "file" line. Followed by suggestion I was able to re-parse "tup dump" output and obtain a valid json file with (probably) correct "file" entry. But solution is really ugly and heavily depends on compilation command syntax.

Moreover playing a while with my solution shows that it is not convenient as on every dependency change I must run "tup dump" again. This requires launching command by hand (same applies to rtags) or wrapping tup in a Makefile for example (which I happily stopped using after found tup). So I think that best solution would be to introduce new Tupfile command like:

dump_to "path to json file"

that would instruct tup to also dump commands while parsing. I am not sure if it would be good idea to do this on whole tup hierarchy level or create a new ^-flag to instruct tup, that this command should be dumped into json.

What do you think?

[Ilya Novozhilov]

Hi everyone,

Another way of providing commands list to rtags is to provide plain-text commands list. I tried changing the patched main.c so that it does exactly that, but then the problem is to substitute input file names to absolute file paths. If we go the JSON way, we need to provide input files list — the problem in both ways is exactly the same: getting raw input file names from a graph node. Does anyone knows a way to do it? I'm about to dive into the code to find it out...

For now, I solved the problem this way: tup dump prints two types of entries for me:

- C command [args...]

- F /path/to/file

Then, using a Python script, I make a set of all the input files and for each argument of each command I check if that argument is in the file list. If it is the case, then I replace it with a full path. Clumsy. And prone to collisions.

суббота, 30 января 2016 г., 9:24:18 UTC+3 пользователь maK написал: