rspfile_content with new lines
Scott Graham
I've just run into the edge-of-edge-case with rspfile_content.
Edge case: command line too long -> move command line arguments into a
response file, via rspfile and rspfile_content.
Edge-of-edge case: rspfile_content = $in, but $in is > 131072
characters due to (very) large number of object files. link.exe errors
out if any individual line is longer than that 128k characters.
I made what I thought was an easy fix, by adding $in_newline which is
the same as $in, except that files are separated by \n rather than
space. This works fine to fix the rspfile.
Unfortunately, the build log uses \n as its separator, so the
concatenation of ;rspfile_content=<contents> results in a broken
.ninja_log as contents contains \n's. I tried using a different
character to separate entries in the build log, but then fgets isn't
appropriate so it got a bit messy to maintain backwards upgradability.
The log becomes hard to inspect in an editor too, which is a bit
annoying.
Instead, I made $in_newline use \r as a separator internally, and then
only when writing the response file to disk, replace \r with \n to
satisfy the linker's line requirement.
It's a bit hokey but seems to work and is a small patch:
https://github.com/martine/ninja/pull/268/files
Seem OK? Or, other suggestions other than "make your project smaller"? ;-)
scott
Nico Weber
rspfile_content? Then .ninja_log wouldn't contain \r characters in
edgeedge cases. I write scripts that process .ninja_log from time to
time, and I'd certainly get this case wrong.
Nico
Scott Graham
> Could $in just always separate its files by newlines when writing to
> rspfile_content?
I think it'd be a little tricky. rspfile_content can contain things
other than $in, and replace(' ', '\n') doesn't work. So, the
evaluation of rspfile_content needs context information that it
doesn't have right now. (Mostly just a larger change... but I'm sure
it'd be possible. Is it better though?)
I'm also not 100% sure on whether it's always valid to always \n
separate either. It works for cl and link, but I'm not sure if other
tools handle \n. Does whoever added rspfile use it for any other tools
that care about that?
> Then .ninja_log wouldn't contain \r characters in
> edgeedge cases. I write scripts that process .ninja_log from time to
> time, and I'd certainly get this case wrong.
How do you process them? You can still split on \n and presumably
you're not running the commands (if they're space-separated they're
not going to be runnable at that point because they'd be too long). I
could use some other character separator (rather than \r) in the log
file, if that's the concern?
Nico Weber
> On Fri, Apr 13, 2012 at 3:49 PM, Nico Weber <tha...@chromium.org> wrote:
>> Could $in just always separate its files by newlines when writing to
>> rspfile_content?
>
> I think it'd be a little tricky. rspfile_content can contain things
> other than $in, and replace(' ', '\n') doesn't work. So, the
> evaluation of rspfile_content needs context information that it
> doesn't have right now. (Mostly just a larger change... but I'm sure
> it'd be possible. Is it better though?)
Hm, good point. Could the program that generates the ninja file write
explicit (escaped) newlines? It probably has enough information to do
so.
> I'm also not 100% sure on whether it's always valid to always \n
> separate either. It works for cl and link, but I'm not sure if other
> tools handle \n. Does whoever added rspfile use it for any other tools
> that care about that?
You'll find out faster if you always have newlines instead of in
0.001% of cases :-)
But if you leave the newline insertion up to the generator, it's a
moot point (for ninja) either way.
>> Then .ninja_log wouldn't contain \r characters in
>> edgeedge cases. I write scripts that process .ninja_log from time to
>> time, and I'd certainly get this case wrong.
>
> How do you process them? You can still split on \n and presumably
> you're not running the commands (if they're space-separated they're
> not going to be runnable at that point because they'd be too long). I
> could use some other character separator (rather than \r) in the log
> file, if that's the concern?
My (minor) concern is mostly that the approach feels a bit hacky.
Nico
Scott Graham
> On Fri, Apr 13, 2012 at 4:05 PM, Scott Graham <sco...@chromium.org> wrote:
>> On Fri, Apr 13, 2012 at 3:49 PM, Nico Weber <tha...@chromium.org> wrote:
>>> Could $in just always separate its files by newlines when writing to
>>> rspfile_content?
>>
>> I think it'd be a little tricky. rspfile_content can contain things
>> other than $in, and replace(' ', '\n') doesn't work. So, the
>> evaluation of rspfile_content needs context information that it
>> doesn't have right now. (Mostly just a larger change... but I'm sure
>> it'd be possible. Is it better though?)
>
> Hm, good point. Could the program that generates the ninja file write
> explicit (escaped) newlines? It probably has enough information to do
> so.
No, unfortunately not since it's just literally "$in" as the generator
writes it. I guess I could try to expand it, and not use variables? I
don't think there's a way to get \n in a .ninja file right now though.
>> I'm also not 100% sure on whether it's always valid to always \n
>> separate either. It works for cl and link, but I'm not sure if other
>> tools handle \n. Does whoever added rspfile use it for any other tools
>> that care about that?
>
> You'll find out faster if you always have newlines instead of in
> 0.001% of cases :-)
Yeah, it's fine for "me", that was a general question to the group.
Note that I'd be using $in_newline it for all link command lines
because I can't know how long the resulting command line will be, so
it won't be infrequently used as it is.
> But if you leave the newline insertion up to the generator, it's a
> moot point (for ninja) either way.
>
>>> Then .ninja_log wouldn't contain \r characters in
>>> edgeedge cases. I write scripts that process .ninja_log from time to
>>> time, and I'd certainly get this case wrong.
>>
>> How do you process them? You can still split on \n and presumably
>> you're not running the commands (if they're space-separated they're
>> not going to be runnable at that point because they'd be too long). I
>> could use some other character separator (rather than \r) in the log
>> file, if that's the concern?
>
> My (minor) concern is mostly that the approach feels a bit hacky.
Can't disagree.
>
> Nico