Re: DXR on FreeBSD

[Erik Rose]

> I have been flooding IRC about this, but just wanted to mention for
> anyone lurking here that I have DXR working on FreeBSD, and am working
> on indexing and analyzing the full FreeBSD source tree (kernel +
> userland.)

I am watching your exploits with rapt interest. :-) And thanks for all the great front-end patches you've been flinging over the wall!

> FreeBSD 10.0 uses Clang 3.3 by default, so I haven't really had any
> problems on that front (well except that I need to compile the sqlite
> trilite extension with clang which does not like a debug macro, which
> is easy to hack around) - I hear trilite going away soon anyway.

Yep. Every day we chip away at one more blocker on the road to elasticsearch. We're almost there: https://bugzilla.mozilla.org/show_bug.cgi?id=1030307. It'll give us easier contribution (as you can see), line-based searching (a la grep) rather than our current weird behavior, a path to easy tree independence and parallel indexing, and it'll make a lot of requested fruit hang lower.

> I am tracking everything I had to change and current status in:
> https://gist.github.com/rhelmer/60bc81c6cee9c507008a

Looking forward to the patches!

> The only problems specific to DXR so far have been:
>
> * "_csv.Error: field larger than field limit (131072)" after build. I
> am testing to see if "csv.field_size_limit(sys.maxsize)" in
> clang/indexer.py helps

Wow, I wonder what was in that field.

> * compiling LLVM doesn't seem to work, I am just skipping it for now
> but as it's part of the BSD base system I'd like to get it indexed and
> analyzed along with everything else. It fails on
> llvm/tools/clang/lib/Lex/ModuleMap.cpp with "clang: error: unable to
> execute command: Illegal instruction (core dumped)" - I saved the code
> and clang command-line used, and the same clang invocation works fine
> if I remove the DXR-added flags.

Sounds like you found a legitimate bug in our clang plugin. It must be a real corner case if you got through all that FreeBSD code without hitting it. gdb time, perhaps.

> Overall I am really happy with how easy it's been to get DXR up and
> running,

That's the first time we've ever been accused of that. ;-) And you're not even using the Vagrant box!

Cheers,
Erik

[Robert Helmer]

On Mon, Oct 20, 2014 at 10:47 AM, Erik Rose <er...@mozilla.com> wrote:
>> The only problems specific to DXR so far have been:
>>
>> * "_csv.Error: field larger than field limit (131072)" after build. I
>> am testing to see if "csv.field_size_limit(sys.maxsize)" in
>> clang/indexer.py helps
>
>
> Wow, I wonder what was in that field.

(per IRC) so that did help, and I uploaded the CSV to a github gist:
https://gist.github.com/rhelmer/c3a2a2defcdb98ed2152

So is this just an insane type? I am not sure if it's a bug... do you
want a patch
to bump up the CSV field_size_limit? Maybe a compromise would be to make
it configurable...

>> * compiling LLVM doesn't seem to work, I am just skipping it for now
>> but as it's part of the BSD base system I'd like to get it indexed and
>> analyzed along with everything else. It fails on
>> llvm/tools/clang/lib/Lex/ModuleMap.cpp with "clang: error: unable to
>> execute command: Illegal instruction (core dumped)" - I saved the code
>> and clang command-line used, and the same clang invocation works fine
>> if I remove the DXR-added flags.
>
>
> Sounds like you found a legitimate bug in our clang plugin. It must be a
> real corner case if you got through all that FreeBSD code without hitting
> it. gdb time, perhaps.

Yep, good times :P

[Erik Rose]

> (per IRC) so that did help, and I uploaded the CSV to a github gist:
> https://gist.github.com/rhelmer/c3a2a2defcdb98ed2152
>
> So is this just an insane type?

Absolutely; that is utterly insane. Wow. That's more like a compiler stress-test than an actual type.

> I am not sure if it's a bug... do you
> want a patch
> to bump up the CSV field_size_limit? Maybe a compromise would be to make
> it configurable...

I was really hoping it was going to be a CSV-emitting bug on our side: that we missed a quotation mark or something. But since it's not, sure, I'll take a field-size-limit-increasing patch. I don't think there's really a need to make it configurable. The crazy values you're running into are 200K. If we made the limit 500K, I don't think anybody would notice the RAM hit. And, if you can do a quick test to make sure the _csv module isn't allocating a static buffer of the max size, larger would be fine as well (though we should stop short of sys.maxsize, which we were kicking around earlier; that's around 7 exabytes ;-)).

Cheers,
Erik