Re: Making DXR a rocking OpenSource project

[Martyn Russell]

On 10/02/12 16:14, Lionel Dricot wrote:
> Hi,

Hello Lionel,

> After discussing with Taras during FOSDEM, I've put some thoughts on how
> to make DXR rock and have some appeal in the community.
>
> Currently, I see some weaknesses that may hurt DXR massive adoption.
> IMHO, this should be fixed before speaking more about DXR.
>
> 1) Indexing consume too much memory, making it impossible to use on less
> than high-end server.

I agree. But this is really a hurdle subject to the project you index at
the moment. If you use glib or not some conglomerate project like
LibreOffice then things are fine.

I think it needs tackling at some point.

> 2) DXR is too tight to mozilla-central

Do you mean "tied" ?
If so, can you elaborate?

> 3) Indexing is currently too slow (is it really a problem?)

Generally in our experience with Trackerน on embedded devices, you
either choose to optimise for querying or indexing. I think querying is
more important here.

There may also be some low hanging fruit to improve the indexing speed
here though that we should look into :)

น http://projects.gnome.org/tracker/

> 4) DXR is hard to install and to use.

I agree. I was discussing this with Carlos/Lionel on Friday. It wouldn't
be much work to make the project properly deploy-able and to have a bit
more documentation to make the process and tools easier to use for first
timers.

> 5) I still don't have a clear picture on how to use one DXR instance with multiple repositories

Well, right now AFAIU, it doesn't matter. You point DXR at a directory
and it will index it. This directory could contain all your project
interests.

I think ultimately, you don't want a separate virtual host or URL per
project, you want one for all.

> Regarding 2), we have identified the HG link and are currently trying to
> put that in a config file.

A solution that covers git and other repositories should also be
considered IMO.

> 1) and 3) looks rather critical because, for now, you need an high-end
> machine and lote of time just to test it.

Well, yes, with mozilla-central. Most people will likely try the project
with smaller projects, but we should expect some famous cases (like the
Linux Kernel) to be tried I agree.

> For 4), Carlos think that DXR should use autotools (or, I would say,
> python-distutils). That would allow easy installation and, even more,
> distribution packages. We should also make some clear shell commands :
> index, deploy. Even a small PyGTK gui might be done quickly. Anyway, lot
> of opportunity there.
>
>
> So, what are, in your opinion, the priorities of DXR as an OpenSource
> project? Should we try to make a roadmap?

I think a wiki with a roadmap makes sense so others can edit it in
place. Taras, is this possible to do here:

https://wiki.mozilla.org/DXR

?

I have some ideas too, I will come back to this email when I have put
them together.

--
Regards,
Martyn

Founder and CEO of Lanedo GmbH.

[Taras Glek]

I want debian/fedora to DXR all of their code, so we can have a
replacement for google code :)

>
>> Regarding 2), we have identified the HG link and are currently trying to
>> put that in a config file.
>
> A solution that covers git and other repositories should also be
> considered IMO.
>
>> 1) and 3) looks rather critical because, for now, you need an high-end
>> machine and lote of time just to test it.
>
> Well, yes, with mozilla-central. Most people will likely try the project
> with smaller projects, but we should expect some famous cases (like the
> Linux Kernel) to be tried I agree.
>
>> For 4), Carlos think that DXR should use autotools (or, I would say,
>> python-distutils). That would allow easy installation and, even more,
>> distribution packages. We should also make some clear shell commands :
>> index, deploy. Even a small PyGTK gui might be done quickly. Anyway, lot
>> of opportunity there.
>>
>>
>> So, what are, in your opinion, the priorities of DXR as an OpenSource
>> project? Should we try to make a roadmap?
>
> I think a wiki with a roadmap makes sense so others can edit it in
> place. Taras, is this possible to do here:
>
> https://wiki.mozilla.org/DXR

Above wiki has a specific structure for Malini and other build&release
people.

Feel free to take over https://wiki.mozilla.org/DXR_Future_Work_Plan .


As long as we have a really appealing dxr.mozilla.org(current iteration
shows potential, but is not yet very appealing) and have a sufficiently
sane deployment story contributors will help us with the rest the
"papercut" issues discussed above.

Taras

[Martyn Russell]

On 10/02/12 16:14, Lionel Dricot wrote:
> Hi,

Hi,

> After discussing with Taras during FOSDEM, I've put some thoughts on how
> to make DXR rock and have some appeal in the community.
>
> Currently, I see some weaknesses that may hurt DXR massive adoption.
> IMHO, this should be fixed before speaking more about DXR.
>
> 1) Indexing consume too much memory, making it impossible to use on less
> than high-end server.

> 2) DXR is too tight to mozilla-central

> 3) Indexing is currently too slow (is it really a problem?)

> 4) DXR is hard to install and to use.

> 5) I still don't have a clear picture on how to use one DXR instance with multiple repositories

Just some ideas I've been thinking of:

* Perhaps choose on one license for the project, currently 4 or so are
used.

* Do frequent releases and have versions. Right now there is no version
for DXR AFAICS.

* Have an easy method to automate indexing projects from the website
for people to try locally and/or on dxr.{lanedo|mozilla}.com. This
may be tricky for projects which have tough build requirements.

* One "dxr" command, this replaces "dxr-index".

* Clean up of the dxr code base, there seems to be a lot of png files
imported from some theme which we don't need. That would help with
deployment and packaging.

* Atomic update support in config/dxr command.

* The "dxr" command covers:

* entering a shell i.e.
dxr --shell

* creating a config (XDG locations, ~/.config/dxr/$profile.config) i.e.
dxr --create-config "foo" $2 $3 ...

* testing a setup is working by profile (e.g. do a HTTP request) i.e.
dxr --test

* add web infrastructure to web server location (in config file) i.e.
dxr --deploy

* add or update index i.e.
dxr --update or dxr --index

* Documentation via a man page for all this so people know how to use
it properly!

* Update the theme/look and feel. The usability could use some love too.

* Refactoring tool?

* Committers/Authors tool? Way to find out which businesses and who
from them is working on the code base. This would help people find
companies that could be offering services around those projects.

* Language limitations (support Java with gcj?)

* No property support (for languages like C#, etc), other languagisms

* Reduce memory footprint required (goal: index LO with dxr)

* Improve indexing time needed

* Provide more statistics about projects perhaps

* Perhaps some status information about the indexing progress for larger
deployments?

--

We should put these on the roadmap.

[Lassi Tuura]

Hi,

> So, what are, in your opinion, the priorities of DXR as an OpenSource
> project? Should we try to make a roadmap?

I'm interested in using DXR, and have spent a little time playing with
it to see if I could install and use it. So far I didn't succeed quick
enough and gave up each time; here's why.

1) The setup-env.sh assumes you use bash. Of course I dropped into bash
to get anywhere to play with it, but seems an unnecessary hurdle.

2) I know it says Linux only for now, but I am mostly playing on OS X.
I'm very comfortable porting large bits of software and would be
happy to contribute the necessary changes, but I actually never got
far enough to contribute anything platform-specific.

3) For the life of mine I can't figure out how to build clang/llvm
for which I could actually build the DXR plugin, let alone use it.
I've tried for example using MacPorts llvm/clang installations, but
they don't appear to install all the parts needed. The instructions
just say "cxx-clang: A copy of clang". $(llvm-config --obj-root)
or the headers/libraries DXR seems to want didn't survive in any
of the more or less obvious install targets I tried.

4) The mixed env setup scripts + python + makefile fragments to
figure out which plugins are enabled is, ahem, creative but
perhaps over-complicated. It's certainly a hurdle when trying
to figure out why the install isn't working.

The (3) is where I given up so far each time I've tried. I'm comfortable
enough working with software, including porting, including multi-million
line code base and all sorts of weird build systems. I'd say DXR at this
point decidedly qualifies as "hard to install". In comparison, I found
dehydra easier to set up to build, even with figuring out how to build
the javascript interpreter it required.

Perhaps at some point I'll find the energy to push through to figure out
the installation permutation which works, but an exact build stanza,
even if it was just an example, would help tremendously. I'm definitely
interested in DXR, but before worrying about ease of use or speed, I
first need to be able to build it :-) Don't underestimate the value of
precise step-by-step of building the whole lot from ground up, including
all the required dependencies!

Regards,
Lassi

[Ehsan Akhgari]

Building clang is actually really easy. Have you tried the instructions
here? http://clang.llvm.org/get_started.html

Cheers,
--
Ehsan
<http://ehsanakhgari.org/>

> _______________________________________________
> dev-static-analysis mailing list
> dev-stati...@lists.mozilla.org
> https://lists.mozilla.org/listinfo/dev-static-analysis
>

[Jean-Marc Desperrier]

Ehsan Akhgari a écrit :

> Building clang is actually really easy. Have you tried the instructions
> here?http://clang.llvm.org/get_started.html

The purpose is not to build clang, but get to the "I have something
working" poitn as fast and as easily as possible.

This page has a downloadable MacOS X/x86-64 build, I hope there's no
hidden dependencies :
http://llvm.org/releases/download.html#3.0

Clang Binaries for MacOS X/x86-64 :
http://llvm.org/releases/3.0/clang+llvm-3.0-x86_64-apple-darwin11.tar.gz

[Lionel Dricot]

I agree with Lassi that we should have some kind of "copy/paste"
instruction set. That would be really helpful.

Le 24/02/12 21:25, Ehsan Akhgari a écrit :

> Building clang is actually really easy. Have you tried the instructions

[Lionel Dricot]

Hi,

I would like to take this remark from Martyn Russell:

> * Perhaps choose on one license for the project, currently 4 or so are
> used.
>

How can we improve the license situation. In an ideal world, the project
would be under only one license. Is that doable? Which license should be
choosen?

I believe it is something we really need to improve.


Lionel

[Martyn Russell]

On 05/03/12 15:01, Lionel Dricot wrote:
> Hi,

I agree. Right not it's a bit confusing what license the project is.

In other Open Source projects, we've done this to make it easier for
people wanting to adopt it in their custom environments.

To give a quick summary of the current situation for DXR, currently
we're using using:

- MPLv2
- CCv2.5
- APLv2.0

But officially, the LICENSE file says the project follows the MIT license.

So,

1. Is MIT intended (not MPL)? If not, what is? AGPL?
2. Can we change the other code portions with the consent of the authors?

Thoughts?

[Martyn Russell]

On 01/03/12 18:17, Jean-Marc Desperrier wrote:
> Ehsan Akhgari a écrit :
>> Building clang is actually really easy. Have you tried the instructions
>> here?http://clang.llvm.org/get_started.html
>
> The purpose is not to build clang, but get to the "I have something
> working" poitn as fast and as easily as possible.

I couldn't agree more. It's something we're trying to focus on right now.

> This page has a downloadable MacOS X/x86-64 build, I hope there's no
> hidden dependencies :
> http://llvm.org/releases/download.html#3.0
>
> Clang Binaries for MacOS X/x86-64 :
> http://llvm.org/releases/3.0/clang+llvm-3.0-x86_64-apple-darwin11.tar.gz

Actually, from what I understand clang's official release of 3.0 is not
the version we're using for dxr.{lanedo|mozilla}.org and the output is
different too from our testing. Lionel/Christian correct me if I am
wrong here.

[Joshua Cranmer]

On 3/5/2012 10:30 AM, Martyn Russell wrote:
> I agree. Right not it's a bit confusing what license the project is.
>
> In other Open Source projects, we've done this to make it easier for
> people wanting to adopt it in their custom environments.
>
> To give a quick summary of the current situation for DXR, currently
> we're using using:
>
> - MPLv2
> - CCv2.5
> - APLv2.0
>
> But officially, the LICENSE file says the project follows the MIT
> license.
>
> So,
>
> 1. Is MIT intended (not MPL)? If not, what is? AGPL?
> 2. Can we change the other code portions with the consent of the authors?

The original code written by David Humphrey and myself was intended to
be released under the MIT license.
The icon set is CC BY 2.5, which (as I understand) is effectively
similar to the MIT/3-clause BSD licenses.
Dojo and ply are both 3-clause BSD licenses, which (as I understand) are
effectively the same as the MIT license.

[Lionel Dricot]

Le 05/03/12 17:32, Martyn Russell a écrit :

> On 01/03/12 18:17, Jean-Marc Desperrier wrote:

>> Ehsan Akhgari a écrit :
>>> Building clang is actually really easy. Have you tried the instructions
>>> here?http://clang.llvm.org/get_started.html
>>
>> The purpose is not to build clang, but get to the "I have something
>> working" poitn as fast and as easily as possible.
>

> I couldn't agree more. It's something we're trying to focus on right now.
>

>> This page has a downloadable MacOS X/x86-64 build, I hope there's no
>> hidden dependencies :
>> http://llvm.org/releases/download.html#3.0
>>
>> Clang Binaries for MacOS X/x86-64 :
>> http://llvm.org/releases/3.0/clang+llvm-3.0-x86_64-apple-darwin11.tar.gz
>

> Actually, from what I understand clang's official release of 3.0 is not
> the version we're using for dxr.{lanedo|mozilla}.org and the output is
> different too from our testing. Lionel/Christian correct me if I am
> wrong here.
>

You are right. At Lanedo, we didn't worked on it anymore because Taras
told it was not a priority before the summer when Joshua would work on it.

Lionel