Static inspection in constant memory

[Michał Kwiatkowski]

Hi,

I've just completed a feature that I've been working on for the last
two weeks. In short, it allows Pythoscope to analyze projects of
practically any size and number of modules. I achieved that by
decoupling modules' ASTs from the rest of Pythoscope data and storing
them separately, in their own pickle files. During test generation
only a handful (exactly one in most cases) of ASTs are modified, so
there's no need to keep all of them in memory at once.

I successfully tested the feature on the Leo editor source code, which
has 320k LOC and more than 10Mb of source code in total.

For those willing to give it a try:

bzr branch lp:~ruby/pythoscope/memory-profiling

Note that you'll have to reinitialize your dot pythoscope directory if
you used other version of pythoscope before.

There are still some corner cases to check, but I hope to merge this
branch into trunk this week.

Cheers,
mk

[Paul Hildebrandt]

This is great.  This was stopping us from using it on large code bases here. 

As far as reinitializing the .pythoscope project directory maybe it would be good to put a version number in the directory, maybe a .__version__ file or something like that.  This way Pythoscope could would notice that the project directory was incompatible and automatically recreate it.   This will hopefully lead to less confusion when upgrading to incompatible projects. 

While I am on the subject of project directories, I still cling to the idea that Pythoscope should automatically create it instead of the user having to issue a separate pythoscope --init command.   It would make editor integration a little easier.

Paul

--

[Michał Kwiatkowski]

On Tue, Mar 24, 2009 at 5:42 AM, Paul Hildebrandt
<Paul.Hil...@disneyanimation.com> wrote:
> This is great.  This was stopping us from using it on large code bases
> here.
>
> As far as reinitializing the .pythoscope project directory maybe it would be
> good to put a version number in the directory, maybe a .__version__ file or
> something like that.  This way Pythoscope could would notice that the
> project directory was incompatible and automatically recreate it.   This
> will hopefully lead to less confusion when upgrading to incompatible
> projects.

Good idea, and already registered as a blueprint:
https://blueprints.launchpad.net/pythoscope/+spec/keep-pythoscope-version-in-pickle-file

> While I am on the subject of project directories, I still cling to the idea
> that Pythoscope should automatically create it instead of the user having to
> issue a separate pythoscope --init command.   It would make editor
> integration a little easier.

The problem remains - how to figure out which directory is the top
directory of a project (in other words, into which directory we should
put dot pythoscope). One possible hint is the presence of setup.py
file, but not all projects have them.

Cheers,
mk