performance of NetworkX vs R igraph

[Hugo Koopmans]

Hi,

I am about to start a pilot in analyzing Telecom CDR data.
It is a lot of data, estimate will be we need around 10^7 CDR records
to analyse and enhance our Churn dataset with SNA features to improve
the churn model.

Any idea NetworkX would be up for the job?

I know R with igraph has been used to do this kind of stuff but i do
not know which tricks have been used or are needed to finetune
performance?
Parrellell processing?
Column database?

Any direction would be great!

hugo

[tcb]

hi,

No reason not to use networkx, but since it's written in python you will certainly find better solutions for performance and (particularly) memory usage. If you have a fast machine and enough memory you might find the performance is adequate for your needs, but it will also depend heavily on which algorithms you are using (so you should try various solutions before going too far). The graph data structure in networkx is implemented on a python dictionary, which has its pros and cons (not just because its a python dict, but because some algorithms are better suited to different data structures).

Depending on the algorithms you are using you might find that you can speed parts of them up by implementing them in cython, but you might still find that the dict data structure is a bottleneck. See discussion here:
http://groups.google.com/group/networkx-discuss/browse_thread/thread/56909bc9d7230f2f/698977f43b742559

Other tools like igraph and graph-tool use a different approach of implementing the basic data structures and functions in c (igraph) and c++ (on top of boost graph library in graph-tool), with a fairly thin python binding. They certainly get better performance and (in general) are a lot more memory efficient, and I find the python interfaces are quite easy to use.

So, why use networkx? I would say the advantages over other tools are that it is (very nicely) written completely in python and well supported too. Many of the algorithms are very easy to read and modify in python (you might like to compare some of the networkx implementations to their equivalents in boost bgl or igraph). This is a huge advantage when you hit a problem or have to implement your own algorithms (quite likely if you are doing things like community finding, which is currently a weak spot in most tools). The barrier to something-which-works in networkx is generally very low- its easy to work with and easy to extend for your own use.