Re: [networkx-discuss] NetworkX for large graphs about 2 Million Nodes and 400 Million edges
1,178 views
Skip to first unread message
Aric Hagberg
Feb 2, 2013, 8:08:33 PM2/2/13
to networkx...@googlegroups.com
On Sat, Feb 2, 2013 at 5:58 PM, Vivek Kulkarni <vive...@gmail.com> wrote:
> Hi,
> Can some one guide me on whether networkx is suitable for a graph on 2
> million nodes and ~400 million edges. What order of nodes and edges is
> typically supported by networkx. I only intend to load the graph and run
> pagerank. I believe page rank should be efficient (perhaps run in the number
> of edges). I do have access to server with 128GB RAM and an Intel Xeon CPU
> with 6 cores. Also any suggestions on how to improve memory footprint would
> be helpful. I am already using integers as node labels and also building the
> graph in 2 pieces (to keep memory in check). I intended to have float edge
> weights but soon saw that I was running out of memory. So had to discard the
> edge weights.
If you have enough memory it would work. But it could be that 128GB
is not enough.
If all you want to do is calculate PageRank it is a linear algebra
problem and you don't need all of the NetworkX machinery. If you want
to use Python you could adapt the NetworkX pagerank_scipy() algorithm
and just load the SciPy sparse matrix directly with your data.
Aric
> Hi,
> Can some one guide me on whether networkx is suitable for a graph on 2
> million nodes and ~400 million edges. What order of nodes and edges is
> typically supported by networkx. I only intend to load the graph and run
> pagerank. I believe page rank should be efficient (perhaps run in the number
> of edges). I do have access to server with 128GB RAM and an Intel Xeon CPU
> with 6 cores. Also any suggestions on how to improve memory footprint would
> be helpful. I am already using integers as node labels and also building the
> graph in 2 pieces (to keep memory in check). I intended to have float edge
> weights but soon saw that I was running out of memory. So had to discard the
> edge weights.
If you have enough memory it would work. But it could be that 128GB
is not enough.
If all you want to do is calculate PageRank it is a linear algebra
problem and you don't need all of the NetworkX machinery. If you want
to use Python you could adapt the NetworkX pagerank_scipy() algorithm
and just load the SciPy sparse matrix directly with your data.
Aric
Vivek Kulkarni
Feb 3, 2013, 9:02:09 AM2/3/13
to networkx...@googlegroups.com
Thanks Aric, One more question. Do you know if the scipy can handle graphs of order of billion edges. I have a larger graph of 2 million nodes and about a billion edges. I was just wondering if the scipy sparse implemenation supports such large matrices. Again, all I need is page rank.
Thanks.
Vivek.
Thanks.
Vivek.
Aric Hagberg
Feb 3, 2013, 9:14:57 AM2/3/13
to networkx...@googlegroups.com
On Sun, Feb 3, 2013 at 7:02 AM, Vivek Kulkarni <vive...@gmail.com> wrote:
> Thanks Aric, One more question. Do you know if the scipy can handle graphs
> of order of billion edges. I have a larger graph of 2 million nodes and
> about a billion edges. I was just wondering if the scipy sparse
> implemenation supports such large matrices. Again, all I need is page rank.
The only limitation is how is how much memory you have. So however
> Thanks Aric, One more question. Do you know if the scipy can handle graphs
> of order of billion edges. I have a larger graph of 2 million nodes and
> about a billion edges. I was just wondering if the scipy sparse
> implemenation supports such large matrices. Again, all I need is page rank.
much it takes to store 10^9 edges is what you need.
Aric
Reply all
Reply to author
Forward
0 new messages