why networkx requires so much memory?

[liu lily]

I have a ajacency list file with 3.65 million nodes which compose a huge network, and then I run the following simple script:


  import networkx as nx
  import sys
  G = nx.read_adjlist(sys.argv[1]+'.adj')
  print nx.number_of_nodes(G)

I notice the script occupies up to 5 GB memory, namely around 14 KB per node.
Why does networkx require so much memory? or in another way, is networkx efficient on memory usage over large networks?

[Aric Hagberg]

Don't forget to count the edges. I'm guessing you have at least as
many as nodes.
So, yes, it may take 5GB of memory. You could use as a rough estimate
about 100bytes per node or edge - the exact size depends on what you
use as a node or store with an edge.

Aric

> --
> You received this message because you are subscribed to the Google Groups
> "networkx-discuss" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to networkx-discu...@googlegroups.com.
> To post to this group, send email to networkx...@googlegroups.com.
> Visit this group at http://groups.google.com/group/networkx-discuss.
> For more options, visit https://groups.google.com/d/optout.

[liu lily]

1 I don't understand this sentence "the exact size depends on what you use as a node or store with an edge. ", what do you mean?

2 yes, it has around 7.8 million edges, so altogher 3.6+7.8 million
so it is around 440 Bytes per node or edge
I'm wondering why each node or edge need 440 Bytes memory?

[Aric Hagberg]

You can store arbitrary (Python hashable) objects as nodes. Integers
will use less memory than long strings or other complicated objects.
Also you can store arbitrary data with edges so the size of that data
matters too. The default data for an edge is an empty dictionary
which is 136 bytes on my machine.

Aric

[Moritz Beber]

Hey,

> 2 yes, it has around 7.8 million edges, so altogher 3.6+7.8 million
> so it is around 440 Bytes per node or edge
> I'm wondering why each node or edge need 440 Bytes memory?
>

This is partially due to the dict of dict structure used by networkx. Maybe this post here is interesting for you http://graph-tool.skewed.de/performance ?