visualizing big graph structures

[Aaron Swartz]

Thanks to a project from the get.theinfo list, I now have a graph of
the Amazon similarity database, comprising around 700K books. It's of
the form

X similarTo Y
X similarTo Z
...

where X and Y and Z are ISBNs. Does anyone know any good tools for
visualizing big graph structures like these? My experience is that
pumping things this big through something like graphviz is a waste of
time.

[Alexandre Rafalovitch]

Why is graphviz a waste of time? Is it the problem with visualization
of large sets or with processing time. If it is the former, project
such as ZGRViewer may help
(http://zvtm.sourceforge.net/zgrviewer.html). Otherwise, there is
Pajek (http://vlado.fmf.uni-lj.si/pub/networks/pajek/), but I have not
worked with it too much.

Regards,
Alex.

--
Personal blog: http://blog.outerthoughts.com/
Research group: http://www.clt.mq.edu.au/Research/

[alex kessinger]

Yea what I have noticed is that the larger your datasets get the more like rats nests they look like. I think if their was something that was more like a node graph exploerer and less like a one shot one time image generator. Although I can say that I like the dot file format for storring node graphs.

On Jan 18, 2008 12:41 PM, Alexandre Rafalovitch <araf...@gmail.com> wrote:

Why is graphviz a waste of time? Is it the problem with visualization
of large sets or with processing time. If it is the former, project
such as ZGRViewer may help

( http://zvtm.sourceforge.net/zgrviewer.html). Otherwise, there is

[Alf Eaton]

GraphViz and aiSee <http://www.aisee.com/> could still work for laying out the large graph - the problem is interacting with it. I wish there was a Flash viewer for dot files that would keep the overall graph layout but selectively display only a part of it at a time.

The TouchGraph Amazon Browser <http://www.touchgraph.com/TGAmazonBrowser.html> is a visualisation of the Amazon similarity data, but loads data dynamically so doesn't keep the overall graph structure.

alf

[eisen...@gmail.com]

I agree that graphviz wouldn't do a great job with so many nodes, but
I wonder whether you'd gain a lot of insight by trying to draw the
entire graph. If I had to guess, I'd bet that there are many cohesive
sub-groups/clusters within your data. Spy novels are probably most
similar to other spy novels, visualization books to other
visualization books. Instead of trying to graph every node, you could
try to identify these clusters in your data and then graph the
relationships between the clusters. Within each cluster, you could
graph the relationships between sub-clusters, and so forth as you zoom
in.

Clustering algorithms are usually pretty computationally intensive,
but I saw an amazing new one in Science a year or two ago (with Matlab
code). I can dig up the reference if you're interested.

Noah

[Ed Summers]

Just ran across this article reviewing various approaches to
large-graph visualization ...

http://www.mkbergman.com/?p=414

I imagine some of you may have run across it as well, but I figured it
was worth mentioning on this thread.

//Ed

[paolo massa]

The following is an interesting paper for an historical overview of
network visualization. However it does not cover too much big graphs.

Visualizing Social Networks
Linton C. Freeman
http://www.cmu.edu/joss/content/articles/volume1/Freeman.html

ABSTRACT: This paper documents the use of pictorial images in social
network analysis. It shows that such images are critical both in
helping investigators to understand network data and to communicate
that understanding to others.
The paper reviews the long history of image use in the field. It
begins with illustrations of the earliest hand-drawn images in which
points were placed by using ad hoc rules. It examines the development
of systematic procedures for locating points. It goes on to discuss
how computers have been used to actually produce drawings of networks,
both for printing and for display on computer screens. Finally, it
illustrates some of the newest procedures for producing web-based
pictures that allow viewers to interact with the network data and to
explore their structural properties.

--
Paolo Massa
Email: ma...@fbk.eu
Blog: http://gnuband.org

[Philip (flip) Kromer]

This thread is stale, so I apologize for the necromancy. A question
about exploring networks over on get.theinfo list made me want to
mention that a) I summarized this thread on the theinfo.org wiki
http://theinfo.org/view/tools
and b) if you haven't looked at the review
(http://www.mkbergman.com/?p=414) Ed Summers linked to, the correct
answer to "Visualizing Big Graph Structures" appears to be Cytoscape:
http://www.mkbergman.com/?p=415
http://www.cytoscape.org/features2.php
which is standalone & interactive and stays interactive to (per their
website) 10^5 nodes+edges.

For a non-interactive app, LGL (Large Graph Layout, C++ + Boost) has
been used to create tree graphs (MST) up to 302,832 nodes, in another
case 32 k nodes + 1.2 million edges:
http://apropos.icmb.utexas.edu/lgl/#apps
http://sourceforge.net/projects/lgl
It's old, but it is still being developed:
http://lgl.svn.sourceforge.net/viewvc/lgl/trunk/
Visualizations at this scale calls for some heavy iron, but as it's
based on boost should continue to scale with your wallet.

I don't know whether there's any good flash/prefuse/whatever based
online tools, and how they'd scale.

If you have experience with big-graph-viz tools, and you let me know
- what you used, and whether it was static / standalone+interactive /
browser-based+interactive
- roughly how many nodes / edges / variance in edges-per-node
- whether it made you happy or sad,
I'll summarize back to the list + wiki.

flip

[Jonathan Gray]

I thought list members might be interested to hear that we're organising a
series of informal, hands-on workshops on open-source visualisation
technologies in London. The first one will be later this month.

http://okfn.org/wiki/OpenVisualisation/Workshop

Warm regards,

Jonathan Gray
The Open Knowledge Foundation

[Jonathan Gray]

Just to let you know, the details of the workshop are as follows:

* When: Saturday 24th May 2008
* Where: Trampoline Systems, 8-15 Dereham Place, London, EC2A 3HJ
* Wiki: http://okfn.org/wiki/OpenVisualisation/Workshop

Warm regards,

Jonathan Gray
The Open Knowledge Foundation

http://okfn.org/