Cytoscape hangs when trying to visualize large network

[George Furbish]

Cytoscape hangs when trying to visualize this network:

https://drive.google.com/open?id=0B7y_oTRt8PCyWEpBVWs1dDRpX00

It loads fine, I click "Yes" to visualize, it works very hard for 2-3 minutes, then the CPU usage goes to zero and RAM sits at ~13 GB. System has 32 GB of RAM. CPU usage for Cytoscape sits at 0% to 0.1% after the first 2-3 minutes. No progress ever appears on the "Prefuse Force Directed Layout" progress bar.

Cytoscape vmoptions:

-Xms2048M

-Xmx31032M

[Barry Demchak]

Hi, George --

Cytoscape can handle larger and more complex networks than this, though one often has to give it lots of memory and time.

It's interesting that the CPU seems to get very quiet after a few minutes ... I'm guessing that it ran out of memory and didn't report the problem gracefully.

Note that the Prefuse algorithm is n^3 ... meaning that it may not finish in a reasonable time when run on a very large number of nodes.

Have you considered running a Grid or Circular layout, just as a test?

Also, for very large networks, it's often unhelpful to do a layout, as most/all layouts produce hairballs at this scale. Have you considered using filters to either exclude unneeded nodes/edges or to select only needed nodes/edges? This will reduce the network to something that may contain useful information after layout. 

Also, as a sanity check ... does your PC actually have 32GB RAM?? ... or are you depending on swapping??

FYI, I was able to load this network in 4.5GB RAM and use 77% of that. Also, once that was loaded, I started a Circular layout ... that was 90 minutes ago, and it's still going. I also see CPU utilization down in the 5% range ... no appreciable disk swapping. I'm going to let this go overnight to see whether it finishes.

I'll continue this thread in the morning ...

[George Furbish]

It's interesting that the CPU seems to get very quiet after a few minutes ... I'm guessing that it ran out of memory and didn't report the problem gracefully.

 

I suspect some kind of exception occurs that gets silenced, but prevents execution from continuing. It should have plenty of memory, so I'm not sure what the actual issue is.

 

Note that the Prefuse algorithm is n^3 ... meaning that it may not finish in a reasonable time when run on a very large number of nodes.

I understand; it's allowed to go all night. I wouldn't mind a long runtime, but the fact that CPU usage cuts after a short while suggests to me that it's not the time complexity that's the issue. (Interestingly, memory usage also dwindles over time. Wonder if there's some sort of garbage collection or something taking place?)

 

Have you considered running a Grid or Circular layout, just as a test?

No; I just used prefuse because it was the default visualization. 

 

Also, for very large networks, it's often unhelpful to do a layout, as most/all layouts produce hairballs at this scale. Have you considered using filters to either exclude unneeded nodes/edges or to select only needed nodes/edges? This will reduce the network to something that may contain useful information after layout. 

I was going to start by visualizing the whole network, and then try to isolate interesting portions or identify communities.

 

 

Also, as a sanity check ... does your PC actually have 32GB RAM?? ... or are you depending on swapping??

The PC has 32 GB RAM in 4x8 GB sticks. Swap space is available on an SSD if necessary, but I believe that should be more than adequate for this application, no?

 

FYI, I was able to load this network in 4.5GB RAM and use 77% of that. Also, once that was loaded, I started a Circular layout ... that was 90 minutes ago, and it's still going. I also see CPU utilization down in the 5% range ... no appreciable disk swapping. I'm going to let this go overnight to see whether it finishes.

I'll continue this thread in the morning ...

Appreciate the help. Think there's something breaking in Cytoscape due to the network size or format. Node names aren't valid int32, so I wonder if it's parsing those and screwing up or something weird like that?

 

[Barry Demchak]

Hi, George …

 

Still running … and not making any progress … and not swapping or using CPU.

 

OK … we definitely have a problem, especially considering that there’s more than enough RAM to hold this network. I’ll look further.

 

Thanks!

--
You received this message because you are subscribed to the Google Groups "cytoscape-helpdesk" group.
To unsubscribe from this group and stop receiving emails from it, send an email to cytoscape-helpd...@googlegroups.com.
To post to this group, send email to cytoscape...@googlegroups.com.
Visit this group at https://groups.google.com/group/cytoscape-helpdesk.
For more options, visit https://groups.google.com/d/optout.

[Barry Demchak]

Hi, George …

 

I haven’t forgotten this … in fact, I have been playing with it so I could characterize it … so far, I can see that the Circle layout causes a hard hang, too.

 

I have also verified that a smaller version (manually edited using a text editor) does not cause a hang. So this seems to be size related. To double-check, I’ll manually edit it into a SIF file to see if we get the same result.

 

You’re quite right that there’s something very rotten in Cytoscape-land.

 

Continuing on …

 

From: cytoscape...@googlegroups.com [mailto:cytoscape...@googlegroups.com] On Behalf Of George Furbish

Sent: Thursday, November 17, 2016 3:51 PM
To: cytoscape-helpdesk

--

[Barry Demchak]

Hi, George –

 

There are a few things going on here.

 

First, the layout base class (on which multiple layouts rely) is calling itself recursively as it traverses various paths through the network. There is apparently a very long path in this network, and a stack overflow is occurring.

 

Second, the stack overflow isn’t being caught and reported properly … and so the progress dialog is never dismissed.

 

This is pretty good news, when you think of it … considering this happens after only a few minutes, there’s a pretty good chance that when this is fixed, your layout won’t take forever.

 

As a workaround for now: Would you know off hand where the long path is?? If so, is it feasible to break the path??

 

 

From: cytoscape...@googlegroups.com [mailto:cytoscape...@googlegroups.com] On Behalf Of George Furbish

Sent: Thursday, November 17, 2016 3:51 PM
To: cytoscape-helpdesk

--

[Mike Kucera]

This only effects layouts that extend AbstractPartitionLayoutTask. The following layouts are non-partitioned and should be OK (although I haven't tested them):

Grid

Group Attributes

Hierarchical

Stacked Node

I'm not sure if the yFiles layouts are partitioned or not.

Mike.

To unsubscribe from this group and stop receiving emails from it, send an email to cytoscape-helpdesk+unsub...@googlegroups.com.
To post to this group, send email to cytoscape-helpdesk@googlegroups.com.

Visit this group at https://groups.google.com/group/cytoscape-helpdesk.
For more options, visit https://groups.google.com/d/optout.

--

You received this message because you are subscribed to the Google Groups "cytoscape-helpdesk" group.

To unsubscribe from this group and stop receiving emails from it, send an email to cytoscape-helpdesk+unsub...@googlegroups.com.
To post to this group, send email to cytoscape-helpdesk@googlegroups.com.

[Scooter Morris]

There is also an option under Settings to not attempt to partition the network before laying it out....

-- scooter

To unsubscribe from this group and stop receiving emails from it, send an email to cytoscape-helpd...@googlegroups.com.
To post to this group, send email to cytoscape...@googlegroups.com.

[George Furbish]

I have no idea where the long path is. This is just a preliminary analysis of the network, so it's as foreign to me as it is to you.

Is this an issue that will be addressed in an upcoming patch? If so, will the update expand the stack or just stop silencing the exception when it overflows? That is, will I be able to use this layout eventually (or even in the near future) for this or similar networks?

[Barry Demchak]

Hi, George--

The fix wouldn't be in the next release ... it's too risky for a release that's very near.

It *would* either not overflow (because a non-recursive technique would be used) or it would overflow gracefully. Either way, the fix would be very disruptive to existing code and would therefore involve risk of breakage.

It *is* possible for us to accelerate the fix into a non-distribution version of Cytoscape ... it wouldn't be well tested.

No promises as to timeframe ... what's your timeframe for this work??

-----Original Message-----
From: cytoscape...@googlegroups.com [mailto:cytoscape...@googlegroups.com] On Behalf Of George Furbish

[George Furbish]

This project I'm doing in the next few weeks, so I'll have to visualize the data by other means, but I intend to continue working with similar datasets for the next few years, so in the long run this would be a very useful tool if it will be compatible with my networks.

-George

[Barry Demchak]

George --

I think I advanced the ball somewhat, though you might be able to tell better.

In the Layout | Settings menu, you'll see that there are settings for each layout ... and settings that declare the default layout.

The settings allow you to indicate that the network shouldn't be partitioned before layout. I'm not sure what partitioning means here, but I tried the setting on your network. My CPU pegged and stayed pegged, as did my memory usage and occasional disk activity.

If you have a ton of memory, you might try this setting to see whether you actually get a layout ...

-----Original Message-----
From: cytoscape...@googlegroups.com [mailto:cytoscape...@googlegroups.com] On Behalf Of George Furbish
Sent: Wednesday, November 23, 2016 12:09 PM
To: cytoscape-helpdesk
Subject: RE: [cytoscape-helpdesk] Re: Cytoscape hangs when trying to visualize large network