Hi,
The ceiling for memory on 32-bit Java seems to be somewhat system-dependent (and hard to correctly predict), which is why you're experiencing problems running 32-bit Cytoscape. For this reason, we're planning on removing 32-bit support from the next major Cytoscape release (3.2).
The best thing to do here is to install 64-bit Java and Cytoscape - without it, Cytoscape can't use much more than 1.2GB of memory. If that is not possible, you can try decreasing the memory allocation in Cytoscape.vmoptions until it runs - however, the memory available to Cytoscape will likely be insufficient to handle large networks.
Hope this helps...
Tim