Hi Vachan,
The debug version of the library on my mac requires about 3.8 GB of memory to link: if you are patient and willing to use all your swap space then you should be able to compile the library. I recommend trying:
1) close all other applications on your machine (if possible, try to boot directly to the command line)
2) if you can, compile deal.II with the bare minimum number of dependencies that you might want to use. Every dependency you can skip will save some memory.
3) make sure you are only compiling with a single processor (i.e., make -j1)
Best,
David Wells