Hello Alexey,
Unfortunately this is a typical situation in large GWT application
which uses a lot of custom Widgets and DOM/native code.
We have found no tooling which would efficiently show us the exact
places of memory leaks and the only approach left is to cut you
application into two halves and see how the memory behaves.
We have succeeded in reducing the memory leak by carefully analyzing
the code and unlinking custom event listeners and object hierarchies
by overriding onUnload() methods of custom widgets. Splitting your
application into several and providing hard navigation by changing
window.url from time to time, instead of relying on GWT History
mechanism also helps.
Memory leak situation is a lot better in Firefox and Opera, but Chrome
went further by providing separate memory space for each tab, thus it
guarantees that if you close the tab everything is "garbage
collected", because process is killed. I guess Google took the only
right way to solve the problem :)
I hope it will help you to deal with your problems.
Dmitry