The applet runs in a Frame in most (all?) browser implementations, so you can get it by traversing up the containment hierarchy from any component, testing "... instanceof Frame" at each level until you reach it.
Modal dialogs are a tricky beast however. In Netscape, for instance, opening a dialog seems to block the opening thread (i.e.the one that does show()), but allows other AWT threads to be created from the applet components. You can mess with focusLost events and window (deactivated) events to try and regain focus on the dialog whenever the user attempts to interact with the applet, but these are not robust and will fail if the user is persistent. You can disable the components of the applet and let this trickle down that component hierarchy to simulate modality. You must remember, however, to restore the previous state of the components after closing the dialog if you do this.
Hope this helps...
Dave Cotton