The DHTML Style Guide suggests that in all dialogs (modal and non-modal) "focus via the tab and shift-tab key must be maintained within the dialog." We currently do this for modal dialogs, but not for non-modal dialogs. For those wondering how you would interact with the rest of the page while keeping the dialog open, they recommend using F6 to toggle focus between the page and the dialog. Any objections to extending this behavior to all dialogs and following the DHTML Style Guide?
One thing I'm not sure about is how the focus toggling would work if there are multiple dialogs open at the same time.