Can I please, please beg of you not to use a popup window. It's ugly.
Most browsers just block them anyway so it's useless.
You've got to remember as well that a popup window is a COMPLETELY
separate browser instance. You can't access any variables at all from
the source window, and prototype/scriptaculous will not be available
unless they're included in the source of the popup.
Much much better is to bring up a fixed positioned div floated above
everything else on the page. Google prototip to see what I mean - and
they don't have to be "tooltips" either, they can be full on windows
over the rest of your page - see the other project by the author of
prototip, think it's called lightbox, to see what I mean.
Other than that advice, your question is way too vague to be able to
answer it. If you have prototype in the source of the popup window,
ajax will be available just as it otherwise would be - either you
haven't included prototype or there's an error in your code, in which
case the fact that it's running in a popup window is completely
irrelevant.