The code is now committed. Here's how my servlet filter looks like:

public void doFilter(final ServletRequest request, final
ServletResponse response, FilterChain chain) throws IOException,
ServletException {
  HttpServletRequest httpReq = (HttpServletRequest) request;
  HttpServletResponse httpRes = (HttpServletResponse) response;
  HttpSession session = httpReq.getSession();
  Facebook fb = new Facebook (httpReq, httpRes,_apiKey, _secretKey);
  String next = httpReq.getServletPath().substring(1);
  if (fb.requireLogin(next)) return;
  session.setAttribute(ServerUtils.FACEBOOK_CLIENT,
fb.getFacebookRestClient());
  chain.doFilter(request, response);

As you can see, the Facebook object is not designed to be kept in the
server-side session. You need to create a new one and then discard it
with every request that comes. It associates itself with a request-
response pair so it can issue a redirect properly if necessary. This
is not to my particular taste, and I know it can be done differently,
but I copied the object from the php library almost line-by-line, and
it was important for me to preserve the same semantics (albeit with
some exception to adapt the code for Java specific features).

I'm using an iframe for my application, not fbml, so I only had a
chance to test this in a stand-alone webapp (accessing it locally at
http://localhost:8080/myapp/) and inside facebook via an iframe.
In both cases the method requireLogin does what it is supposed to do.
It returns a boolean flag indicating whether or not a redirect to
Facebook's "login to this app" page was issued.

My application happens to also have a servlet that is serving ajax
requests. It requires access to FacebookRestClient, so I keep it in
the session. As you can see, it happens to be that this object in the
session gets overwritten with every request. I think this is not the
ideal design, but I doubt if this is a big deal, unless you are facing
serious performance issues.

The magic is of course in the constructor. It does the following:
1) Parses the various fb_sig_ parameters and validates their signature
using md5 and the app's secret. It then saves the user_id and the
session_key in the server-side session. When using an iframe inside
facebook, this is typically the case for the first request.
2) If there are no such parameters, it checks the server-side session.
When using an iframe, this is typically the case for all the
subsequent ajax requests, or for subsequent requests that originate
from links inside the frame. This alleviates the need to pass all
those parameters with each request. (in the php code this is done with
cookies. I changed it so it is done with the server-side session)
3) If there is no such information in the session, then it looks for
the auth_token parameter, and uses it to create a user_id and
session_key. This is the case when the request is the result of a
facebook login page redirect.
4) When all else fails, do nothing. At this state, the object is not
associated with a user_id or a session_key at all. This is where
methods like requireLogin(), requireAdd(), isAdded(), etc. kick in.

I hope this helps,

Yoni