My idea goes along the lines of this:

Have a helper that defines a post_amf method, using this from your
functional tests would be just like using the standard post method.

Inside the post_amf method it switches out the controller your testing
for the amf controller and sets things up e.g.:

  amf_post :index, [ :param = 'something' ]

 1. Serializes the request, with the source set to the name of the
controller your testing & the method name to the one you passed in and
the params put into the body.
 2. Switches out the @controller for the amf controller
 3. Set anything else up set as the content type on the request
 4. Calls the gateway method on the amf controller with the params
(which should now be serialized to amf)
 5. Switches back the @controller to what it was

If that works then that 'should' be calling the amf gateway with
exactly what it needs and everything 'should' be doing things as if
they were a normal request.

Then you would also need a helper method to test that the response was
AMF (e.g. used the format.amf) and something to convert that (or do it
after the end of the send) back to something you can test.

I know I'm over-simplifying things there (as I have spent the past
couple of hours trying to get my head around how Rails functional
tests mock out the requests and RubyAMF works to do the above) and I
haven't gotten very far see the pastie code: http://pastie.caboo.se/158549

That code has it calling the gateway how I envisioned it, but I'm
really stuck on trying to serialize something that the gateway would
accept as a valid amf request.