For that I have been downloading the builds and creating my own local copy of the Javadoc files. I use the Subversion support in my ID to do that and simply browse to the correct tag for the release. There is no downloadable copy of the Javadoc files on the site that I am aware of.
Note, I noticed that for the latest builds you need the android-14.jar file from trunk/third_party/java/android to resolve some android external info, so I include that in my compile-time library list just for creating the Javadoc. You do not want to compile using it however, because it is only stub routines.
Mike