Reaping an object with references

[Craig R Hughes]

Rama,

I just hit this same weird bug. First time it's happened to me was this
morning in a server that's been running fine for months. The server where this
happened is an RMI server which gets RMI requests and then dumps to a file (it's
an RMI parallel for syslogd on unix). This morning is stopped processing
requests and dumped the same stack trace you had:

java.lang.Error: Reaping an object with references
at sun.rmi.transport.ObjectTable.reap(ObjectTable.java:137)
at sun.rmi.transport.Reaper.run(ObjectTable.java:200)
at java.lang.Thread.run(Thread.java)

It looks like somehow the garbage collector is getting confused about certain
remote objects under certain conditions, and then reaping objects that it
shouldn't be reaping. What I don't get is that if it knows this object is
referenced in order to dump this stack trace, then why the heck can't it figure
out not to try reaping it in the first place.

> Hi,This problem is related to Remote Objects(persistence).
> Here is my applet description.I have an applet that runs using IE4.0 browser.
> Client Communication to the backend is via
> TCP/IP(since IE does not support RMI) to a
> proxy server and from the proxy server using RMI
> to the backend servers and then to the database.How it works.
> I have my own Central Registry and this is
> started first. The backend server is then
> started and it registers itself with the
> Central Registry. Next the proxy server is
> started and it begins to listen for incomingclient calls.
> Every time a client logs on(talks to proxy server),
> the proxy server gets the backend server
> reference from the Central Registry and the
> Backend server in turn talks to the database to
> verify if the logon is valid. If the logon is valid
> the backend server then creates a remote object
> instance and exports this object and also returns
> a reference to this object back to the client via
> the proxy server. Note that there is one remote object
> instance per client and this is created during logon.
> The client then uses this remote object reference
> to communicate further with the backend and perform
> database accesses. A list of all such remote instances
> is stored by the Proxy Server along with a client
> identification for each. All calls after the logon
> from clients are then routed through their respectiveremote objects.
> (The Proxy Server retrieves the remote object for a client
> from its list whenever the client makes a call)The Problem.
> Very often the Proxy Server throws up the following
> exception when it tries to access the remote object.
> java.rmi.NoSuchObjectException: no such object in table
> At around the same time the Backend server throws up thefollowing.
> java.lang.Error: Reaping an object with references
> at sun.rmi.transport.ObjectTable.reap(ObjectTable.java:137)
> at sun.rmi.transport.Reaper.run(ObjectTable.java:200)
> at java.lang.Thread.run(Thread.java:474)
> After this the client has to close down Internet Explorer and
> restart in order to get a new remote object reference.(We
> however don't have to restart our backend processes).
> Has anyone had a similar problem? Is this a bug?ThanksRama
>

[Craig R Hughes]

Having just sent that other message, I now find at the JDC site this very
informative report:

http://developer.javasoft.com/developer/bugParade/bugs/4066643.html

So, it looks like there's a bug in the VM where strong references aren't getting
generated by the RMI system in some cases when they should be. I'm myself not
serializing any objects with remote references (except that RMI is doing it for
me to pass them over the TCP stream). Due to the infrequency of my bug (once in
several months of uptime), I would guess there's a race condition between the
object ref being dropped on the server and the ref being created on the client
side. If the network connection is slow and loaded (which was the case when my
server crashed), then it might even help the timing out to facilitate the
crash. Here's my theory:

1. Client calls remote method X()
2. Server does X() which returns a Remote object R1 with ref to remote object
R2
3a. Server throws away its ref to R1 and R2
3b. Client deserializes the result of remote X() call and creates ref to R1 and
R2
4. Crash or fly

The key is the ordering of 3a and 3b, and whether the GC thread kicks in between
3a and 3b. If it does, and it marks the objects at the end of R1 and R2 as
collectable, then we're screwed. This is what I suspect it happening. I'll try
getting in touch with Javasoft to confirm this.

C