Issues with Key serialization

[dilbert]

First of all sorry if this is the second topic with a similar issue. I already posted a similar topic but I can't see it in the group (it has already been several hours) so I'm posting again. I had many problems porting my GAE app to Objectify 3.0 from 2.2.3 with Key serialization when com.googlecode.objectify.Key (Objectify Key) started using com.google.appengine.api.datastore.Key (GAE Key).

I use an application on a non-GAE server to backup some data (via a Hessian-service implemented on GAE) from my APP engine application datastore. The data is loaded from the datastore as objects which are then serialized by Hessian and sent to the non-GAE service. The data is then stored in an object database. The data can be restored (read from the local object database sent to the GAE application and stored to the datastore). The restoring step fails with the message:
java.lang.NullPointerException
        at com.google.appengine.api.datastore.KeyTranslator.convertToPb(KeyTranslator.java:55)
        at com.google.appengine.api.datastore.EntityTranslator.convertToPb(EntityTranslator.java:34)
        at com.google.appengine.api.datastore.AsyncDatastoreServiceImpl.put(AsyncDatastoreServiceImpl.java:429)
        at com.googlecode.objectify.impl.AsyncObjectifyImpl.put(AsyncObjectifyImpl.java:255)

I analyzed the decompiled Key code and came to the conclusion that the problem lies in the fact that the appId private variable is not properly retrieved from the server when the GAE Key instance is serialized by Hessian. In fact the appId variable is only set upon Java serialization (the writeObject method).

Another issue is that Java serialization of Key relies on a static object being previously initialized. For example you cannot write this:

Key key = KeyFactory.createKey("banana", (5L));

in a non-GAE application if you have not initialized a LocalServiceTestHelper instance. It dies in the following manner:
java.lang.NullPointerException: No API environment is registered for this thread.
    at com.google.appengine.api.datastore.DatastoreApiHelper.getCurrentAppId(DatastoreApiHelper.java:108)
    at com.google.appengine.api.datastore.DatastoreApiHelper.getCurrentAppIdNamespace(DatastoreApiHelper.java:118)
    at com.google.appengine.api.datastore.Key.<init>(Key.java:51)
    at com.google.appengine.api.datastore.Key.<init>(Key.java:37)
    at com.google.appengine.api.datastore.KeyFactory.createKey(KeyFactory.java:46)
    at com.google.appengine.api.datastore.KeyFactory.createKey(KeyFactory.java:31)

This serialization issue is significant on Android applications that use classes containing Key instances which are shared between Activities since the transfer of objects between Activities uses Java Serialization. I believe that the problem lies with the Gae Key but the question remains: Was it really necessary to include GAE Key in Objectify Key? Can something be done about it? Before the change everything worked fine, of course.

Also, if anyone has the same problems, she can star this issue:
http://code.google.com/p/googleappengine/issues/detail?id=4966

Best regards.

[Jeff Schnitzer]

Wow, interesting.

The first issue (incomplete serialization of datastore native Key) you
should be able to fix with a custom Hessian serializer.

The second issue looks pretty bad. Honestly, I would cut out the Key
and use your own class instead. This is another compelling argument
for the "unactivated entity" feature.

Jeff

[dilbert]

Thanks for the advice, I already used Hessian Serializers for solving the first issue. If someone is interested here is the solution:

public class KeySerializer extends AbstractSerializer {
    private final JavaSerializer serializer = new JavaSerializer(Key.class);

    @Override
    public void writeObject(Object obj, AbstractHessianOutput out) throws IOException {
        Key key = (Key) obj;
        ObjectOutputStream oos = new ObjectOutputStream(new ByteArrayOutputStream());
        try {
            oos.writeObject(key);
        } finally {
            oos.close();
        }
        serializer.writeObject(key, out);
    }
}

public class KeyDeserializer extends AbstractDeserializer {
    private final JavaDeserializer deserializer = new JavaDeserializer(Key.class);

    @Override
    public Class getType() {
        return Key.class;
    }

    @Override
    public Object readObject(AbstractHessianInput in, Object[] fieldNames) throws IOException {
        Key key = (Key) deserializer.readObject(in, fieldNames);
        return setupKey(key);
    }

    @Override
    public Object readObject(AbstractHessianInput in, String[] fieldNames) throws IOException {
        Key key = (Key) deserializer.readObject(in, fieldNames);
        return setupKey(key);
    }

    @Override
    public Object readObject(AbstractHessianInput in) throws IOException {
        Key key = (Key) deserializer.readObject(in);
        return setupKey(key);
    }

    @Override
    public Object readList(AbstractHessianInput in, int length) throws IOException {
        Key key = (Key) deserializer.readList(in, length);
        return setupKey(key);
    }

    @Override
    public Object readLengthList(AbstractHessianInput in, int length) throws IOException {
        Key key = (Key) deserializer.readLengthList(in, length);
        return setupKey(key);
    }

    @Override
    public Object readMap(AbstractHessianInput in) throws IOException {
        // This method gets invoked.
        Key key = (Key) deserializer.readMap(in);
        return setupKey(key);
    }

    private Object setupKey(Key key) throws IOException {
        ByteArrayOutputStream bos = new ByteArrayOutputStream();
        ObjectOutputStream oos = new ObjectOutputStream(bos);
        try {
            oos.writeObject(key);
        } finally {
            oos.close();
        }

        ObjectInputStream ois = new ObjectInputStream(new ByteArrayInputStream(bos.toByteArray()));
        try {
            return ois.readObject();
        } catch (ClassNotFoundException e) {
            throw new RuntimeException(e);
        } finally {
            ois.close();
        }
    }
}

Of course these should be registered with the Hessian infrastructure. I do not like this solution though cause it relies on an undocumented GAE feature (namely that the appId member variable is written to when serialization is invoked). It could be changed at any time. As for the second issue I tweaked a bit the building of the objectify-client.jar which I use on Android. You use the Key.class from the GAE SDK for building the library. I replaced it with a version that I produced by decompiling the original Key and removing the readObject and writeObject methods. It works but it sure isn't pretty.

Best regards.

[Jeff Schnitzer]

Since we're on the subject of Hessian, others might also find this code helpful:

http://code.google.com/p/unsuck/source/browse/trunk/src/unsuck/hessian/ThrowableSerializerFactory.java

One problem with Hessian (client) on appengine is that exceptions
cannot be reconstituted properly - the private msg & stacktrace fields
cannot be set on JDK classes like Throwable. This fixes that problem.
Mostly. It doesn't completely work - I think there's a problem
getting the full stacktrace. But it reconstitutes the proper
exception and message, which was enough info to solve the bug that was
plaguing me.

Jeff