Java Protocol Buffers classes serialization

[yahro]

Hi,

Is there something preventing Protocol Buffer Java classes from being
serializable e.g. using java.io.Externalizable? I would like to use
Protocol Buffer classes to store data in caches, data stores etc.
Currently I have to write awkward custom serializers while it seems,
that it would be enough to add add two methods, implement
java.io.Externalizable interface and classes would be efficiently
serializable. Am I missing something?

Best regards,
Jarek

[Kenton Varda]

Message objects are immutable.  Unfortunately, readExternal() -- specified by the Externalizable interface -- is a mutating method.  Implementing it would destroy the immutability guarantee, possibly introducing bugs or even security problems.  Is there a way around this?  I admit I'm no expert on Java serialization (as you might guess, I don't use it! :) ).

--
You received this message because you are subscribed to the Google Groups "Protocol Buffers" group.
To post to this group, send email to prot...@googlegroups.com.
To unsubscribe from this group, send email to protobuf+u...@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/protobuf?hl=en.

[Jaroslaw Odzga]

I'll have to disagree :-) Serialization is not a way of mutating objects and Externalizable is thought to be just an efficient Serialization. readExternal() is not a public method anyway. Obviously, if you want you can use it to mutate the object, but if you like you can mutate even perfectly final and immutable objects in Java (using some trickery). I think Serialization is just alternative way of creating objects and it is perfectly ok to serialize immutable objects e.g. String is immutable and Serializable. I would go ahead and make Protocol Buffer classes serializable ;-)

Regards,
Jarek

[Kenton Varda]

On Sun, Feb 14, 2010 at 4:36 AM, Jaroslaw Odzga <jarek...@gmail.com> wrote:

I'll have to disagree :-) Serialization is not a way of mutating objects and Externalizable is thought to be just an efficient Serialization. readExternal() is not a public method anyway.

Since Externalizable is an interface, all its methods are necessarily public.

 

Obviously, if you want you can use it to mutate the object, but if you like you can mutate even perfectly final and immutable objects in Java (using some trickery).

I'm pretty sure that's not true.  Effective Java, for example, suggests using immutable classes as a security measure, which would be a terrible suggestion if they were not really immutable.

 

I think Serialization is just alternative way of creating objects and it is perfectly ok to serialize immutable objects e.g. String is immutable and Serializable. I would go ahead and make Protocol Buffer classes serializable ;-)

As it turns out, someone inside Google is working on this now, so if it is reasonably possible it should be in the next release.

[yahro]

On Feb 16, 9:25 pm, Kenton Varda <ken...@google.com> wrote:

> On Sun, Feb 14, 2010 at 4:36 AM, Jaroslaw Odzga <jarek.od...@gmail.com>wrote:
>
> > I'll have to disagree :-) Serialization is not a way of mutating objects
> > and Externalizable is thought to be just an efficient Serialization.
> > readExternal() is not a public method anyway.
>
> Since Externalizable is an interface, all its methods are necessarily
> public.
>

You are right :-)

> > Obviously, if you want you can use it to mutate the object, but if you like
> > you can mutate even perfectly final and immutable objects in Java (using
> > some trickery).
>
> I'm pretty sure that's not true.  Effective Java, for example, suggests
> using immutable classes as a security measure, which would be a terrible
> suggestion if they were not really immutable.
>

I'm sure that's true ;-) Here you have a perfectly final and immutable
class:

public final class Person {
private final String name;
private final int age;
private final int iq = 110;
private final Object country = "South Africa";

public Person(String name, int age) {
this.name = name;
this.age = age;
}

public String toString() {
return name + ", " + age + " of IQ=" + iq + " from " + country;
}
}

Now, try to guess what the following program prints:

import java.lang.reflect.Field;

public class FinalFieldChange {
private static void change(Person p, String name, Object value)
throws NoSuchFieldException, IllegalAccessException {
Field firstNameField = Person.class.getDeclaredField(name);
firstNameField.setAccessible(true);
firstNameField.set(p, value);
}
public static void main(String[] args) throws Exception {
final Person heinz = new Person("Heinz Kabutz", 32);
change(heinz, "name", "Ng Keng Yap");
change(heinz, "age", new Integer(27));
change(heinz, "iq", new Integer(150));
change(heinz, "country", "Malaysia");
System.out.println(heinz);
}
}

And here is what is really printed (I executed it using Java 6_14):

Ng Keng Yap, 27 of IQ=110 from Malaysia

I "borrowed" this example from Dr. Heinz M. Kabutz's blog:
http://www.javaspecialists.eu/archive/Issue096.html. I only added few
"final" keywords to make it even more dramatic :-)

This does not invalidate recommendation from Effective Java to use
immutable objects (for thread safety, security etc.).

> > I think Serialization is just alternative way of creating objects and it is
> > perfectly ok to serialize immutable objects e.g. String is immutable and
> > Serializable. I would go ahead and make Protocol Buffer classes serializable
> > ;-)
>
> As it turns out, someone inside Google is working on this now, so if it is
> reasonably possible it should be in the next release.
>

Great. In my opinion adding serialization to PB is a good move. Most
immutable classes from JDK are serializable i.e. String.

>
>
>
>
> > Regards,
> > Jarek
>
> > On Fri, Feb 12, 2010 at 9:47 PM, Kenton Varda <ken...@google.com> wrote:
>
> >> Message objects are immutable.  Unfortunately, readExternal() -- specified
> >> by the Externalizable interface -- is a mutating method.  Implementing it
> >> would destroy the immutability guarantee, possibly introducing bugs or even
> >> security problems.  Is there a way around this?  I admit I'm no expert on
> >> Java serialization (as you might guess, I don't use it! :) ).
>

> >> On Fri, Feb 12, 2010 at 2:32 AM, yahro <jarek.od...@gmail.com> wrote:
>
> >>> Hi,
>
> >>> Is there something preventing Protocol Buffer Java classes from being
> >>> serializable e.g. using java.io.Externalizable? I would like to use
> >>> Protocol Buffer classes to store data in caches, data stores etc.
> >>> Currently I have to write awkward custom serializers while it seems,
> >>> that it would be enough to add  add two methods, implement
> >>> java.io.Externalizable interface and classes would be efficiently
> >>> serializable. Am I missing something?
>
> >>> Best regards,
> >>> Jarek
>
> >>> --
> >>> You received this message because you are subscribed to the Google Groups
> >>> "Protocol Buffers" group.
> >>> To post to this group, send email to prot...@googlegroups.com.
> >>> To unsubscribe from this group, send email to

> >>> protobuf+u...@googlegroups.com<protobuf%2Bunsubscribe@googlegroups.c om>

[Oliver Jowett]

yahro wrote:

> firstNameField.setAccessible(true);

throws SecurityException

-O

[David Dabbs]

 
>Obviously, if you want you can use it to mutate the object, but if you like
you can mutate even
>perfectly final and immutable objects in Java (using some trickery).

>I'm pretty sure that's not true.  Effective Java, for example, suggests
using immutable classes as a
>security measure, which would be a terrible suggestion if they were not
really immutable.
 

Actually, unless there's a SecurityManager in place any member (even final)
is accessible. For instance,
I've used code similar to the following (valid on Sun JVM only, I think) to
directly read a String's char[] to avoid data copying.
I *could* have modified the value.

Cheers,

david

///////////////////////////////////////

import java.lang.reflect.*;

public final class ReflectionUtils {

public static final Field STRING_VALUE_FIELD =
getFieldAccessible(String.class, "value");
public static final Field STRING_OFFSET_FIELD =
getFieldAccessible(String.class, "offset");
public static final Constructor<?> STRING_PP_CTOR =
getConstructor(String.class, 0, int.class, int.class, char[].class);

public static char[] getChars(final String s) {
//
// use reflection to read the char[] value from the string. . .
//
try {
return (char[]) STRING_VALUE_FIELD.get(s);
} catch (Throwable e) {
//
}
return null;
}

public static String sharedString(final char[] chars, final int offset,
final int length) {
try {

return (String) STRING_PP_CTOR.newInstance(offset, length,
chars);

} catch (InstantiationException e) {
e.printStackTrace();
} catch (IllegalAccessException e) {
e.printStackTrace();
} catch (InvocationTargetException e) {
e.printStackTrace();
} catch (Exception e) {
e.printStackTrace();
}
return null;
}

public static Field getFieldAccessible(final Class<?> clazz, final
String fieldName) {
Field fld = null;
try {

fld = clazz.getDeclaredField(fieldName);
fld.setAccessible(true);

} catch (NoSuchFieldException e) {
e.printStackTrace();
} catch (SecurityException e) {
e.printStackTrace();
}
return fld;
}

public static Constructor<?> getConstructor(final Class<?> clazz, final
int searchModifier, Class<?>... paramTypes) {
//
// N.B. there is no explicit value for "package-private," so pass 0.
//
try {

Constructor<?>[] allConstructors =
clazz.getDeclaredConstructors();

for (Constructor<?> ctor : allConstructors) {

if (searchModifier == (ctor.getModifiers() &
(Modifier.PUBLIC | Modifier.PRIVATE | Modifier.PROTECTED))) {
//
// access modifier match. . .
//
final Class<?>[] parameterTypes =
ctor.getParameterTypes();
if (parameterTypes.length == paramTypes.length) {
//
// same number of parameters. . .
//
for (int i = 0; i < parameterTypes.length; i++) {
if (!parameterTypes[i].equals(paramTypes[i])) {
ctor = null;
break;
} else {
// Type[] gpType =
ctor.getGenericParameterTypes();
// for (int j = 0; j < gpType.length; j++) {
// char ch =
(gpType[j].equals(paramTypes[i]) ? '*' : ' ');
// System.out.format("%7c%s[%d]: %s%n",
ch, "GenericParameterType", j, gpType[j]);
// }
}
}
if (ctor != null) {
// System.out.format("%s%n",
ctor.toGenericString());
// System.out.format(" [ synthetic=%-5b
var_args=%-5b ]%n", ctor.isSynthetic(), ctor.isVarArgs());

ctor.setAccessible(true);
return ctor;
}
}
}
}

// production code should handle this exception more gracefully
} catch (Exception x) {
x.printStackTrace();
}
return null;
}

public static void main(String[] args) {
getConstructor(String.class, 0, int.class, int.class, char[].class);
}

public static boolean setAccessible(AccessibleObject o) {
try {
o.setAccessible(true);
return true;
} catch (SecurityException e) {
e.printStackTrace();
}
return false;
}
}

[HeinzMK]

Just need to add my 10c worth, seeing that my name was mentioned :-)

Final fields are considered immutable by the JMM. The fact that you
CAN change them does not matter to that definition. Final fields are
allowed to be inlined at runtime, which means that changing them might
cause visibility issues.

Serializable also modifies final fields. The ObjectInputStream first
constructs an object, using a special method in sun.reflect (see
http://www.javaspecialists.eu/archive/Issue175.html). It then sets
the final fields.

Incidentally in Java 1.1 you could NOT set final fields. Then in 1.2
you COULD. In 1.3 and 1.4 you could NOT. In 1.5+ you CAN. That is
with just standard reflection. It's actually bad, because setting a
final field could cause serious concurrency issues, but the reflection
API does not give you a warning when you try to do it.

Externalizable is quite dangerous as a general solution and does not
offer many performance benefits over Serializable with custom read/
writeObject methods. Have a look at my talk from JavaZone:
http://www.javaspecialists.eu/talks/oslo09/ReflectionMadness.pdf and
search for Externalizable Hack.

Heinz

[Kenton Varda]

For the record, someone who understands Java serialization better than me has come up with a very reasonable way to make GeneratedMessage and GeneratedMessageLite implement Serializable, so that should end up in the next release.  Unfortunately DynamicMessage will still not implement Serializable since it would be tricky to do efficiently in the case of many DynamicMessages having the same or overlapping types, and I don't think this is a use case people really need.

--
You received this message because you are subscribed to the Google Groups "Protocol Buffers" group.
To post to this group, send email to prot...@googlegroups.com.

To unsubscribe from this group, send email to protobuf+u...@googlegroups.com.