A class contains map type serialization problem

215 views
Skip to first unread message

liang Chen

unread,
May 13, 2020, 4:15:55 AM5/13/20
to protostuff

I need to perform RPC communication with .net.
We have defined a class containing the map type:
public class TestMapDto { public int TestNumber; public Map<String, String> Pairs; }
I assign a value to this class and use ProtobufIOUtil for serialization:
Map<String, String> hashMap = new TreeMap<>(); 

hashMap.put("test", "handsome"); 

TestMapDto testMapDto = TestMapDto.builder().TestNumber(111).Pairs(hashMap).build();
The resulting byte array is as follows:

8,111,18,18,10,16,10,4,116,101,115,116,18,8,104,97,110,100,115,111,109,101

But the result after serializing the native Protobuf file is as follows,

8,111,18 ,16,10,4,116,101,115,116,18,8,104,97,110,100,115,111,109,101

They have different results after serialization,Causes .net not to deserialize properly
Hope for help,Sincerely

Message has been deleted

mohammadreza pasha

unread,
May 13, 2020, 5:11:17 AM5/13/20
to proto...@googlegroups.com
hi 
protobuf net have some change to native protostuf for better performance.
if you check your byte array difference consider that in bytes 4 and 5 you dont have 18,10 in native serializer . 10 is id of inner class and 18 is byte lengths of inner class . you must simulate protobuf serialization that generate exact what you need in your native .
my recommend is first to compile your c# protobuf class that generate .proto file and base on this generated file keep your trying.

--
You received this message because you are subscribed to the Google Groups "protostuff" group.
To unsubscribe from this group and stop receiving emails from it, send an email to protostuff+...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/protostuff/44dd478f-9a12-4f21-8a03-196545c6ac3f%40googlegroups.com.
Message has been deleted

liang Chen

unread,
May 13, 2020, 9:58:51 PM5/13/20
to protostuff
Thanks for your help sincerely  mohammadreza,
My situation now is that using protostuff serialize a POJO Java class which is contains map type,
the bytes[] I got is diffent from  serializing by the Java class generated by proto.file,

English is not my mother tongue,The expression may not be clear.Let me show you codes:

First,the proto.file
syntax = "proto3";
option java_package = "com.frxs.remoting.demo.protobuf";
option java_outer_classname = "PbArrayModule";

message PbArray {
  map<string,string> dic =1;
}

The POJO class:
@Data
public class PbArray implements Serializable {
    @Tag(1)
    public HashMap<String, String> dic;
}

The Java class generated by proto.file I omitted.

The test method:
public static void main(String[] args) throws InvalidProtocolBufferException {
        // the proto.file generate java class serialization
        PbArrayModule.PbArray.Builder builder = PbArrayModule.PbArray.newBuilder();
        builder.putDic("test", "cc");
        PbArrayModule.PbArray pbArray = builder.build();
        byte[] bytes = pbArray.toByteArray();
        System.out.println("proto.file serialize:----- " + Arrays.toString(bytes));

        // using ProtobufIOUtil
        HashMap<String, String> aMap = new HashMap<>(1);
        aMap.put("test", "cc");
        PbArray array = new PbArray();
        array.setDic(aMap);
        RuntimeSchema<PbArray> schema = RuntimeSchema.createFrom(PbArray.class);
        LinkedBuffer buffer = LinkedBuffer.allocate(LinkedBuffer.DEFAULT_BUFFER_SIZE);
        byte[] bytes1 = ProtobufIOUtil.toByteArray(array, schema, buffer);
        System.out.println("ProtobufIOUtil serialize: " + Arrays.toString(bytes1));

        PbArrayModule.PbArray pbArray1 = PbArrayModule.PbArray.parseFrom(bytes1);
        System.out.println("using proto.file's way deserialize ProtobufIOUtil bytes1:");
        System.out.println(pbArray1.toString());
    }

Test method console output:
proto.file serialize:----- [10, 10, 10, 4, 116, 101, 115, 116, 18, 2, 99, 99]
ProtobufIOUtil serialize: [10, 12, 10, 10, 10, 4, 116, 101, 115, 116, 18, 2, 99, 99]
using proto.file's way deserialize ProtobufIOUtil bytes1:
dic {
  key: "\n\004test\022\002cc"
  value: ""
}

As you can see, 
The extra bytes are caused by ProtobufIOUtil.
And the bytes serialized by ProtobufIOUtil cannot be deserialized normally by native protobuf.
Therefore, net probuf cannot be deserialized normally.

在 2020年5月13日星期三 UTC+8下午5:11:17,mohammadreza pasha写道:
To unsubscribe from this group and stop receiving emails from it, send an email to proto...@googlegroups.com.

mohammadreza pasha

unread,
May 14, 2020, 12:26:26 PM5/14/20
to proto...@googlegroups.com
hi
if your issue is communicate between protobufnet library in c# and protostuff library in java you must clone github protostuff and change it to generate byte array as exact protobufnet generated.
this release of protostuff that i attach for you will help you . please try it and report me the result

To unsubscribe from this group and stop receiving emails from it, send an email to protostuff+...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/protostuff/dd3774ee-48a5-4462-9ea5-e45c39de2dd3%40googlegroups.com.
ProtobufNet.zip

Wayne Chan

unread,
Jun 23, 2020, 7:05:16 AM6/23/20
to protostuff
Hello ,I'm the poster's colleague , and this question actually does not relate to c# .

When I use Original Protobuf Serializer In Java (proto file way)  and Protostuff Serializer to serialize same Pojo , I can't get the same bytes content . 

Here is the code :

Origninal Proto file way:

User.Proto
syntax = "proto3";

message User {
   string userName = 1;
   map<string,string> testMap = 2;
}



demo:
UserOuterClass.User user = UserOuterClass.User.newBuilder()
               
.setUserName("BB")
               
.putTestMap("aa","AA")
               
.build();
byte[] bytes = user.toByteArray();
//output bytes :10,2,66,66,18,8,10,2,97,97,18,2,65,65


Protostuff Way:

Pojo:
@Data
public class ProtostuffUser implements Serializable {  
 
@Tag(1)
 
private String userName;
 
@Tag(2)
 
private Map<String, String> testmap;
}

demo:
ProtostuffUser user = new ProtostuffUser();
user
.setUserName("BB");
user
.setTestmap(new HashMap<>());
user
.getTestmap().put("aa","AA");
byte[] output= ProtoStuffUtil.serializer(user);
//output bytes :10,2,66,66,19,11,10,2,97,97,18,2,65,65,12,20


output bytes: 
origin:        10,2,66,66,18,8,  10,2,97,97,18,2,65,65
protostuff : 10,2,66,66,19,11,10,2,97,97,18,2,65,65,12,20

So I can't communicate protostuff with origin Protobuf when Pojo contains map
 
Plz help ...   :-)

Appendix:

version:
<dependency>
           
<groupId>com.google.protobuf</groupId>
           
<artifactId>protobuf-java</artifactId>
           
<version>3.5.1</version>
       
</dependency>
       
<dependency>
           
<groupId>io.protostuff</groupId>
           
<artifactId>protostuff-core</artifactId>
           
<version>1.7.2</version>
       
</dependency>
       
<dependency>
           
<groupId>io.protostuff</groupId>
           
<artifactId>protostuff-runtime</artifactId>
           
<version>1.7.2</version>
       
</dependency>




ProtoStuffUtil
public class ProtoStuffUtil {
    private static Logger log = LoggerFactory.getLogger(ProtoStuffutil.class);
    private static Map<Class<?>, Schema<?>> cachedSchema = new ConcurrentHashMap<Class<?>, Schema<?>>();

    private static Objenesis objenesis = new ObjenesisStd(true);

    private static <T> Schema<T> getSchema(Class<T> cls) {
        Schema<T> schema = (Schema<T>) cachedSchema.get(cls);
        if (schema == null) {
            schema = RuntimeSchema.createFrom(cls);
            if (schema != null) {
                cachedSchema.put(cls, schema);
            }
        }
        return schema;
    }

    public static <T> byte[] serializer(T obj) {
        Class<T> cls = (Class<T>) obj.getClass();
        LinkedBuffer buffer = LinkedBuffer.allocate(LinkedBuffer.DEFAULT_BUFFER_SIZE);
        try {
            Schema<T> schema = getSchema(cls);
            return ProtostuffIOUtil.toByteArray(obj, schema, buffer);
        } catch (Exception e) {
            log.error("protobuf serialize fail");
            throw new IllegalStateException(e.getMessage(), e);
        } finally {
            buffer.clear();
        }
    }

    public static <T> T deserializer(byte[] bytes, Class<T> clazz) {
        try {
            T message = (T) objenesis.newInstance(clazz);
            Schema<T> schema = getSchema(clazz);
            ProtostuffIOUtil.mergeFrom(bytes, message, schema);
            return message;
        } catch (Exception e) {
            log.error("protobuf deserialize fail");
            throw new IllegalStateException(e.getMessage(), e);
        }
    }

}







David Yu

unread,
Jun 23, 2020, 7:06:31 AM6/23/20
to protostuff
Use ProtobufIOUtil

--
You received this message because you are subscribed to the Google Groups "protostuff" group.
To unsubscribe from this group and stop receiving emails from it, send an email to protostuff+...@googlegroups.com.


--
When the cat is away, the mouse is alone.
dyuproject.com
Reply all
Reply to author
Forward
0 new messages