A class contains map type serialization problem

[liang Chen]

I need to perform RPC communication with .net.
We have defined a class containing the map type：
public class TestMapDto { public int TestNumber; public Map<String, String> Pairs; }
I assign a value to this class and use ProtobufIOUtil for serialization：
Map<String, String> hashMap = new TreeMap<>(); 

hashMap.put("test", "handsome"); 

TestMapDto testMapDto = TestMapDto.builder().TestNumber(111).Pairs(hashMap).build();
The resulting byte array is as follows：

8,111,18,18,10,16,10,4,116,101,115,116,18,8,104,97,110,100,115,111,109,101

But the result after serializing the native Protobuf file is as follows，

8,111,18 ,16,10,4,116,101,115,116,18,8,104,97,110,100,115,111,109,101

They have different results after serialization，Causes .net not to deserialize properly
Hope for help,Sincerely

[mohammadreza pasha]

hi 

protobuf net have some change to native protostuf for better performance.

if you check your byte array difference consider that in bytes 4 and 5 you dont have 18,10 in native serializer . 10 is id of inner class and 18 is byte lengths of inner class . you must simulate protobuf serialization that generate exact what you need in your native .

my recommend is first to compile your c# protobuf class that generate .proto file and base on this generated file keep your trying.

--
You received this message because you are subscribed to the Google Groups "protostuff" group.
To unsubscribe from this group and stop receiving emails from it, send an email to protostuff+...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/protostuff/44dd478f-9a12-4f21-8a03-196545c6ac3f%40googlegroups.com.

[liang Chen]

Thanks for your help sincerely  mohammadreza，

My situation now is that using protostuff serialize a POJO Java class which is contains map type,

the bytes[] I got is diffent from  serializing by the Java class generated by proto.file,

English is not my mother tongue,The expression may not be clear.Let me show you codes:

First,the proto.file

syntax = "proto3";

option java_package = "com.frxs.remoting.demo.protobuf";

option java_outer_classname = "PbArrayModule";

message PbArray {

  map<string,string> dic =1;

}

The POJO class:

@Data
public class PbArray implements Serializable {

    @Tag(1)

    public HashMap<String, String> dic;

}

The Java class generated by proto.file I omitted.

The test method:

public static void main(String[] args) throws InvalidProtocolBufferException {

        // the proto.file generate java class serialization

        PbArrayModule.PbArray.Builder builder = PbArrayModule.PbArray.newBuilder();

        builder.putDic("test", "cc");

        PbArrayModule.PbArray pbArray = builder.build();

        byte[] bytes = pbArray.toByteArray();

        System.out.println("proto.file serialize:----- " + Arrays.toString(bytes));

        // using ProtobufIOUtil

        HashMap<String, String> aMap = new HashMap<>(1);

        aMap.put("test", "cc");

        PbArray array = new PbArray();

        array.setDic(aMap);

        RuntimeSchema<PbArray> schema = RuntimeSchema.createFrom(PbArray.class);

        LinkedBuffer buffer = LinkedBuffer.allocate(LinkedBuffer.DEFAULT_BUFFER_SIZE);

        byte[] bytes1 = ProtobufIOUtil.toByteArray(array, schema, buffer);

        System.out.println("ProtobufIOUtil serialize: " + Arrays.toString(bytes1));

        PbArrayModule.PbArray pbArray1 = PbArrayModule.PbArray.parseFrom(bytes1);

        System.out.println("using proto.file's way deserialize ProtobufIOUtil bytes1:");

        System.out.println(pbArray1.toString());

    }

Test method console output：

proto.file serialize:----- [10, 10, 10, 4, 116, 101, 115, 116, 18, 2, 99, 99]

ProtobufIOUtil serialize: [10, 12, 10, 10, 10, 4, 116, 101, 115, 116, 18, 2, 99, 99]

using proto.file's way deserialize ProtobufIOUtil bytes1:

dic {

  key: "\n\004test\022\002cc"

  value: ""

}

As you can see, 
The extra bytes are caused by ProtobufIOUtil.

And the bytes serialized by ProtobufIOUtil cannot be deserialized normally by native protobuf.

Therefore, net probuf cannot be deserialized normally.

在 2020年5月13日星期三 UTC+8下午5:11:17，mohammadreza pasha写道：

To unsubscribe from this group and stop receiving emails from it, send an email to proto...@googlegroups.com.

[mohammadreza pasha]

hi

if your issue is communicate between protobufnet library in c# and protostuff library in java you must clone github protostuff and change it to generate byte array as exact protobufnet generated.

this release of protostuff that i attach for you will help you . please try it and report me the result

To unsubscribe from this group and stop receiving emails from it, send an email to protostuff+...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/protostuff/dd3774ee-48a5-4462-9ea5-e45c39de2dd3%40googlegroups.com.

[Wayne Chan]

Hello ,I'm the poster's colleague , and this question actually does not relate to c# .

When I use Original Protobuf Serializer In Java (proto file way)  and Protostuff Serializer to serialize same Pojo , I can't get the same bytes content . 

Here is the code :

Origninal Proto file way:

User.Proto

syntax = "proto3";

message User {

   string userName = 1;

   map<string,string> testMap = 2;

}

demo:

UserOuterClass.User user = UserOuterClass.User.newBuilder()
                .setUserName("BB")
                .putTestMap("aa","AA")
                .build();
byte[] bytes = user.toByteArray();
//output bytes :10,2,66,66,18,8,10,2,97,97,18,2,65,65

Protostuff Way:

Pojo:

@Data
public class ProtostuffUser implements Serializable {  
  @Tag(1)
  private String userName;
  @Tag(2)
  private Map<String, String> testmap;
}

demo:

ProtostuffUser user = new ProtostuffUser();
user.setUserName("BB");
user.setTestmap(new HashMap<>());
user.getTestmap().put("aa","AA");
byte[] output= ProtoStuffUtil.serializer(user);
//output bytes :10,2,66,66,19,11,10,2,97,97,18,2,65,65,12,20

output bytes: 

origin:        10,2,66,66,18,8,  10,2,97,97,18,2,65,65

protostuff : 10,2,66,66,19,11,10,2,97,97,18,2,65,65,12,20

So I can't communicate protostuff with origin Protobuf when Pojo contains map

 

Plz help ...   :-)

Appendix：

version:

<dependency>
            <groupId>com.google.protobuf</groupId>
            <artifactId>protobuf-java</artifactId>
            <version>3.5.1</version>
        </dependency>
        <dependency>
            <groupId>io.protostuff</groupId>
            <artifactId>protostuff-core</artifactId>
            <version>1.7.2</version>
        </dependency>
        <dependency>
            <groupId>io.protostuff</groupId>
            <artifactId>protostuff-runtime</artifactId>
            <version>1.7.2</version>
        </dependency>

ProtoStuffUtil

public class ProtoStuffUtil {

    private static Logger log = LoggerFactory.getLogger(ProtoStuffutil.class);

    private static Map<Class<?>, Schema<?>> cachedSchema = new ConcurrentHashMap<Class<?>, Schema<?>>();

    private static Objenesis objenesis = new ObjenesisStd(true);

    private static <T> Schema<T> getSchema(Class<T> cls) {

        Schema<T> schema = (Schema<T>) cachedSchema.get(cls);

        if (schema == null) {

            schema = RuntimeSchema.createFrom(cls);

            if (schema != null) {

                cachedSchema.put(cls, schema);

            }

        }

        return schema;

    }

    public static <T> byte[] serializer(T obj) {

        Class<T> cls = (Class<T>) obj.getClass();

        LinkedBuffer buffer = LinkedBuffer.allocate(LinkedBuffer.DEFAULT_BUFFER_SIZE);

        try {

            Schema<T> schema = getSchema(cls);

            return ProtostuffIOUtil.toByteArray(obj, schema, buffer);

        } catch (Exception e) {

            log.error("protobuf serialize fail");

            throw new IllegalStateException(e.getMessage(), e);

        } finally {

            buffer.clear();

        }

    }

    public static <T> T deserializer(byte[] bytes, Class<T> clazz) {

        try {

            T message = (T) objenesis.newInstance(clazz);

            Schema<T> schema = getSchema(clazz);

            ProtostuffIOUtil.mergeFrom(bytes, message, schema);

            return message;

        } catch (Exception e) {

            log.error("protobuf deserialize fail");

            throw new IllegalStateException(e.getMessage(), e);

        }

    }

}

[David Yu]

Use ProtobufIOUtil

--
You received this message because you are subscribed to the Google Groups "protostuff" group.
To unsubscribe from this group and stop receiving emails from it, send an email to protostuff+...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/protostuff/3bc08290-0ad0-4200-b1d9-e1af3bfa4865o%40googlegroups.com.

--

When the cat is away, the mouse is alone.
dyuproject.com