De-serialization stops for unknown field in JSON

332 views
Skip to first unread message

Ricardo De Lemos

unread,
Oct 28, 2014, 8:16:16 AM10/28/14
to proto...@googlegroups.com
Hi,

We are using Prostuff for JSON serialization/deserialization, for communication between our server and our different clients. These clients share a common Business logic written in Javascript, and we have used Protostuff to compile the .proto files for iOS and Android (Java).
The issue we have in Java, is that the Business Logic (Javascript) adds some fields to the JSON object received from the server, before sending this data to the Java layer, which uses the POJO generated by the .proto compiled.
This data added to the JSOn is not really needed by the Java layer, and can be ignored, but the de-serialization is throwing an exception, and stopping the de-serialization of the object.

I would expect that if the deserialization cannot find a match for a field, would not populate the field -basically ignoring it, but continue to the next field.

I could not find any way to "force" the de-serialization to ignore the unknown field, but it did not work. The only way I could make this work was to add the new field to the .proto and re-compile, but this is impractical in our case, because the business logic layer adds several fields in different messages for its internal use.

There is any way to make the JsonIOUtil.mergeFrom method more lenient, and simply ignore "unknown" fields, but continue the de-serialization of the object?


## this is the partial code used to de-serialize the messages (JSON objects)
   public static <T> T parseProtoJson(Class<T> cls, Schema<T> schema, String json) throws JSONException {
        try {
            T parsedProtostuffObj = cls.newInstance();
            JsonIOUtil.mergeFrom(json.getBytes(), parsedProtostuffObj, schema, false);                    // Dump Parsed object
            return parsedProtostuffObj;
        } catch (IOException e) {
            Log.e(LOG_TAG, "## JsonIOUtil FAILED TO PARSE PROTO_JSON for %s", cls.toString());
            e.printStackTrace();
            throw new JSONException("Unable to parse JSON");
        }
  }

## JSONIoUtil throws an exception during de-serializatiion

10-27 17:10:25.935  14661-14830/com.unify.ansible I/java﹕ [PubSubService] ## Received PROTO_JSON_EVENT '/localUser/init' --> {"isExtended":true,"shortName":"Lemos R.","isGuest":false,"hasSupportRole":false,"isTenantAdmin":false,"isSuspended":false,"isDeleted":false,"isInactive":false,"avatar":"content/images/icon-general-default-avatar-yellow.png","avatarLarge":"content/images/icon-general-default-avatar-yellow-XL.png","exchangeData":{},"locale":"en-US","userId":"52","lastName":"Ricardo","firstName":"Lemos","displayName":"Lemos Ricardo","state":"AVAILABLE","largeImageUri":"","smallImageUri":"","emailAddress":"some email","userPresenceState":{"userId":"52","state":"AVAILABLE","timeZoneOffset":240,"mobile":false,"inTransit":false,"latitude":1000,"longitude":1000},"tenantId":"4","roles":["USER"],"userState":"ACTIVE","clientId":"a5d15c27-dde6-40e8-81ff-c09b3b2dcf8f","company":"","jobTitle":"","updatedTimeStamp":0,"phoneNumbers":[],"emailAddresses":[]}
10-27 17:10:25.945  14661-14661/com.unify.ansible D/java﹕ [ConversationSelectorFragment] register on event bus
10-27 17:10:25.955  14661-14830/com.unify.ansible E/java﹕ [JSONParsingManager] ## JsonIOUtil FAILED TO PARSE PROTO_JSON for class net.ansible.protobuf.user.User
10-27 17:10:25.955  14661-14830/com.unify.ansible W/System.err﹕ com.dyuproject.protostuff.JsonInputException: Unknown field: exchangeData on message net.ansible.protobuf.user.User
10-27 17:10:25.955  14661-14830/com.unify.ansible W/System.err﹕ at com.dyuproject.protostuff.JsonInput.readFieldNumber(JsonInput.java:213)
10-27 17:10:25.955  14661-14830/com.unify.ansible W/System.err﹕ at com.dyuproject.protostuff.JsonInput.readFieldNumber(JsonInput.java:137)
10-27 17:10:25.955  14661-14830/com.unify.ansible W/System.err﹕ at net.ansible.protobuf.user.User.mergeFrom(User.java:424)
10-27 17:10:25.955  14661-14830/com.unify.ansible W/System.err﹕ at net.ansible.protobuf.user.User.mergeFrom(User.java:20)
10-27 17:10:25.955  14661-14830/com.unify.ansible W/System.err﹕ at com.dyuproject.protostuff.JsonIOUtil.mergeFrom(JsonIOUtil.java:340)
10-27 17:10:25.955  14661-14830/com.unify.ansible W/System.err﹕ at com.dyuproject.protostuff.JsonIOUtil.mergeFrom(JsonIOUtil.java:257)
10-27 17:10:25.955  14661-14830/com.unify.ansible W/System.err﹕ at com.dyuproject.protostuff.JsonIOUtil.mergeFrom(JsonIOUtil.java:240)
10-27 17:10:25.955  14661-14830/com.unify.ansible W/System.err﹕ at com.unify.ansible.common.JSONParsingManager.parseProtoJson(JSONParsingManager.java:54)

David Yu

unread,
Oct 30, 2014, 4:06:29 AM10/30/14
to protostuff
On Tue, Oct 28, 2014 at 8:16 PM, Ricardo De Lemos <ricardo...@gmail.com> wrote:
Hi,

We are using Prostuff for JSON serialization/deserialization, for communication between our server and our different clients. These clients share a common Business logic written in Javascript, and we have used Protostuff to compile the .proto files for iOS and Android (Java).
The issue we have in Java, is that the Business Logic (Javascript) adds some fields to the JSON object received from the server, before sending this data to the Java layer, which uses the POJO generated by the .proto compiled.
This data added to the JSOn is not really needed by the Java layer, and can be ignored, but the de-serialization is throwing an exception, and stopping the de-serialization of the object.

I would expect that if the deserialization cannot find a match for a field, would not populate the field -basically ignoring it, but continue to the next field.

I could not find any way to "force" the de-serialization to ignore the unknown field, but it did not work. The only way I could make this work was to add the new field to the .proto and re-compile, but this is impractical in our case, because the business logic layer adds several fields in different messages for its internal use.

There is any way to make the JsonIOUtil.mergeFrom method more lenient, and simply ignore "unknown" fields, but continue the de-serialization of the object?
First off, sorry for the late reply.  If your unknown fields are scalar fields, it is ignored by default.
If they are messages, the JsonInput class will need some tweaking.  It is doable though. 


## this is the partial code used to de-serialize the messages (JSON objects)
   public static <T> T parseProtoJson(Class<T> cls, Schema<T> schema, String json) throws JSONException {
        try {
            T parsedProtostuffObj = cls.newInstance();
            JsonIOUtil.mergeFrom(json.getBytes(), parsedProtostuffObj, schema, false);                    // Dump Parsed object
            return parsedProtostuffObj;
        } catch (IOException e) {
            Log.e(LOG_TAG, "## JsonIOUtil FAILED TO PARSE PROTO_JSON for %s", cls.toString());
            e.printStackTrace();
            throw new JSONException("Unable to parse JSON");
        }
  }

## JSONIoUtil throws an exception during de-serializatiion

10-27 17:10:25.935  14661-14830/com.unify.ansible I/java﹕ [PubSubService] ## Received PROTO_JSON_EVENT '/localUser/init' --> {"isExtended":true,"shortName":"Lemos R.","isGuest":false,"hasSupportRole":false,"isTenantAdmin":false,"isSuspended":false,"isDeleted":false,"isInactive":false,"avatar":"content/images/icon-general-default-avatar-yellow.png","avatarLarge":"content/images/icon-general-default-avatar-yellow-XL.png","exchangeData":{},"locale":"en-US","userId":"52","lastName":"Ricardo","firstName":"Lemos","displayName":"Lemos Ricardo","state":"AVAILABLE","largeImageUri":"","smallImageUri":"","emailAddress":"some email","userPresenceState":{"userId":"52","state":"AVAILABLE","timeZoneOffset":240,"mobile":false,"inTransit":false,"latitude":1000,"longitude":1000},"tenantId":"4","roles":["USER"],"userState":"ACTIVE","clientId":"a5d15c27-dde6-40e8-81ff-c09b3b2dcf8f","company":"","jobTitle":"","updatedTimeStamp":0,"phoneNumbers":[],"emailAddresses":[]}
10-27 17:10:25.945  14661-14661/com.unify.ansible D/java﹕ [ConversationSelectorFragment] register on event bus
10-27 17:10:25.955  14661-14830/com.unify.ansible E/java﹕ [JSONParsingManager] ## JsonIOUtil FAILED TO PARSE PROTO_JSON for class net.ansible.protobuf.user.User
10-27 17:10:25.955  14661-14830/com.unify.ansible W/System.err﹕ com.dyuproject.protostuff.JsonInputException: Unknown field: exchangeData on message net.ansible.protobuf.user.User
10-27 17:10:25.955  14661-14830/com.unify.ansible W/System.err﹕ at com.dyuproject.protostuff.JsonInput.readFieldNumber(JsonInput.java:213)
10-27 17:10:25.955  14661-14830/com.unify.ansible W/System.err﹕ at com.dyuproject.protostuff.JsonInput.readFieldNumber(JsonInput.java:137)
10-27 17:10:25.955  14661-14830/com.unify.ansible W/System.err﹕ at net.ansible.protobuf.user.User.mergeFrom(User.java:424)
10-27 17:10:25.955  14661-14830/com.unify.ansible W/System.err﹕ at net.ansible.protobuf.user.User.mergeFrom(User.java:20)
10-27 17:10:25.955  14661-14830/com.unify.ansible W/System.err﹕ at com.dyuproject.protostuff.JsonIOUtil.mergeFrom(JsonIOUtil.java:340)
10-27 17:10:25.955  14661-14830/com.unify.ansible W/System.err﹕ at com.dyuproject.protostuff.JsonIOUtil.mergeFrom(JsonIOUtil.java:257)
10-27 17:10:25.955  14661-14830/com.unify.ansible W/System.err﹕ at com.dyuproject.protostuff.JsonIOUtil.mergeFrom(JsonIOUtil.java:240)
10-27 17:10:25.955  14661-14830/com.unify.ansible W/System.err﹕ at com.unify.ansible.common.JSONParsingManager.parseProtoJson(JSONParsingManager.java:54)

--
You received this message because you are subscribed to the Google Groups "protostuff" group.
To unsubscribe from this group and stop receiving emails from it, send an email to protostuff+...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.



--
When the cat is away, the mouse is alone.
- David Yu

Ricardo De Lemos

unread,
Oct 30, 2014, 11:15:24 AM10/30/14
to proto...@googlegroups.com
Hi David,

Thanks for your feedback. The fields added are not scalar, but some times messages or lists.  Do you have an example how could i change/overwrite the JsonInput so that it can ignoer the default?

David Yu

unread,
Oct 30, 2014, 11:35:55 AM10/30/14
to protostuff
On Thu, Oct 30, 2014 at 11:15 PM, Ricardo De Lemos <ricardo...@gmail.com> wrote:
Hi David,

Thanks for your feedback. The fields added are not scalar, but some times messages or lists.  Do you have an example how could i change/overwrite the JsonInput so that it can ignoer the default?
Take a look at the methods handleUnknownField and readFieldNumber.
You would basically check if the current token is START_OBJECT, then you set a flag where you ignore everything (recursively) until you see an END_OBJECT token.

Vijay Mohan

unread,
Jul 31, 2019, 12:12:42 AM7/31/19
to protostuff
Hi Ricardo,

  Were you able to solve the above issue, if Yes can you help me with some sample code changes.

-Vijay
To unsubscribe from this group and stop receiving emails from it, send an email to proto...@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.
Reply all
Reply to author
Forward
0 new messages