Re: [jackson-user] Unexpected next token on entry to a deserializer

[Tatu Saloranta]

(note: cc:ing the new Jackson-user list  -- Codehaus listed will be phased out in near future).

On Tue, Jan 28, 2014 at 9:42 AM, Benson Margulies <bimar...@gmail.com> wrote:

Here we have a class that has been serialized polymorphically, so the
json looks like:

 "@class" : "com.basistech.dm.ListAttribute",
      "itemClass" : "com.basistech.dm.EntityMention",
      "items" : [ {
        "startOffset" : 16,
        "endOffset" : 39,
        "entityType" : "ORGANIZATION",
        "source" :
"gazetteer:/Users/benson/x/trunk/rlp/rlp/dicts/en-all-gazetteer-LE.bin",
        "normalized" : "Samsung Electronics"
      },

On entry to a custom deserializer, the next token is not the object
start, nor neither the field after the @class field. Line 9 has the
@class, but there's no column 50.

Using Jackson 2.3.0. I'm guessing I'm entered pointing at the value of @class.

com.fasterxml.jackson.databind.JsonMappingException: Unexpected token
(VALUE_STRING), expected FIELD_NAME: Expected itemClass
 at [Source: [B@6335605a; line: 9, column: 50]


public class ListAttributeDeserializer extends JsonDeserializer<ListAttribute> {
    @Override
    @SuppressWarnings("unchecked")
    public ListAttribute deserialize(JsonParser jp,
DeserializationContext ctxt) throws IOException {
        if (!jp.nextFieldName(new SerializedString("itemClass"))) {
            throw ctxt.wrongTokenException(jp, JsonToken.FIELD_NAME,
"Expected itemClass");
        }
        String itemClassName = jp.nextTextValue();
        Class<? extends BaseAttribute> itemClass = null;
        try {
            itemClass = (Class<? extends BaseAttribute>)
ctxt.findClass(itemClassName);
        } catch (ClassNotFoundException e) {
            throw new JsonMappingException("Failed to find class " +
itemClassName);
        }
        if (!jp.nextFieldName(new SerializedString("items"))) {
            throw ctxt.wrongTokenException(jp, JsonToken.FIELD_NAME,
"Expected items");
        }
        List<BaseAttribute> items = Lists.newArrayList();
        if (jp.nextToken() != JsonToken.START_ARRAY) { // what about nothing?
            throw ctxt.wrongTokenException(jp, JsonToken.START_ARRAY,
"Expected array of items");
        }
        while (jp.nextToken() != JsonToken.END_ARRAY) {
            items.add(jp.readValueAs(itemClass));
        }

        ListAttribute.Builder<BaseAttribute> builder = new
ListAttribute.Builder<BaseAttribute>((Class<BaseAttribute>)
itemClass);
        builder.setItems(items);
        return builder.build();
    }
}

I am guessing that the problem could be that JsonDeserializer does not implement (override):

deserializeWithType(...)

which is the method needed to handle polymorphic type case. So that may be what you will need to do here.

The reason this is needed is simply because type id inclusion with different JSON shapes (array/object/scalar) vary; while handling is mostly delegated, it can not be implemented in generic manner.

But you;ll want something like:

    @Override
    public final Object deserializeWithType(JsonParser jp, DeserializationContext ctxt,
            TypeDeserializer typeDeserializer)
        throws IOException, JsonProcessingException
    {
        return typeDeserializer.deserializeTypedFromObject(jp, ctxt);
   }

and see if that helps?

-+ Tatu +-

ps. If you want to use nextFieldName(), make sure to pre-construct SerializedName instances -- they are bit costly; or just use 'nextToken()' with 'getCurrentName()' instead. nextXxx() methods are for performance optimization more than convenience (they can do exact byte matching if ordering is expected).

 

[Benson Margulies]

Tatu, did you mean to write that the problem is that JsonDeserializer
does not override or that my class does not? I appreciate that the fix
is for me to add this in, but I am just curious as to where the fix
belongs in the first place. I'll take your advice about the
nextFieldName.

This sure was a 'fight with generic' party, to boot.

[Benson Margulies]

Your proposed change did not change the behavior at all. 

[Tatu Saloranta]

On Tue, Jan 28, 2014 at 11:38 AM, Benson Margulies <ben...@basistech.com> wrote:

Tatu, did you mean to write that the problem is that JsonDeserializer
does not override or that my class does not? I appreciate that the fix
is for me to add this in, but I am just curious as to where the fix

I meant that your deserializer should override it. JsonDeserializer can't implement it, since JSON shape is not known. It could, however, throw an exception to indicate that handling is not implemented (the reason it's not abstract is legacy backwards compatibility -- method itself was added in 1.5).

 

-+ Tatu +-

[Tatu Saloranta]

Ok. The next question is that of how deserialization is handled -- is this a root value, or a Collection of such values?

-+ Tatu +-

On Tue, Jan 28, 2014 at 11:41 AM, Benson Margulies <ben...@basistech.com> wrote:

Your proposed change did not change the behavior at all. 

--
You received this message because you are subscribed to the Google Groups "jackson-user" group.
To unsubscribe from this group and stop receiving emails from it, send an email to jackson-user...@googlegroups.com.
To post to this group, send email to jackso...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

[Benson Margulies]

Starting from the top of the entire doc:

{

  "attributes" : {

    

    "com.basistech.dm.EntityMention" : {

      "@class" : "com.basistech.dm.ListAttribute",

      "itemClass" : "com.basistech.dm.EntityMention",

      "items" : [ {

        "startOffset" : 16,

        "endOffset" : 39,

        "entityType" : "ORGANIZATION",

        "source" : "gazetteer:/Users/benson/x/trunk/rlp/rlp/dicts/en-all-gazetteer-LE.bin",

        "normalized" : "Samsung Electronics"

      }

    ]

 }

}

}

[Tatu Saloranta]

Thanks. Can you share the class definitions for things that do not have custom deserializer? Main pojo that has 'attributes' property I guess.

Alternatively a simple self-contained test would work. That might be simplest way for me to see what is causing the issue. There are no known problems in polymorphic type handling, so it is something with usage or interaction of features.

-+ Tatu +-

[Benson Margulies]

Let me see what I can do for you in the way of a testcase.

[Benson Margulies]

https://github.com/benson-basis/jackson-test-case

If the test 'passes' it means that the deserializer was called pointed to the unexpected location.

[Tatu Saloranta]

Ok, so JsonParser points to FIELD_NAME, because default type deserializer has handled the type id:

"@class" : "com.basistech.dm.SentenceBoundaries",

and this is why current token is not START_OBJECT which would otherwise be the case. This is an oddity due to the way type ids are 'peeled off', and standard types compensate for it, specifically BeanDeserializer.

So as with regular BeanDeserializer, code should do roughly following:

1. If current token is START_OBJECT, advance to next token

2. Loop over properties of the object; either verify that current token is FIELD_NAME, or that it is not END_OBJECT (either way works).

so for the particular deserializer what is missing is advancing of parser to next token; it is still pointing to name of field, "itemClass".

Addition of:

        if (jp.getCurrentToken() == JsonToken.START_OBJECT) {
            jp.nextToken();
        }
        jp.nextToken();
 

in ListAttributeDeserializer.deserialize() makes test pass. Code probably should do sanity checking to verify the order of properties is as expected, but other than that I think this is the fix needed.

Hope this helps,

-+ Tatu +-

[Benson Margulies]

Tatu,

Tatu, the code as written works: the first nextToken() skips the "itemClass": field name,  so that the subsequent getText() gets the value. I'm bugging you because it surprised me.

The 'if' you suggest would make it work even if the type didn't happen to be used in a context with @class.

Or did the test fail for you?

Anyway, if you remind me of which repo contains JsonDeserializer, I'll give you a patch with javadoc containing what you just wrote :-)

--benson

[Tatu Saloranta]

On Wed, Jan 29, 2014 at 3:55 AM, Benson Margulies <ben...@basistech.com> wrote:

Tatu,

Tatu, the code as written works: the first nextToken() skips the "itemClass": field name,  so that the subsequent getText() gets the value. I'm bugging you because it surprised me.

The 'if' you suggest would make it work even if the type didn't happen to be used in a context with @class.

Correct, since JsonParser typically points to START_OBJECT, not the first FIELD_NAME, unless type id handling needed to shuffle things around. I forget the full context here; while it is possible to reconstruct stream with buffering (which is done when actual reordering is needed, or type id exposed in addition to being used) there was something that caused this not to be done for this case.

So that if statement is needed to cover both polymorphic and regular use case. And that it is both an ugly and necessary thing to do. :-)

 

Or did the test fail for you?

Yes, test actually failed as it was written, wasn't skipping FIELD_NAME.

 

Anyway, if you remind me of which repo contains JsonDeserializer, I'll give you a patch with javadoc containing what you just wrote :-)

It's in jackson-databind, and improvements to Javadoc would be useful. The challenge is that what is really needed is how-to manual for writing "perfect custom (de)serializers"; this is one aspect of it.

-+ Tatu +-

[Benson Margulies]

Indeed. i forgot that I deleted a nextToken call from my working version. 

[Tatu Saloranta]

Ok. That makes sense then.

-+ Tatu +-