How to filter out fields based on a dynamic condition?

[Clément Poulain]

Hi there,

I'm using Jackson 2.7.5 with Jersey 2.22.1 (run on Grizzly 2.3.23) to build a REST API.

I have a class that looks like:

public class Task {

   List<Update> updates = new ArrayList<>();

}

public class Update {

    public final long timestamp; // in seconds
    public final String message;

}

When serializing it during a GET request, I would like to filter out somes updates, based on a query parameter.
Something like: "GET /task/id/?minTimestamp=1234" would serialize a Task with only updates which timestamp is > 1234.

For example "GET /task/id/" would return:
{
   updates: [ {
      timestamp: 0000,
      message: "message 1"
   }, {
      timestamp: 0123,
      message: "message 2"
   }, {
      timestamp: 4567,
      message: "message 3"
   } ]
}

and "GET /task/id/?minTimestamp=1234" would return only:
{
   updates: [ {
      timestamp: 4567,
      message: "message 3"
   } ]
}

I have registered a module to my ObjectMapper with a BeanSerializerModifier to achieve that, but I'm stuck in propagating "minTimestamp=1234" from the query parameter to my BeanSerializerModifier.
I tried to use some @Context injection, but it does not looks to work, as the BeanSerializerModifier is not managed by Jersey.

What would be the best way to achieve that?

Thanks a lot for any ideas!

[Tatu Saloranta]

BeanSerializerModifier would probably not help here either because it only gets called once when constructing serializer for a type, and them it is cached (new instances may be created via ContextualSerializer, but that's still just once per property serializer is to be used for).

It would allow you to wrap serializer, but not do dynamical filtering itself.

 

What would be the best way to achieve that?

Of facilities provided, JsonFilter could help, explained f.ex here:

http://www.baeldung.com/jackson-serialize-field-custom-criteria

but otherwise what is usually done is, I think, two-pass processing: first serialize from POJO into JsonNode or Map, and then explicitly modify result (remove entries or change).

-+ Tatu +

 

Thanks a lot for any ideas!

--
You received this message because you are subscribed to the Google Groups "jackson-user" group.
To unsubscribe from this group and stop receiving emails from it, send an email to jackson-user+unsubscribe@googlegroups.com.
To post to this group, send email to jackso...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Clément Poulain]

BeanSerializerModifier would probably not help here either because it only gets called once when constructing serializer for a type, and them it is cached (new instances may be created via ContextualSerializer, but that's still just once per property serializer is to be used for).

It would allow you to wrap serializer, but not do dynamical filtering itself.

 

OK, so BeanSerializerModifier will not work here...
 

Of facilities provided, JsonFilter could help, explained f.ex here:

http://www.baeldung.com/jackson-serialize-field-custom-criteria

I tried the BeanSerializerModifier after reading this link :-).
I'm already using JsonFilter for something else (fields filtering base on field name like in "GET /task?fields=name,id"), but I found it a bit complex to set-up (again it needs to be dynamic) and was looking for another solution for this new filtering.

Maybe I should use JsonFilter again for this, but I was afraid of pilling up filters every time I need such a thing. Do you think would be OK?
 

but otherwise what is usually done is, I think, two-pass processing: first serialize from POJO into JsonNode or Map, and then explicitly modify result (remove entries or change).

The problem with this approach is that it breaks the streaming behavior of Jersey. You have to serialize the whole object in memory before modifying it and send it trough the pipe. This is what I tried initially for the name filtering.
In my case it was eliminatory because my objects are big (collections of nested objects), resulting in very high memory consumption and bad performances.

Thanks for you answer, I'll keep you informed!

[Tatu Saloranta]

On Sat, Sep 10, 2016 at 6:45 AM, Clément Poulain <plo...@gmail.com> wrote:

BeanSerializerModifier would probably not help here either because it only gets called once when constructing serializer for a type, and them it is cached (new instances may be created via ContextualSerializer, but that's still just once per property serializer is to be used for).

It would allow you to wrap serializer, but not do dynamical filtering itself.

 

OK, so BeanSerializerModifier will not work here...
 

Of facilities provided, JsonFilter could help, explained f.ex here:

http://www.baeldung.com/jackson-serialize-field-custom-criteria

I tried the BeanSerializerModifier after reading this link :-).
I'm already using JsonFilter for something else (fields filtering base on field name like in "GET /task?fields=name,id"), but I found it a bit complex to set-up (again it needs to be dynamic) and was looking for another solution for this new filtering.

Maybe I should use JsonFilter again for this, but I was afraid of pilling up filters every time I need such a thing. Do you think would be OK?

I think there are many ways to build filters, including cases where you have a small number of filters, but each with complicated logic. One possible problem is that of passing settings/configuration into filters, regarding options/parameters set. It may be necessary to resort to using something like ThreadLocal, or, possibly `ContextAttributes` (key/value pairs you can define for ObjectReader/ObjectWriter, accessible via DeserializationContext/SerializerProvider, respectively).

So that may well be your best option I think.

 

 

but otherwise what is usually done is, I think, two-pass processing: first serialize from POJO into JsonNode or Map, and then explicitly modify result (remove entries or change).

The problem with this approach is that it breaks the streaming behavior of Jersey. You have to serialize the whole object in memory before modifying it and send it trough the pipe. This is what I tried initially for the name filtering.
In my case it was eliminatory because my objects are big (collections of nested objects), resulting in very high memory consumption and bad performances.

True, it would require full in-memory representation.

 

Thanks for you answer, I'll keep you informed!

--

 

Good luck!

-+ Tatu +-

[Clément Poulain]

I think there are many ways to build filters, including cases where you have a small number of filters, but each with complicated logic. One possible problem is that of passing settings/configuration into filters, regarding options/parameters set. It may be necessary to resort to using something like ThreadLocal, or, possibly `ContextAttributes` (key/value pairs you can define for ObjectReader/ObjectWriter, accessible via DeserializationContext/SerializerProvider, respectively).

So that may well be your best option I think.

I was not aware of ContextAttributes, many thanks!!
I managed to get my filtering working by transmitting the dynamic parameter to my custom serializer using the ContextAttributes; it is exactly what I was looking for :-)

One pitfall was that calling "withPerCallAttribute()" on an empty ContextAttributes (ContextAttributes.getEmpty()) is not a "mutator" as the javadoc pretends. It returns a new ContextAttributes, which took me a while to discover.
Maybe something to improve here? Either always initializing the "_nonShared" field (maybe a performance hit), or just reflect this in the documentation?

One last question...
I'm registering my custom serializer using "@JsonSerialize(using = UpdateSerializer.class)" on top of my "Update" class:

    final static class UpdateSerializer extends StdSerializer<Update> {

        @Override
        public void serialize(Update value, JsonGenerator jgen, SerializerProvider provider) throws IOException, JsonProcessingException {
            long maxTime = provider.getAttribute(MAX_TIME_KEY);
            if (value.getTimestamp() > maxTime) {
                // jgen.writeObject(value); // <= create stack overflow by calling UpdateSerializer.serialize() again
                jgen.writeStartObject();
                jgen.writeObjectField("timestamp", value.getTimestamp());
                jgen.writeObjectField("message", value.getMessage());
                jgen.writeEndObject();
            }
        }

    }

In the serialize method, I had to do the serialization "by hand", because "jgen.writeObject(value);" would result in a stack overflow exception:
    ...
    Update$UpdateSerializer.serialize(Object, JsonGenerator, SerializerProvider) line: 1   
    DefaultSerializerProvider$Impl(DefaultSerializerProvider).serializeValue(JsonGenerator, Object) line: 130   
    ObjectMapper.writeValue(JsonGenerator, Object) line: 2444   
    UTF8JsonGenerator(GeneratorBase).writeObject(Object) line: 355   
    Update$UpdateSerializer.serialize(Update, JsonGenerator, SerializerProvider) line: 44   
    Update$UpdateSerializer.serialize(Object, JsonGenerator, SerializerProvider) line: 1   
    DefaultSerializerProvider$Impl(DefaultSerializerProvider).serializeValue(JsonGenerator, Object) line: 130   
    ObjectMapper.writeValue(JsonGenerator, Object) line: 2444   
    UTF8JsonGenerator(GeneratorBase).writeObject(Object) line: 355   
    Update$UpdateSerializer.serialize(Update, JsonGenerator, SerializerProvider) line: 44   
    Update$UpdateSerializer.serialize(Object, JsonGenerator, SerializerProvider) line: 1
    ...

I understand what is going on (UpdateSerializer is registered, so it is used; normal), but I don't see how to call the "default" serializer here. I tried to use
                JsonSerializer<Object> ser = provider.findTypedValueSerializer(Object.class, true, null);
                ser.serialize(value, jgen, provider);
but it resulted in empty object, because "UnknownSerializer" is returned by "findTypedValueSerializer(Object.class, true, null)".

Do you miss something simple here?

Thanks again for you time and your precious help!

[Tatu Saloranta]

On Mon, Sep 12, 2016 at 9:30 AM, Clément Poulain <plo...@gmail.com> wrote:

I think there are many ways to build filters, including cases where you have a small number of filters, but each with complicated logic. One possible problem is that of passing settings/configuration into filters, regarding options/parameters set. It may be necessary to resort to using something like ThreadLocal, or, possibly `ContextAttributes` (key/value pairs you can define for ObjectReader/ObjectWriter, accessible via DeserializationContext/SerializerProvider, respectively).

So that may well be your best option I think.

I was not aware of ContextAttributes, many thanks!!
I managed to get my filtering working by transmitting the dynamic parameter to my custom serializer using the ContextAttributes; it is exactly what I was looking for :-)

One pitfall was that calling "withPerCallAttribute()" on an empty ContextAttributes (ContextAttributes.getEmpty()) is not a "mutator" as the javadoc pretends. It returns a new ContextAttributes, which took me a while to discover.
Maybe something to improve here? Either always initializing the "_nonShared" field (maybe a performance hit), or just reflect this in the documentation?

Documentation should reflect that this is a "mutant factory" method -- idiom of "withXxx()" is consistently used for this within Jackson, they are newer mutators -- and not claim that instance itself is changed.

Thanks, I'll see how to improve javadoc.

Not necessarily simple, but you did describe the obvious problem that once you register custom (de)serializer, that's what will be found.

It would also be tricky to know what the "original" deserializer would be, considering that all deserializers are pluggable by any number of modules; and lookup is in general via callbacks (ask module if it has deserializer for given type).

But there is a way to get your deserializer used via different mechanism: `BeanDeserializerModifier`. This is post-processing that occurs after databind has found deserializer to use. What you can do from this method:

    public JsonDeserializer<?> modifyDeserializer(DeserializationConfig config,
            BeanDescription beanDesc, JsonDeserializer<?> deserializer);

is to pass `deserializer` to custom deserializer, and then delegate to it when necessary.

If you do that just make sure to delegate `createDelegate` (so need to implement `ContextualDeserializer`), and make sure to use return value (possibly a new deserializer instance).

And I think you also better implement `resolve()` of `ResolvableDeserializer` similarly.

It may help to look at `StdDelegatingDeserializer` which does this; it's not complicated but you need to be aware of this since not delegating these methods may lead the original deserializer configuration in incomplete state (just for certain deserializers, or annotation overrides, but when that happens it takes a while find the root cause).

 

Thanks again for you time and your precious help!

You are welcome!

-+ Tatu +-

 

--

[Clément Poulain]

Not necessarily simple, but you did describe the obvious problem that once you register custom (de)serializer, that's what will be found.

It would also be tricky to know what the "original" deserializer would be, considering that all deserializers are pluggable by any number of modules; and lookup is in general via callbacks (ask module if it has deserializer for given type).

True. Even more true now that I've discovered my current implementation loose the first filter feature (the one based on names) :-)
 

But there is a way to get your deserializer used via different mechanism: `BeanDeserializerModifier`. This is post-processing that occurs after databind has found deserializer to use. What you can do from this method:

    public JsonDeserializer<?> modifyDeserializer(DeserializationConfig config,
            BeanDescription beanDesc, JsonDeserializer<?> deserializer);

is to pass `deserializer` to custom deserializer, and then delegate to it when necessary.

If you do that just make sure to delegate `createDelegate` (so need to implement `ContextualDeserializer`), and make sure to use return value (possibly a new deserializer instance).

And I think you also better implement `resolve()` of `ResolvableDeserializer` similarly.

It may help to look at `StdDelegatingDeserializer` which does this; it's not complicated but you need to be aware of this since not delegating these methods may lead the original deserializer configuration in incomplete state (just for certain deserializers, or annotation overrides, but when that happens it takes a while find the root cause).

 

So back to BeanSerializerModifier to register my custom serialize instead of "@JsonSerialize()":

    MAPPER.registerModule(new SimpleModule().setSerializerModifier(new BeanSerializerModifier() {
            @Override
            public JsonSerializer<?> modifySerializer(SerializationConfig config, BeanDescription desc, JsonSerializer<?> serializer) {
                if (Update.class.isAssignableFrom(desc.getBeanClass())) {
                    return new UpdateSerializer(serializer);
                }
                return serializer;
            }
        }));

I assume that you mean `createContextual` instead of `createDelegate`, so my custom serializer is now:

     public final static class UpdateSerializer extends JsonSerializer<Update> implements ContextualSerializer, ResolvableSerializer {

        private final JsonSerializer<Object> delegate;

        public UpdateSerializer(JsonSerializer<?> serializer) {
            this.delegate = (JsonSerializer<Object>) serializer;

        }

        @Override
        public void serialize(Update value, JsonGenerator jgen, SerializerProvider provider) throws IOException, JsonProcessingException {

            Object maxTimeObj = provider.getAttribute(UPDATE_MIN_TIMESTAMP_PARAMETER);
            if (maxTimeObj == null || (value.getTimestamp() > (long) maxTimeObj)) {
                delegate.serialize(value, jgen, provider);
            }
        }

        @Override
        public void resolve(SerializerProvider provider) throws JsonMappingException {
            if (delegate instanceof ResolvableSerializer) {
                ((ResolvableSerializer) delegate).resolve(provider);
            }
        }

        @Override
        public JsonSerializer<?> createContextual(SerializerProvider prov, BeanProperty property) throws JsonMappingException {
            JsonSerializer<?> delSer = delegate;
            if (delSer instanceof ContextualSerializer) {
                delSer = prov.handleSecondaryContextualization(delSer, property);
            }
            if (delSer == delegate) {
                return this;
            }
            return new UpdateSerializer(delSer);
        }

    }

It works as expected with both filters honored, this is great! :-)
Does it looks correct to you?

[Tatu Saloranta]

On Tue, Sep 13, 2016 at 5:28 AM, Clément Poulain <plo...@gmail.com> wrote:

Not necessarily simple, but you did describe the obvious problem that once you register custom (de)serializer, that's what will be found.

It would also be tricky to know what the "original" deserializer would be, considering that all deserializers are pluggable by any number of modules; and lookup is in general via callbacks (ask module if it has deserializer for given type).

True. Even more true now that I've discovered my current implementation loose the first filter feature (the one based on names) :-)
 

But there is a way to get your deserializer used via different mechanism: `BeanDeserializerModifier`. This is post-processing that occurs after databind has found deserializer to use. What you can do from this method:

    public JsonDeserializer<?> modifyDeserializer(DeserializationConfig config,
            BeanDescription beanDesc, JsonDeserializer<?> deserializer);

is to pass `deserializer` to custom deserializer, and then delegate to it when necessary.

If you do that just make sure to delegate `createDelegate` (so need to implement `ContextualDeserializer`), and make sure to use return value (possibly a new deserializer instance).

And I think you also better implement `resolve()` of `ResolvableDeserializer` similarly.

It may help to look at `StdDelegatingDeserializer` which does this; it's not complicated but you need to be aware of this since not delegating these methods may lead the original deserializer configuration in incomplete state (just for certain deserializers, or annotation overrides, but when that happens it takes a while find the root cause).

 

So back to BeanSerializerModifier to register my custom serialize instead of "@JsonSerialize()":

    MAPPER.registerModule(new SimpleModule().setSerializerModifier(new BeanSerializerModifier() {
            @Override
            public JsonSerializer<?> modifySerializer(SerializationConfig config, BeanDescription desc, JsonSerializer<?> serializer) {
                if (Update.class.isAssignableFrom(desc.getBeanClass())) {
                    return new UpdateSerializer(serializer);
                }
                return serializer;
            }
        }));

Yes, exactly like this.

 

I assume that you mean `createContextual` instead of `createDelegate`, so my custom serializer is now:

Yes.

 

     public final static class UpdateSerializer extends JsonSerializer<Update> implements ContextualSerializer, ResolvableSerializer {

        private final JsonSerializer<Object> delegate;

        public UpdateSerializer(JsonSerializer<?> serializer) {
            this.delegate = (JsonSerializer<Object>) serializer;
        }

        @Override
        public void serialize(Update value, JsonGenerator jgen, SerializerProvider provider) throws IOException, JsonProcessingException {
            Object maxTimeObj = provider.getAttribute(UPDATE_MIN_TIMESTAMP_PARAMETER);
            if (maxTimeObj == null || (value.getTimestamp() > (long) maxTimeObj)) {
                delegate.serialize(value, jgen, provider);
            }

Actually you can not omit serialization here: at this point a value generally has to be written (if within JSON Object), since name has been written.

So you would need to serialize something here; suppression of a property (name + value) can not be achieved from within serializer itself unfortunately.

 

        }

        @Override
        public void resolve(SerializerProvider provider) throws JsonMappingException {
            if (delegate instanceof ResolvableSerializer) {
                ((ResolvableSerializer) delegate).resolve(provider);
            }
        }

        @Override
        public JsonSerializer<?> createContextual(SerializerProvider prov, BeanProperty property) throws JsonMappingException {
            JsonSerializer<?> delSer = delegate;
            if (delSer instanceof ContextualSerializer) {
                delSer = prov.handleSecondaryContextualization(delSer, property);
            }
            if (delSer == delegate) {
                return this;
            }
            return new UpdateSerializer(delSer);
        }
    }

Yes that looks correct.

 

It works as expected with both filters honored, this is great! :-)
Does it looks correct to you?

Yes, other than the problem with suppressing from within serialize() method.

[Clément Poulain]

Actually you can not omit serialization here: at this point a value generally has to be written (if within JSON Object), since name has been written.

So you would need to serialize something here; suppression of a property (name + value) can not be achieved from within serializer itself unfortunately.

In my very precise case, this is actually working because Update are always members of a list, so there is no name.
Still this is good to know, I will keep that in a corner of my mind for the future.

I consider I'm done with my problem, so thanks again, this was very valuable!

[Tatu Saloranta]

Excellent!