local event bus without serializations or codec

[Gadi Eichhorn]

Hi All,

Is there a way to work with event bus locally only?

1. no serialization of messages (POJO)

2. not defining codecs for each message (pojo ) type because I want the same "codec" for all messages

3. local send / publish (non cluster)

4. local consuming 

I have a vertx project on ARM (Raspberry PI) and the serialization is killing the performance.

I don't need cluster on polyglot features. just local message us as thin as possible.

Can you create a generic codec for all messages?

vertx.eventBus().registerDefaultCodec(CheckHealthCommand.class, new CheckHealthCommandCodec());

This forces you to still use a default codec per POJO ?

Thanks in advance.

G.

[Prasad C]

May not be exact answer, but I will share my experience, 

We were in similar situation a while ago and ended up drawing three options 

1) Have a separate codec for each type,

2) Use JavaScript every where in project, so that we save time of map from/ map to

3) and POJOs

Finally, we have chosen POJOs and we have plan to write Codecs in next version depending on the requirements

Thanks

..pC

--
You received this message because you are subscribed to the Google Groups "vert.x" group.
To unsubscribe from this group and stop receiving emails from it, send an email to vertx+un...@googlegroups.com.
Visit this group at https://groups.google.com/group/vertx.
To view this discussion on the web, visit https://groups.google.com/d/msgid/vertx/7ea5aa20-aa6a-45c6-9ad9-9d0b639345ef%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Asher Tarnopolski]

well, working with the eventbus requires codecs. local delivery leverages local eventbus consumers, not bypassing it.
there is no problem to register the same codec for all pojos you have on vertx startup, so i guess this is not something that will affect the latency in runtime.
which i guess leaves you with a need to find the fastest way for your pojos serialization. a couple of years ago we did some benchmarking to compare java default serialization and jackson and ended up with jackson. if you go with json and don't need features like Jackson's custom de/serializaers, you can for sure find faster solutions than jackson.

[Gadi Eichhorn]

So in the end, I found the solution I was after.

create a default codec for some generic class and only implement the transform method

public class LocalMessageCodec implements MessageCodec<Message, Message> {

    @Override
    public void encodeToWire(Buffer buffer, Message obj) {
        throw new UnsupportedOperationException(obj.toString());
    }

    @Override
    public Message decodeFromWire(int pos, Buffer buffer) {
        throw new UnsupportedOperationException(buffer.toString());
    }

    @Override
    public Message transform(Message command) {
        return command;
    }

    @Override
    public String name() {
        return Message.class.getSimpleName();
    }

    @Override
    public byte systemCodecID() {
        return -1;
    }

}

only interested in local messages...

MessageConsumer<T> consumer = vertx.eventBus()
    .localConsumer(type.getName(),

Now all you have to do is add a codec for each message options

private final DeliveryOptions options = new DeliveryOptions()
    .setCodecName(Message.class.getSimpleName())

vertx.eventBus().publish(event.getClass().getName(), event, options);

as you can see my codec will throw an exception if there is an attempt to send anything cluster-wide.

Hope this helps anyone else...

[Asher Tarnopolski]

pay attention that you are ending up sharing objects between event loops. if your consumer changes your message, it's changed also in your producer.  

the thread safety is gone. 

[Gadi Eichhorn]

Good point Asher!

We have no choice, performance is so crap on the RPI we had to go this way. We use messages as immutable objetcs.  Now that I dont need to use Json. encode and decode I can make all the fields final.

[Julien Viet]

interesting way, thanks for sharing that, I'll keep it in mind if the same question is asked again !

-- 
You received this message because you are subscribed to the Google Groups "vert.x" group.
To unsubscribe from this group and stop receiving emails from it, send an email to vertx+un...@googlegroups.com.
Visit this group at https://groups.google.com/group/vertx.

To view this discussion on the web, visit https://groups.google.com/d/msgid/vertx/6559c8e8-27be-4572-b721-0530005d2e6e%40googlegroups.com.

[Julien Viet]

indeed,

objects should be correctly published before being sent in the bus.

--
You received this message because you are subscribed to the Google Groups "vert.x" group.
To unsubscribe from this group and stop receiving emails from it, send an email to vertx+un...@googlegroups.com.
Visit this group at https://groups.google.com/group/vertx.

To view this discussion on the web, visit https://groups.google.com/d/msgid/vertx/1edf87e5-27d3-49b1-8d95-17921717a643%40googlegroups.com.

[Gadi Eichhorn]

Hi Julien,

It will be interesting to have a full local bus. with no codec or message transformations. On some devices, the transformation is a killer. We moved from blocking event-loop for 2,4,6,8, .. sec to nothing just by avoiding JSON.    I know this is not what the bus was designed for but we had to make a serious workaround to make it work at all. If you only interested in local messages we should have a more direct path.

Another issue I found on the RPI is the bus message headers. it only supports Strings and that is another conversion we had to avoid. maybe it is the multimap performance that is blocking. I ended up making a wrapper class with the headers as properties. wrap each message with this class and deliver the properties to a custom message handler.

[Ronoel Junior]

Hi Gadi,

I needed to develop a similar solution, but I also had to use it in a cluster.

Although, I didn't need to preserve the polyglot feature, so I just serialized my object to make it faster.

Maybe it can improve your solution.

@Log
@RequiredArgsConstructor(staticName = "of")
public class GenericCodec<T extends Object> implements MessageCodec<T, T> {

    private final Class<T> entityClass;
    private final String codecName;

    @Override
    public void encodeToWire(Buffer buffer, T s) {
        ObjectOutput out = null;
        ByteArrayOutputStream bos = new ByteArrayOutputStream();

        try {
            out = new ObjectOutputStream(bos);
            out.writeObject(s);
            out.flush();
            buffer.appendInt(bos.size());
            buffer.appendBytes(bos.toByteArray());

        } catch (IOException ex) {
            log.log(Level.SEVERE, "Encode problem", ex);
        } finally {
            try {
                bos.close();
            } catch (IOException ex) {
                log.log(Level.SEVERE, "Close encode problem", ex);
            }
        }
    }

    @Override
    public T decodeFromWire(int pos, Buffer buffer) {

        // Length of JSON
        int length = buffer.getInt(pos);

        // Jump 4 because getInt() == 4 bytes
        ByteArrayInputStream bis = new ByteArrayInputStream(buffer.getBytes(pos += 4, pos += length));
        ObjectInput in = null;

        T u = null;
        try {
            in = new ObjectInputStream(bis);
            u = (T) in.readObject();
        } catch (Exception ex) {
            log.log(Level.SEVERE, "Decode problem", ex);
        } finally {
            try {
                if (in != null) {
                    in.close();
                }
            } catch (IOException ex) {
                log.log(Level.SEVERE, "Close decode problem", ex);
            }
        }

        return u;
    }

    @Override
    public T transform(T t) {
        return t;
    }

    @Override
    public String name() {
        return this.codecName;

[Julien Viet]

Hi,

maybe we could try to move the serialization/transformation logic into an event bus interceptor and this interceptor could be removed in case like your ?

Julien

To view this discussion on the web, visit https://groups.google.com/d/msgid/vertx/081dfab3-ceb5-49aa-bcaa-d20538e9bf2e%40googlegroups.com.

[Gadi Eichhorn]

Hi,

I don't know much about interceptor.

Any String search, write, read or other operations should be limited unless needed. 

1. the default decode should be a default. no need to add a codec to each message. I think this may improve search for codec per message. If you need a specific codec then add it to the message options. 

2. Separate local codec from cluster decode. Have a different codec class for the two  paths. It will make the API clear too. At the moment is not clear which serialization method is used and when.

 

3. maybe add send and publish to local as a more specific way of working. publishLocal & sendLocal 

Thanks,

G.