reducing garbage in benchmark

[rik...@ngs.hr]

I'm interested in reducing garbage created by the benchmark.

There are two sore points I think should be improved:

 1) object cloning - https://github.com/eishay/jvm-serializers/blob/master/tpc/src/serializers/JavaBuiltIn.java#L95

 2) byte[] result - https://github.com/eishay/jvm-serializers/blob/master/tpc/src/serializers/Serializer.java#L15

I don't see the reason for cloning since this is usually required only for external schema generators which can't work on existing POJOs.

byte[] result requires (without doing anything fishy) a new allocation, which is rather unfortunate. Better way to deal with it is to provide OutputStream argument and copy the result there.

This way benchmark should not be changed too much, since it will still use byte[] input.

I don't want to go overboard and reduce garbage during deserialization (in a sense that I expect a different instance of the MediaContent to be returned).

While this won't completely remove objections such as this: https://github.com/real-logic/simple-binary-encoding/issues/55

it will provide more fair comparison if someone want's to pay the penalty for conversion.

Any thoughts on those changes?

Regards,

Rikard

[Tatu Saloranta]

I think that sounds reasonable; not sure what the role of copying in that place is.

The only concern I have is that there were some frameworks/libs that cheated by reusing resulting of past serializations. It's debatable whether it should require this handling, or just kicking out codec implementations that use what amounts to benchmark-optimizations.

I agree in that retaining deser side to use byte[] makes sense; and on output, use of OutputStream makes sense.

But there may be a challenge with codecs that do not support use of OutputStreams -- but, if passing a bogus OutputStream (no op), there shouldn't be much overhead on just using byte[] result, writing that to bogus OutputStream.

-+ Tatu +-

--
You received this message because you are subscribed to the Google Groups "java-serialization-benchmarking" group.
To unsubscribe from this group and stop receiving emails from it, send an email to java-serialization-be...@googlegroups.com.
To post to this group, send email to java-serializat...@googlegroups.com.
Visit this group at http://groups.google.com/group/java-serialization-benchmarking.
For more options, visit https://groups.google.com/d/optout.

[Kannan Goundan]

The copying is because some serialization tools do things lazily.  At one point, the benchmark included a protobuf implementation that "deserialized" in zero time, but did the actual deserialization of each field only when it was needed.  That's a potentially useful feature, but the benchmark results were useless.

CapnProto is similar -- you pay the deserialization cost on every field access instead of up front (I think Google's FlatBuffers might do this as well).  The cost is low, but it's probably higher than a regular Java field access.

So we changed the benchmark to read all the fields after deserialization -- something that will probably happen anyway in a real-world use case.  The "read all the fields" operation was accomplished by having each serializer implement a copy function (the Transformer interface).  This was convenient because we needed this copy operation anyway: we used it to provide the source object for serializers that didn't use POJOs.

To avoid allocations, we could have each serializer implement a comparison function.  The downside is that it's more error prone -- if the copy function is buggy (e.g. forgetting to copy some fields) this error would be detected because the round-tripping would fail.  If the comparison function were buggy, it's harder to know :-\

We could go even further and allow serializers to recycle objects (assuming the objects are mutable), but I worry that this will be misleading.  People might see the results and say "hey, XXX format is twice as fast as the others" not realizing that you have to recycle objects to get that performance.  (This is the issue with trying to provide results that apply broadly vs trying to benchmark the absolute maximum performance possible.  Could be mitigated with better presentation.)

[Rikard Pavelic]

On Tue, 3 Nov 2015 14:39:42 -0800
Kannan Goundan <kan...@cakoose.com> wrote:

> To avoid allocations, we could have each serializer implement a
> comparison function. The downside is that it's more error prone --
> if the copy function is buggy (e.g. forgetting to copy some fields)
> this error would be detected because the round-tripping would fail.
> If the comparison function were buggy, it's harder to know :-\

I don't think there is a need for equals in each serializers.
This is the other issue I did not want to get into.
Basically I found it strange that serializers first defined
transformation function and then the serialization function.

Instead if they were provided with POJO object directly into
serialization function (no need for transformer), the ones which did
not supported POJO would have to implement transformation themselves
anyway.

Of course this is only applicable to the current version. It might be
that in the past this didn't make too much sense.

Anyway, that way only the bench POJOs would need to have comparison
function (and they already do).

> We could go even further and allow serializers to recycle objects
> (assuming the objects are mutable), but I worry that this will be
> misleading. People might see the results and say "hey, XXX format is
> twice as fast as the others" not realizing that you have to recycle
> objects to get that performance. (This is the issue with trying to
> provide results that apply broadly vs trying to benchmark the
> absolute maximum performance possible. Could be mitigated with better
> presentation.)

Yeah, I mentioned I didn't want to go overboard ;)
But thats a useful feature, although it could not be implemented with
zero allocations for this bench, since it has nested collections.

Regards,
Rikard

--
Rikard Pavelic
https://dsl-platform.com/
http://templater.info/

[Kannan Goundan]

Instead if they were provided with POJO object directly into
serialization function (no need for transformer), the ones which did
not supported POJO would have to implement transformation themselves
anyway.

Usually, when you use something like Protobuf, your code works with the Protobuf-generated classes directly.  There's no conversion to/from POJOs.  It would be unfair to have the benchmark include that conversion time.

[Sam Pullara]

I would probably have a few benchmarks as i think that we should be aware of the advantage of per-field deserialization. Having an object that has 10 fields and grabbing 10, 5 and 1 of them i think would be interesting. I think it would show the various strategies that each system uses. 

Sam

[Kannan Goundan]

We used to have two benchmarks.  One that pulled all the fields and one that pulled just the top-level fields.  That's why the Transformer class has a "shallowReverse" function.

I forget exactly why it was removed, but it could have been because it wasn't that typical of a use case and it was adding clutter to the results page.  (I feel like we've removed many sort-of-useful things to reduce the clutter.)

I think spending some time on improving the UI/UX of the results page would make it easier to include more data.  A while back I took a stab at trying to make it easier to filter things, but never really polished it: http://eishay.github.io/jvm-serializers/prototype-results-page/

[rik...@ngs.hr]

TLDR: without garbage 20% speed for serialization is possible and create will be nullified.

So I did some initial testing of changes and here are the results:

Old Protostuff:

min:74.54

max:4899.673

average:86.04783316078326ms deviation:8.080333160783255ms

-----------------------------------------------------------------------------

min:429.057

max:2079.188

average:465.73040217391167ms deviation:17.9744021739117ms

-----------------------------------------------------------------------------

min:643.0495

max:5948.684

average:694.1958037129713ms deviation:27.579303712971296ms

-----------------------------------------------------------------------------

Old json/dsl-platform:

min:44.296

max:6901.2685

average:49.83894883720957ms deviation:3.9214488372095744ms

-----------------------------------------------------------------------------

min:610.2865

max:5257.732

average:656.3769124178714ms deviation:22.289412417871404ms

-----------------------------------------------------------------------------

min:826.356

max:15202.3155

average:892.2987436997346ms deviation:36.36324369973454ms

-----------------------------------------------------------------------------

Old json/dsl-platform/databind (not currently included in bench)

min:55.5685

max:3643.4955

average:62.5230743507717ms deviation:4.6225743507717ms

-----------------------------------------------------------------------------

min:610.268

max:4916.0535

average:652.778143735286ms deviation:20.228643735286028ms

-----------------------------------------------------------------------------

min:818.764

max:13084.1555

average:885.4443308497441ms deviation:32.60333084974411ms

-----------------------------------------------------------------------------

After that I've added few changes discussed here.

In Serializer I've changed input signature from 

    S deserialize(byte[])

    byte[] serialize(S)

into 

    S deserialize(byte[], int)

    void serialize(S, OutputSteam)

I've introduced inherited ByteArrayOutputStream for access to underlying buffer

I've removed copy from JavaBuiltIn.mediaTransformer so it doesn't create useless garbage.

Rerun the tests for those three codecs...

Protostuff remained almost the same (+15 for serialization)

json/dsl-platform remained almost the same (-25 for serialization)

New json/dsl-platform/databind codec which can bind to an existing POJO improved a lot (this is equivalent of json/dsl-platform but instead of translating from existing POJO to DSL POJO, it uses existing POJO directly)

min:0.007

max:4.379

average:0.00952261257168601ms deviation:5.226125716860115E-4ms

-----------------------------------------------------------------------------

min:550.659

max:1193.018

average:578.8223589951363ms deviation:12.746358995136234ms

-----------------------------------------------------------------------------

min:812.383

max:16727.2635

average:878.1641930719179ms deviation:36.50219307191787ms

So it still generates garbage, but only on deserialization - I also tried deserializing into an existing instance, but there was no major difference, it still generated garbage (which is ok since it contains nested collections).

My take on this changes...

I think bench is certainly improved, since we can now introduce other libraries, such as SBE, FlatBuff and CNP without too much objections. The resulting comparison is much closer to the real world use case when somebody has an existing POJO and wants to use different codec for transport.

This also makes bench more POJO friendly and code-gen unfriendly, but I think that represents majority of use cases. Since conversion is being tracked and timed, it's fine to display results as such. We can reduce code-gen unfriendliness by allowing transformation into existing instances... but this too will create garbage (unless someone goes to the extreme).

Regards,

Rikard

To unsubscribe from this group and stop receiving emails from it, send an email to java-serialization-benchmarking+unsubscribe@googlegroups.com.
To post to this group, send email to java-serialization-benchm...@googlegroups.com.

Visit this group at http://groups.google.com/group/java-serialization-benchmarking.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "java-serialization-benchmarking" group.

To unsubscribe from this group and stop receiving emails from it, send an email to java-serialization-benchmarking+unsubscribe@googlegroups.com.
To post to this group, send email to java-serialization-benchm...@googlegroups.com.