JsonObject and BigDecimal support

[KevinC]

Hi,
We are in the process of migrating from 2.1.6 to 3.0.0 and have found that JsonObject no longer supports BigDecimal as a Number.
The code in Json.java method checkAndCopy has a specific check for this "val instanceof Number && !(val instanceof BigDecimal)" on line 84.

It seems it is by design that BigDecimal is not supported. Is there a reason for this?

To get around this we currently extend it and removed the object checks (not a great workaround).

Regards,
Kevin.

[bytor99999]

That is correct, It does NOT support BigDecimal anymore. Not sure why not, but you could take how many decimals you are taking BigDecimal to and multiply it to make it Long/Int size, put it in JsonObject, then on the other end divide to get it back into BigDecimal on the receiving side. Just a suggestion. I think I actually had posted here the same question a month or so ago in the Google Group here..

Mark

[bytor99999]

Yep, found it.

https://groups.google.com/forum/#!searchin/vertx/BigDecimal/vertx/uns_ltkEOtA/gvAoVS60gRsJ

Mark

[Tim Fox]

I think this is probably because having a very large number from a BigDecimal in a JSON object is going to have problems being deserialized in other languages, as there's no BD there and it would need to be mapped to a long type (or equivalent)

--
You received this message because you are subscribed to the Google Groups "vert.x" group.
To unsubscribe from this group and stop receiving emails from it, send an email to vertx+un...@googlegroups.com.
Visit this group at http://groups.google.com/group/vertx.
To view this discussion on the web, visit https://groups.google.com/d/msgid/vertx/d493410e-6012-4947-b894-33e9f7d764c0%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Krzysztof Kowalczyk]

Hi,

BigDecimal is explicitly discarded but BigInteger is not so it does not seem consistent. Jackson can be configured to allow them both and would be nice to have this option in Java as in vertx 2.

I guess if this check is really needed you could make it configurable and on by default, throwing bit more meaningful exception.

Regards,

Krzysztof

[Paulo Lopes]

On Wednesday, August 26, 2015 at 11:07:19 AM UTC+2, Krzysztof Kowalczyk wrote:

Hi,

There is a request in the backlog for plugable json API:

https://github.com/vert-x3/issues/issues/44

when and if this gets implemented then we cannot make any assumptions on Jackson, probably we need to give a round of discussion on defining a API that is not related to Jackson.

As Tim already said, BigInteger/BigDecimal can cause interop issues with other languages/libraries, for example on browsers javaScript max integer number is usually less the the max long from java which means that even using long or BigInteger will cause problems if the json is to be consumed by a browser.

Same for decimail values where a JavaScript Number is a 64bit floating point value using a BigDecimal can lead to precision errors or even parsing errors.

[Krzysztof Kowalczyk]

Hi Paulo,

I agree with everything you written, that's why I said the check is inconsistent. If the aim of this line (Json.java):
} else if (val instanceof Number && !(val instanceof BigDecimal)) {

is to assure interoperability - it is not achieved. We still allow BigInteger and do not check long values...
So there should be more checks or no check at all - if someone is using BigDecimal they must be aware that they are hard to consume by external world in general.

Regards,
Krzysztof

[Guido Medina]

What I'm doing is to just treat messages as String and at the very end using Jackson instead, I totally forgot JsonObject from Vert.x exists.

Most applications just want to return the BigDecimal value in Java, how the client deals with such value, say "0.1" is up to the client but limiting the server side to completely evaporate such important type as BigDecimal, not a good decision, not sure what type of applications you guys think Vert.x 3 is used for?

Any financial application will use BigDecimal and most likely to be in Java for the server side so ... what do we do, convert back and forth or just say ... you know what, give me a JsonNode from that String and then getDecimal value, problem solved, at least for us, now for the server side where decimals are important you just rendered JsonObject useless, nobody is going to use JsonOject on one half of the application and JsonNode in the other half? why would a developer put himself thru such inconsistency?

Sorry for my rant but I just find this to be a bad decision, moving forward, treat everything as String and use JsonNode from Jackson, problem solved.

[Guido Medina]

Adding to this subject, is there a plugin/codec to send JsonNode messages between nodes as JsonObject without BigDecimal is not useful anymore for some applications, not without too much hassle.

[Tim Fox]

Maybe I'm missing something here but can't you just put/get the big decimals as strings in the JsonObject? I.e. something like:

BigDecimal myBigDecimal = ...

jsonObject.put("foo", myBigDecimal.toString());

and to get it:

BigDecimal bd = new BigDecimal(jsonObject.getString("foo"));

To view this discussion on the web, visit https://groups.google.com/d/msgid/vertx/672218b2-1b13-4c02-b58e-7ab0f3834b24%40googlegroups.com.

[Guido Medina]

I thought about that as eventually I will need to send inter-node messages, now for inter-operability not sure how client side will see "foo":"0.1" vs "foo":0.1 if it is expecting a number in general, server side wise I can do that indeed, with a small correction to keep a clean visual representation:

new BigDecimal().toPlainString()

Say the client receives a decimal value as string and sends it back as number as it can be the case for some client side JS frameworks like data tables, also the value could be use for sorting so representing a decimal as a string might not be wise, getting that json back might not be that straight forward, say if the string coming from the client is { ...., "foo":0.1}

For that case if I want my big decimal back I have so avoid JsonObject.

Best regards,

Guido.

[Krzysztof Kowalczyk]

You can create subclass:

public class J extends io.vertx.core.json.JsonObject { public J() { super(); } public J(Map<String, Object> map) { super(map); } public J(String json) { super(json); } @Override public io.vertx.core.json.JsonObject put(String key, Object value) {     //Skipping any validation present in JsonObject     getMap().put(key, value); return this;   } } And use it. You will need to turn on BigDecimals/BigIntegers in mapper as usual.

[Paulo Lopes]

Hi,

If your main concern is interop then you should avoid BigDecimal at all, this question has already been asked before:

https://groups.google.com/d/msg/vertx/uns_ltkEOtA/wByb35eIwm0J

If you want interop with floating point precision you should be aware that ECMAScript spec says that floating point values in javascript have a max magnitude of 2^53 bits therefore cannot handle the full 64bit of a java double range. When you pick big decimal you can go with even higher precision than 64 bits.

Alternatively if you want to use the full power of jackson, it may sound a bit strange but you could write your own eventbus codec that just uses jackson to encode your java objects to json the way you want:

http://vertx.io/docs/apidocs/io/vertx/core/eventbus/MessageCodec.html

[Guido Medina]

You are looking at it from very theoretical point of view, meaning, pragmatically speaking that's not how applications are designed, by design applications will need BigDecimals on the server side regardless of JavaScript limitation, the chances of using all the bits in practice are from slim to none as most applications use BigDecimal to represent a human value, a sum, or a factor and the only way to make sure you are not losing precision is by representing that as a BigDecimal in the server side, say, factors, I wan't 0.5 to be 0.5, not because of some weirdness to be 0.4999999999, that would be catastrophic for most applications that deal with money.

Ask developers around if they would risk money just because of "JavaScript" limitations, why would recommend any sane developer not to use BigDecimal on the server side? Because of some improbable theory?

I know what you are saying, I understand the limitation, but in practice it is more harmful than anything else you are trying to avoid.

As a side note, I still need a nice visual representation on the client side of a decimal value, like 0.003 and not 3^10-e3 or something similar so for client side Strings will be likely to be used, not for server side inter-node communication it will be smaller and faster to send a BigDecimal than converting back and forth to String.

My question again, why do you limit Json from storing a BigDecimal in Java? Because of JavaScript? The weight of such argument isn't enough.

Hope that helps,

Guido.

[Krzysztof Kowalczyk]

Hi Paulo, Tim, 

First, BigInteger is allowed in Vertx 3 JsonObject, only BigDecimal is throwing exception:

https://github.com/eclipse/vert.x/blob/master/src/main/java/io/vertx/core/json/Json.java

So as I mention some time ago the interop argument is not valid as you still allow all longs, doubles and BigInteger. Thus there is no value added by restricting only BigDecimal other than confusion. 

If you provide validation against specs it should actually validate something, it would be slower that's why it should be optional.

If you use Groovy  "(​1 / 2 )​.class" is BigDecimal, so using JsonObject in Groovy is particularity painful.

Regards,

Krzysztof

[Jez P]

Keep in mind you can write verticles (i.e. server-side code) in Javascript in vert.x so interop is not to be ignored here. That said, I do understand Krzystof's point that BigInteger is permitted and presumably is also problematic in Javascript. I don't understand why permit one but not the other. 

However, saying "it will have to be BigDecimal on the server" is rather ignoring the use of Javascript to write verticles. Presumably whatever technology is used for Javascript verticles is subject to the same limitations as Javascript, unless they just ignore that bit of the Javascript specification.

[Krzysztof Kowalczyk]

Hi Jez,

Javascirpt verticles can use BigDecimal without problem, just not as nice as Groovy:

http://www.tutorialspoint.com/java8/java8_nashorn.htm

It would be problematic if someone is sending BigDecimal value to web browser, but if someone is using BigDecimal with uknown external world, they are asking for troubles.

But that should be an option.

Regards.

Krzysztof

[Paulo Lopes]

I think i did not explain myself clear, when i mean interop i am not just considering running javascript on a JVM using nashorn I am also considering running JS on nodejs, google chrome, firefox, even internet explorer for the kicks :)

since we can bridge vert.x with browsers and soon with the TCP bridge to anything that can read and write json to a socket... our concern is not only how things work on the JVM but trully interop...

But as all of you already said maybe we should review the policy on allow BigInteger vs not BigDecimal and come to a better solution but we need to consider all this points of view not just verticles running on the JVM...

[Jez P]

Is internet explorer still a thing? ;)

[bytor99999]

On Friday, September 25, 2015 at 8:27:02 AM UTC-7, Jez P wrote:

Is internet explorer still a thing? ;)

Yes, it is that thing that we now hide in the closet so no one can see that we use it. But, yes, unfortunately there are still companies that that is all they use. One day it will be a figment of our imagination, like Unicorns are now.

But, as someone who was writing an app that was going to be gambling with real money. The GLI, or the Gaming licensing people require that we used BigDecimal for representing money. So we had to use it. Just as an FYI here, not really adding anything new to this discussion, so just ignore me.

Mark 

[Chris Kaminski]

On Friday, September 25, 2015 at 9:48:12 AM UTC-4, Guido Medina wrote:

Ask developers around if they would risk money just because of "JavaScript" limitations, why would recommend any sane developer not to use BigDecimal on the server side? Because of some improbable theory?

I thought popular wisdom was that you don't do money transactions in floating point simply due to the imprecise nature of floating point operations? 

 

[bytor99999]

Hence why people use BigDecimal. :D It doesn't have that problem. 

However, I do find myself that it has been easier to use ints as pennies and just move the decimal over two when I needed it to be in dollars.

Mark

[Jez P]

I'm not saying they _can't_ use them via the Java classes, however, surely the decoding from JsonObject into Javascript types shouldn't rely on underlying Java types being exposed like that. That rather defeats the aim of polyglot, if you can only write Javascript verticles if to deal with the default mode of passing data round you have to interact with the Java core libs directly. 

Personal view, for what it's worth, is that the Java impl of Json object should decode into BigDecimal, the Javascript impl should decode into whatever the largest JS decimal type is and if that explodes with a runtime error, then ask the Javascripters what they'd do with that value in node :)

[Gumpta Shirvan]

Hence why people use BigDecimal. :D It doesn't have that problem. 

However, I do find myself that it has been easier to use ints as pennies and just move the decimal over two when I needed it to be in dollars.

Mark

That's a trick they use in gaming.  Coinage in ad&d based games stored as copper.  Parse value into copper/silver/gold whatever for display only.

I think node.js people use this

https://www.npmjs.com/package/big-decimal 

[Jez P]

Does the big-decimal package for node provide a mean to decode a large (bigdecimal) value from Json into a big instance? This is what the original question is about, not using BigDecimal internally, but about support in JsonObject for bigdecimal. So what happens when node receives a JSON object containing a large decimal value and tries to extract that value? How about when it sends Json containing a big value? Does it send a whole object structure containing the members of a big?

Like I said, I think we should allow writing of BigDecimal values into Json objects. Json doesn't mandate size/precision limits for the numerics, so in theory you can manually send a piece of Json to one of these other languages which it would have a problem deserializing (or if talking system to system communication from any other Java application using BigDecimals). In other words, sorry but that's a problem the user of that language has to solve regardless. I don't think that limiting JsonObject is the answer. Saying "we can send unconsumable values back to the browser" runs into exactly the same concern with any other application which supports BigDecimal values (e.g. through using Jackson encoding to convert objects to Json). 

[Krzysztof Kowalczyk]

JsonObject is used for different things and thats why it should be flexible. On one side it is used to read config, parse JSON from internet, store values and share them internally. It is allso used to output JSON to "internet". 

Now, if I want to read config file, I want it to be without quotes and with comments and read numbers. It is not a valid JSON at all but it is widely supported and Vertx actually can read that.

If I want to use it internally, I want it to be able accept things like BigDecimal.

If I want it to output JSON and share it with external world, I would like to have option to assert that it will be correctly parsed by modern browsers and NodeJs, this includes checks for "double" etc.

This could be achived by having validation during printing not creating the object. It could still accept BigDecimal as long as precision is small enough.

This can also be achived during whole lifecycle of the object (which is not exactly the case now as you can parse JsonObject that will contain BigDecimals but you can't put them there yourself), but then it should be optional. So there are different options to support interoperability and flexibility.

Regards,

Krzysztof

[Paulo Lopes]

Hi everyone, lets try to transform this thread into something we can learn from, let just start by writing down what we would like to see in the JsonObject/JsonArray class definitions, it seems to be clear for everyone that BigDecimal should be allowed, so what is the expected behavior for decoding, also allow BigDecimal, try to match the smallest numerical type say 0.5 should it be parsed as a float, double, bigdecimal?

[Guido Medina]

Hi Paulo,

I'm not sure 100% of how Jackson treats the values, all I know is that they have the concept of a JsonNode which can be obtained as any other type, the problem I see you will face with Vert.x Json object is that it contains a Map with the final types so it would be too late to decide what to get from such key/value from the user perspective, Vert.x Json is one step further down the row compared to Jackson Json which allows to get the value as the type you decide, so maybe refactoring the internal Map inside Vert.x Json object might be needed to accomplish such thing.

Though an intermediary type will affect memory and performance compared to specific types as it is now. It will be hard for you to try to decide what to return so I would leave such decision to the user: getFloat(...), getDouble(...), getDecimal(...), etc.

Representing everything as String and converting internally maybe is not that ideal?, I would look at Jackson source code and see what can be copied from their way of doing it which is proven to be fast enough? I'm not suggesting to make an exact copy as I love the Vert.x Json class simplicity compared to Jackson's, but such simplicity I know is not free.

Just my thoughts on it, this will need more brains to add their two cents to this whole equation.

Best regards,

Guido.

[Gumpta Shirvan]

On Sunday, September 27, 2015 at 10:11:52 AM UTC-4, Paulo Lopes wrote:

Hi everyone, lets try to transform this thread into something we can learn from, let just start by writing down what we would like to see in the JsonObject/JsonArray class definitions, it seems to be clear for everyone that BigDecimal should be allowed, so what is the expected behavior for decoding, also allow BigDecimal, try to match the smallest numerical type say 0.5 should it be parsed as a float, double, bigdecimal?

Jackson defaults to double for the precision.

http://www.davismol.net/2015/04/22/jackson-set-bigdecimal-as-type-for-json-deserialization-of-decimal-numbers/ 

[Guido Medina]

There is a serialization feature that would treat all decimals as big decimal if set on the Object Mapper, it is explained on the same article you linked.