bson binary user defined codes

[Marcel Timmerman]

Hello,

I'm writing a perl6 driver and wanted to add some types for which there are no BSON representations. This could be done in the binary type using the user definable codes 128 to 255.

But thinking this over, a driver could make mistakes when another driver is using the same codes for other types on the same database. The question is, is there any agreement on using these user-definable codes?

Some idea I have is that the user code first represents a programming language and after that, any coding for some type defined by the driver. If that also is documented on some shared place, then drivers could use some of the data written by other drivers.

Regards,
Marcel

[David Golden]

Hi, Marcel.

I haven't seen any such conventions in the wild.  I have to admit I'm skeptical a convention getting widespread adoption.  See https://xkcd.com/927/ :-)

What specific types are you interested in representing?  If there is really a type that might be useful for multiple languages, we could consider whether there's enough user interest to consider adding it as a formal BSON type.

Regards,

David

[Marcel Timmerman]

Hi David,

There are a lot of types in perl6 which might be interesting to save in a database. In a discussion with Dan Zwell (lefth on github)  we noted that it is perhaps needed to store a Rat or FatRat. The Rat can be converted to a Num with a test on precision loss, all controlled by several flags but the Num cannot be converted back to Rat. For these and some others, there are no representations and the only way is to store them in binary. I do not know if other languages have these types too so to create a formal type just for perl6 would be a bit too much. Also, having a formal type, it implies that the server can handle it too. So I think it is better to convert it to binary and think about a way to prevent confusion among different drivers.

Other interesting types are maybe Set, Bag, Seq. etc. Convert them to an Array can only return an Array and therefore loses the special behavior of the types.

Marcel

[David Golden]

Marcel,

Hypothetically, what about something like this:

• Add a new binary subtype like 0x06 for "language-specific type" where the first byte of the binary payloads is a tag representing a specific language
• Add a registry of language tags to the spec.  E.g. hypothetically imagine Perl 6 would be 0x0C
• Driver writers for a language are free to define a format for everything after that tag.  E.g. Perl 6 could define leading byte 0x01 to be a rational, composed of 2 int64's stored little-endian.

So, for example, assuming 0x06, 0x0C and 0x01 and two uint64s for the numerator and denominator of a rational number, a document with key "a" and value of Rat(1/3) could look like this:

1f00000005610012000000060c010100000000000000030000000000000000

Broken up for visibility:

1f000000 05   "a"00 12000000   06      0C   0101000000000000000300000000000000 00

length   type key   datalength subtype lang lang-specific-payload              term

Is that sort of what you had in mind?

David

--
You received this message because you are subscribed to a topic in the Google Groups "mongodb-dev" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/mongodb-dev/uYoSjz2VwI0/unsubscribe.
To unsubscribe from this group and all its topics, send an email to mongodb-dev+unsubscribe@googlegroups.com.
To post to this group, send email to mongo...@googlegroups.com.
Visit this group at https://groups.google.com/group/mongodb-dev.
To view this discussion on the web visit https://groups.google.com/d/msgid/mongodb-dev/9446c8f0-76ed-4458-8f23-260de95461a9%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

[Marcel Timmerman]

Hi David,

That's exactly what I had in mind, although I had the user definable codes 128-255 as a language code in mind which makes the encoding one byte shorter but that doesn't matter nowadays. In your example you will keep these codes free for other uses.

Regards,
Marcel

On Monday, October 30, 2017 at 9:02:28 PM UTC+1, David Golden wrote:

Marcel,

Hypothetically, what about something like this:

• Add a new binary subtype like 0x06 for "language-specific type" where the first byte of the binary payloads is a tag representing a specific language
• Add a registry of language tags to the spec.  E.g. hypothetically imagine Perl 6 would be 0x0C
• Driver writers for a language are free to define a format for everything after that tag.  E.g. Perl 6 could define leading byte 0x01 to be a rational, composed of 2 int64's stored little-endian.

So, for example, assuming 0x06, 0x0C and 0x01 and two uint64s for the numerator and denominator of a rational number, a document with key "a" and value of Rat(1/3) could look like this:

1f00000005610012000000060c010100000000000000030000000000000000

Broken up for visibility:

1f000000 05   "a"00 12000000   06      0C   0101000000000000000300000000000000 00

length   type key   datalength subtype lang lang-specific-payload              term

Is that sort of what you had in mind?

David

On Mon, Oct 30, 2017 at 6:31 AM, Marcel Timmerman <mt1...@gmail.com> wrote:

Hi David,

There are a lot of types in perl6 which might be interesting to save in a database. In a discussion with Dan Zwell (lefth on github)  we noted that it is perhaps needed to store a Rat or FatRat. The Rat can be converted to a Num with a test on precision loss, all controlled by several flags but the Num cannot be converted back to Rat. For these and some others, there are no representations and the only way is to store them in binary. I do not know if other languages have these types too so to create a formal type just for perl6 would be a bit too much. Also, having a formal type, it implies that the server can handle it too. So I think it is better to convert it to binary and think about a way to prevent confusion among different drivers.

Other interesting types are maybe Set, Bag, Seq. etc. Convert them to an Array can only return an Array and therefore loses the special behavior of the types.

Marcel

--
You received this message because you are subscribed to a topic in the Google Groups "mongodb-dev" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/mongodb-dev/uYoSjz2VwI0/unsubscribe.

To unsubscribe from this group and all its topics, send an email to mongodb-dev...@googlegroups.com.

To post to this group, send email to mongo...@googlegroups.com.
Visit this group at https://groups.google.com/group/mongodb-dev.

[David Golden]

Marcel, I've filed https://jira.mongodb.org/browse/SERVER-31771 to track this feature request.  Please follow it, vote for it, and/or comment on it with anything you think I've left out.

Regards,

David

To unsubscribe from this group and all its topics, send an email to mongodb-dev+unsubscribe@googlegroups.com.

To post to this group, send email to mongo...@googlegroups.com.
Visit this group at https://groups.google.com/group/mongodb-dev.

To view this discussion on the web visit https://groups.google.com/d/msgid/mongodb-dev/9a708ea9-8625-4e37-a904-d3202d1616b4%40googlegroups.com.

[Marcel Timmerman]

Hi David,

Thanks very much for making this feature request. I've looked there but could not find links or buttons to follow it or vote for it.
Regards,
Marcel

[David Golden]

Marcel,

You'll need to create a Jira account, I think.  https://jira.mongodb.org/secure/Signup!default.jspa

Regards,

David

--
You received this message because you are subscribed to a topic in the Google Groups "mongodb-dev" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/mongodb-dev/uYoSjz2VwI0/unsubscribe.
To unsubscribe from this group and all its topics, send an email to mongodb-dev+unsubscribe@googlegroups.com.
To post to this group, send email to mongo...@googlegroups.com.
Visit this group at https://groups.google.com/group/mongodb-dev.

To view this discussion on the web visit https://groups.google.com/d/msgid/mongodb-dev/a48f9f87-a457-4b14-a011-45c519079a7a%40googlegroups.com.