Will MongoDB Go Driver automatically convert ‘uint64’ to bson type? Overflows error returned.

[sammy Ma]

As the title shows, I have a struct defined an uint64 field, but error returned when I set its value as math.MaxUint64.

This is my code:

type MyDoc struct {

    Number    uint64 `bson:"_id"`

    Timestamp int64  `bson:"time"`

}

// I just want to know whether uint64 overflows or not.
func main() {
    mydoc := &MyDoc{

    Number: math.MaxUint64,

    Timestamp: time.Now().UnixNano(),

}

v, err := bson.Marshal(mydoc)

if err != nil {

panic(err)

}

fmt.Println(v)

}

After executed, error is following:

panic: 18446744073709551615 overflows int64 [recovered]

panic: 18446744073709551615 overflows int64

Obviously, uint64 types of data are processed as int64 which is not I expect.

So, how to store an uint64 data but not overflows in MongoDB?? I can not use string type instead, because I need to compare the size of number so that sort documents.

Thanks in advance!

[Divjot Arora]

Hi,

All numbers in BSON are signed, so it is not possible to represent math.MaxUint64. The largest number you can represent is math.MaxInt64. Also, this Google Group is being archived soon and we ask that you post any new questions in https://community.mongodb.com/ instead. For Go Driver questions, please use the go-driver tag when asking questions on that site.

Thanks,

Divjot

[sammy Ma]

Hi Divjot, 

Thanks for your reply. I had tried to post my question in https://community.mongodb.com/ yesterday, but this webpage is so slow to respond. (It may be related to my network, everything is ok today)

Do you known how to store uint64 in MongoDB?? In other words, do I need to convert uint64 to other type(which type?) before storing??

Uint64 type is a very common type in my system, and now I would like to use MongoDB as a sub system to store a part of data. So I can not change data type of my system, otherwise, it may bring disaster. 

Best regards,

Sammy
 
在 2020年3月4日星期三 UTC+8下午11:36:27，Divjot Arora写道：

[Divjot Arora]

You can store the uintt64 in MongoDB as long as it's between math.MinInt64 and math.MaxInt64, inclusive. Do you absolutely need to store numbers outside of this range? If so, I see 2 ways around this issue:

1. Write a custom BSON codec for the uint64 type that ignores the requirement. Note that this will cause the numbers to appear as negative in the database because a number larger than math.MaxInt64 will overflow. This can cause issues if you need to do comparisons for your data in MongoDB (e.g. do a find with a $gte query).

2. Write a custom BSON codec to store your uint64 instances as strings. Once again, though, I'm not sure how we can make this work if you need to do comparisons on the data because it'll be a string in the database.

Can you provide some more information about your use case and constraints? In the meantime, I'll talk to the team tomorrow to see if anyone else has a good way around this and post here with more information once I have it.

-- Divjot

[Matthew Zimmerman]

I compare long strings just fine, just pad with leading 0s...

There's also https://github.com/mongodb/mongo-go-driver/blob/master/bson/primitive/decimal.go

--
You received this message because you are subscribed to the Google Groups "mongodb-go-driver" group.
To unsubscribe from this group and stop receiving emails from it, send an email to mongodb-go-dri...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/mongodb-go-driver/2a72b45e-5372-4840-9bb3-464fc2024565%40googlegroups.com.

[sammy Ma]

Hi, my use case is following: 

Part of the data in my application is stored in the KV database(e.g. LevelDB) and part is stored in the file. And data structure is like:

type Data struct{
  Version []byte `protobuf:"bytes,1,opt,name=version,proto3" json:"version,omitempty"`
  Number  uint64 `protobuf:"varint,2,opt,name=number,proto3" json:"number,omitempty"`
  Timestamp int64 `protobuf:"varint,3,opt,name=timestamp,proto3" json:"timestamp,omitempty"`
  Content []byte `protobuf:"bytes,4,opt,name=content,proto3" json:"content,omitempty"`
}

The actual data structure is more complicated than listed here. Field `Number` is unique and monotonically increasing. So in the KV database, Key is `Number`, Value is the result of Data instance protobuf marshal. 

And now, I want to store part of fields mentioned above to MongoDB, so that supports "SQL" query in the MongoDB, and then get the final detailed result from KV database using the primary key of document. The document structure is like:

type Doc struct{
  Number uint64   `bson:"_id"`        // This is primary key.
  Timestamp int64 `bson:"timestamp"`  // The reason why stores this field is that user can query data within a specified time range,
                                      // and KV database only support key query.
}

`Number` is the primary key, and I suppose it may exceed math.MaxInt64 one day. Another requirement is that user can specify the start position of querying through giving a `Number` value, so I need to compare `Number`.

In conclusion:

1. In my application developed in Go, `Number` should be uint64 type.

2. I have to store `Number` into MongoDB, and will do comparisons for `Number`.

Thanks, sorry for my english,

Sammy

在 2020年3月5日星期四 UTC+8上午10:33:30，Divjot Arora写道：

[sammy Ma]

Hi,

I agree it, 

but in my application, MongoDB is just a small part of it. Will overly complex type conversions affect system performance and cause more harm than good? If so, I might consider using other database supporting SQL or Like-SQL query.

Thanks,

Sammy

在 2020年3月5日星期四 UTC+8下午12:47:02，mongouser写道：

To unsubscribe from this group and stop receiving emails from it, send an email to mongodb-...@googlegroups.com.

[Divjot Arora]

Hi,

Given your use case, I don't think it's likely for your records to overflow math.MaxInt64. You'd need to insert approximately 9*10^18 records (2^63-1) for that to happen. If you are concerned about this, you can store numbers as strings and use the $strcasecmp aggregation stage (https://docs.mongodb.com/manual/reference/operator/aggregation/strcasecmp/) or use Decimal128 if needed.

-- Divjot