Is there a way to add/override additional logic for certain decode operations?

[Christopher Sterling]

In mgo.v2, if something was a *time.Time, mgo would attempt to do a reflection of the mongo value into the time.Time struct.

Now, it appears that mongo-go-driver will only try to parse it if it's a date, even though it's a string in valid ISOFormat.

I get why you might want to add that boundary type check as a rule rather than possibly unmarshalling a bad value for the sake of performance, but some of our data (mostly test data) has that field as a string type.

Is it possible to add overrides for certain types (without having to reimplement the whole unmarshaller) so that we could (in rare occasion) force the decode of a proper ISODate string type into a time.Time?

[Divjot Arora]

Yes, you can add custom marshalling/unmarshalling logic through our BSON library's codec system. We are currently working on a documentation project that will add more docs and examples about how to do this. I believe what you're looking for is a custom unmarshaller for the time.Time type. You can try to create a custom BSON registry with this special unmarshalling logic and pass that registry in as an option when creating a new Client using ClientOptions.SetRegistry.

[Christopher Sterling]

Would you be willing to share just a single example of any custom unmarshalling? It doesn't need to be documented or anything like that, I was just having a hard time getting the code to work.

[Divjot Arora]

The default decoder for time.Time (which enforces that the value must be of type bsontype.DateTime) is defined at https://github.com/mongodb/mongo-go-driver/blob/master/bson/bsoncodec/default_value_decoders.go#L603. To override this, you could create a new decoder with the same signature and use that to create a new registry as follows:

timeType := reflect.TypeOf(time.Time{})
registry := bson.NewRegistryBuilder().RegisterDecoder(timeType, newTimeDecodeFunction)

To create a new Client with this registry, you can use

mongo.NewClient(options.Client().SetRegistry(registry))

[Divjot Arora]

Let me know if you need more help with this and I can try to write up a custom decoder that accomplishes your gaol of decoding a time.Time from a string.

[Christopher Sterling]

Sweet! Love it!

Thanks for the pointer.

[Divjot Arora]

Hi Christopher,

I re-read the question and realized you mentioned that mgo does this by default. Just to let you know, we are currently working on a project that will allow users to opt into mgo-compatible encoding/decoding behavior with a custom registry. I'll bring up this specific behavior to the team to make sure it's handled in that project. In the mean time, I attached a file that has a working example for a custom decoder and how to register it when creating a new Client. Can you take a look and see if this satisfies your requirements?

[Christopher Sterling]

Yes! This is exactly what I needed! :-)

I think I get how the custom decoder works now, I'll keep playing with this, but I think having this template (and the time.Time string implementation) solves all of our use cases for this issue.

Thanks so much for putting that together Divjot - very much appreciated.

- Topher

[Christopher Sterling]

I've been playing with this for a while now, and I'm having a hard time around custom types.

We have one library that maintains/wraps our DB connection. We then have many other libraries that consume that DB library.

In order to write our custom decode operations, we need to register our custom types with the decoder, but if we try to do that, it causes an import cycle.

Is there some method in my other libraries that I can register the decoder?

e.g. with the json library, you implement "func (s myStruct) UnmarshalJSON()" and the json library uses interfaces in order to properly call the correct function.

Is there an UnmarshalBSON() equivalent we can implement so we can consume this functionality without import cycles?

Would you have any recommendations for extending this for custom types?

There are literally hundreds of data structures that would need to move to shift them into the DB library and it doesn't really make sense to factor client/library specific structs into the DB library. I just can't figure out how people are consuming this at scale and I'd love some advice.

[Divjot Arora]

Hi Chris,

Like JSON, we do have Marshaler and Unmarshaler interfaces that custom types can define. Our BSON library will defer any marshalling/unmarshalling to those functions if it finds a type that implements the interfaces. You can find the interface definitions at https://github.com/mongodb/mongo-go-driver/blob/master/bson/marshal.go#L23 and https://github.com/mongodb/mongo-go-driver/blob/master/bson/unmarshal.go#L21.

[Freddy Martinez]

Joining

=============================================
Freddy Martínez García
Software Engineer
B.S. Computer Science
LinkedIn: https://ar.linkedin.com/in/freddy-martinez-garcia-47157259

“If you give someone a program, you will frustrate them for a day;
if you teach them how to program, yo will frustrate them for a lifetime.”
 
David Leinweber

--
You received this message because you are subscribed to the Google Groups "mongodb-go-driver" group.
To unsubscribe from this group and stop receiving emails from it, send an email to mongodb-go-dri...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/mongodb-go-driver/190ce7e5-3584-4d86-af20-3ce60e1f3fc9%40googlegroups.com.

[Christopher Sterling]

Very cool Divjot - 

I'm assuming just like JSON, I just need to implement the standard type for the marshal and the pointer to the type for the unmarshal?

[Divjot Arora]

Yes, I believe that's correct. We have a couple of examples of how these functions can be written in our code base if you're interested. The bsonx.Doc type implements both of these interfaces. Both the type definition and marshal/unmarshal functions can be found in x/bsonx/document.go.

[Christopher Sterling]

I keep trying to use this functionality and failing.

Alright, so we want to mimic the behavior we used to have in mgo.v2, where if an array is nil, don't store it as a null, instead store it as an array type so that subsequent $push operations to that slice pass. In order to do that, I assumed we would want to implement a custom Marshal and Unmarshal function

We have some insanely complex structs I'm trying to write custom behavior for. So we have let's say this struct:

type CFResponse struct {

    ID                      *primitive.ObjectID `json:"_id" bson:"_id,omitempty"`

    KTUUID                  string              `json:"ktuuid" bson:"ktuuid"`

    Created                 time.Time           `json:"created,required" bson:"created" description:"Time response was loaded"`

    KtScore                 bool                `json:"ktScore,required" bson:"ktScore" description:"true if client would like KT score on response"`

    Tags                    map[string]string   `json:"tags" bson:"tags" description:"Optional metadata"`

    //KTTags                 map[string]string          `json:"ktTags" bson:"ktTags" description:"Contains ktTags that kt may send to epen on a request.for human scores"`

    Jobs                   []CFJobLabel               `json:"jobs" bson:"jobs" description:"List of Jobs that score has be associated with"`

    Scores                 []CFResponseScore          `json:"scores" bson:"scores"`

    Updated                time.Time                  `json:"updated" bson:"updated"`

...

}

// MarshalBSON controls the marshal behavior of the CFResponse struct

func (resp CFResponse) MarshalBSON() ([]byte, error) {

    if len(strings.TrimSpace(resp.KTUUID)) == 0 {

        ruuid, _ := uuid.NewV4()

        resp.KTUUID = ruuid.String()

    }

    if resp.Created.IsZero() {

        // Set the created time if we weren't given one

        resp.Created = time.Now()

    }

    if resp.ID == nil {

        resp.ID = utils.NewBSONObjectIDPointer()

    }

    if resp.Events == nil {

        resp.Events = []CFResponseEventDetail{}

    }

    if resp.Jobs == nil {

        resp.Jobs = []CFJobLabel{}

    }

    return bson.Marshal(resp)

}

// UnmarshalBSON controls the behavior of the CFResponse as it's

// unmarshalled from the DB

func (resp *CFResponse) UnmarshalBSON(bytes []byte) error {

    if err := bson.Unmarshal(bytes, resp); err != nil {

        return err

    }

    if len(strings.TrimSpace(resp.KTUUID)) == 0 {

        ruuid, _ := uuid.NewV4()

        resp.KTUUID = ruuid.String()

    }

    if resp.Created.IsZero() {

        // Set the created time if we weren't given one

        resp.Created = time.Now()

    }

    if resp.ID == nil {

        resp.ID = utils.NewBSONObjectIDPointer()

    }

    if resp.Scores == nil {

        resp.Scores = []CFResponseScore{}

    }

    if resp.Events == nil {

        resp.Events = []CFResponseEventDetail{}

    }

    if resp.Jobs == nil {

        resp.Jobs = []CFJobLabel{}

    }

    return nil

}

Keep in mind that ALL of these values were properly initialized by the old mgo library, so we either have to modify hundreds of functions or add a central decoder.

I basically just need to call the standard encode/decode operation with some pre/post processing logic. That's all I'm trying to do is mimic what the old library was doing for us automatically.

What I've pasted above has the issue of being infinitely recursive. How can I call out to the standard decode/encode hooks?

[Divjot Arora]

Hi Chris,

The driver's RegistryBuilder type has a RegisterDefaultEncoder that allows you to register an encoder for a reflect.Kind, which is more general than a reflect.Type. This would allow you to register an encoder that would be used for all slices. You can see the default slice encoder being registered at https://github.com/mongodb/mongo-go-driver/blob/master/bson/bsoncodec/default_value_encoders.go#L106. I've attached a Go file that registers a slice encoder to encode nil slices as empty BSON arrays instead of BSON null.

Note that the code I've linked actually registers 2 encoders: one for slices and one specifically for []byte. This is because the Go driver (like mgo) encodes []byte as BSON binary rather than a BSON array. The custom encoder in the linked code changes it so a nil []byte is encoded as a BSON binary of subtype 0 and empty data rather than BSON null. I believe this matches the behavior of the globalsign mgo fork

[Christopher Sterling]

We can't use the serializer you keep referencing because our DB package is factored and consumed by other packages. We can't define serializer behavior without causing import cycles.

Do you have the same example but using only Marshal/Unmarshal?

[Divjot Arora]

I'm not sure I understand. You could do this using only Marshal/Unmarshal but then you'd have to implement Marshal/Unmarshal for every struct that has a slice field. Is there a single place that a mongo.Client is created in your codebase? If so, would it be possible to create a registry using the encode functions I linked in that spot? Those functions aren't type-dependent; they'll encode nil slices as empty BSON arrays for all slice fields in all structs that are marshalled using the driver.

[Christopher Sterling]

Ahhh - I misread. My bad :-)

I'll try to reimplement the generic slice kind.

Thanks for the tip and code pointer.