how to store/query deeply-nested golang structs into mongodb with mgo

[David Marceau]

Good afternoon,

I read this presentation "Tutorial and examples with mgo Rich MongoDB driver for Go":
http://denvergophers.com/2013-04/mgo.article
about using mgo.  "Example 6" is particularly useful.  Hats off to the authors Mr. Cook and Mr. LaNou:

I appreciate the above presentation's mgo with golang examples.  It gave me a clearly demonstrated method to store deeply-nested golang structs into mongodb.   Thank you for that. 
There aren't too many mongodb/golang examples out there with nestedness.  I see plenty in mongodb shell(a form of javascript) and php, but I didn't find any for mgo.

"Example 6" stores the entire golang struct, EVERYTHING, as one json object into mongodb.  Extending on this, you can make a deeply nested struct in golang, then marshal it as a json object and finally store(upsert it in the example 6 case) it into the mongodb.  I can understand that and do that.    The presenters didn't emphasize that, but what they did was brilliant.  Better examples showing the nestedness in golang struct would definitely show the brilliance of your strategy.  I'm working on some, but they're not done and working yet. 

For example, a potential customer reaches your web site is just a "session" with no userid.  The session creates a "system build" using different "component"'s.  When the session wants to buy, that's when a requirement to login/register happens.  The "session" becomes a "user" struct nested with systembuilds nested with components.  Your json object takes care of all these automatically.  All you have to do is upsert the json object. 

There are ways to store deeply nested structs documented in mongo's javascript shell and php, but no clear examples for nested golang structs directly to mongodb exist.

The one disadvantage of storing json object which represents the entire nested golang struct:  it's difficult to make queries/aggregations within the json object without unmarshalling first.  For the user.systembuilds.components.merchants.prices use case mentioned however, having the user.userid is enough because we are always focused on one particular user being served anyways. 

For creating reports however involving aggregation, that would mean another golang routine could create another database with another set of collections to restructure all the data captured within the user collection.   For example here is a query:
For every merchant, for a particular product, which users purchased it and show the purchase count. 

For a query like that when storing under json objects, there is no way we could use the mongo shell query syntax within mgo to filter and iterate on the specific results data because they are hidden within the json objects.  We would need to code a golang routine to complete query traversing all the json object nestedness.

Could someone here more knowledgeable please provide a clear example for inserting/storing a deeply-nested golang struct
(with at least 4 levels deep with a few arrays/maps in each level) with mgo into mongodb without resorting to json.

Here are some example nested structs which the json storing method overcomes, but fails to make it easy for querying/updating different fields using mongo query addToSet syntax is something like below:

//not sure but fits ok into json objects.
type Customer struct {
    FirstName string
    FamilyName string
    AddressAptNumber string
    AddressStreet string
    City string
    Province string
    PostalCode string
    PhoneNumber string
    Email string
    IsDifferentDeliveryAddress string
    DeliveryPersonContact string
    DeliveryAptNumber string
    DeliveryAddressStreet string
    DeliveryCity string
    DeliveryProvince string
    DeliveryPostalCode string
    DeliveryPhoneNumber string
    DeliveryEmail string
    PurchaseHistory  []*ShoppingCart
}

//not sure but fits ok into json objects.
type ShoppingCart struct {
    ShoppingItems  []*ShoppingItem
    PaymentInfo PaymentDetails
}

//this should be embedded into a transaction.
//php/javascript examples are well documented, but not documented in mgo
type PaymentDetails struct {
    PaymentMethod string
    TimeStamp string
    ConfirmationNumber string
}

//not sure but fits ok into json objects.
type ShoppingItem struct {
    selectedConfigurationSpecification  ConfigurationSpecification
    Quantity int
}

//not sure but fits ok into json objects.
type ConfigurationSpecification struct {
    Name string //SystemBuild name
    SelectedParts []*Part //parts to make this build
    Weight float64
    Price float64 
    IsOffered string
}

//this could be a parts collection
type Part struct {
    Manufacturer string
    ManufacturerModelName string
    ManufacturerModelNumber string
    PartType string //HARD DRIVE | SOLID STATE DRIVE |
                    // DDR3 RAM | DDR2 RAM | DDR4 RAM | DDR5 RAM | CPU | GPGPU |
                // MOTHERBOARD | UNINTERRUPTIBLE POWER SUPPLY | CABLE | DC POWER SUPPLY  ...etc
    Capacity string
    Weight float64 //Kilograms
    ManufacturerPrice float64
    DistributorPrice float64
    CustomsCostChina float64
    CustomsCostUS float64
    CustomsCostCanada float64
}

//not sure but fits ok into json objects.
type MyProductBundles struct {
    ConfigurationSpecifications []*ConfigurationSpecification
}

////not sure but fits ok into json objects.
//parts I find interesting, but won't necessarily offer/buy them.
type PartsList struct {
    Parts []*Part
}

Thank you and Cheers

[Gustavo Niemeyer]

Hi David,

On Mon, Jul 29, 2013 at 4:52 PM, David Marceau
<uticdmar...@gmail.com> wrote:
> Could someone here more knowledgeable please provide a clear example for
> inserting/storing a deeply-nested golang struct
> (with at least 4 levels deep with a few arrays/maps in each level) with mgo
> into mongodb without resorting to json.

mgo can handle nested Go structures naturally. Have you tried to just
save a value in the database?

This works fine, for example:

type A struct { I int }
type B struct { MyA A }
err := collection.Insert(&B{A{42}})

Just try it out and you'll be pleasantly surprised. :-)


gustavo @ http://niemeyer.net

[David Marceau]

>    type A struct { I int }
>    type B struct { MyA A }
>    err := collection.Insert(&B{A{42}})

It's the querying part that gets difficult and especially with nested structures.  Later I will provide a full working example using your suggestions and then come back to clarify my nested structure queries.  Thank you for taking the time.

[David Marceau]

> db.customers.insert({"FirstName":"David","FamilyName":"Marceau","PurchaseHistory":"nil"})
> db.customers.find()
{ "_id" : ObjectId("51f98272cb285502da80d468"), "FirstName" : "David", "FamilyName" : "Marceau", "PurchaseHistory" : "nil" }
> db.customers.update({},{$set: {PurchaseHistory: "blah"} },{multi:true})
> db.customers.find()
{ "FamilyName" : "Marceau", "FirstName" : "David", "PurchaseHistory" : "blah", "_id" : ObjectId("51f98272cb285502da80d468") }
> db.customers.update({},{$set: {PurchaseHistory: { cart1:1, cart2:2, cart3:3} } },{multi:true})
> db.customers.find()
{ "FamilyName" : "Marceau", "FirstName" : "David", "PurchaseHistory" : { "cart1" : 1, "cart2" : 2, "cart3" : 3 }, "_id" : ObjectId("51f98272cb285502da80d468") }
> db.customers.update({},{$set: {PurchaseHistory: { cart1: { shopitem1: 1, shopitem2:2, shopitem3:3}, cart2:{shopitem1:4, shopitem2:5, shopitem3:6}, cart3:{shopitem7:7,shopitem8:8,shopitem9:9}} } },{multi:true})
> db.customers.find()
{ "FamilyName" : "Marceau", "FirstName" : "David", "PurchaseHistory" : { "cart1" : { "shopitem1" : 1, "shopitem2" : 2, "shopitem3" : 3 }, "cart2" : { "shopitem1" : 4, "shopitem2" : 5, "shopitem3" : 6 }, "cart3" : { "shopitem7" : 7, "shopitem8" : 8, "shopitem9" : 9 } }, "_id" : ObjectId("51f98272cb285502da80d468") }

Here is where I have difficulty.  How do I replace cart3?
> db.customers.update({},{$set: {cart3:{shopitem7:10,shopitem8:11,shopitem9:12}} },{multi:false})
> db.customers.find()
{ "FamilyName" : "Marceau", "FirstName" : "David", "PurchaseHistory" : { "cart1" : { "shopitem1" : 1, "shopitem2" : 2, "shopitem3" : 3 }, "cart2" : { "shopitem1" : 4, "shopitem2" : 5, "shopitem3" : 6 }, "cart3" : { "shopitem7" : 7, "shopitem8" : 8, "shopitem9" : 9 } }, "_id" : ObjectId("51f98272cb285502da80d468"), "cart3" : { "shopitem7" : 10, "shopitem8" : 11, "shopitem9" : 12 } }
That results with two cart3's.  Is that an instance of $set: PurchaseHistory.cart3 {"shopitem7" : 10, "shopitem8" : 11, "shopitem9" : 12}?

> db.customers.find()
{ "FamilyName" : "Marceau", "FirstName" : "David", "PurchaseHistory" : { "cart1" : { "shopitem1" : 1, "shopitem2" : 2, "shopitem3" : 3 }, "cart2" : { "shopitem1" : 4, "shopitem2" : 5, "shopitem3" : 6 }, "cart3" : { "shopitem7" : 7, "shopitem8" : 8, "shopitem9" : 9 } }, "_id" : ObjectId("51f98272cb285502da80d468"), "cart3" : { "shopitem7" : 10, "shopitem8" : 11, "shopitem9" : 12 } }
> db.customers.find(ObjectId("51f98272cb285502da80d468"))
{ "FamilyName" : "Marceau", "FirstName" : "David", "PurchaseHistory" : { "cart1" : { "shopitem1" : 1, "shopitem2" : 2, "shopitem3" : 3 }, "cart2" : { "shopitem1" : 4, "shopitem2" : 5, "shopitem3" : 6 }, "cart3" : { "shopitem7" : 7, "shopitem8" : 8, "shopitem9" : 9 } }, "_id" : ObjectId("51f98272cb285502da80d468"), "cart3" : { "shopitem7" : 10, "shopitem8" : 11, "shopitem9" : 12 } }
> db.customers.update({_id:ObjectId("51f98272cb285502da80d468")},{$set: {cart3:{shopitem7:13,shopitem8:14,shopitem9:15}} },{multi:false})
> db.customers.find(ObjectId("51f98272cb285502da80d468"))
{ "FamilyName" : "Marceau", "FirstName" : "David", "PurchaseHistory" : { "cart1" : { "shopitem1" : 1, "shopitem2" : 2, "shopitem3" : 3 }, "cart2" : { "shopitem1" : 4, "shopitem2" : 5, "shopitem3" : 6 }, "cart3" : { "shopitem7" : 7, "shopitem8" : 8, "shopitem9" : 9 } }, "_id" : ObjectId("51f98272cb285502da80d468"), "cart3" : { "shopitem7" : 13, "shopitem8" : 14, "shopitem9" : 15 } }
> db.customers.update({_id:ObjectId("51f98272cb285502da80d468")},{$unset: {cart3:{shopitem7:13,shopitem8:14,shopitem9:15}} },{multi:false})
> db.customers.find(ObjectId("51f98272cb285502da80d468"))
{ "FamilyName" : "Marceau", "FirstName" : "David", "PurchaseHistory" : { "cart1" : { "shopitem1" : 1, "shopitem2" : 2, "shopitem3" : 3 }, "cart2" : { "shopitem1" : 4, "shopitem2" : 5, "shopitem3" : 6 }, "cart3" : { "shopitem7" : 7, "shopitem8" : 8, "shopitem9" : 9 } }, "_id" : ObjectId("51f98272cb285502da80d468") }

> db.customers.update({_id:ObjectId("51f98272cb285502da80d468")},{$unset: cart3 },{multi:false})
Wed Jul 31 17:58:09 ReferenceError: cart3 is not defined (shell):1
What's wrong with doing this? Why do I need to restate the values that were stored there when I want to remove them?

> db.customers.update({_id:ObjectId("51f98272cb285502da80d468")},{$unset: {PurchaseHistory} },{multi:false})
Wed Jul 31 18:00:02 SyntaxError: missing : after property id (shell):1
What's wrong with doing this? Why do I need to restate the values that were stored there when I want to remove them?

The only thing I succeeded in doing was to rewrite the entire contents for the specified objectid.
> db.customers.update({_id:ObjectId("51f98272cb285502da80d468")},{"FamilyName" : "Marceau", "FirstName" : "David", "PurchaseHistory" : { "cart1" : { "shopitem1" : 1, "shopitem2" : 2, "shopitem3" : 3 } } },{multi:false})
> db.customers.find(ObjectId("51f98272cb285502da80d468"))
{ "_id" : ObjectId("51f98272cb285502da80d468"), "FamilyName" : "Marceau", "FirstName" : "David", "PurchaseHistory" : { "cart1" : { "shopitem1" : 1, "shopitem2" : 2, "shopitem3" : 3 } } }
>

Thanks again for taking the time.