Nested Arrays in MongoEngine

[Joel Johnston]

I'm a newb to both Mongo and MongoEngine.  Coming from RDB land, I'm trying to wrap my head around nested data structures.

My Example: 

I have a collection of cloud accounts. 

Each cloud account has (n) number of regions

Each region has (n) number of VPCS

Each VPC has (n) number of Subnets. 

So:

Account:

    <Region>

        <VPCs>

            <Subnets>

I have 3 questions.

Whats the best practice for creating such a structure in Mongo and how is that expressed in a mongoengine class(es)? 

What's the best practice for querying such a structure with mongoengine?

What's the best practice for updating individual fields with mongoengine? 

If there's a tutorial you can point me to that explains this vs. writing some epic response, I'm game. I'm having a difficult time searching for the right explanation online due to semantics (and likely my lack of knowledge of mongo terminology.)

Thanks for your help. 

Joel

[Stefan Wójcik]

Hi Joel! Here's how I'd suggest modeling it:

class Subnet(EmbeddedDocument):
    cidr = StringField()


class VPC(EmbeddedDocument):
    subnets = ListField(EmbeddedDocumentField(Subnet))


class Region(EmbeddedDocument):
    vpcs = ListField(EmbeddedDocumentField(VPC))


class Account(Document):
    regions = ListField(EmbeddedDocumentField(Region))

With that schema, you can create a new account like so:

account = Account(
    regions=[
        Region(
            vpcs=[
                VPC(
                    subnets=[
                        Subnet(cidr="10.0.0.0/16"),
                        Subnet(cidr="10.0.1.0/16"),
                        Subnet(cidr="10.0.2.0/16"),
                    ]
                )
            ]
        )
    ]
)

If you persist this doc, the following data will be stored in your MongoDB:

{
    "_id": ObjectId("some-autogenerated-id"),
    "regions": [
        {
            "vpcs": [
                {
                    "subnets": [
                        {
                            "cidr": "10.0.0.0/16"
                        },
                        {
                            "cidr": "10.0.1.0/16"
                        },
                        {
                            "cidr": "10.0.2.0/16"
                        }
                    ]
                }
            ]
        }
    ]
}

> What's the best practice for querying such a structure with mongoengine?

This really depends on what you'll be querying by.

> What's the best practice for updating individual fields with mongoengine?

You can access any field, modify it, and then call `save()` on the whole document, e.g. with the example above, you could do:

account.regions[0].vpcs[0].subnets.append("10.0.3.0/16")
account.save()

Hope this gives you a good jumping off point.

Cheers,

Stefan

PS Note that if you'll have thousands of VPCs/subnets within a single account, it might be better to split those into a separate collection due to MongoDB's limit of 16MB for a single document.

[Joel Johnston]

Stefan,

Many Thanks. Your jumpstart worked very well.  I now have these classes defined as my document and I'm able to store arrays properly in Mongo and query them.

class Subnet(EmbeddedDocument):
    subnetid = StringField()

class VPC(EmbeddedDocument):
    vpcid = StringField()

    subnets = ListField(EmbeddedDocumentField(Subnet))

class Region(EmbeddedDocument):

    region = StringField()
    vpcs = ListField(EmbeddedDocumentField(VPC))

class CloudAccount(Document):
    account_id = StringField()
    cloud_name = StringField(max_length=25)
    account_number = IntField(required=True)
    account_name = StringField()
    regions = ListField(EmbeddedDocumentField(Region))

Now the conundrum I face is that I need to be able to intelligently add regions, vpcs, and subnet ids. By intelligently I mean I need to test if a region exists in a document and if it doesn't add a new region. If a vpc already exists in a document, then move on and test for existing subnets, etc. etc.

I'm trying to append the regions intelligently first, but I'm not understanding the append object syntax.

    def append_subnet(self,account_name,db_host,db_port,db_username,db_password,db_name,region,vpcid,subnetid):
        ''' Append a Region, VPC, and Subnet to any existing account record'''
        accounts = inv.db_connect(self, db_host, db_port, db_username, db_password, db_name)
        account = CloudAccount.objects.get(account_name=account_name)

        region_counter = 0
        region_matches = 0
        for r in account.regions:
            if r == account.regions[region_counter].region:
                region_matches = 1
                print("matched")
                break
        print("Region_Matches = %s" % (region_matches))

        if region_matches is 0:
            regions = account.regions
            regions.region.append(region)
            regions.save()

This append attempt errors with BaseList object has not attribute 'region'

If you don't mind adding one more sample, I'm sure you have a much more elegant approach in mind.   If not, Thank You Greatly for your jumpstart.  Really helped out.