Nested Arrays in MongoEngine

39 views
Skip to first unread message

Joel Johnston

unread,
Jul 13, 2019, 11:41:21 AM7/13/19
to MongoEngine Users
I'm a newb to both Mongo and MongoEngine.  Coming from RDB land, I'm trying to wrap my head around nested data structures.

My Example: 

I have a collection of cloud accounts. 

Each cloud account has (n) number of regions

Each region has (n) number of VPCS

Each VPC has (n) number of Subnets. 

So:
Account:
    <Region>
        <VPCs>
            <Subnets>

I have 3 questions.

Whats the best practice for creating such a structure in Mongo and how is that expressed in a mongoengine class(es)? 

What's the best practice for querying such a structure with mongoengine?

What's the best practice for updating individual fields with mongoengine? 

If there's a tutorial you can point me to that explains this vs. writing some epic response, I'm game. I'm having a difficult time searching for the right explanation online due to semantics (and likely my lack of knowledge of mongo terminology.)

Thanks for your help. 

Joel

Stefan Wójcik

unread,
Jul 14, 2019, 3:47:38 AM7/14/19
to MongoEngine Users
Hi Joel! Here's how I'd suggest modeling it:

class Subnet(EmbeddedDocument):
    cidr
= StringField()


class VPC(EmbeddedDocument):
    subnets
= ListField(EmbeddedDocumentField(Subnet))


class Region(EmbeddedDocument):
    vpcs
= ListField(EmbeddedDocumentField(VPC))


class Account(Document):
    regions
= ListField(EmbeddedDocumentField(Region))

With that schema, you can create a new account like so:

account = Account(
    regions
=[
       
Region(
            vpcs
=[
                VPC
(
                    subnets
=[
                       
Subnet(cidr="10.0.0.0/16"),
                       
Subnet(cidr="10.0.1.0/16"),
                       
Subnet(cidr="10.0.2.0/16"),
                   
]
               
)
           
]
       
)
   
]
)

If you persist this doc, the following data will be stored in your MongoDB:

{
   
"_id": ObjectId("some-autogenerated-id"),
   
"regions": [
       
{
           
"vpcs": [
               
{
                   
"subnets": [
                       
{
                           
"cidr": "10.0.0.0/16"
                       
},
                       
{
                           
"cidr": "10.0.1.0/16"
                       
},
                       
{
                           
"cidr": "10.0.2.0/16"
                       
}
                   
]
               
}
           
]
       
}
   
]
}

> What's the best practice for querying such a structure with mongoengine?

This really depends on what you'll be querying by.

> What's the best practice for updating individual fields with mongoengine?

You can access any field, modify it, and then call `save()` on the whole document, e.g. with the example above, you could do:

account.regions[0].vpcs[0].subnets.append("10.0.3.0/16")
account
.save()

Hope this gives you a good jumping off point.

Cheers,
Stefan

PS Note that if you'll have thousands of VPCs/subnets within a single account, it might be better to split those into a separate collection due to MongoDB's limit of 16MB for a single document.

Joel Johnston

unread,
Jul 15, 2019, 6:12:20 PM7/15/19
to MongoEngine Users
Stefan,

Many Thanks. Your jumpstart worked very well.  I now have these classes defined as my document and I'm able to store arrays properly in Mongo and query them.

class Subnet(EmbeddedDocument):
    subnetid
= StringField()

class VPC(EmbeddedDocument):
    vpcid
= StringField()

    subnets
= ListField(EmbeddedDocumentField(Subnet))

class Region(EmbeddedDocument):

    region
= StringField()
    vpcs
= ListField(EmbeddedDocumentField(VPC))

class CloudAccount(Document):
    account_id
= StringField()
    cloud_name
= StringField(max_length=25)
    account_number
= IntField(required=True)
    account_name
= StringField()
    regions
= ListField(EmbeddedDocumentField(Region))

Now the conundrum I face is that I need to be able to intelligently add regions, vpcs, and subnet ids. By intelligently I mean I need to test if a region exists in a document and if it doesn't add a new region. If a vpc already exists in a document, then move on and test for existing subnets, etc. etc.

I'm trying to append the regions intelligently first, but I'm not understanding the append object syntax.

    def append_subnet(self,account_name,db_host,db_port,db_username,db_password,db_name,region,vpcid,subnetid):
       
''' Append a Region, VPC, and Subnet to any existing account record'''
        accounts
= inv.db_connect(self, db_host, db_port, db_username, db_password, db_name)
        account
= CloudAccount.objects.get(account_name=account_name)

        region_counter
= 0
        region_matches
= 0
       
for r in account.regions:
           
if r == account.regions[region_counter].region:
                region_matches
= 1
               
print("matched")
               
break
       
print("Region_Matches = %s" % (region_matches))

       
if region_matches is 0:
            regions
= account.regions
            regions
.region.append(region)
            regions
.save()


This append attempt errors with BaseList object has not attribute 'region'

If you don't mind adding one more sample, I'm sure you have a much more elegant approach in mind.   If not, Thank You Greatly for your jumpstart.  Really helped out.
Reply all
Reply to author
Forward
0 new messages