Writing to Nested ModelSerializer

[Jason Guy]

After spending a week re-reading the docs, digging into the source, searching this forum, and testing and tracing with pdb, I decided it is time to ask for help. I finally feel like I have isolated what I don't understand about the DRF and implementing a flexible way to write to my nested serializer. Reads are working great, but I want to use the same nested JSON (returned by the read), as my write schema. 

I have tried a variety of ideas from the forum, (like this, this, and this) with varying levels of partial success. 

Let's use a generic example (since my code is a mess) and it is easier for others to associate with:

#models.py
class Color(models.Model):
    name = models.CharField(max_length=100, unique=True)
    finish = models.CharField(max_length=100, null=True, blank=True)


class Car(models.Model):
    make = models.CharField(max_length=100, unique=True)
    model = models.CharField(max_length=100, unique=True)
    color = models.ForeignKey(Color)

#serializer.py
class CarSerializer(serializers.ModelSerializer):
    class Meta:
        model = models.Color
        fields = ('name', 'finish')


class CarSerializer(serializers.ModelSerializer):
    color = ColorSerializer()
    class Meta:
        model = models.Car
        fields = ('make', 'model', 'color')

#data to write
car_data = {
    'make': 'Dodge',
    'model': 'Daytona',
    'color': {'name': 'black', 'finish': 'gloss'}
}


# instantiate a serializer with the data to write
mycar = serializers.CarSerializer(data=car_data)
# runs the to_internal_value, and validation methods on the fields specified
mycar.is_valid()
# Assuming the data is valid, this writes the instance to the database
mycar.save()

Goal: Create a new car and specify its color, IF the color exists, it should simply return the existing object, ELSE create the new Color object, and return it. 

At a high level, the logic seems simple, but translating this to the serializer logic has been difficult. I am clearly not the only one who does this based on the number of questions in the forum related to writable nested serializers. So rather than seeking a solution, I am looking to understand the building blocks involved in validating data, creating nested objects, and saving the final result. Apologies if this is explained elsewhere, but I think I have researched as much as I could and still cannot quite understand how it all works. The following questions are meant to clear up my confusion (and likely others are scratching their heads).  Perhaps we can create a section in the docs or a tutorial on writable nested deserialization:

1) When the parent serializer mycar.is_valid() is called, it appears to follow the nested relationship, and validates the color.to_internal_value().  

• So the validation process appears to automatically follow the nested relationship, which I would assume would continue (perhaps the nested depth was 5). 
• The confusion is for validation or data manipulation, I would specify the to_internal_value or validate methods under each serializer class. This is good as it keeps the logic under the serializer class.

2) In the docs, it states, "If you want to implement a read-write relational field, you must also implement the .to_internal_value(self, data) method." 

• I don't really understand what to do in the to_internal_value() method. 
• The examples I have seen use it for a variety of things, but it is not clear if this is meant strictly for validation.
• Would it be used to access the self.Meta.model manager in the Color serializer, to lookup an existing Color object, and return the instance to the parent. 
  

3) The validation automatically occurs for the nested data, by drilling down into each serializer class, but I found the create method DOES NOT appear to follow the relationship.

• When it comes to saving the validated data, the create() method is called for the serializer associated to the parent Car object.
• To handle the nested create, it seems this all needs to be done from the parent (Car) serializer; using the model manager methods (.get(), .create(), etc), for BOTH the Car and Color objects. 
• Why can't the parent serializer (Car) see that a nested relationship exists, and call the nested create method?
• It seems cleaner to have the child Color.create() handle the lookup for existing or create new (Color.objects.get_or_create), and return the instance.
• Then the parent Car serializer create can finish creating the Car object with the instance returned from the Color serializer.
  
• This way each serializer locally handles the validation, lookup, and creation.
• If the nested depth is 5, the serializer at level1 would have a really complicated create() method.

4) ModelSerializer is nice for simple models like the example, so why would I want to convert it to a Serializer?

• I have read Tom mention mention many times to re-write the ModelSerializer as a Serializer, but I want to understand the real differences with ModelSerializer vs Serializer.
  

Thanks in advance,

Jason

[Xavier Ordoquy]

Hi,

Feels overcomplicated.

Why don’t you defer the get or create to the actual create / update of the root serializer ?

You get all the data validated according to the schema as it is, no need to override anything.

You add your business logic (get or create for the colors) when you create/update on the CarSerializer.

2) In the docs, it states, "If you want to implement a read-write relational field, you must also implement the .to_internal_value(self, data) method." 

• I don't really understand what to do in the to_internal_value() method. 
• The examples I have seen use it for a variety of things, but it is not clear if this is meant strictly for validation.
• Would it be used to access the self.Meta.model manager in the Color serializer, to lookup an existing Color object, and return the instance to the parent. 

Here, you’re fighting on two things at the same time. On one side it’s a nested serializer, on the other it’s a custom field. Let’s make things simpler as proposed in my previous answer.

3) The validation automatically occurs for the nested data, by drilling down into each serializer class, but I found the create method DOES NOT appear to follow the relationship.

• When it comes to saving the validated data, the create() method is called for the serializer associated to the parent Car object.
• To handle the nested create, it seems this all needs to be done from the parent (Car) serializer; using the model manager methods (.get(), .create(), etc), for BOTH the Car and Color objects. 
• Why can't the parent serializer (Car) see that a nested relationship exists, and call the nested create method?
• It seems cleaner to have the child Color.create() handle the lookup for existing or create new (Color.objects.get_or_create), and return the instance.
• Then the parent Car serializer create can finish creating the Car object with the instance returned from the Color serializer.
  
• This way each serializer locally handles the validation, lookup, and creation.
• If the nested depth is 5, the serializer at level1 would have a really complicated create() method.

Create/update is performed in the root serializer because this allows much more complex use cases to be handled by developers.

We tried to make nested serializers automated back in 2.x and it didn’t work. There was too many use case to deal with and each fixed bug added another 2 to 5 edge cases.

One of the key issue is the order in which the nested data has to be created.

For a FK, you need to create the nested element first and then the root one with the FK to the nested.

For a reversed FK, it’s the other way round.

For many to many, both should be created before joining them.

Try to figure how to deal with those cases where you have several nesting levels mixing the previous cases ?

If you isolate the creation / update the serializer needs to figure the right order and that’s extremely complex and fragile.

Another reason is more simple to get.

We says you need to call the nested serializer’s create when create is called on the root ?

It may just be an update or just a description of an existing nested instance. How could the framework figure it by himself ?

And yes, for nested level above 2, the create/update are complex. One should challenge its desire or need to have such use case.

4) ModelSerializer is nice for simple models like the example, so why would I want to convert it to a Serializer?

• I have read Tom mention mention many times to re-write the ModelSerializer as a Serializer, but I want to understand the real differences with ModelSerializer vs Serializer.
  

The thing being that most - if not all - of us have boxed thoughts and it’s hard sometime to think out of the box.

ModelSerializer is nice to get started and may go quite far. However, while processing the various project requirements we get caught in moving forward with a ModelSerializer where it doesn’t make sense. We bend it and try to make it fit something it’s not a good fit for. Then we blame the framework for being hard to deal with if not worst.

ModelSerializers are Serializers that performs some interaction on a Model.

The issue is resource representations should not be contraints by the storage - i.e. the Model.

Thanks in advance,

Jason

Hope this will help,

Xavier Ordoquy,

Linovia

[Jason Guy]

Xavier,

Thanks for the explanation. I agree, it feels like I am stuck in my thinking, and trying to fully understand how this should be done. I did a bit for reading, and found in the 3.0 announcement much of what you had said (wish I had found that earlier). I also came across this blog post Tom wrote, and it helped me to understand what it is I am struggling with here. Let me explain my approach, and perhaps it will help to expose the confusion.

I already created all of the models for my app, many of them with save() and clean() functions, as I had planned to add a set of forms to the app for user interaction. However, I needed/wanted the writeable API, so I could efficiently push all of the initial content to the database. The data is all produced by automation and this seemed the best approach. 

My struggle now is why would I want to maintain the validation logic in the model AND the serializers? As the blog post mentions, the best approach in a Django App is keeping all of the logic central to the managers or models. So is the serializer considered "view logic"? I would say yes and no. I think of it as being more similar to a form in the case of writing to it. 

With all of that said, how should I proceed? I think I would prefer to just have the serializer create/update utilize the model managers for creation and updating. This is mentioned in the docs for nested writes, but I wanted to explore how the serializer methods do it. Regardless, I am trying to learn/understand what is generally accepted as the "proper way" to meld the Django framework with both the DRF and eventually interactive forms. I would assume this will also be the most maintainable approach. 

Seriously, thanks for the help,

Jason