PUTting null data

[Salvatore Iovene]

Hi,

I have some difficulties placing PUT request for data that is `null`.

Here's a field in my model:

    parent = models.ForeignKey(                                                                                  

        'self',

        null = True,

        blank = True,

        related_name = 'children',

        on_delete = models.SET_NULL,                                                                             

    )

As you can see, null = True.

This is how I place a PUT request:

        undelete: function() {

            var self = this,

                data = {

                    id: self.get('id'),                                                                          

                    author: self.get('author'),

                    content_type: self.get('content_type'),                                                      

                    object_id: self.get('object_id'),

                    text: self.get('text'),

                    created: self.get('created'),

                    updated: self.get('updated'),

                    deleted: false

                };                                                                                               

    

            if (self.get('parent'))                                                                              

                data['parent'] = self.get('parent');                                                             

    

            $.ajax({

                type: 'put',

                url: nc_app.baseApiURL + 'nestedcomments/' + self.get('id') + '/',                               

                data: data,

                timeout: 10000,

                success: function() {                                                                            

                    self.set('deleted', false);                                                                  

                }

            });

        } 

If `self.get('parent')` is `null`, in Javascript, it will be omitted from the data. This gives me the following error in the response:

{"parent": ["Invalid pk 'None' - object does not exist."]}

If I put `parent: null` in my `data`, I will get an Error 500, because Django at some point tries to get an object from the database with the empty string as PK.

Note that my djangorestframework web interface won't allow me to put nothing as the `parent`, when I try to perform a PUT request there, while I can do it in Django's admin interface.

What am I missing?

Thanks!

[Salvatore Iovene]

After some research, I see that this is related to this:

https://github.com/tomchristie/django-rest-framework/issues/445

Is there any known workaround or will the bug be fixed soon?

[Steven Kane]

Salvatore,

Here is a code snippet that I have been passing around this forum that will help you with this issue.  The PrimaryKeyRelatedField (and most likely the other RelatedFields) does not accept null values due to its use of self.queryset.get.  Get throws an ObjectDoesNotExist exception if it finds nothing matching the query.  You can subclass the RelatedField and overwrite the from_native method to return "None" when the queryset finds no matches.  This is not probably the best way (it's quite obviously not actually) to get around this problem but it should get you back on track.  

You could look into re-writing my code using a conditional statement such as: (if data is not None) or perhaps (if data) in order to more directly analyze the content of data.

Overwriting the exception ObjectDoesNotExist to simply return None is not a particularly elegant solution in the long-run.

class PKRelatedFieldAcceptNull(fields.PrimaryKeyRelatedField):
	#overwrite from native to change the logic for handling object lookups that return nothing
	def from_native(self, data):
		if self.queryset is None:
			raise Exception('Writable related fields must include a `queryset` argument')
		try:
			return self.queryset.get(pk=data)	#if no object is found, then set the value to "None" instead of raising exception
		except ObjectDoesNotExist:
			return None

Best of luck,

Steve Kane
Co-Founder 4-south Studios

[Tom Christie]

There is this outstanding pull req:

https://github.com/tomchristie/django-rest-framework/pull/443

I havnt reviewed if it's sufficient or if the same fix is needed on other related fields,

and I believe there might be at least on other open related ticket.

Any input on triaging tickets around this issue and reviewing the pull req would be fab.

  - Tom

[Salvatore Iovene]

Tom,

I hope you'll find time to figure this out soon, as it's rather important.

It's quite the missing feature, if it's impossible to save null fields via the API.

The PR looks good to me, and the tests pass, but I don't know anything about DRF's internals to comment any further.

Salvatore.

[Salvatore Iovene]

Thanks Steven.

Could you please post an example on how to use this with a ModelSerializer?

Salvatore.

[Ian Strachan]

I'm not entirely certain that this is the same problem but I did some work on fixing something similar: https://github.com/tomchristie/django-rest-framework/pull/356

My fix worked but only for fields that were defined on the serializer explicitly - I wasn't certain of the best approach to get this to work when the related field was implicit from a model.  I'm not sure if it's still valid on the latest code, it was written a while ago and things have probably moved on since then.

[Steven Kane]

For a ModelSerializer your code will look like the following:

class FooModelSerializer ( serializers.ModelSerializer ):

  class Meta:

    model = Foo

  #put the name of your foreign key field that may be null below

  BarForeignKeyField  =  PKRelatedFieldAcceptNull()

Be sure to include the field as defined in my post above and this should allow your modelserializer to use this custom version of the RelatedField class to accept null values.  

I have not tested it yet, but you should also be able to use the serializer flag (partial = True) somewhere in that definition in order to accept partial JSON payloads without throwing an error.  

My guess would be that this belongs on the meta class but I cannot say for certain at this time.  

Remember, you can always explicitly declare fields in a Model Serializer if you don't wish to use the defaults that the class will assign them.  

Steve 

[Salvatore Iovene]

Thanks Steven, that worked.

--
Salvatore Iovene

Personal website: http://iovene.com/
Founder of AstroBin: http://astrobin.com/

[Tom Christie]

> Is there any known workaround or will the bug be fixed soon?

Should be fixed as of 2.1.7

http://django-rest-framework.org/topics/release-notes.html#217

On Tuesday, 4 December 2012 12:51:16 UTC, Salvatore Iovene wrote:

[Salvatore Iovene]

Thanks Tom!

It appears to be working on related keys, but my null Datetime fields
with auto_now set to True still behave like non-null, mandatory
fields.

I can send a fake date in the request, like ´1970-01-01', because the
fields are auto_now, but it's inelegant. Got a fix for that too?


TIA,
Salvatore.

[Salvatore Iovene]

Also, your changes seem to work only on PUT requests, the problem
persists with POST requests.

At some point django-rest-framework attempts to get something from the
queryset with id = "", and I get an exception about converting the
empty string to int.

So not only do I still need Steven work around above, I also need to
set my related field to 0, when I mean null.

A similar problem appears with the fact that I need to include "id" in
my "fields". When I post, I need to set that to 0.

Salvatore.

[Tom Christie]

> At some point django-rest-framework attempts to get something from the 
queryset with id = "", and I get an exception about converting the 
empty string to int.

Okay that was a bug that you'd see in the Browseable API, but not when using JSON.  Should be fixed in 2.1.8 which is now released.

> A similar problem appears with the fact that I need to include "id" in 
my "fields". When I post, I need to set that to 0. 

I'm not sure I follow that.  POST or PUT?  Which generic view?  When would you want to set the id of an object to null?

> I can send a fake date in the request, like ´1970-01-01', because the 
fields are auto_now, but it's inelegant. Got a fix for that too? 

The best way to help me out and get it fixed quickly would be to submit a failing test case in a pull req.

Cheers,

  Tom

[Salvatore Iovene]

On Sat, Dec 8, 2012 at 9:14 PM, Tom Christie <christ...@gmail.com> wrote:
>> At some point django-rest-framework attempts to get something from the
> queryset with id = "", and I get an exception about converting the
> empty string to int.
>
> Okay that was a bug that you'd see in the Browseable API, but not when using
> JSON. Should be fixed in 2.1.8 which is now released.

I see that using JSON, I'll try one more time to make sure.

>> A similar problem appears with the fact that I need to include "id" in
> my "fields". When I post, I need to set that to 0.
>
> I'm not sure I follow that. POST or PUT? Which generic view? When would
> you want to set the id of an object to null?

I'm not trying to POST or PUT null as an id. I have fields = ('id',
...,) in my ModelSerializer, because I want to retrieve the id during
GET requests. But when I do a POST to create a new object, and I omit
the id, rest-framework will raise an exception because the empty
string could not be converted to int, during a .objects.get(id = id)
query.

So I have to resort to setting a fake id = 0 in my POST.

>> I can send a fake date in the request, like ´1970-01-01', because the
> fields are auto_now, but it's inelegant. Got a fix for that too?
>
> The best way to help me out and get it fixed quickly would be to submit a
> failing test case in a pull req.

I'll give it a try soon, unless you have time to look into it first.

Thanks!

[Steven Kane]

I believe the way this should be tackled is by inspecting your model fields "require" option to determine if the field is required for model creation.  If it is not, then gets and posts should, be default, not require those fields to exist.  Else, you should have to explicitly declare a field as optionally present.  I think Django forms may have a similar layered approach to required fields though model fields probably do what I suggested with regards to inspecting the model's flags to determine "requiredness".  

When you are "posting" data to your server I would not expect you to have an ID in that post message as the ID is traditionally set by the server.  If you would like to set the primary key field on the client for whatever reason (perhaps it is a key generated as a combination of model properties such that it should be unique) then you may need to overwrite several methods and it may not be advised to use a model serializer at all.  The basic structure of the serializer's main methods are all pretty clear and easy to override (credit to Tom for this readability, separation of concerns, and good use of expressions).  You can then also write your own class-based-view and explicitly define what you would like to happen on "post" "put" "get" etc.  I have done this in my test project and it's been remarkably transparent.  

In summary:

-create custom view that extends the closest fitting CBV

  -CBV inheriting from ListCreateAPIView accepting "Post"/"Get" at "api/modelname/"

  -CBV inheriting from RetrieveUpdateDestroyAPIView accepting "Post"/"Get"/"Delete"

-create your own methods that control what data gets passed along to your serializer

-create custom serializer (if needed) that can handle additional changes/omissions to your data before saving the model to your DB)

  -define your own save method

  -define your own restore_object method

  -define your fields to take advantage of the custom fields provided above or write your own custom fields that better handle cases where data might be "null"

Some of these solutions are going to overlap and will be redundant but I think it will easily get you into the inner-working of these objects and allow you to articulate exactly what is or isn't working while also solving your particular issue.  

Steve

[Steven Kane]

Several Typos in that previous post but I think the intent is clear.  Can anyone point out to me how to edit my post in google groups....lol.  Pretty sad that I cannot locate this option.