Add an optional parameter to values() that returns a nested dictionary for foreign keys

[Moenad]

Currently, after calling values() and the query executes, the output is a single level dictionary, including foreign keys. I propose adding an extra parameter for values, or at least values_list, where if it's set to true, a nested dictionary will be returned when there's a foreign key.

Example:

Person model with the following fields: first_name, last_name and hometown (foreign key)

Hometown model with the following fields: name

A single record from Person.objects.values() will looks like this

{"id": 1
 "first_name": "first name",
 "last_name": "last name",
 "hometown__id": 1,
 "hometown__name": "town name",
}

I propose adding a nested optional parameter to values, where a single record from Person.objects.values(nested=True) will look like

{"id": 1
 "first_name": "first name",
 "last_name": "last name",
 "hometown": {
      "id": 1,
      "name": "town name"
  }
}

This feature is needed given that most APIs these days are nested, while it's simple to implement, I think it's much better to have it a built-in django feature.

[Bobby Mozumder]

I could also use a couple of enhancement to this:

1) Allow renaming of keys, instead of using the database column names.  

2) Allow callbacks functions (or lambdas) to convert output values to another format if needed.

With this, I could send the queries results right to JSON outputs.

-bobby

--
You received this message because you are subscribed to the Google Groups "Django developers (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email to django-develop...@googlegroups.com.
To post to this group, send email to django-d...@googlegroups.com.
Visit this group at http://groups.google.com/group/django-developers.
To view this discussion on the web visit https://groups.google.com/d/msgid/django-developers/8e5cbc9a-0317-40d3-8038-5b4300738b90%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Moenad]

100%, that would be great also. I thought of just posting the basic requirement that might be useful to most.

[Tim Graham]

There's an accepted ticket for adding aliasing to values(): https://code.djangoproject.com/ticket/16735

The current patch there hijacks values() **kwargs for the mapping of renamed fields which would prevent adding other kwargs like "nested" without disallowing those values as aliases. I guess we may want to rethink that approach.

[Bobby Mozumder]

A useful overall target for the next Django version would be to try and get all these feature up so that high-speed REST API development becomes easier.  (really need to be able to push thousands of requests per second)

I’d like to directly go from Query to Response:  

response = JsonResponse(Articles.objects.get(id=1).values(’title’, ’author’, ‘body'))

While still having things like aliased fields, nested trees, and value formatting.

-bobby

To view this discussion on the web visit https://groups.google.com/d/msgid/django-developers/369843ca-7b82-49b9-b7f5-e2f7fa578ca7%40googlegroups.com.

[Moenad]

Well, switch the field name aliasing to a dictionary without hijacking **kwargs ?

I prefer the following:

Articles.objects.get(id=1).values(’title’, ’author’, ‘body', alias={"title": "my_custom_title"}, nested=True)

[Marten Kenbeek]

I think it'd be more consistent with other parts of the ORM to use **kwargs to specify aliases. For nested data you can use an object, say N, similar to Q and F objects:

Articles.objects.filter(id=1).values('body', N('author'), my_custom_title='title')

I'm no ORM expert, but could something like this be possible by allowing expressions in values() and using custom output fields?

[Moenad]

I like the idea but what about multiple nesting, multiple foreign keys?

We end up with something like N('author__book')? a bit confusing no?

[Joachim Jablon]

Marten's suggestion is quite interesting for providing a way to tell which data you want nested and which data you don't. Plus, this form might be interesting to solve another problem : how would Django know if we want :

{"id": 1
 "first_name": "first name",
 "last_name": "last name",
 "hometown": {
      "id": 1,

      "name": "town name",
      "country": 3
  }
}


# or

{"id": 1
 "first_name": "first name",
 "last_name": "last name",
 "hometown": {
      "id": 1,

      "name": "town name",
      "country": {
        "id": 3,
        "name": "country name"
      }
  }
}

Limiting the nesting to a single level would be an arbitrary decision and users should be able to control this (IMHO)

So we could have a "level" argument that would say how many levels deep it will search but then what if you want SOME nesting in some branches, not in others, like : 

{"id": 1
 "first_name": "first name",
 "last_name": "last name",
 "hometown": {
      "id": 1,

      "name": "town name",
      "country": {
        "id": 3,
        "name": "country name"
      }
  },
  "father": 4
}

(here, "father" is another FK that we don't want expanded ?

Maybe a syntax like :

N("person", "person__hometown", "person__hometown__country")

Note : this might not be equivalent to N("person__hometown__country"), that you could use if you want ONLY the nested "country"

I'd like that.

And it's compatible with the suggestion of using **kwargs for aliasing (for the top level element of the dict, at least)

[Moenad]

I think I wasn't clear from the beginning, the idea of "nested" is to nest all possible levels, not just a single level. I like the idea of "N", given that you can have more control, but having something like N("person", "person__hometown", "person__hometown__country") which will be different than N("person__hometown__country") is confusing.

I have another idea, why not make the alias + nest possible with a single parameter, where it takes a dictionary and expect how the final structure and aliasing are?

For example:

{"id": "custom_id"
 "first_name": "custom_first_name",
 "last_name": "custom_last_name",
 "hometown": {
      "__alias__": "custom_hometown"
      "id": "custom_hometown_id",
      "name": "custom_name",
      "country": "custom_country"
  }
}

or to make it a standard in someway,

{"id": "custom_id"
 "first_name": "custom_first_name",
 "last_name": "custom_last_name",
 "hometown": {
      "__alias__": "custom_hometown"
      "hometown__id": "custom_hometown_id",
      "hometown__name": "custom_name",
      "hometown__country": "custom_country"
  }
}

as you noticed, if there's a foreign key for example, a new key "__alias__" or something should be added in the dict. Also, no need for values() *args, the dict structure would be more than enough?

[Marc Tamlyn]

I can see a use for this, but the API is unsure. Given that from a performance point of view it should be possible to do this as a transform after a values query (in most cases using a similar lazy sequence-like object will maintain the performance you need), can I propose implementing it as an external app to find a good API. Once this has been done, we can look at how buildable that API is at a lower level to get the maximum performance.

To view this discussion on the web visit https://groups.google.com/d/msgid/django-developers/551b126b-8d72-42de-9eff-45d14e8a7372%40googlegroups.com.

[Shai Berger]

On Wednesday 25 November 2015 20:34:11 Marc Tamlyn wrote:
> I can see a use for this, but the API is unsure. Given that from a
> performance point of view it should be possible to do this as a transform
> after a values query (in most cases using a similar lazy sequence-like
> object will maintain the performance you need), can I propose implementing
> it as an external app to find a good API. Once this has been done, we can
> look at how buildable that API is at a lower level to get the maximum
> performance.
>

That sounds like a good plan to me.

I would just like to add that, since we're talking about values() queries
where no model instances are constructed, one possible API is to have just a
single boolean `nested` argument, where the selection of which FKs to expand
is implied by select_related() calls.

Shai.

[Josh Smeaton]

I would really like two things for values to support.

1. Aliases .values(alias='field');

2. Expressions .values(alias=F('field'))

I think these two features are absolute must haves, and the syntaxes above are already standard in other parts of the ORM.

If someone can come up with a way to support nested relations while supporting the above syntax, then I'd be OK with that. But at the moment, I'm firmly in the "this is the responsibility of a serialiser" camp. I'm not convinced Django needs to support nested objects at all. Is this something you could implement with your own queryset method on a manager? Is this maybe something we could look at creating a new queryset method called .values_dict() ?

If it weren't for backwards compatibility, I'd suggest that referencing the related object would automatically nest that object. That would differentiate between the id and the field values('related_id', 'related') -> '{"related_id": 1, "related": {"id": 1, ..}}'.

If there's (rough) consensus on having nested objects, then we could allow something like: .values(..., ..., nested=('related', 'related__otherrelated')). If the value of nested is an iterable then assume we're nesting, otherwise nested is an alias for the field. I don't particularly like overloaded kwargs, but we're just guarding against someone wanting to alias as "nested" which we could call out in docs anyway.

The more I think about this the more I think nesting and aliases within a nest should probably be done in a different queryset method. Or just handled by a serialiser. If you want more requests per second, then add some more backends.

[Tim Graham]

We now have support for expressions in values()/values_list() -- thanks Ian! With the new commit [0], aliases can be created like this: .values(alias=F('field'))

Ian has offered an additional commit in the pull request [1] to allow .values(alias='field') (without the F() expression) to automatically wrap the string in an F() expression to create an alias. I'm not sure whether or not to accept that patch as I think I prefer the look of the explicit F() rather than magically treating strings as F() expressions. What do you think?

[0] https://github.com/django/django/commit/39f35d4b9de223b72c67bb1d12e65669b4e1355b
[1] https://github.com/django/django/pull/7088

[Loïc Bistuer]

I prefer enforcing .values(alias=F(’something’)), to me .values(alias=‘something’) reads as the equivalent of .values(alias=Value(‘something’)).

--
Loïc

> --
> You received this message because you are subscribed to the Google Groups "Django developers (Contributions to Django itself)" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to django-develop...@googlegroups.com.
> To post to this group, send email to django-d...@googlegroups.com.

> Visit this group at https://groups.google.com/group/django-developers.
> To view this discussion on the web visit https://groups.google.com/d/msgid/django-developers/00d4305c-175e-415c-b446-a53c7d15c00d%40googlegroups.com.

[Constantine Covtushenko]

Agree with Loïc on 100%.

And also it opens more options in the future.

Regards,

On Fri, Aug 19, 2016 at 9:58 PM, Loïc Bistuer <loic.b...@gmail.com> wrote:

I prefer enforcing .values(alias=F(’something’)), to me .values(alias=‘something’) reads as the equivalent of .values(alias=Value(‘something’)).

--
Loïc

> On Aug 20, 2016, at 1:04 AM, Tim Graham <timog...@gmail.com> wrote:
>
> We now have support for expressions in values()/values_list() -- thanks Ian! With the new commit [0], aliases can be created like this: .values(alias=F('field'))
>
> Ian has offered an additional commit in the pull request [1] to allow .values(alias='field') (without the F() expression) to automatically wrap the string in an F() expression to create an alias. I'm not sure whether or not to accept that patch as I think I prefer the look of the explicit F() rather than magically treating strings as F() expressions. What do you think?
>
> [0] https://github.com/django/django/commit/39f35d4b9de223b72c67bb1d12e65669b4e1355b
> [1] https://github.com/django/django/pull/7088
>
> On Wednesday, November 25, 2015 at 7:24:22 PM UTC-5, Josh Smeaton wrote:
> I would really like two things for values to support.
>
> 1. Aliases .values(alias='field');
> 2. Expressions .values(alias=F('field'))
>
> I think these two features are absolute must haves, and the syntaxes above are already standard in other parts of the ORM.
>
> If someone can come up with a way to support nested relations while supporting the above syntax, then I'd be OK with that. But at the moment, I'm firmly in the "this is the responsibility of a serialiser" camp. I'm not convinced Django needs to support nested objects at all. Is this something you could implement with your own queryset method on a manager? Is this maybe something we could look at creating a new queryset method called .values_dict() ?
>
> If it weren't for backwards compatibility, I'd suggest that referencing the related object would automatically nest that object. That would differentiate between the id and the field values('related_id', 'related') -> '{"related_id": 1, "related": {"id": 1, ..}}'.
>
> If there's (rough) consensus on having nested objects, then we could allow something like: .values(..., ..., nested=('related', 'related__otherrelated')). If the value of nested is an iterable then assume we're nesting, otherwise nested is an alias for the field. I don't particularly like overloaded kwargs, but we're just guarding against someone wanting to alias as "nested" which we could call out in docs anyway.
>
> The more I think about this the more I think nesting and aliases within a nest should probably be done in a different queryset method. Or just handled by a serialiser. If you want more requests per second, then add some more backends.
>
> --
> You received this message because you are subscribed to the Google Groups "Django developers (Contributions to Django itself)" group.

> To unsubscribe from this group and stop receiving emails from it, send an email to django-developers+unsubscribe@googlegroups.com.
> To post to this group, send email to django-developers@googlegroups.com.

> Visit this group at https://groups.google.com/group/django-developers.
> To view this discussion on the web visit https://groups.google.com/d/msgid/django-developers/00d4305c-175e-415c-b446-a53c7d15c00d%40googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "Django developers  (Contributions to Django itself)" group.

To unsubscribe from this group and stop receiving emails from it, send an email to django-developers+unsubscribe@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.

Visit this group at https://groups.google.com/group/django-developers.

To view this discussion on the web visit https://groups.google.com/d/msgid/django-developers/C7D98DF5-5F48-4E89-849F-D5E28A93D1DE%40gmail.com.

[Josh Smeaton]

Just as an additional data point - most expressions that accept strings internally convert them to F(), so it's not entirely inconsistent with other behaviour. I don't really have a preference here though, so happy to go with what the majority prefer.

[Артём Клименко]

may be useful https://github.com/aklim007/DjangoNestedValues

[Ian Foote]

I think the consensus here is to not add the extra commit, so I've closed the ticket as wontfix. (https://code.djangoproject.com/ticket/16735#comment:26)

Ian

To view this discussion on the web visit https://groups.google.com/d/msgid/django-developers/80d0aa4c-bdac-433b-96b5-cbde00bec257%40googlegroups.com.