API for introspecting models
sebastien piquemal
So this might be a stupid idea, or it might have been answered
before ... actually it seems so obvious to me that you have probably
thought of it before and rejected it !!! But let's try ...
I was wondering, why isn't there a user API for introspecting models ?
OK, there is an API, through '_meta', which is not documented (unless
you read the source), which is not addressed to users, and is
therefore likely to change, and therefore unreliable.
On the other hand, there are a lot of cases where introspecting your
models enables much more dryer code (ModelForm, admin, but also lots
of 3rd-party applications, APIs - serialization, automatic templating,
and so on). So why not helping the users to do that ?
I am myself working on some serializers and I need a list of ALL the
fields, (ALL as in "as seen by a user"). A lot of things are actually
not straight-forward, for example : many2many handled separately
(which as a user you don't really care about), MTI which messes-up
everything by adding pointer fields (which as a user you don't really
care about either), and hiding the real pk.
So, right now introspecting models is pretty much detective work,
while I think providing a simple interface for it could enable a lot
of powerful things to be written.
Does this make sense ? Any thoughts ?
Sébastien
PS : here is the code I came up with to return a dict of all fields
(like I said ... detective work) : http://dpaste.com/hold/619453/
PS3 : I can totally work on that / submit a patch if some are
interested
Russell Keith-Magee
> Hi all !
>
> So this might be a stupid idea, or it might have been answered
> before ... actually it seems so obvious to me that you have probably
> thought of it before and rejected it !!! But let's try ...
>
> I was wondering, why isn't there a user API for introspecting models ?
>
> OK, there is an API, through '_meta', which is not documented (unless
> you read the source), which is not addressed to users, and is
> therefore likely to change, and therefore unreliable.
See https://code.djangoproject.com/ticket/12663
_meta is in a weird limbo state. It isn't *technically* stable API,
and it isn't documented, but it's *effectively* a stable API -- both
because it hasn't changed in a long time, and there is lots of code in
the wild that would break hard if we changed it.
So - it's one of those things that *should* be documented and
fomalized; it's just waiting on someone with the spare time to do the
job.
There's also a light cleanup required. There are some parts of the
_meta api -- like get_field() -- that are completely uncontroversial,
and would clearly be stabilized as is. However, there are also API
entry points that are more esoteric, and in some cases, are represent
potentially duplicated functionality in the _meta structure. What is
needed is for a motivated someone to do an audit, cleanup, and
documentation of the code that is there.
Yours,
Russ Magee %-)
sebastien piquemal
> fomalized; it's just waiting on someone with the spare time to do the
> job.
documentation task can go forever :)
And what about the extra features I suggested ? A more user-friendly
way to get all -data- fields (including m2m, excluding pointers), an
easy way to get the name of the pk field of the first parent.
This would allow the user to forget that there is some multi-table
inheritance under the hood. Most of the time - when using the models -
you just use them transparently, e.g.
class Animal(models.Model):
code = models.IntegerField(primary_key=True)
name = models.CharField(max_length=100)
class Mouse(Animal):
lab = models.CharField(max_length=100)
It is great that you can do :
>>> my_mouse.lab, my_mouse.name
('acme', 'pinky')
instead of :
>>> my_mouse.lab, my_mouse.animal_ptr.name
('acme', 'pinky')
so MTI is transparent, if you know what I mean. Thing is, there is no
equivalent when you introspect your model ... which gives extra-work
to the developer, and makes introspecting more complicated. Instead of
having :
>>> [f.name for f in Mouse._meta.fields]
['code', 'name', 'animal_ptr', 'lab']
I'd love to have something in the following (not to mention the m2m
fields)
>>> [f.name for f in Mouse._meta.get_fields(exclude_ptrs=True)] #
Or why not get_all_field_names(exclude_ptrs=False)
['code', 'name', 'lab']
Because that's all you need to know in most of the cases.
Same pb with the primary key. The only way I found to get the name of
the pk field is to do :
>>> Animal._meta.pk.name
'code'
Which with Mouse will become :
>>> Mouse._meta.pk.name
'animal_ptr'
Once again, if you want to know the name of the pk field, you would
expect (MTI being transparent from the user perspective), to get
'code' instead.
Or did I miss some methods/properties from the _meta API ?
Cheers,
Sébastien
On Sep 23, 12:10 pm, Russell Keith-Magee <russ...@keith-magee.com>
wrote:
> > Hi all !
>
> > So this might be a stupid idea, or it might have been answered
> > before ... actually it seems so obvious to me that you have probably
> > thought of it before and rejected it !!! But let's try ...
>
> > I was wondering, why isn't there a user API for introspecting models ?
>
> > OK, there is an API, through '_meta', which is not documented (unless
> > you read the source), which is not addressed to users, and is
> > therefore likely to change, and therefore unreliable.
>
Russell Keith-Magee
>> So - it's one of those things that *should* be documented and
>> fomalized; it's just waiting on someone with the spare time to do the
>> job.
>
> Sounds good ! Any estimation on how big is the cleaning task ? I know
> documentation task can go forever :)
About as long as this piece of string that I'm holding. Oh, wait - you
can't see the string? Let me hold it up for you... :-)
Seriously -- I can't really answer this. It's not a completely trivial
task, but it's not complex on the order of "multi-db" or "no-sql
support" either. The first step really is an audit -- a rough cut at
documenting everything that is there at the moment. Once we have that,
we can look at how the existing API is being used, and evaluate the
cleanup opportunities.
Once that's done, we'll be in a much better position to evaluation how
much work is involved.
> And what about the extra features I suggested ? A more user-friendly
> way to get all -data- fields (including m2m, excluding pointers), an
> easy way to get the name of the pk field of the first parent.
...
> Or did I miss some methods/properties from the _meta API ?
Off the top of my head, I can't answer this. It's *possible* that
there might be some way to do this, but I'd need to go digging to work
out for sure, and I can't refer to the documentation to find out
easily :-)
However, I wouldn't be at all surprised if it isn't possible. The
_meta API is mostly designed for Django's own internal needs, and
Django needs to make a very clear distinction between fields and
m2m's, forward and reverse fields, and so on; hence the distinctions
that exist in the current API. "Iterating over all fields" doesn't
make much sense from an internal perspective, because different types
of fields need to be handled differently. Similarly, the "hidden" _ptr
fields might not be important from a public visibility perspective,
but they're very important from an internal perspective.
That said, just because Django doesn't have a use for the API doesn't
mean it wouldn't be useful. If you can propose a clean way to augment
the _meta API to provide this sort of capability, I don't see any
fundamental reason why it shoudn't be included.
Yours,
Russ Magee %-)
Justin Holmes
definitely have some of the "break hard" projects that Russell is
talking about and I'd love to have a more straightforward and
sustainable way to do introspection.
Let me know how I can help.
> --
> You received this message because you are subscribed to the Google Groups "Django developers" group.
> To post to this group, send email to django-d...@googlegroups.com.
> To unsubscribe from this group, send email to django-develop...@googlegroups.com.
> For more options, visit this group at http://groups.google.com/group/django-developers?hl=en.
>
>
--
Justin Holmes
Head Instructor, SlashRoot Collective
SlashRoot: Coffee House and Tech Dojo
60 Main Street
New Paltz, NY 12561
845.633.8330
sebastien piquemal
Honestly, I don't have that much time, so I can't promise to look
actively into that (but I guess that's no big deal). Of course, if
Justin helps it sounds like a more reasonable task :) Anyways, I could
perfectly start with the first step :
> a rough cut at documenting everything that is there at the moment.
offer exactly, and therefore give me a clearer idea on how to
implement those "gimme all the fields" functions.
That's for the theory !
> If you can propose a clean way to augment the _meta API to provide this sort of capability
What would be an "unclean" way for you ? Are you talking about (a) the
beauty of the API itself, or (b) its implementation ?
Assuming (a) : from what I understood from your previous message,
_meta API's fate is -one day- to be a public API that everybody could
use in peace and harmony, right ? So with that in mind ... is it clean
to have methods for public use and and methods for internal use in the
same object ? -> I guess in that case the best would be to make
private all methods for internal use, and public all methods for
public API, but that's probably not realistic since that would require
massive renamings.
> That said, just because Django doesn't have a use for the API doesn't mean it wouldn't be useful.
am pretty sure we are not the only ones !
Cheers,
Sébastien
Simon Charette
Simon
Mathieu AGOPIAN
seems my previous message got eaten up, so I'll try a repost :
I've done a small app that I use to inspect models, it lives here: https://github.com/magopian/django-inspect-model
I'm afraid this isn't rocket science, and is probably very ugly. I'm using the "inspect" module from the python stdlib.
Please feel free to comment on my piece of crap^Wcode and let me know if there's anything that could help or if I need to tweak/modify anything to make it useful.
Regards
Mathieu