[Django] #30818: Django can't handle m2m relations well sometimes

3 views

Skip to first unread message

Django

unread,

Sep 29, 2019, 11:02:58 PM9/29/19

to django-...@googlegroups.com

#30818: Django can't handle m2m relations well sometimes
-------------------------------------+-------------------------------------
Reporter: AvTiMp | Owner: nobody
Type: Bug | Status: new
Component: Database | Version: master
layer (models, ORM) | Keywords: Django 1.9 Python
Severity: Normal | 2.7.10
Triage Stage: | Has patch: 0
Unreviewed |
Needs documentation: 0 | Needs tests: 0
Patch needs improvement: 0 | Easy pickings: 0
UI/UX: 0 |
-------------------------------------+-------------------------------------
In my case, we have hundreds of pods who run gunicorn to handle requests
and controlled by k8s hpa, we set that gunicorn should restart one
worker's thread who has handled 10000 requests, so there could be
thosands of restart per day. but something strange happens then,
sometimes a gunicorn wroker thread report an error like this:

{{{
host : dp-b6b59027f6-57f89f9547-4qpv5
username:
args : ()
kwargs : {}
exc : Traceback (most recent call last):
File "/usr/local/lib/python2.7/dist-packages/rest_framework/views.py",
line 471, in dispatch
response = handler(request, *args, **kwargs)
File
"/opt/tiger/tce_platform/src/platform_site/service/views/service_v3.py",
line 715, in get_by_psm
result = serializer.data
File "/usr/local/lib/python2.7/dist-
packages/rest_framework/serializers.py", line 507, in data
ret = super(Serializer, self).data
File "/usr/local/lib/python2.7/dist-
packages/rest_framework/serializers.py", line 239, in data
self._data = self.to_representation(self.instance)
File "/usr/local/lib/python2.7/dist-
packages/rest_framework/serializers.py", line 476, in to_representation
ret[field.field_name] = field.to_representation(attribute)
File "/usr/local/lib/python2.7/dist-
packages/rest_framework/serializers.py", line 615, in to_representation
iterable = data.all() if isinstance(data, models.Manager) else data
File "/usr/local/lib/python2.7/dist-
packages/django/db/models/manager.py", line 223, in all
return self.get_queryset()
File "/usr/local/lib/python2.7/dist-
packages/django/db/models/fields/related_descriptors.py", line 792, in
get_queryset
return qs._next_is_sticky().filter(**self.core_filters)
File "/usr/local/lib/python2.7/dist-packages/django/db/models/query.py",
line 790, in filter
return self._filter_or_exclude(False, *args, **kwargs)
File "/usr/local/lib/python2.7/dist-packages/django/db/models/query.py",
line 808, in _filter_or_exclude
clone.query.add_q(Q(*args, **kwargs))
File "/usr/local/lib/python2.7/dist-
packages/django/db/models/sql/query.py", line 1243, in add_q
clause, _ = self._add_q(q_object, self.used_aliases)
File "/usr/local/lib/python2.7/dist-
packages/django/db/models/sql/query.py", line 1269, in _add_q
allow_joins=allow_joins, split_subq=split_subq,
File "/usr/local/lib/python2.7/dist-
packages/django/db/models/sql/query.py", line 1149, in build_filter
lookups, parts, reffed_expression = self.solve_lookup_type(arg)
File "/usr/local/lib/python2.7/dist-
packages/django/db/models/sql/query.py", line 1035, in solve_lookup_type
_, field, _, lookup_parts = self.names_to_path(lookup_splitted,
self.get_meta())
File "/usr/local/lib/python2.7/dist-
packages/django/db/models/sql/query.py", line 1330, in names_to_path
"Choices are: %s" % (name, ", ".join(available)))
FieldError: Cannot resolve keyword u'service' into field. Choices are: id,
is_primary, port, type
}}}

then all same kind of requests proxying to this failed with the same
error, but other threads work just as excepted.

To find the reason, I echo the field info of all models at our handler
func when error occurs, like this:

{{{
from django.core.exceptions import FieldError
try:
##handler logic here
##just try to get one instance of model service who has many
ManyToManyFields(with model B, C, D ...). they all belongs to app service
except FieldError as e:
from django.apps import apps
for v in
apps.get_app_config('service').get_models(include_auto_created=True,
include_deferred=True):
print('-----------------', v.__name__)
# <class 'django.utils.datastructures.ImmutableList'>
print(' ----------', v._meta.get_fields())
raise e
}}}

then I found that model B has no field `<ManyToManyRel:
service.service>`, so we have the previous error `Cannot resolve keyword
u'service' into field`.
but in other healthy threads they all have this field. I think the func
'''_populate_directed_relation_graph''' generate all relate fields then
store them in `__dict__` which leading to the errors of all coming
requests. So why does this happen?

--
Ticket URL: <https://code.djangoproject.com/ticket/30818>
Django <https://code.djangoproject.com/>
The Web framework for perfectionists with deadlines.

Django

unread,

Sep 30, 2019, 2:15:02 AM9/30/19

to django-...@googlegroups.com

#30818: Django can't handle m2m relations well sometimes.

-------------------------------------+-------------------------------------
Reporter: AvTiMp | Owner: nobody

Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------

Changes (by felixxm):

* status: new => closed
* resolution: => invalid

Comment:

Thanks for this report, however Django 1.9 is not supported anymore, we
can consider this as a bug if you will be able to reproduce this in Django
2.2+. Moreover you use case looks niche and can be hard to reproduce, so a
sample project is also required.