[Django] #26600: map says a queryset is not iterable
Django
----------------------------------------------+----------------------------
Reporter: ihucos | Owner: nobody
Type: Bug | Status: new
Component: Database layer (models, ORM) | Version: 1.8
Severity: Normal | Keywords: queryset
| iterator map
Triage Stage: Unreviewed | Has patch: 0
Easy pickings: 0 | UI/UX: 0
----------------------------------------------+----------------------------
So I got a funny bug with iterators map and Django that I can not localize
or reproduce.
In our production server we get this sometimes, mainly for only one
customer and it only fails on a very small percentage of all requests.
Sorry for only chunks of information, not sure if it is enough to process
this bug request:
Th error message:
{{{
TypeError: argument 2 to map() must support iteration
}}}
Chunk of the code where this happens:
{{{#!python
if filter(
lambda obj:
self.pk in verbose_map(attrgetter('pk'),
obj.resources.all()) and
obj.begins < end and obj.ends > dt,
resource_override_qs):
return 0
}}}
Trying to reconstruct resource_override_qs I come up with:
{{{#!python
facility = Facility.objects.\
prefetch_related('resources',
'opening_hours',
'openinghour_overrides',
'slow_hours').\
get(pk=facility.pk)
resource_override_qs =
facility.facilityresourceoverride_set.prefetch_related('resources')
}}}
What I did was to replace map with verbose_map in our production
environment.
{{{#!python
def verbose_map(function, iterable):
try:
return map(function, iterable)
except TypeError, exc:
raise TypeError('map failed: "{}". iterable: {}, type: {}, attrs:
{}'.format( # NOQA
exc, iterable, type(iterable), dir(iterable)))
}}}
After two weeks or so the error occurred again:
{{{#!python
map failed: "argument 2 to map() must support iteration". iterable:
[<FacilityResource: 4>, <FacilityResource: 5>, <FacilityResource: 6>,
<FacilityResource: 7>], type: <class 'django.db.models.query.QuerySet'>,
attrs: ['__and__', '__bool__', '__class__', '__deepcopy__', '__delattr__',
'__dict__', '__doc__', '__format__', '__getattribute__', '__getitem__',
'__getstate__', '__hash__', '__init__', '__iter__', '__len__',
'__module__', '__new__', '__nonzero__', '__or__', '__reduce__',
'__reduce_ex__', '__repr__', '__setattr__', '__setstate__', '__sizeof__',
'__str__', '__subclasshook__', '__weakref__', '_add_hints', '_as_sql',
'_batched_insert', '_clone', '_create_object_from_params', '_db',
'_earliest_or_latest', '_extract_model_params', '_fetch_all',
'_filter_or_exclude', '_for_write', '_has_filters', '_hints', '_insert',
'_known_related_objects', '_merge_known_related_objects',
'_merge_sanity_check', '_next_is_sticky', '_populate_pk_values',
'_prefetch_done', '_prefetch_related_lookups',
'_prefetch_related_objects', '_prepare', '_raw_delete', '_result_cache',
'_setup_aggregate_query', '_sticky_filter', '_update', 'aggregate', 'all',
'annotate', 'as_manager', 'bulk_create', 'complex_filter', 'count',
'create', 'dates', 'datetimes', 'db', 'defer', 'delete', 'distinct',
'earliest', 'exclude', 'exists', 'extra', 'filter', 'first', 'get',
'get_or_create', 'in_bulk', 'is_compatible_query_object_type', 'iterator',
'last', 'latest', 'model', 'none', 'only', 'order_by', 'ordered',
'prefetch_related', 'query', 'raw', 'reverse', 'select_for_update',
'select_related', 'update', 'update_or_create', 'using',
'value_annotation', 'values', 'values_list']
}}}
As we can see map really gets an QuerySet object, that queryset object
does reveal its content in __str__ but python's built in map says its not
iterable nevertheless
Alone for the customer where this happens the most that part of the code
should be called at least a couple of times per day.
I am going with a workaround like calling list() on the queryset before
mapping it or so.
But my company should be ok, to try this again with another verbose_map
that may provide more debugging information. My efforts to reproduce this
in our code base failed.
--
Ticket URL: <https://code.djangoproject.com/ticket/26600>
Django <https://code.djangoproject.com/>
The Web framework for perfectionists with deadlines.
Django
Reporter: ihucos | Owner: nobody
Type: Bug | Status: closed
Component: Database layer | Version: 1.8
(models, ORM) |
Severity: Normal | Resolution: needsinfo
Keywords: queryset iterator | Triage Stage:
map | Unreviewed
Has patch: 0 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
Changes (by timgraham):
* status: new => closed
* needs_better_patch: => 0
* resolution: => needsinfo
* needs_tests: => 0
* needs_docs: => 0
Comment:
No, I can't figure out if or where a bug might in Django based on the
report.
--
Ticket URL: <https://code.djangoproject.com/ticket/26600#comment:1>
Django
Reporter: ihucos | Owner: nobody
Type: Bug | Status: closed
Component: Database layer | Version: 1.8
(models, ORM) |
Severity: Normal | Resolution: needsinfo
Keywords: queryset iterator | Triage Stage:
map | Unreviewed
Has patch: 0 | Needs documentation: 0
Easy pickings: 0 | UI/UX: 0
Comment (by shaib):
FWIW, the original traceback (which your `verbose_map()` hides) may be
much more informative. Guessing that you're on Python 2, you may want to
rewrite it as something like:
{{{#!python
def verbose_map(function, iterable):
try:
return map(function, iterable)
except TypeError:
e_cls, e_inst, traceback = sys.exc_info()
raise TypeError, (
'map failed: "{}". iterable: {}, type: {}, attrs:
{}'.
format(e_inst, iterable, type(iterable),
dir(iterable))
), traceback
}}}
--
Ticket URL: <https://code.djangoproject.com/ticket/26600#comment:2>
Django
Reporter: ihucos | Owner: nobody
Type: Bug | Status: closed
Component: Database layer | Version: 1.8
(models, ORM) |
Severity: Normal | Resolution: needsinfo
Keywords: queryset iterator | Triage Stage:
map | Unreviewed
Has patch: 0 | Needs documentation: 0
Easy pickings: 0 | UI/UX: 0
Comment (by ihucos):
We got where it comes from.
This exception occurs after Heroku times out the request, but apparently
it does not kill it: https://devcenter.heroku.com/articles/request-timeout
So this happens in some funny Heroku environment that I really can't
reproduce.
--
Ticket URL: <https://code.djangoproject.com/ticket/26600#comment:3>
Django
Reporter: ihucos | Owner: nobody
Type: Bug | Status: closed
Component: Database layer | Version: 1.8
(models, ORM) |
Severity: Normal | Resolution: needsinfo
Keywords: queryset iterator | Triage Stage:
map | Unreviewed
Has patch: 0 | Needs documentation: 0
Easy pickings: 0 | UI/UX: 0
Comment (by Brian Hahn):
I am also seeing this issue, and have to force the queryset into a list to
workaround this error. Any plausible idea why this might be happening?
--
Ticket URL: <https://code.djangoproject.com/ticket/26600#comment:4>
Django
Reporter: ihucos | Owner: nobody
Type: Bug | Status: closed
Component: Database layer | Version: 1.8
(models, ORM) |
Severity: Normal | Resolution: needsinfo
Keywords: queryset iterator | Triage Stage:
map | Unreviewed
Has patch: 0 | Needs documentation: 0
Easy pickings: 0 | UI/UX: 0
Changes (by Sergey Fedoseev):
* cc: Sergey Fedoseev (added)
--
Ticket URL: <https://code.djangoproject.com/ticket/26600#comment:5>
Django
Reporter: ihucos | Owner: nobody
Type: Bug | Status: closed
Component: Database layer | Version: 1.11
(models, ORM) |
Severity: Normal | Resolution: needsinfo
Keywords: queryset iterator | Triage Stage:
map | Unreviewed
Has patch: 0 | Needs documentation: 0
Easy pickings: 0 | UI/UX: 0
Changes (by Vitaliy):
* version: 1.8 => 1.11
Comment:
I found a way to reproduce the error.
Class **A** is a simplified version of **QuerySet**
{{{#!python
class A(object):
def __init__(self):
self._result_cache = None
def _fetch_all(self):
if self._result_cache is None:
self._result_cache = list(self.iterator())
def iterator(self):
for x in range(10):
if x == 5:
raise MemoryError # the type of exception doesn't matter
yield x
def __iter__(self):
self._fetch_all()
return iter(self._result_cache)
}}}
{{{#!python
In [2]: map(str, A())
---------------------------------------------------------------------------
TypeError Traceback (most recent call
last)
/home/user/workspace/project/service/models.pyc in <module>()
----> 1 map(str, A())
TypeError: argument 2 to map() must support iteration
}}}
When using python 3:
{{{#!python
In [6]: map(str, A())
---------------------------------------------------------------------------
MemoryError Traceback (most recent call
last)
<ipython-input-6-aad9944017be> in <module>()
----> 1 map(str, A())
<ipython-input-5-4a8af6429c4d> in __iter__(self)
15
16 def __iter__(self):
---> 17 self._fetch_all()
18 return iter(self._result_cache)
19
<ipython-input-5-4a8af6429c4d> in _fetch_all(self)
6 def _fetch_all(self):
7 if self._result_cache is None:
----> 8 self._result_cache = list(self.iterator())
9
10 def iterator(self):
<ipython-input-5-4a8af6429c4d> in iterator(self)
11 for x in range(10):
12 if x == 5:
---> 13 raise MemoryError
14 yield x
15
MemoryError:
}}}
--
Ticket URL: <https://code.djangoproject.com/ticket/26600#comment:6>
Django
Reporter: ihucos | Owner: nobody
Type: Bug | Status: new
(models, ORM) |
Severity: Normal | Resolution:
map | Unreviewed
Has patch: 0 | Needs documentation: 0
Easy pickings: 0 | UI/UX: 0
Changes (by Carlton Gibson):
* status: closed => new
* resolution: needsinfo =>
Comment:
I'm going to reopen so we review the reproduce.
--
Ticket URL: <https://code.djangoproject.com/ticket/26600#comment:7>
Django
Reporter: ihucos | Owner: nobody
Type: Bug | Status: closed
Component: Database layer | Version: 1.11
(models, ORM) |
Keywords: queryset iterator | Triage Stage:
map | Unreviewed
Has patch: 0 | Needs documentation: 0
Easy pickings: 0 | UI/UX: 0
Changes (by Simon Charette):
* status: new => closed
* resolution: => wontfix
Comment:
I think we can close as ''wontfix'' given `map` doesn't turn `__iter__()`
exceptions into `TypeError` on Py3k and we don't support Py2k anymore.
Thanks for the investigation Vitaliy.
--
Ticket URL: <https://code.djangoproject.com/ticket/26600#comment:8>
Django
Reporter: ihucos | Owner: nobody
Type: Bug | Status: closed
Component: Database layer | Version: 1.11
(models, ORM) |
Severity: Normal | Resolution: wontfix
Keywords: queryset iterator | Triage Stage:
map | Unreviewed
Has patch: 0 | Needs documentation: 0
Easy pickings: 0 | UI/UX: 0
Comment (by Vitaliy):
There are at least two ways to workaround the error on Py2k:
* force the queryset into a list when using `map`
* use `imap` (from `itertools`) instead of `map`
--
Ticket URL: <https://code.djangoproject.com/ticket/26600#comment:9>
Django
Reporter: ihucos | Owner: nobody
Type: Bug | Status: closed
Component: Database layer | Version: 1.11
(models, ORM) |
Severity: Normal | Resolution: wontfix
Keywords: queryset iterator | Triage Stage:
map | Unreviewed
Has patch: 0 | Needs documentation: 0
Easy pickings: 0 | UI/UX: 0
Comment (by pro100filipp):
The problem is not only with the map. We faced the same issue when using
code like this:
{{{
if model_instance in model.objects.all():
...
}}}
Replying to [comment:8 Simon Charette]:
> I think we can close as ''wontfix'' given `map` doesn't turn
`__iter__()` exceptions into `TypeError` on Py3k and we don't support Py2k
anymore.
>
> Thanks for the investigation Vitaliy.
--
Ticket URL: <https://code.djangoproject.com/ticket/26600#comment:10>
Django
Reporter: ihucos | Owner: nobody
Type: Bug | Status: closed
Component: Database layer | Version: 1.11
(models, ORM) |
Severity: Normal | Resolution: wontfix
Keywords: queryset iterator | Triage Stage:
map | Unreviewed
Has patch: 0 | Needs documentation: 0
Easy pickings: 0 | UI/UX: 0
Comment (by Simon Charette):
Philipp, if you hit a similar exception that means `map` must be involved
somehow.
If you didn't call it yourself then it's likely caused by a usage of `map`
internally in the queryset iteration code.
In this case the ticket resolution still stands, Python 2 `map`'s
shadowing of the underlying exception makes it impossible to determine its
true nature and thus we cannot conclude Django is at fault. As mentioned
previously this should all be solved on Python 3 where `map` doesn't
exhibit this behaviour.
By the way an `in` operation on a queryset you are disposing off is likely
to trigger a `MemoryError` just like in the original report. I'd suggest
you opt for something along `queryset.filter(pk=instance.pk).exists()`
instead.
--
Ticket URL: <https://code.djangoproject.com/ticket/26600#comment:11>
Django
Reporter: ihucos | Owner: nobody
Type: Bug | Status: closed
Component: Database layer | Version: 1.11
(models, ORM) |
Severity: Normal | Resolution: wontfix
Keywords: queryset iterator | Triage Stage:
map | Unreviewed
Has patch: 0 | Needs documentation: 0
Easy pickings: 0 | UI/UX: 0
Comment (by Philipp Kuznetsov):
I'm really sorry, of course in case of `in` exception message is `argument
of type 'QuerySet' is not iterable`. This has nothing to do with `map` and
ofc there's an obvious workaround with `exists()`. What I am afraid of is
if there are more of this hidden cases where queryset's `__iter__` raises
an exception and it's shadowed by the outer code exception.
Replying to [comment:11 Simon Charette]:
> Philipp, if you hit a similar exception that means `map` must be
involved somehow.
>
> If you didn't call it yourself then it's likely caused by a usage of
`map` internally in the queryset iteration code.
>
> In this case the ticket resolution still stands, Python 2 `map`'s
shadowing of the underlying exception makes it impossible to determine its
true nature and thus we cannot conclude Django is at fault. As mentioned
previously this should all be solved on Python 3 where `map` doesn't
exhibit this behaviour.
>
> By the way an `in` operation on a queryset you are disposing off is
likely to trigger a `MemoryError` just like in the original report. I'd
suggest you opt for something along
`queryset.filter(pk=instance.pk).exists()` instead.
--
Ticket URL: <https://code.djangoproject.com/ticket/26600#comment:12>
Django
Reporter: ihucos | Owner: nobody
Type: Bug | Status: closed
Component: Database layer | Version: 1.11
(models, ORM) |
Severity: Normal | Resolution: wontfix
Keywords: queryset iterator | Triage Stage:
map | Unreviewed
Has patch: 0 | Needs documentation: 0
Easy pickings: 0 | UI/UX: 0
Comment (by Abhijeet):
I'm unsure if bumping this up is against forum rules. However, I get this
Error in Py3k in production. I'm not using `map` in my code, though I'm
unsure if Django is internally using it.
This is the code block that produces the error. The save method is
overriding the save method of a Mode called `Transaction`.
{{{
def save(self, *args, **kwargs):
obj = self
if not obj.ref_number: # Set the reference number before saving
transactions = Transaction.objects.values_list("ref_number",
flat=True) # cache the list
while True:
ref_number = random.randint(1000000001, 9999999999)
if ref_number not in transactions:
obj.ref_number = ref_number
super(Transaction, obj).save(*args, **kwargs)
return
else:
super(Transaction, obj).save(*args, **kwargs)
}}}
The erroring line of code is: `if ref_number not in transactions:`
This piece of code was working fine until we had modified another
unrelated part of the source code.
{{{
items =
OrderItem.objects.prefetch_related('toppings').select_related('item').select_related('item__category')
order = Order.objects.filter(uuid=order_uuid).prefetch_related(
Prefetch(
lookup='items',
queryset=items
)
).select_related('outlet').select_related('user').select_related('outlet__group')
\
.select_for_update(of=('self', 'outlet'))
}}}
What we did was add the call to `select_for_update`, followed by a
transaction which modifies the `Order` and `Outlet` model. Here, the
`Order` model is a child model of the `Transaction` model.
My guess as to what the problem is is that `select_for_update` locks the
relevant rows/table which results in the line of code in the `save` method
to fail internally, ultimately being caught as a cryptic type error. I'll
be modifying
`transactions = Transaction.objects.values_list("ref_number", flat=True)`
to
`transactions = Transaction.objects.values_list("ref_number",
flat=True).list()` to see if that'll fix the problem.
PS: I'm sorry if my formatting/post/bump is against norms. I'm new here,
with this being my first comment.
Replying to [comment:8 Simon Charette]:
> I think we can close as ''wontfix'' given `map` doesn't turn
`__iter__()` exceptions into `TypeError` on Py3k and we don't support Py2k
anymore.
>
> Thanks for the investigation Vitaliy.
--
Ticket URL: <https://code.djangoproject.com/ticket/26600#comment:13>