QuerySet not iterable

[Abhijeet Viswa]

Hey guys,

I need a help with a quirky bug while iterating over a QuerySet:

TypeError: argument of type 'QuerySet' is not iterable

The following is the block of code that produces the error: (the save method overrides the default save method in a Model called Transaction)

def save(self, *args, **kwargs):
        obj = self
        if not obj.ref_number:  # the line of code throwing the error
            transactions = Transaction.objects.values_list("ref_number", flat=True)     # cache the list
            while True:
                ref_number = random.randint(1000000001, 9999999999)
                if ref_number not in transactions:
                    obj.ref_number = ref_number
                    super(Transaction, obj).save(*args, **kwargs)
                    return
        else:
            super(Transaction, obj).save(*args, **kwargs)

This piece of code was working fine until we had modified (what we thought to be) an unrelated part of the source: (this is the modified line of code)

items = OrderItem.objects.prefetch_related('toppings').select_related('item').select_related('item__category')
order = Order.objects.filter(uuid=order_uuid).prefetch_related(
            Prefetch(
                lookup='items',
                queryset=items
            )
        ).select_related('outlet').select_related('user').select_related('outlet__group') \
         .select_for_update(of=('self', 'outlet'))

We basically added a call to the select_for_update. This was required by us since we are modifying both the Order as well as Outlet models. (Also, the Transaction model is a super-model for the Order model)

My best guess is that the select_for_update locks certain rows of the the Order, Transaction and a bunch of other models. This results in the line of code failing because the query to list the value fails. However, I tested this hypothesis out by running a select_for_update, populating the QuerySet returned and then sleeping for a while while I ran a different query on the Model for which rows were locked. This ran perfectly fine, with no problems.

We are using PostgreSQL as our backend database. Any help would be greatly appreciated.

Thanks

[Simon Charette]

Hello there,

From looking at your code (super() calls) it seems like your are using Python 2.

 We've seen similar reports about stdlib functions hiding system level exceptions

instead of surfacing them[0] so that might it.

It's hard to tell without the full traceback though.

Best,

Simon

[0] https://code.djangoproject.com/ticket/26600

[Abhijeet Viswa]

Hello,

Thank your for your reply. I'm not using Python 2. I have attached the traceback as is emailed by Django to the Admin.

Traceback:

File "/home/user/api/env/lib/python3.6/site-packages/django/core/handlers/exception.py" in inner
  34.             response = get_response(request)

File "/home/user/api/env/lib/python3.6/site-packages/django/core/handlers/base.py" in _get_response
  115.                 response = self.process_exception_by_middleware(e, request)

File "/home/user/api/env/lib/python3.6/site-packages/django/core/handlers/base.py" in _get_response
  113.                 response = wrapped_callback(request, *callback_args, **callback_kwargs)

File "/home/user/api/env/lib/python3.6/site-packages/django/views/decorators/csrf.py" in wrapped_view
  54.         return view_func(*args, **kwargs)

File "/home/user/api/env/lib/python3.6/site-packages/django/views/generic/base.py" in view
  71.             return self.dispatch(request, *args, **kwargs)

File "/home/user/api/env/lib/python3.6/site-packages/rest_framework/views.py" in dispatch
  505.             response = self.handle_exception(exc)

File "/home/user/api/env/lib/python3.6/site-packages/rest_framework/views.py" in handle_exception
  465.             self.raise_uncaught_exception(exc)

File "/home/user/api/env/lib/python3.6/site-packages/rest_framework/views.py" in raise_uncaught_exception
  476.         raise exc

File "/home/user/api/env/lib/python3.6/site-packages/rest_framework/views.py" in dispatch
  502.             response = handler(request, *args, **kwargs)

File "/usr/lib/python3.6/contextlib.py" in inner
  52.                 return func(*args, **kwds)

File "/home/user/api/project/cashless/views/order.py" in post
  143.             order.save()

File "/home/user/api/project/payment_gateway/models.py" in save
  79.                 if ref_number not in transactions:

Exception Type: TypeError at [ENDPOINT]
Exception Value: argument of type 'QuerySet' is not iterable

I have did further testing and determined that my earlier hypothesis of select_for_update is not responsible for this error. The error persisted even after having removed that call.

I also figured that it the TypeError was mapping some internal system call and I did further testing and I realized the error is a TransactionManagementError: An error occurred in the current transaction. You can't execute queries until the end of the 'atomic' block.

That save happens inside of an atomic transaction. There are a couple of other select statements that are executed before this one (also inside the transaction). Though, I haven't been able to figure out what's wrong though.

Any leads would be appreciated.

Thank you.

[Abhijeet Viswa]

Hey there guys,

I'm happy to say that we've finally figured out the bug. It was never an error with the underlying DBMS or Django itself. It was our code. It's sad that we didn't figure it out sooner and had to spend so many man hours on this.

As I said, the select_for_update was never a problem to begin with. The TransactionManagementError was due to us executing queries inside Transaction block even after the transaction had rollbacked due to some reason (which we couldn't figure out). After having PSQL log all queries, we realized someone was sending an explicit rollback message after we do a SELECT COUNTER(*) (etither Django or our code). We thought it was some hidden bug with Django while using aggregation.

We test the result of the SELECT statement and deny the request (or so we thought we did). We properly rollback and then instead of returning a Response, we just continued forward issuing more queries. Thankfully, the TransactionManagementError helped us and resulted in us not losing any data. Bottom line, if you guys face this error, make sure your user code is correct and you are doing everything as expected. Also, try logging the DBMS to see whether error is originating so you can narrow it down further.

Thanks to all those who helped. I did learn a whole lot over this process, and for that I'm glad.