Weird Database behavior (Can't access object from a view after directly saving it from inside another view)

[Ahmed Shahwan]

Hi,

I have a very strange problem that I don't know why it happens.

I'm creating a REST API service that wraps another REST API payment service, let's call the other services XPAY. It has various payment options (Card, eWallet, Kiosk, and Cash). The steps required to use the services (to make a transaction)

1. authentication 2. order registration 3. payment key 4. prepare frontend based on payment type 5. Callback URL 6. Redirect URL

and each step depends on its predecessor. The first 4 steps are performed through HTTP Requests that I issue from my service. I store some of the information I receive in those steps in a Model called Transaction. 5th and 6th steps are Callback endpoints that I create to receive further information about the status of the transaction that I started in the first 4 steps.

My endpoints design is:

myservice/payment/card

myservice/payment/wallet

myservice/payment/kiosk

myservice/payment/cash

myservice/payment/callback-url

myservice/payment/redirect-url

These endpoints are wrappers for the payment methods available in XPAY's service that my frontend apps will use to provide those payment options for users.

I implemented the first 3 endpoints with no problem. inside those endpoints, I create a transaction object with a unique id and then update it from inside the callback-url endpoint after I receive the information from XPAY's service. Note that it takes some time between creating the Transaction object and saving it into the database from inside my endpoint, and updating it from inside the callback url endpoint.

The weird thing happens in the 4th endpoint `cash` endpoint. XPAY's service responds very quickly on my callback-url endpoint and when it gets to the updating step, I get Transaction.DoesNotExist Exception while getting the transaction object using Transaction.objects.get(id=id).

After some debugging time using ipdb I found that the instance has already been saved to the database! and I can retrieve it with its id. but when I let it run ordinarily the saving operation doesn't finish until that time (I got to know that using some print() statements), which is very strange, it's impossible that my laptop is that slow! 

Solutions I tried:

1. atomic transaction

2. time.sleep some time after making the request and saving the transaction object to the db

3. time.sleep in the callback-url before getting the transaction from DB

I checked any possible logic error related to it and there's nothing illogical.

the issue is related to Postgres and how it operates, and also with Django's ORM. I don't know.

Main technologies I use:

Django 2.2

Postgres Docker container

DRF

I hope someone gives me an insight into why this is happening and how to work around it.

Thanks

[mohammed habib]

Are you sure the callback request passes the right parameters ?

Could you share some of your views code from the 6 steps, and your Transaction model ?

Sent from my iPhone

--
You received this message because you are subscribed to the Google Groups "Django users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to django-users...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/django-users/e43be322-bab9-4162-9fbc-05c6c9b95c94%40googlegroups.com.

[Ahmed Shahwan]

Transaction model:

class WeacceptTransaction(Transaction):

   """Weaccept's Transaction model"""

    # before frontend interaction

   auth_token = models.TextField()  # got from step 1

   order_id = models.IntegerField(verbose_name="Accept's Order ID")  # id from step 2

   order_url = models.CharField(max_length=255, null=True)  # order_url from step 2

   billing_data = JSONField()  # provided in request for paymet key, step 3

   payment_key = models.TextField()  # payment_token from step 3

   hmac = models.TextField(null=True)

   # after (from callback)

   success_status = models.BooleanField(default=False)

   is_refunded = models.BooleanField(default=False)

   reference_number = models.IntegerField(null=True)

   processed_clbk_json = JSONField(null=True)

The cash_payment View:

class CashPayment(APIView):

   """Cash Payment endpoint"""

    serializer_class = serializers.CardPaymentSerializer

   required_fields = ["amount_cents", "billing_data"]

    def post(self, request):

       # Validate required fields

       utils.validate(self.required_fields, request)

       payment = utils.CashPayment(**request.data)

       payment.prepare()

       payment.pay_request()

       # create a new transaction

       models.WeacceptTransaction.objects.create(

           merchant_order_id=payment.merchant_order_id,

           amount=payment.amount_cents,

           auth_token=payment.auth_res.get("token"),

           order_id=payment.order_reg_res.get("id"),

           order_url=payment.order_reg_res.get("order_url"),

           billing_data=payment.billing_data,

           payment_key=payment.payment_key_res.get("token"),

           hmac=payment.pay_req_res.get("hmac"),

       )

        return Response(

           {

               "message": "Our representative will go to the address you provided "

               "to collect the cash from you",

               **payment.pay_req_res,

           }

       )

Please note: the first 3 steps are wrapped into the method `prepare()`

the 4th step is `pay_request()`

the callback-url endpoint (where the 5th step happens) 

class TransactionProcessedCallback(APIView):

   """Processed callback that will recieve "TRANSACTION", "TOKEN", "ORDER",

      "DELIVERY_STATUS" objects"""

    def post(self, request):

       # XXX extend to handle TOKEN, DELIVERY_STATUS objects... later

       t_obj = request.data.get("obj")  # transaction object

       incoming_hmac = request.query_params.get("hmac")

       calculated_hmac = utils.calculate_hmac_transaction(t_obj)

        # if not equal hmac, not coming from Accept!

       if incoming_hmac != calculated_hmac:

           return Response(

               {"message": "invalid data"}, status=status.HTTP_400_BAD_REQUEST

           )

        import ipdb

        ipdb.set_trace()
        # XXX: The error happens here

        transaction = models.WeacceptTransaction.objects.get(

           merchant_order_id=t_obj.get("order").get("merchant_order_id")

       )

       transaction.success_status = bool(t_obj.get("success"))

       transaction.is_refunded = bool(t_obj.get("is_refunded"))

       transaction.reference_number = int(t_obj.get("data").get("transaction_no") or 1)

       transaction.processed_clbk_json = t_obj

       transaction.save(

           update_fields=[

               "success_status",

               "is_refunded",

               "reference_number",

               "processed_clbk_json",

           ]

       )

        email = t_obj.get("order").get("shipping_data").get("email")

       # TODO: send mail based upon success status

       try:

           send_mail(

               "Transaction Processed",

               "Your transaction is processed",

               "SERNDER MAIL",

               [email],

               fail_silently=False,

           )

       except Exception as ex:

           print(">>>>>>>>>>FAILED TO SEND MAIL")

           print(ex)

        return Response({"message": "Transaction Updated!"}, status=status.HTTP_200_OK)

The 6th step (redirect-url) is irrelevant here, and not included in the XPAY's service for this payment option.

To unsubscribe from this group and stop receiving emails from it, send an email to django...@googlegroups.com.

[mohamed habib]

Wild guess in the dark here, as I am unfamiliar with this API. you are doing:

 `payment.pay_request()` 

before creating the WeacceptTransaction object. 

Maybe this line needs to move below: `models.WeAcceptTransaction.objects.create(...)` ? 

At what point is the callback request triggered ?

To unsubscribe from this group and stop receiving emails from it, send an email to django-users...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/django-users/a379e013-a01c-4c58-bf2b-0b4cd01f0fae%40googlegroups.com.

--

Best regards,
Mohammed M. Habib, PhD

[Ahmed Shahwan]

I tried, same happened. it's related to how Postgres works internally with the Django-ORM.

To view this discussion on the web visit https://groups.google.com/d/msgid/django-users/a379e013-a01c-4c58-bf2b-0b4cd01f0fae%40googlegroups.com.

[mohamed habib]

I highly doubt its related to ORM or postgres, the creation of the record is synchronous so you can be sure that the record exists after the create() call.

Since you are sure that you are fetching the data properly we can rule out this possibility

Its probably a race condition of the callback arriving too soon *before* you hit the create() call. So do you know at what point your payment gateway will make the callback request and what would trigger it? Make sure you are 

To unsubscribe from this group and stop receiving emails from it, send an email to django-users...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/django-users/987c6ce7-70b6-49a9-ab3e-506e369d060e%40googlegroups.com.

[mohamed habib]

if you are able to share more code from the `utils.CashPayment` call that may help us

[Ahmed Shahwan]

the response to my callback quicker than saving the object?! it's highly unlikely! there's something else I don't understand. The step that `pay_request()` is the last before getting the response from XPAY's server. I tried to put it after saving the transaction object. same error occurred!

I also tried to put the transaction save operation inside the `with transaction.atomic()` statement, but same error.

also, I tried refresh_from_db() method on the transaction after saving it, and saving it again! and using some print statements

The weird thing is I get the transaction object printed before and after XPAY's server hitting my callback-url (with the same error)

using the ipdb and fetching the object from DB I get the same error, after trying the same statement for 2 or 3 times, it works!

To view this discussion on the web visit https://groups.google.com/d/msgid/django-users/987c6ce7-70b6-49a9-ab3e-506e369d060e%40googlegroups.com.

[Ahmed Shahwan]

That's the class, nothing different other than HTTP requests

class CashPayment(BasePayment):

"""Cash Payment Wrapper Class"""

integration_id = CASH_INTEGRATION_ID

def __init__(self, amount_cents: int, billing_data: dict):

super().__init__(amount_cents, billing_data)

def pay_request(self):

"""Step 4. issue Pay request"""

payload = {

"source": {"identifier": "cash", "subtype": "CASH"},

"payment_token": self.payment_key_res.get("token"),

}

self.pay_req_res = self.hit_accept(

self._api_url + "acceptance/payments/pay", payload=payload

)

On Wednesday, August 21, 2019 at 3:39:13 PM UTC+2, mohamed habib wrote:

if you are able to share more code from the `utils.CashPayment` call that may help us

To view this discussion on the web visit https://groups.google.com/d/msgid/django-users/987c6ce7-70b6-49a9-ab3e-506e369d060e%40googlegroups.com.

--

Best regards,
Mohammed M. Habib, PhD

[mohammed habib]

What is the Transaction class which your model inherits from ? Is it just a base model ?

Let’s do an experiment 

Print(“object created”) right after the create() method

Print(“callback requested”) on top of your callback method

Which one do you see first ? 

Sent from my iPhone

To unsubscribe from this group and stop receiving emails from it, send an email to django-users...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/django-users/044d543f-5b6e-40d7-9ee6-36b7e84f5647%40googlegroups.com.

[Ahmed Shahwan]

The Transaction is just a base class with 2 fields to inherit, nothing more.

class Transaction(models.Model):

merchant_order_id = models.CharField(

max_length=255, blank=False, null=False, default=""

)

# this should afford 99,999,999 (amount is in cents (piasters))

amount = models.DecimalField(

verbose_name="money amount", max_digits=8, decimal_places=0

)

I see "callback requested" first, somewhat expected.

I moved the "pay_request()" function call (which tells XPAY's backend to call mine) after saving the transaction, and:

I see:

object created

callback requested

STRANGELY enough, I get the same error which is:

this is definitely related to django-orm and postgres!

To view this discussion on the web visit https://groups.google.com/d/msgid/django-users/044d543f-5b6e-40d7-9ee6-36b7e84f5647%40googlegroups.com.

[Ahmed Shahwan]

I solved the problem.

It's related to the atomic requests. I had "ATOMIC_REQUESTS": True in the database settings and XPAY's service responded so fast while the first atomic-request hadn't finished yet, so it prevented the other callback-url from accessing the same instance.

Thank you very much for your time and effort

To view this discussion on the web visit https://groups.google.com/d/msgid/django-users/044d543f-5b6e-40d7-9ee6-36b7e84f5647%40googlegroups.com.

[mohamed habib]

Wow that is interesting, thank you for this update. I will be reading more about this. 

To unsubscribe from this group and stop receiving emails from it, send an email to django-users...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/django-users/79ed22d5-675d-4362-834d-5f70f5e23856%40googlegroups.com.