urlfetch maximum interactive timeout why 10 seconds and not 20?

[PK]

I have integrated my site with a cool online payments service. They provide an API for online payments that I access from the server side using the urlfetch() API. Unfortunately, once in a while their service takes a little bit more than 10 seconds to respond on the pay API, so the payment goes through but my code gets a timeout and reports an error to the user which is very frustrating.

I am trying to get the online payments vendor to improve their response times but I was wondering why did you pick 10 instead of 20 seconds for the urlfetch maximum timeout. In 30 seconds there is enough time to get in, do a 20 second urlfetch and return in worst case.

Thanks,

PK

PS Yes I have thought of doing the urlfetch from a task but the code gets pretty complicated pretty fast.

[Gwyn Howell]

it's annoying isn't it. could you catch the error and query the api again to check payment status?

[Ronoaldo José de Lana Pereira]

Hi PK,

We build a 500 qps website that do handle the payments from two payment gateways: PayPal and PagSeguro (from Brazil). Both suffer for those problems, and we only was able to solve this by using Task Queues. What we do is to proccess using this workflow:

1. User is redirected to payment gateway
2. Gateway charges the user
3. User goes back to our site, and we show a message like "Thanks for buying, your order is processing. Click here to see the order details."
4. Asynchronously, the gateway notifies the payment status (sometimes it takes a few minutes or even ours to the order be approved entirely from PagSeguro).
1. We receive the notification, and schedule a task queue with all the parameters to be processed in background. This idea that just forward the post from the gateway to be a task queue, and avoid deadlines.

Usually, the overall process (except when the payment isn't approved instantaneously) is very fast, and we don't have the user-facing problem.

Hope this help,

- Ronoaldo

[GAEfan]

Not sure what your are using for the urlfetch method, but it does have
a deadline parameter. Some syntax examples:

urlfetch.create_rpc(deadline=20)

urlfetch.fetch('http://....'% domain, method=urlfetch.GET, deadline=20)

[stevep]

If you are referring to Paypal url calls made under their test
environment -- these ran much more slowly (variably) for me vs.
production. Overall, GAE is difficult to ensure PP completion. I'd
made several comments a long time ago (b/4 pull queues & instances)
for a single, high-performance TQ to address key weaknesses. Note: My
only frame of reference is PP's digital goods purchase process.

HTH,
stevep

[Jason Collins]

For PayPal integration on App Engine, you need to ensure that you
split your PreAuth/Capture phases (i.e., never do a straight up
Purchase), and you have to implement their IPN mechanism to know the
true state of things. A PreAuth takes longer than a Capture, but if a
PreAuth times out, the consequences are less: on your IPN, if you get
a PreAuth transaction that you don't know about (i.e., because it
timed out), simply Void it. If your PreAuth succeeds, you can issue a
Capture which is typically quite quick.

It's a pain given the short(ish) url fetch timeout (or, if you look at
it another way, the long time PayPal takes to process cards
sometimes), but it can be done.

j

[P K]

My integration is with a new service
called Dwolla. It was particularly
embarrassing because the sandbox
always takes less than 10 seconds so
the issues arose only with real
customers in production.

Yesterday night, I implemented the
urlfetch part using a task so
now I should be good for up to
10 minutes.

PK

> --
> You received this message because you are subscribed to the Google Groups "Google App Engine" group.
> To post to this group, send email to google-a...@googlegroups.com.
> To unsubscribe from this group, send email to google-appengi...@googlegroups.com.
> For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
>

[PK]

The 1.5.5 pre-release that was just announced today includes:

" We have increased the URLFetch maximum deadline from 10 seconds to 60 seconds."

So I am thinking: I posted this topic four days ago. I gave it two days to see somebody from Google reply that they will do it or that they will never do it or something. Then I spent a night working around the issue. I am wondering: Couldn't somebody from Google reply to this thread  and mention that this will be coming in the next release??????? 

PK

www.gae123.com

[Brandon Wirtz]

If they promise, then cut it, you get mad. If they don’t promise and deliver you get mad. 

 

What was the right solution?

 

Generally I think this support was added to make things like Pay Pal work, but you need to be careful using it, waiting 58 seconds for your URLFetch will burn a lot of instance time.

--

You received this message because you are subscribed to the Google Groups "Google App Engine" group.

To view this discussion on the web visit https://groups.google.com/d/msg/google-appengine/-/vQMX34ZWUb8J.

[PK]

Brandon,

Who asked for a promise? I asked for an answer to a very reasonable question. Some guidance on why they picked 10 seconds in the first place and that they are considering changing it for this or next release would have been more than enough. The fact that 4 days later they changed it means that they have been thinking about it for some time. 

Java already has multithreading and when MT comes to python the instance time issue you are raising will be no issue. While the thread sleeps waiting for the urlfetch to complete other threads will make use of the instance.

PK

PS regarding the "you get mad" language you probably confuse me with somebody else or you have not seen somebody being really mad. If you try a bit harder with this kind of language and attitude you might make me mad but I doubt it :-)