UrlRequest's wait() "waits" forever in case of error

[Omar Little]

Hi,

I posted an issue around the use of UrlRequest on StackOverflow (http://stackoverflow.com/questions/39640695/kivy-urlrequest-called-as-a-class-methods-not-executed) and somebody there kindly helped me figuring out a possible way forward with my issue, but since I have no real resolution I was thinking about posting it here as well to have it looked at.

I have setup a class to host UrlRequest so I can call it to fetch both the status of the request (success/fail) and the results themselves. The function needs to wait for the request to complete before the call returns to the main app. My issue is, that I´m using req.wait() to achieve this (which is as per the documentation), BUT req.wait hangs if the request was unsuccesful (error/exception). If I use a URL that simply doesn't exist, UrlRequest correctly calls the on_failure/on_error function, and the is_finished flag is set to True as a result of that. Despite that, req.wait wouldn't let the method finish.

I´m looking for a way to make the class exit anyway to prevent any kind of looping.

from kivy.app import App
from kivy.uix.floatlayout import FloatLayout
from kivy.network.urlrequest import UrlRequest
from kivy.logger import Logger
from kivy.config import Config
from kivy.clock import Clock

class WebExplorer():
  
    def test_connection(self, path):
        self.path = path
        req = UrlRequest(self.path,on_failure=self.connectionFailure,on_error=self.connectionFailure,debug=True,on_success=self.connectionSuccess,on_redirect=self.connectionSuccess)
        req.wait()
        return (self._return_value)

    def connectionSuccess(self, req, results):
        print ("Success")
        self._return_value = [0,results]

    def connectionFailure(self, req, results):
        print ("Failure")
        self._return_value = [1,results]

class MainScreen(FloatLayout):
    def __init__(self, **kwargs):
        super(MainScreen, self).__init__(**kwargs)
        self.URLtest = ['http://www.ikea.com/','https://www.google.com','https://www.sdfwrgaeh.com']
        for URL in self.URLtest:
            self.returnCode = WebExplorer().test_connection(URL)
            if self.returnCode[0] == 0:
                print ("Correct URL")
            else:
                print ("Wrong URL")
  
class App(App):
    def build(self):
        return MainScreen()

if __name__ == "__main__":
    App().run()

[ZenCODE]

You don't need to wait on a URLRequest. It's asynchronous, so the callback happens later...

https://kivy.org/docs/api-kivy.network.urlrequest.html#kivy.network.urlrequest.UrlRequest

[Omar Little]

Thanks for commenting on this, I know the call is asynchronous, and that's exactly the reason why I need to wait for it to complete, because the main script operates based on the result of the request.

[ZenCODE]

Then try using a callback. Waiting will (probably) block the normal event loop. Is there some reason you cannot use a callback? You could use a state machine type model and simply resume what you need to do after the callback?

[Omar Little]

I use callbacks from UrlRequest to set the status of the request accordingly. Is this what you mean? Sure, I want to resume the process as soon as the callback is done since that's what sets the return value, but I need to wait for the response from the callback in the meantime. How to wait for it is the question I´m asking, because any method I could try did not work in the scenario when the request returns an error.

I can't use sleep or setup a kivy clock holding the process while it's waiting.

[ZenCODE]

You can't use sleep because that blocks the event loop. Of course you could use a clock, but that's exactly the same as using a callback in the first place. Why not use it directly?

I suspect this is simpler than you think. If you put together a minimal example of what you are trying to do, I'll try and show you how to do it via a callback?

[Omar Little]

The example provided in the OP is the minimal example. Okey, minus the print commands. E.g. I am trying to setup UrlRequest as a class I can call from the main script. I have setup my callbacks and my return values to the main script. I can't think of a shorter version of what I wrote (which is a tiny subset of the real app I´m working on)

[ZenCODE]

from kivy.app import App
from kivy.uix.floatlayout import FloatLayout
from kivy.network.urlrequest import UrlRequest

class WebExplorer():
    def test_connection(self, path, cb):
        self.path = path
        UrlRequest(self.path, on_failure=cb,
                   on_error=cb, debug=True,
                   on_success=cb,
                   on_redirect=cb)
        return self

class MainScreen(FloatLayout):
    def __init__(self, **kwargs):
        super(MainScreen, self).__init__(**kwargs)
        self.URLtest = ['http://www.ikea.com/', 'https://www.google.com',
                        'https://www.sdfwrgaeh.com']

        explorers = []  # In case you want to use them
        for URL in self.URLtest:
            explorers.append(WebExplorer().test_connection(URL,
                                                           self.got_result))

    def got_result(self, *args):
        print("got results {0}".format(args))

[Omar Little]

Thanks for the example.  I don't understand everything so I have some research to do to figure out how your code works in details. Mainly how you can execute the call back of UrlRequest outside the WebExplorer class.

There's one caveat with this solution however, which is that it sorts of defeats the purpose of using the WebExplorer class as a one liner to check URL and then resume the main script. Using your solution you can only resume starting from a new callback function. The call to UrlRequest can only be the last command you run. So waiting for the request to complete is basically achieved by not having more code to execute in the main loop and resume once the call back function has been called. You effectively need one new callback function every time you want to call UrlRequest (assuming you can't always group all requests in one single call like you did in your example)

E.g. I cannot effectively design a call to UrlRequest handling all events in one line and carry on the execution on the next following line, like:

[stuff]

[call to UrlRequest, the class handles the waiting]

[more stuff, like checking the success of the above call]

[ZenCODE]

On Thursday, September 29, 2016 at 10:44:35 PM UTC+2, Omar Little wrote:

Thanks for the example.  I don't understand everything so I have some research to do to figure out how your code works in details. Mainly how you can execute the call back of UrlRequest outside the WebExplorer class.

 

Yes, the WebExplorer class does not do anything here. It's really not required. See example following.

There's one caveat with this solution however, which is that it sorts of defeats the purpose of using the WebExplorer class as a one liner to check URL and then resume the main script. Using your solution you can only resume starting from a new callback function. The call to UrlRequest can only be the last command you run. So waiting for the request to complete is basically achieved by not having more code to execute in the main loop and resume once the call back function has been called. You effectively need one new callback function every time you want to call UrlRequest (assuming you can't always group all requests in one single call like you did in your example)

No, you can give it large batches and it calls the calback as it completes each download.
 

E.g. I cannot effectively design a call to UrlRequest handling all events in one line and carry on the execution on the next following line, like:

 

[stuff]

[call to UrlRequest, the class handles the waiting]

[more stuff, like checking the success of the above call]

Not using that code, because it synchronous. You need to think more event driven and use the callback to react and continue what you doing.

See the example below. You can whatever you want in the callbacks. You can design code to carry on execution where you left off. This example demonstrates that.

from kivy.app import App
from kivy.uix.floatlayout import FloatLayout
from kivy.network.urlrequest import UrlRequest

class MainScreen(FloatLayout):
    def __init__(self, **kwargs):
        super(MainScreen, self).__init__(**kwargs)
        self.URLtest = ['http://www.ikea.com/', 'https://www.google.com',
                        'https://www.sdfwrgaeh.com']

        for URL in self.URLtest:
            UrlRequest(URL, on_failure=self.got_fail,
                       on_error=self.got_error, debug=True,
                       on_success=self.got_success,
                       on_redirect=self.got_redirect)

    def got_success(self, req, *args):
        print("got success {0}".format(req.url))

    def got_error(self, req,  *args):
        print("got error {0}".format(req.url))

    def got_fail(self, req, *args):
        print("got fail {0}".format(req.url))

    def got_redirect(self, req, *args):
        print("got redirect {0}".format(req.url))

[Omar Little]

I appreciate the feedback, thanks. What you presented is the way I´ve been using UrlRequest earlier.

I don't feel like I managed to present my point correctly, though, judging by your feedback. Apologies for that. I think I need to be more specific. Basically I want to be able to dig into a URL structure, with a root URL, folders underneath (several levels) and finally a list of files. To do that, I am initiating a first UrlRequest to the root to identify the folders. Etc. until I get the list of files I´m looking after, based on user UI input and so on. Therefore, I typically need to call UrlRequest, process the results, then call it again based on user input, and then possibly calling it again to "explore" the file structure. So the calls are not directly subsequent. My problem is not that I need to handle X calls simultaneously, it's that I need to handle X calls that require some processing between the calls.

My initial feeling based on that requirement (which I implemented, it's just that it doens't handle on_failure as it hangs on it, which is the object of me posting here), and given my background, is that I don't see myself create 5 functions in my App just to handle one call to UrlRequest. 4 callbacks, and 1 new function to carry on the main process. Then either from same function or another one, call UrlRequest again, fork into 4 new callbacks, and carry on in again a new function. That's a bit much for the same basic request? 

That's why I wanted to design it as a class call to handle all the events, therefore streamlining my script by limiting the number of functions, however still preserving the event handling. But maybe this way of thinking is not compliant to the philosophy behind Kivy, or is technically not feasible?

[ZenCODE]

On Friday, September 30, 2016 at 12:24:17 AM UTC+2, Omar Little wrote:

I appreciate the feedback, thanks. What you presented is the way I´ve been using UrlRequest earlier.

I don't feel like I managed to present my point correctly, though, judging by your feedback. Apologies for that. I think I need to be more specific. Basically I want to be able to dig into a URL structure, with a root URL, folders underneath (several levels) and finally a list of files. To do that, I am initiating a first UrlRequest to the root to identify the folders. Etc. until I get the list of files I´m looking after, based on user UI input and so on. Therefore, I typically need to call UrlRequest, process the results, then call it again based on user input, and then possibly calling it again to "explore" the file structure. So the calls are not directly subsequent. My problem is not that I need to handle X calls simultaneously, it's that I need to handle X calls that require some processing between the calls.

Of course that possible. On the return from your initial inquiry, you process it and then create new UrlRequest for the next part? If you need user input, get it and then do your next request. I can't see a problem here?
 

My initial feeling based on that requirement (which I implemented, it's just that it doens't handle on_failure as it hangs on it, which is the object of me posting here), and given my background, is that I don't see myself create 5 functions in my App just to handle one call to UrlRequest. 4 callbacks, and 1 new function to carry on the main process. Then either from same function or another one, call UrlRequest again, fork into 4 new callbacks, and carry on in again a new function. That's a bit much for the same basic request? 

Of course you can use one if you want.

            UrlRequest(URL, on_failure=self.got_result,
                       on_error=self.got_result, debug=True,
                       on_success=self.got_result,
                       on_redirect=self.got_result)

Of, if you want to know which result it is, something like

UrlRequest(URL, on_failure=lambda *args: self.got_result("failed", *args),
                   lambda *args: self.got_result("success", *args)

 There are various ways to do what you require and not limitation in the framework.
 

That's why I wanted to design it as a class call to handle all the events, therefore streamlining my script by limiting the number of functions, however still preserving the event handling. But maybe this way of thinking is not compliant to the philosophy behind Kivy, or is technically not feasible?

It's perfectly technically feasible, and once you wrap your head around it, I'm sure you'll be surprised at home simple it actually is...:-)

 

[Tom Grundy]

Posting a reply to this old thread.  I'm having a similar problem: I'd like to make the call synchronous, which the docs say can be accomplished by using the request.wait() method.

After reading this thread, I understand that synchronous event chain can be accomplished by making a chain of request->callback->next-request->callback->next-request->etc - leading to funky function names like _part1 and _part1cb and _part2 and _part2cb which isn't really a problem but seems fairly odd - however I think the issue here is that the wait() method doesn't perform as documented or as expected: if the result is a success, request_stat gets set to something other than None and the wait() method ends.  If, however, the result is a failure or error, request_stat remains None and you have an endless loop.

So, on the one hand is looks like the response on this thread is 'go async or go home' but the docs say 'use wait to make it syncronous'.  One option for cleaning it up would be to eradicate the wait() function from the code base and from the docs, but I'd suggest a preferred way is to give the user the option of making it sync, by modifying wait() to actually end if there is a failure or an error, and also to have a failsafe timeout of 10 seconds or something, to prevent endless loops.

[João Teixeira]

try to declare timeout on your request.

IE: UrlRequest(url, on_success=vfcallbacksuccess, timeout=3)

if you don't declare timeout on your request and the server doesn't respond it will not even trigger the on_failure and on_error on asyncronous requests