feature suggestion: "intuitive" waits after non-javascript GET or POST

[Harry Percival]

Hi there,

following on from a discussion in the bug tracker, http://code.google.com/p/selenium/issues/detail?id=6497, barancev suggested I ask here.

I'm wondering whether there's a way of providing more "intuitive" wait behaviour for selenium when after it does a "normal" (ie non-ajax) GET or POST request.

This came up when I switched from webdriver.Firefox to webdriver.Phantomjs, and found a lot of changes in behaviour.  It seems the fact that Firefox takes a while to load was masking a load of race conditions.

An example is:  I have a normal HTML POST form on a page, which also displays a table of items.  submitting a new item via the form causes a POST, redirect, and reload of the page whose table now has an extra item.  With webdriver.Firefox, I can do a rows = browser.find_elements_by_css_selector('table tr') or whatever, but switching to PhantomJS I suddenly hit some StaleElement exceptions, because selenium manages to get some table rows from before the browser refreshes.

So the standard answer is: write some wait-based code, with lots of WebDriverWaits and stuff.

But that's a hassle! I wonder if there's a way to provide a more intuitive behaviour? We already very intuitive behaviour on "browser.get", whereby the call effectively blocks until the page is loaded, so subsequent calls to find_element won't pick up anything stale from the previous page.

IF selenium can tell the difference between a "normal" GET caused by clicking on a hyperlink, or a "normal" POST caused by submitting a form, couldn't we also then trigger some sort of block, or some sort of invalidation of the cache?

For basic, non-javascript-heavy websites, I think that would provide a very intuitive way of working...

I know I've said intuitive a bunch of times now, and I'm aware that it only feels intuitive to me possibly, but does that appeal to anyone else?

HP

[Luke Inman-Semerau]

What you call "a hassle" I call a Best Practice™ :)

Most selenium implementation do a best effort to wait for the page to
load and it is explicitly not guaranteed nor will it likely be. If you
are seeing a form post not wait for load using ghostdriver / phantomjs
I would recommend logging an issue on that project:
https://github.com/detro/ghostdriver

But in general I would follow the best practice and add a
WebDriverWait for the table to become stale and then be on my merry
way. (That could is in a Page Object, right?!)

> --
> You received this message because you are subscribed to the Google Groups
> "Selenium Developers" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to selenium-develo...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/selenium-developers/b06ce961-31c3-4bef-ae0c-2e712e356bcc%40googlegroups.com.
> For more options, visit https://groups.google.com/groups/opt_out.

[Harry Percival]

Thanks for the reply Luke!  I failed to set email notifications on this, thought everyone was ignoring me :-/  

So I guess the problem is that this "best practice" is

a) not mentioned explicitly in the docs  (correct me if I'm wrong)
b) a big hassle

We're basically saying "you must always have an explicit wait after any interaction".  So, even for the very simplest of use cases, users have to figure out how to use the webdriverwait api, or roll their own based on time.sleeps and lambda functions.  That's a sucky thing to have to explain to newbie selenium users...  (as I'll shortly reveal, this is a pressing concern of mine)

And it's frustrating because it seems "so easy" (and i know it always seems that way to the ill-informed outsider).  "why can't selenium just tell when I'm doing a normal browser GET or POST, and wait for me?".  OK if I'm doing clever ajax stuff then I'm going to have to be a grown-up and be careful with waits, but if I'm using boring old HTML with no javascript, surely selenium could be made to do some intuitive, implicit waits?

moan moan moan.  On the one hand I'm agitating for what I would see as an improvement to the API.  But I suspect my real psychological motivation here is that I don't want to have to rewrite 17 chapters of my TDD-with-Selenium book...

> email to selenium-developers+unsub...@googlegroups.com.

[Luke Inman-Semerau]

inline

On Thu, Nov 21, 2013 at 8:03 AM, Harry Percival
<harry.p...@gmail.com> wrote:
> Thanks for the reply Luke! I failed to set email notifications on this,
> thought everyone was ignoring me :-/
>
> So I guess the problem is that this "best practice" is
>
> a) not mentioned explicitly in the docs (correct me if I'm wrong)

Maybe not exactly detailing this point, but yes:
http://www.seleniumhq.org/docs/04_webdriver_advanced.jsp#explicit-and-implicit-waits


> b) a big hassle

using the expected conditions class (and even writing your own) isn't
very complex and I prefer doing that and training newbies on your team
isn't very hard (and they'll learn quickly as their tests start
failing ;) )

>
> We're basically saying "you must always have an explicit wait after any
> interaction". So, even for the very simplest of use cases, users have to
> figure out how to use the webdriverwait api, or roll their own based on
> time.sleeps and lambda functions. That's a sucky thing to have to explain
> to newbie selenium users... (as I'll shortly reveal, this is a pressing
> concern of mine)
>

I would highly recommend they learn how to use WebDriverWait.
time.sleep and lambda is not necessary (in fact try to not tell them
about it until they absolutely need them).

> And it's frustrating because it seems "so easy" (and i know it always seems
> that way to the ill-informed outsider). "why can't selenium just tell when
> I'm doing a normal browser GET or POST, and wait for me?". OK if I'm doing
> clever ajax stuff then I'm going to have to be a grown-up and be careful
> with waits, but if I'm using boring old HTML with no javascript, surely
> selenium could be made to do some intuitive, implicit waits?
>

If you have a reproducible sample of where a "simple" GET or POST does
not wait for the form to do it's thing and load the resulting page.
Please log a bug with the test case to reproduce it. We certainly try
to handle these cases in the code, but what one user thinks is
"simple" often ends up more complex with javascript. And no the
selenium project does not have every possible use case of html+JS
handled in our test cases :) (also in your case you're using a
driver implementation I would very much call in *beta* and there's not
very many people getting their hands dirty contributing to it and it
is not in the 'selenium' project)

> moan moan moan. On the one hand I'm agitating for what I would see as an
> improvement to the API. But I suspect my real psychological motivation here
> is that I don't want to have to rewrite 17 chapters of my TDD-with-Selenium
> book...
>

hmm, well, umm... if you wrote a book about using Selenium please do
mention 'waits' as appropriate. With modern websites being highly
complex, one should not assume selenium will be able to detect when
*your* page has finished loading widget X on the screen.

>> > email to selenium-develo...@googlegroups.com.

>> > To view this discussion on the web visit
>> >
>> > https://groups.google.com/d/msgid/selenium-developers/b06ce961-31c3-4bef-ae0c-2e712e356bcc%40googlegroups.com.
>> > For more options, visit https://groups.google.com/groups/opt_out.
>

> --
> You received this message because you are subscribed to the Google Groups
> "Selenium Developers" group.
> To unsubscribe from this group and stop receiving emails from it, send an

> email to selenium-develo...@googlegroups.com.

> To view this discussion on the web visit

> https://groups.google.com/d/msgid/selenium-developers/900172e2-44c5-421c-babc-e26e0ec816ca%40googlegroups.com.

[Harry Percival]

I see what you're saying... Still, I feel like, yes, "the modern web" may be complex, but I believe there's a lot of people out there building simple, static-HTML-only sites which use boring old GET and POSTs, with no clever Ajax or things like that.  And for them, implicit waits that worked would be great.

I think there's quite a neat distinction you could draw here:

* If you build a simple site with no javascript, implicit waits are fine

* If you use javascript of any kind, you must use explicit waits.

Now I might be kidding myself about whether it's possible to make implicit waits really work.  I've been exchanging tweets on the subject with David Burns, and he's given me some hints at the fact that the whole thing may be unrealistic (CSS as a source of dynamic behaviour?)

But I still feel like it *should* be possible for selenium to tell the difference between "normal" GET and POSTs generated by normal user clicks on hyperlinks and forms, and for it to always do an implicit wait after one of those.  I appreciated that Javascript-initiated behaviour is much harder to handle, but I would put that out of scope.  So I'd like to know if this really is an impossible dream?  It seems like it would be such a good solution!  Best of both worlds!  Intuitive in simple cases, complex and flexible when it's needed!   Surely?  Surely?  (and a pony!)

You suggest a bug report, I did actually make one, cf my first post on the subject. It is here:
http://code.google.com/p/selenium/issues/detail?id=6497

If I understand you correctly, maybe it should be logged against Phantom?  Ie, phantom really should be waiting at that point, but it isn't, so they're not implementing correctly against what the webdriver "spec"?   What is the spec?  Where, if anywhere, should I expect implicit waits to work?

Re: documentation, I take back what I said.  You guys totally do mention waits, right there in just about the very first example:  http://docs.seleniumhq.org/docs/03_webdriver.jsp#introducing-webdriver

# type in the search
inputElement.send_keys("cheese!")

# submit the form (although google automatically searches now without submitting)
inputElement.submit()

# the page is ajaxy so the title is originally this:
print driver.title

try:
    # we have to wait for the page to refresh, the last thing that seems to be updated is the title
    WebDriverWait(driver, 10).until(EC.title_contains("cheese!"))

    # You should see "cheese! - Google Search"
    print driver.title

So, am I really in the situation where I have to tell people to use waits, even for a totally simple, bare bones, no-javascript no-css, 1 form 1 field 1 GET request 1 POST request site?  :-(

I do want to make sure I talk about the wait pattern... I just hoped I could do it a little later on in the book...

You received this message because you are subscribed to a topic in the Google Groups "Selenium Developers" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/selenium-developers/hA_jTx4vrDM/unsubscribe.
To unsubscribe from this group and all its topics, send an email to selenium-develo...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/selenium-developers/CAL97Zu7zMwNSWp_p4T3odcKMCSYjm%3D-6oUrpt9RUH1WsU%2BA_Gg%40mail.gmail.com.

For more options, visit https://groups.google.com/groups/opt_out.

--
------------------------------
Harry J.W. Percival
------------------------------
Twitter: @hjwp
Mobile:  +44 (0) 78877 02511
Skype:         harry.percival

[Luke Inman-Semerau]

No, you shouldn't have to use waits for 'normal' html pages. Your
issue stems from phantomjs / ghostdriver so you should be logging a
bug there:
https://github.com/detro/ghostdriver

Remember that the Selenium project is now (mostly) in the business of
implementing the Client side of the JSON Wire Protocol (w3c spec in
working draft). Implementations of it (which is where the implicit
waits problem you are describing would come to play) will be in the
Browser side (consider it 'server' side). Chrome has chromedriver,
PhantomJS has ghostdriver, Android / iOS has Selendroid, ios-driver,
appium. Ones that we still control are IE, Safari and Firefox (but
Mozilla is implementing theirs with 'marionette' which will replace
our implementation hopefully with FF31).

So, for the most part if you have a reproducible issue with a simple
html page with IE, Safari, Firefox. Then log it with the Selenium
project. Otherwise you need to log it with the respective implementors
(all should be linked from http://docs.seleniumhq.org/download/ )

-Luke

On Fri, Nov 22, 2013 at 9:39 AM, Harry Percival

> https://groups.google.com/d/msgid/selenium-developers/CACFvh98%2BFZCKd6odTdctYm9a_9_0AbqT-henuh-xyNBJkkoTMQ%40mail.gmail.com.

[Jim Evans]

I understand that you feel like it should be possible to make "intuitive" waits work, but the fact of the matter is that it's really not that simple. When you use so-called "native events", you're using OS-level mechanisms to shove input into the browser. These are great at guaranteeing that the browser fires all of the correct events when a click is encountered, and it's an important part of the WebDriver methodology. But they can also make what you call "detecting simple GET and POST" operations more challenging.

I'm going to talk about IE and Windows, because I'm intimately familiar with the architecture there. When you're using native events with IE, that means you're using one of two Windows APIs to simulate user input, SendMessage or SendInput. I'll discuss the SendMessage case here, but the issues are similar in both instances. When the user requests a click, the IE driver sends a message to the IE window being automated. The SendMessage API will wait until the message is successfully posted to the target window's message queue, but there's no guarantee that the message will be processed before the API call returns. Nevertheless, the operation of sending the message to the window was successfully completed.

Incidentally, this is why I get aggravated when users say, "Why doesn't the click() method throw an exception when the click fails?" It's because from the point of view of the driver, the click succeeded just fine. The message was successfully sent to the target window. Not every click will trigger a navigation, so always waiting for a navigation to begin after a click would negatively impact performance for those cases.

The second issue is a little more subtle. There is a list of conditions that the IE driver uses to determine whether a navigation has started and whether one has completed. Since native events are, by definition, asynchronous, what happens if those conditions are all checked before the browser has the chance to process the click and begin navigation? It's a classic race condition. IE does not expose events that would allow the driver to completely synchronize with the click. The best we can hope for is a "best effort" in attempting to detect the navigation and waiting for it to complete. This is, in fact, what the IE driver does, but there's still a race condition, and the driver sometimes loses.

So now that you have a better insight into the architecture of the IE driver and browser, I'm going to turn the tables a little. How would you implement "intuitive" waits in IE, knowing what you know now? I'm not trying to be snarky; I'm genuinely interested. I've been pretty close to the IE code for a good while now, and it's possible I'm just too close to it to see any other approaches, but I assure you, the approach we use wasn't undertaken without a good deal of research and experience. If you have another approach that would do what you want without negatively impacting the other cases, feel free to send me a pull request.

--Jim

>> > email to selenium-developers+unsub...@googlegroups.com.

>> > To view this discussion on the web visit
>> >
>> > https://groups.google.com/d/msgid/selenium-developers/b06ce961-31c3-4bef-ae0c-2e712e356bcc%40googlegroups.com.
>> > For more options, visit https://groups.google.com/groups/opt_out.
>
> --
> You received this message because you are subscribed to the Google Groups
> "Selenium Developers" group.
> To unsubscribe from this group and stop receiving emails from it, send an

> email to selenium-developers+unsub...@googlegroups.com.

> To view this discussion on the web visit

> https://groups.google.com/d/msgid/selenium-developers/900172e2-44c5-421c-babc-e26e0ec816ca%40googlegroups.com.
>
> For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to a topic in the Google Groups "Selenium Developers" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/selenium-developers/hA_jTx4vrDM/unsubscribe.

To unsubscribe from this group and all its topics, send an email to selenium-developers+unsub...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/selenium-developers/CAL97Zu7zMwNSWp_p4T3odcKMCSYjm%3D-6oUrpt9RUH1WsU%2BA_Gg%40mail.gmail.com.

For more options, visit https://groups.google.com/groups/opt_out.

[Alexei Barantsev]

We're basically saying "you must always have an explicit wait after any interaction". 

You never should do so. You should wait before an interaction or a check, not after one. That's what users do. They don't wait until "a page is completely loaded" (whatever this mean), they don't wait until "an action caused by a click is finished". A user waits until "a button I want to click is clickable" and immediately click it.

Regards,

-- 

Alexei Barantsev

Software-Testing.Ru

Selenium2.Ru

[Harry Percival]

@ Jim, Luke, thanks for clarifying that. I think I had a mental model whereby you guys had your hooks much deeper into the native browser -- I see that's not the case, or at least not the case for IE.  I can see how it's much harder if you don't have any visibility of what's going on inside the browser -- has it received the click or not?  Is it "loading stuff" or not?

Actually, that's a question -- do you have any way of asking the browser whether it's currently "doing something"?  loading a page, processing some javascript? Does that last even make sense?

@ Alexei - yes, I can see why you would wait before an interaction.  But I'm also concerned with waiting after -- in automated tests, the pattern I tend to follow is "do something, then check its effect".  In the particular case, I enter some text into an input form, hit enter (to submit the form), and I want to see the effect, which is that the page should refresh, and should now contain some updated text.  I'm expecting that implicit wait will take care of "waiting until the submit completes and the page refreshes and is completely loaded".

So that's one case where I thought implicit waits should work. All, is there a spec somewhere of when they're supposed to work?

If it's all a bit vague, maybe the simplest thing, as David says, really would be to just throw them out altogether!

More chats on twitter: https://twitter.com/shs96c/status/404178266178990080

--
You received this message because you are subscribed to a topic in the Google Groups "Selenium Developers" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/selenium-developers/hA_jTx4vrDM/unsubscribe.
To unsubscribe from this group and all its topics, send an email to selenium-develo...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/selenium-developers/1ee36d2f-b722-471c-a063-d5de7972f953%40googlegroups.com.

For more options, visit https://groups.google.com/groups/opt_out.

[Jim Evans]

There's not really any standardized way to ask those questions of just any browser. You might get some mileage out of checking document.readyState, but that doesn't give you full fidelity of what you want.

Just as a point of information, the IE wait algorithm is described in pseudocode in a comment to issue 232 in the tracker[1]. The actual implementation can be found in Browser.cpp in the project source code in the Browser::Wait method[2].

[1] https://code.google.com/p/selenium/issues/detail?id=232#c11
[2] https://code.google.com/p/selenium/source/browse/cpp/iedriver/Browser.cpp#401

[Simon Stewart]

On Fri, Nov 22, 2013 at 7:10 PM, Alexei Barantsev <bara...@gmail.com> wrote:

We're basically saying "you must always have an explicit wait after any interaction". 

You never should do so. You should wait before an interaction or a check, not after one. That's what users do. They don't wait until "a page is completely loaded" (whatever this mean), they don't wait until "an action caused by a click is finished". A user waits until "a button I want to click is clickable" and immediately click it.

Better yet, there are certain interactions that you expect to mutate the DOM via JS or cause a page to reload, and there are those that you don't expect to cause those changes. Examples of the former include clicking on links, or perhaps executing some password validation using JS on a form. Examples of the latter include simply filling out forms and examining text content.

For those cases where you expect the DOM to change, the sensible thing to do is to wait for the element you want to interact with _after_ the mutation has completed. In that case, an explicit wait really spells out what you're doing. As a test author, my expectation is that you're aware of how the AUT works and updates, and this doesn't seem like an overly onerous thing to do.

In the case where the DOM doesn't change, it's senseless to use a wait of any kind, as you want failure cases to be obvious and fast. That's why I favour cranking implicit waits down to zero and using explicit waits. I'm also fully aware that my expectations of how familiar people are with the AUT can be grossly inaccurate, and (sadly) in IME, implicit waits help smooth over some of the rough edges.

The best bit of advice about using implicit waits? Never, ever, do a test for the _lack_ of presence. Always frame expectations positively. Otherwise, you'll be in a whole world of pain as you mix-and-match implicit and explicit waits.

Cheers,

Simon

[Harry Percival]

So, Simon, in your opinion, implicit waits are to be avoided, and we should advise people to *always* use an explicit wait after any interaction that will cause a page refresh or dom update -- whether javascript is involved or not?

If you're hinting that implicit waits will be deprecated in the next few months, or even the next year, then do let me know, and I'll stop telling people to use them, and get cracking on re-writing all the relevant bits in my book...  I really would rather not though -- i'm keen to introduce waits as  "best practice" in the second half of the book, but i feel like they would be a rather unfortunate increase in the learning curve for the early chapters...  especially since things seem to more or less "just work" as it is (as long as we stick to using FF and not phantom)...

--
You received this message because you are subscribed to a topic in the Google Groups "Selenium Developers" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/selenium-developers/hA_jTx4vrDM/unsubscribe.
To unsubscribe from this group and all its topics, send an email to selenium-develo...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/selenium-developers/CAOrAhYHiAt8an1RAUxT2KP356Y8GbF%3D4k53%2B%2BgsByMH9GZBHiA%40mail.gmail.com.

For more options, visit https://groups.google.com/groups/opt_out.

[Simon Stewart]

My personal style is to always use explicit waits, and it's a style that I recommend others follow. If I were you, and advising people on how to write their tests in an idiomatic way, I'd also be advocating this approach.

However, as I said before, I know that not every user is aware of how their AUT works (or just doesn't care, or is using a library that abstracts the horror away, or....) In those cases, implicit waits are a perfectly workable solution, with the caveat that if you're mixing implicit and explicit waits using any ExpectedCondition that checks either for the lack of presence or lack visibility of an element then You're Not Going To Have A Great Time.

To be clear, there's no intention of removing implicit waits.

Simon

You received this message because you are subscribed to the Google Groups "Selenium Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to selenium-develo...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/selenium-developers/CACFvh9_MiH5y6gu6%3DQyqTPeXHfJ%3DGSdwdkTqZXhViczc9NGQEQ%40mail.gmail.com.

[Harry Percival]

> if you're mixing implicit and explicit waits using any ExpectedCondition that checks either for the lack of presence or lack visibility of an element then You're Not Going To Have A Great Time.

What do you mean by this?  do the implicit waits throw off the timing on the explicit waits somehow?  ie, if i say "wait for something to disappear within 10 seconds", and it actually disappears within 1 second, but i have an implicit wait set at 5 seconds, then it will block for 5 seconds?

To view this discussion on the web visit https://groups.google.com/d/msgid/selenium-developers/CAOrAhYEC3QjwWiX2qe8hUkR-Jbcez37KZq5EiTgewXiqb5NOUA%40mail.gmail.com.

For more options, visit https://groups.google.com/groups/opt_out.

[Simon Stewart]

On Mon, Nov 25, 2013 at 12:03 PM, Harry Percival <harry.p...@gmail.com> wrote:

> if you're mixing implicit and explicit waits using any ExpectedCondition that checks either for the lack of presence or lack visibility of an element then You're Not Going To Have A Great Time.

What do you mean by this?  do the implicit waits throw off the timing on the explicit waits somehow?  ie, if i say "wait for something to disappear within 10 seconds", and it actually disappears within 1 second, but i have an implicit wait set at 5 seconds, then it will block for 5 seconds?

This goes to the heart of why people don't like implicit waits. Let's strip it back to basics and build up from there.

An explicit wait (typically) uses an implementation of the Wait interface (normally WebDriverWaits), and takes an operation to determine whether some condition has been met.

An implicit wait could be modeled as an explicit wait where the operation is either "wait until the element is on the DOM" (for passive operations, such as reading an element attribute) or "wait until the element is visible" if an interaction (such as "click" or "sendKeys") is about to occur. However, the implicit wait doesn't require the test author to put in the explicit wait. It's as if every lookup and operation is implicitly wrapped with a Wait (which explains the name :) 

So, if you're using an ExpectedCondition that checks for the opposite of either of the conditions that implicit waits are handling, you'll cause your tests to take longer than they should, as you'll need the implicit timeout to occur before the explicit checks can be made. That means that if you set your implicit timeout to a value greater than the explicit timeout, you tests will appear to hang, since the implicit timeout is wrapped around _every call_.

The way to avoid that being problematic, if you're mixing and matching, is to never use an ExpectedCondition that doesn't also meet the requirements placed by the implicit waits. 

Simon

[Harry Percival]

It occurs to me that selenium, if it detects it's doing an explicit wait, could maybe switch off implicit waiting temporarily.  is that a silly idea?

--

You received this message because you are subscribed to a topic in the Google Groups "Selenium Developers" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/selenium-developers/hA_jTx4vrDM/unsubscribe.
To unsubscribe from this group and all its topics, send an email to selenium-develo...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/selenium-developers/CAOrAhYGf4EV7mhWduDcN%3DVtQ75z-t11e1vNwo0iVprLC_4YNQw%40mail.gmail.com.

For more options, visit https://groups.google.com/groups/opt_out.

[Simon Stewart]

How would you do this detection? The APIs are layered the opposite way you'd want for that to be simple to implement.

Simon

--

You received this message because you are subscribed to the Google Groups "Selenium Developers" group.

To unsubscribe from this group and stop receiving emails from it, send an email to selenium-develo...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/selenium-developers/CACFvh98CeC0QB%3Dx8v53Yvp5_m2TQ-Z6-w%2B0OFfF%3Dvy_0WLuS%3DA%40mail.gmail.com.

[Harry Percival]

I'm looking at the Python version of the API. Could you do it maybe by wrapping the `until` and `until_not` methods with something to un-set and then re-set implicity wait?

http://code.google.com/p/selenium/source/browse/py/selenium/webdriver/support/wait.py

I mean, I'm not sure this would work with nested try/excepts and all, but something a bit like this:

    def until(self, method, message=''):
        """Calls the method provided with the driver as an argument until the \
        return value is not False."""

        self.implicitly_wait(0)

        try:

        end_time = time.time() + self._timeout
        while(True):
            try:
                value = method(self._driver)
                if value:
                    return value
            except self._ignored_exceptions:
                pass
            time.sleep(self._poll)
            if(time.time() > end_time):
                break
        raise TimeoutException(message)

finally:

    self.implictly_wait(wait_from_before)

To view this discussion on the web visit https://groups.google.com/d/msgid/selenium-developers/CAOrAhYG4tYxkwubD8Ut8ZtMuw-D_VYY7JQOMkt%3DNqBgwtgTUyg%40mail.gmail.com.

For more options, visit https://groups.google.com/groups/opt_out.

[David Burns]

How is wait_from_before populated. The client bindings(local end) don't keep this anywhere and goes against their principles of being a light weight wrapper speaking to the remote end where the information is kept.

I personally would be -1 on adding getSearchTimeout as that will open the API up for abuse from people not writing deterministic code, something we suggest people do regularly.

David

David Burns
Email: david...@theautomatedtester.co.uk
URL: http://www.theautomatedtester.co.uk/

To view this discussion on the web visit https://groups.google.com/d/msgid/selenium-developers/CACFvh998vRVH%3DZa6ZqWyU_ypp11M2F%2BCXQN%2BZk4xaE0v8DRQeg%40mail.gmail.com.

[Harry Percival]

OK.  I'm not sure I understand that, but I shan't waste any more of your time.  If someone is really keen to work around the implicit/explicit timing clashes, then they can always write their own implict-switcher-offer by manually stashing away the before value...

Thanks very much everyone, for your patient and thoughtful responses...

In the meantime, I think I'm going to stick with telling people to use implicit at the beginning of the book, and maybe emphasize a little more strongly that explicit waits are best practice somewhere later in the book.  Here's where I currently explain it, maybe I'll make a bigger deal out of it at that point...

To view this discussion on the web visit https://groups.google.com/d/msgid/selenium-developers/CAPmg_ctDXMH4cPS7QWSW_X3D%3DMt_yz3cdVDPi0YyRLz3XWED8g%40mail.gmail.com.

For more options, visit https://groups.google.com/groups/opt_out.

[Simon Stewart]

If you're writing a book, the official party line is that (to steal a phrase from the Agile manifesto): "Prefer explicit waits over implicit waits. That is, although we see value in the things on the right, we strongly prefer the things on the left" :)

Put another, I'd only mention explicit waits, but have a sidenote talking a little about implicit waits. Otherwise, you're setting readers up for hard to understand behaviours.

Simon

To view this discussion on the web visit https://groups.google.com/d/msgid/selenium-developers/CACFvh9-8QFzSORqceu9b4cs9fmkCHo06BpN0eossN%3DzfAHdzzA%40mail.gmail.com.

[Harry Percival]

My problem is that this is a book for beginners.  Compare the learning curve on teaching this:

browser.find_element_by_name('new_item').send_keys('a new item for the list')
browser.find_element_by_name('new_item').send_keys('\n')

items_table = browser.find_element_by_id('id_list_table')

assert 'a new item for the list' in items_table.text

vs

from selenium.webdriver.common.by import By

from selenium.webdriver.support.ui import WebDriverWait

from selenium.webdriver.support import expected_conditions as EC

browser.find_element_by_name('new_item').send_keys('a new item for the list')
browser.find_element_by_name('new_item').send_keys('\n')

         WebDriverWait(self.browser, timeout=20).until(
             EC.text_to_be_present_in_element(

                (By.ID, "id_list_table"),

      "a new item for the list"
    )

)

I mean, it's not the end of the world, but the first requires understanding the "find_elements_by_xyz" concept, and the second requires 3 extra imports, the concept of a locator, the concept of a wait, the concept of a condition function... I'd rather leave all that stuff until a bit later, when readers are a bit more comfortable with the whole selenium thing...

To view this discussion on the web visit https://groups.google.com/d/msgid/selenium-developers/CAOrAhYGYvJcL2B6n57p02hEmwn5%2BR%2BBuZLKYopd4d5J1zZemgQ%40mail.gmail.com.

For more options, visit https://groups.google.com/groups/opt_out.

[Simon Stewart]

I agree that it's a bit more code, but the beginner walks away with a firmer understanding of how things work in the end. Perhaps you could start with implicit waits but preface with some comment that this is just to lower the learning curve, and it isn't recommended "best practice"?

Simon

To view this discussion on the web visit https://groups.google.com/d/msgid/selenium-developers/CACFvh99_mhBr-W1F46moBBBnLdcopsib6i6e9-AhfmRRA27JfQ%40mail.gmail.com.

[Alexei Barantsev]

If you only want to check for presense of elements you're safe (and fast) with implicit waits.

But as soon as you want to wait for more complex conditions you're ready to get rid of impicit waits and be more explicit.

And it is not too much of code there. For example in Java it is

driver.findElement(By.id("test")).click();

vs

wait.until(presenseOfElementLocated(By.id("test"))).click();

or even better

wait.until(visibilityOfElementLocated(By.id("test"))).click();

(where 'wait' variable is initialized somewhere in setup code)

[Bill Ross]

Harry Percival wrote (as he stepped into the cannibals' pot):

> I'm expecting that implicit wait
> will take care of "waiting until the submit completes and the page
> refreshes and is completely loaded".

Sorry for the late, unhelpful response, but welcome to the hell of
unsatisfied expectations. Many have gone before on this one, I have
seen interesting attempts to do something at the test level, but if
javascript is involved, you may need to have the page developer set
a variable you can inspect.

Bill

Harry Percival <harry.p...@gmail.com> wrote:

> @ Jim, Luke, thanks for clarifying that. I think I had a mental model
> whereby you guys had your hooks much deeper into the native browser -- I
> see that's not the case, or at least not the case for IE. I can see how
> it's much harder if you don't have any visibility of what's going on inside
> the browser -- has it received the click or not? Is it "loading stuff" or
> not?
>
> Actually, that's a question -- do you have any way of asking the browser
> whether it's currently "doing something"? loading a page, processing some
> javascript? Does that last even make sense?
>
> @ Alexei - yes, I can see why you would wait before an interaction. But
> I'm also concerned with waiting after -- in automated tests, the pattern I
> tend to follow is "do something, then check its effect". In the particular
> case, I enter some text into an input form, hit enter (to submit the form),
> and I want to see the effect, which is that the page should refresh, and
> should now contain some updated text. I'm expecting that implicit wait
> will take care of "waiting until the submit completes and the page
> refreshes and is completely loaded".
>
> So that's one case where I thought implicit waits should work. All, is
> there a spec somewhere of when they're supposed to work?
>
> If it's all a bit vague, maybe the simplest thing, as David says, really
> would be to just throw them out altogether!
>
> More chats on twitter: https://twitter.com/shs96c/status/404178266178990080
>
>
>
>
> On 22 November 2013 19:10, Alexei Barantsev <bara...@gmail.com> wrote:
>
> >

> >> We're basically saying "you must always have an explicit wait after *any
> >> *interaction".
> >>
> >
> > You never should do so. You should wait *before* an interaction or a
> > check, not *after* one. That's what users do. They don't wait until "a

> You received this message because you are subscribed to the Google Groups "Selenium Developers" group.

> To unsubscribe from this group and stop receiving emails from it, send an email to selenium-develo...@googlegroups.com.
> To view this discussion on the web visit https://groups.google.com/d/msgid/selenium-developers/CACFvh98VfUZd%3DP66Khbcj5ETxN_B-YaNjj8-_zZsbQOMF1TZ-g%40mail.gmail.com.

[Harry Percival]

Hi everyone,

dragging up an old thread I know, but I was trying to explain to someone about the fact that selenium doesn't wait on click for pages to load, since it can't tell a page-load-inducing click from a non-page-loading one, as per our discussion.   person i was talking to referred me to the docs, which seem to say otherwise:

https://selenium.googlecode.com/git/docs/api/java/org/openqa/selenium/WebElement.html#click%28%29

Do I take it the docs are misleading, and need updating?

PS - my recent post on the subject: http://www.obeythetestinggoat.com/how-to-get-selenium-to-wait-for-page-load-after-a-click.html

> > selenium-developers+unsub...@googlegroups.com.

> > To view this discussion on the web visit
> > https://groups.google.com/d/msgid/selenium-developers/1ee36d2f-b722-471c-a063-d5de7972f953%40googlegroups.com
> > .
> >
> > For more options, visit https://groups.google.com/groups/opt_out.
> >
>
>
>
> --
> ------------------------------
> Harry J.W. Percival
> ------------------------------
> Twitter: @hjwp
> Mobile:  +44 (0) 78877 02511
> Skype:         harry.percival
>
> --
> You received this message because you are subscribed to the Google Groups "Selenium Developers" group.

> To unsubscribe from this group and stop receiving emails from it, send an email to selenium-developers+unsub...@googlegroups.com.

[Luke Inman-Semerau]

I think you didn't read the whole thing there?

" If click() causes a new page to be loaded via an event or is done by
sending a native event then the method will *not* wait for it to be
loaded and the caller should verify that a new page has been loaded."

>> > > selenium-develo...@googlegroups.com.

>> > > To view this discussion on the web visit
>> > >
>> > > https://groups.google.com/d/msgid/selenium-developers/1ee36d2f-b722-471c-a063-d5de7972f953%40googlegroups.com
>> > > .
>> > >
>> > > For more options, visit https://groups.google.com/groups/opt_out.
>> > >
>> >
>> >
>> >
>> > --
>> > ------------------------------
>> > Harry J.W. Percival
>> > ------------------------------
>> > Twitter: @hjwp
>> > Mobile: +44 (0) 78877 02511
>> > Skype: harry.percival
>> >
>> > --
>> > You received this message because you are subscribed to the Google
>> > Groups "Selenium Developers" group.
>> > To unsubscribe from this group and stop receiving emails from it, send

>> > an email to selenium-develo...@googlegroups.com.

>> > To view this discussion on the web visit
>> > https://groups.google.com/d/msgid/selenium-developers/CACFvh98VfUZd%3DP66Khbcj5ETxN_B-YaNjj8-_zZsbQOMF1TZ-g%40mail.gmail.com.
>> > For more options, visit https://groups.google.com/groups/opt_out.
>

> --
> You received this message because you are subscribed to the Google Groups
> "Selenium Developers" group.
> To unsubscribe from this group and stop receiving emails from it, send an

> email to selenium-develo...@googlegroups.com.

> To view this discussion on the web visit

> https://groups.google.com/d/msgid/selenium-developers/b8042bc5-cd83-4b2d-892c-80b645c323ea%40googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.

[Harry Percival]

wait, what?  I thought the whole point was that click couldn't tell what was going to happen, whether it was going to be a native event or not?

To view this discussion on the web visit https://groups.google.com/d/msgid/selenium-developers/CAL97Zu7HAvzCpaD_yWprVV5LgSRhqWq7he5Fw%2BTY1JdHBCcptw%40mail.gmail.com.

For more options, visit https://groups.google.com/d/optout.

[Harry Percival]

ah, i see, "native event" refers to the way the click is done, not to what happens after it.

so which clicks are done by sending a "native event"?  is that something the user can know in advance?  is it "all of them" in the case of most webdriver implementations?

On Wednesday, 3 September 2014 23:52:45 UTC+1, Harry Percival wrote:

wait, what?  I thought the whole point was that click couldn't tell what was going to happen, whether it was going to be a native event or not?

On 3 September 2014 23:49, Luke Inman-Semerau <> wrote:

I think you didn't read the whole thing there?

" If click() causes a new page to be loaded via an event or is done by
sending a native event then the method will *not* wait for it to be
loaded and the caller should verify that a new page has been loaded."

>> > > selenium-developers+unsub...@googlegroups.com.

>> > > To view this discussion on the web visit
>> > >
>> > > https://groups.google.com/d/msgid/selenium-developers/1ee36d2f-b722-471c-a063-d5de7972f953%40googlegroups.com
>> > > .
>> > >
>> > > For more options, visit https://groups.google.com/groups/opt_out.
>> > >
>> >
>> >
>> >
>> > --
>> > ------------------------------
>> > Harry J.W. Percival
>> > ------------------------------
>> > Twitter: @hjwp
>> > Mobile:  +44 (0) 78877 02511
>> > Skype:         harry.percival
>> >
>> > --
>> > You received this message because you are subscribed to the Google
>> > Groups "Selenium Developers" group.
>> > To unsubscribe from this group and stop receiving emails from it, send

>> > an email to selenium-developers+unsub...@googlegroups.com.

>> > To view this discussion on the web visit
>> > https://groups.google.com/d/msgid/selenium-developers/CACFvh98VfUZd%3DP66Khbcj5ETxN_B-YaNjj8-_zZsbQOMF1TZ-g%40mail.gmail.com.
>> > For more options, visit https://groups.google.com/groups/opt_out.
>
> --
> You received this message because you are subscribed to the Google Groups
> "Selenium Developers" group.
> To unsubscribe from this group and stop receiving emails from it, send an

> email to selenium-developers+unsub...@googlegroups.com.

> To view this discussion on the web visit

> https://groups.google.com/d/msgid/selenium-developers/b8042bc5-cd83-4b2d-892c-80b645c323ea%40googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to a topic in the Google Groups "Selenium Developers" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/selenium-developers/hA_jTx4vrDM/unsubscribe.

To unsubscribe from this group and all its topics, send an email to selenium-developers+unsub...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/selenium-developers/CAL97Zu7HAvzCpaD_yWprVV5LgSRhqWq7he5Fw%2BTY1JdHBCcptw%40mail.gmail.com.

For more options, visit https://groups.google.com/d/optout.

[Harry Percival]

I found this:  https://code.google.com/p/selenium/wiki/AdvancedUserInteractions#Native_events_versus_synthetic_events

So it seems like most browsers on most platforms will default to native events, which means that the docs, while accurate, do feel a bit misleading?  Most readers won't know what "native events" are, so they may not read past that second sentence, "this method will attempt to block until the page has loaded", which it turns out is just not true under most circumstances...

https://selenium.googlecode.com/git/docs/api/java/org/openqa/selenium/WebElement.html#click%28%29

To unsubscribe from this group and all its topics, send an email to selenium-develo...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/selenium-developers/eef8c0d5-2fd7-446a-80d3-49019d480a77%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

[Harry Percival]

Here's a patch, not sure it's my place to submit this sort of thing, but want to show willing to do more than gripe!

[Luke Inman-Semerau]

The javadoc is auto generated from source, here's where you'd really
want to make your patch:

https://github.com/SeleniumHQ/selenium/blob/master/java/client/src/org/openqa/selenium/WebElement.java#L32-L45

(also, we accept pull requests from github or if you want to attach a
patch file please do so in a new issue on the issue tracker)

Thanks!

>>>>> >> > > selenium-develo...@googlegroups.com.
>>>>> >> > > To view this discussion on the web visit
>>>>> >> > >
>>>>> >> > >

>>>>> >> > > https://groups.google.com/d/msgid/selenium-developers/1ee36d2f-b722-471c-a063-d5de7972f953%40googlegroups.com
>>>>> >> > > .
>>>>> >> > >
>>>>> >> > > For more options, visit
>>>>> >> > > https://groups.google.com/groups/opt_out.
>>>>> >> > >
>>>>> >> >
>>>>> >> >
>>>>> >> >
>>>>> >> > --
>>>>> >> > ------------------------------
>>>>> >> > Harry J.W. Percival
>>>>> >> > ------------------------------
>>>>> >> > Twitter: @hjwp
>>>>> >> > Mobile: +44 (0) 78877 02511
>>>>> >> > Skype: harry.percival
>>>>> >> >
>>>>> >> > --
>>>>> >> > You received this message because you are subscribed to the Google
>>>>> >> > Groups "Selenium Developers" group.
>>>>> >> > To unsubscribe from this group and stop receiving emails from it,
>>>>> >> > send

>>>>> >> > an email to selenium-develo...@googlegroups.com.
>>>>> >> > To view this discussion on the web visit
>>>>> >> >

>>>>> >> > https://groups.google.com/d/msgid/selenium-developers/CACFvh98VfUZd%3DP66Khbcj5ETxN_B-YaNjj8-_zZsbQOMF1TZ-g%40mail.gmail.com.
>>>>> >> > For more options, visit https://groups.google.com/groups/opt_out.
>>>>> >
>>>>> > --
>>>>> > You received this message because you are subscribed to the Google
>>>>> > Groups
>>>>> > "Selenium Developers" group.
>>>>> > To unsubscribe from this group and stop receiving emails from it,
>>>>> > send an

>>>>> > email to selenium-develo...@googlegroups.com.
>>>>> > To view this discussion on the web visit
>>>>> >

>>>>> > https://groups.google.com/d/msgid/selenium-developers/b8042bc5-cd83-4b2d-892c-80b645c323ea%40googlegroups.com.
>>>>> > For more options, visit https://groups.google.com/d/optout.
>>>>>
>>>>> --
>>>>> You received this message because you are subscribed to a topic in the
>>>>> Google Groups "Selenium Developers" group.
>>>>> To unsubscribe from this topic, visit
>>>>> https://groups.google.com/d/topic/selenium-developers/hA_jTx4vrDM/unsubscribe.
>>>>> To unsubscribe from this group and all its topics, send an email to

> --
> You received this message because you are subscribed to the Google Groups
> "Selenium Developers" group.
> To unsubscribe from this group and stop receiving emails from it, send an

> email to selenium-develo...@googlegroups.com.
> To view this discussion on the web visit

> https://groups.google.com/d/msgid/selenium-developers/CACFvh9-grd5-OStpeK3g1M4quixc3i5UfQD-qY%3DGYz4A6z7B%3DQ%40mail.gmail.com.

[Harry Percival]

Here you go! 

https://github.com/SeleniumHQ/selenium/pull/253

I signed the agreement thingie, but I don't have java here so I couldn't rebuild the html docs from the javadoc change...

To view this discussion on the web visit https://groups.google.com/d/msgid/selenium-developers/CAL97Zu6LVktrMeGyhgKiN02T6TWtqM6Ej0j4zjUwwtJFC75Zjw%40mail.gmail.com.

For more options, visit https://groups.google.com/d/optout.