waitForSelector does not find AJAX-loaded DOM element

[randal...@gmail.com]

Hi everyone,

I'm having a real problem getting casperjs to locate a DOM element that is loaded asynchronously after page load. The element's id is "dl_url", and I'm absolutely sure that the element is loaded via AJAX into the page a few seconds after initial page load.

Note: I am using casperjs version 1.1.0-DEV.

Here is my JS file:

function loadError()
{
    casper.echo("error");
    casper.exit();
}

var casper = require('casper').create({
    pageSettings: {
        loadImages: false,
        userAgent: 'Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1)'
    },
    waitTimeout: 38000,
    timeout: 40000,
    stepTimeout: 38000,
    verbose: true,
    logLevel: "debug",
    onLoadError: loadError
});

casper.start(casper.cli.args[0]);
casper.then(function() {
    this.waitForSelector('#dl_url', function() {
        this.echo('loaded');
        var ua = this.evaluate(function() {
            return document.getElementById('dl_url').getElementsByTagName('a')[0].href;
        });
        this.echo(ua);
    },
    function()
    {
        this.echo('not loaded');
    });
});
casper.run(function() {
    this.exit();
});

And here is the command and CLI output [some details have been changed to protect the innocent ;-)]:

root@ns5000918:~# casperjs --proxy=00.000.000.000:8800 --proxy-type=none /home/xxxxx/public_html/phantom/getDlink2.js http://www.mySite.com
[info] [phantom] Starting...
[info] [phantom] Execution timeout set to 40000ms
[info] [phantom] Running suite: 3 steps
[debug] [phantom] opening url: http://www.mySite.com, HTTP GET
[debug] [phantom] Navigation requested: url=http://www.mySite.com, type=Other, willNavigate=true, isMainFrame=true
[debug] [phantom] url changed to "http://www.mySite.com"
[debug] [phantom] Successfully injected Casper client-side utilities
[debug] [phantom] start page is loaded
[info] [phantom] Step anonymous 3/3 http://www.mySite.com (HTTP 0)
[info] [phantom] Step anonymous 3/3: done in 3133ms.
[info] [phantom] Step _step 4/4 http://www.mySite.com (HTTP 0)
[info] [phantom] Step _step 4/4: done in 3154ms.
[error] [phantom] Script timeout of 40000ms reached, exiting.
Script timeout of 40000ms reached, exiting.

From where I'm standing, it appears that casper.waitForSelector never finds the element with id "dl_url". And I can't for the life of me figure out why. I feel like I've tried everything, and I've poured over the casperjs api docs.

It should be noted that with plain, old phantomjs, I WAS able to find this element in the DOM using a custom waitFor function. It's only since I've switched to casperjs that I've been having this problem.

Can anyone help shed some light on this issue for me? Thanks very much in advance.

RJ

[Daniel Zwicker]

Hi,

have you solved this problem? It seam to me I have the same problem.

greetings

[Ryan Whaley]

I'm running into the same problem. Anybody have any ideas?

[James Bryant]

I ran into this problem trying to scrape a very dynamic set of pages.  I experimented with using casper.reload() before trying to find DOM elements, which worked sometimes, but not every time.  I then mixed in a few casper.wait() calls ... which helped, but those are brittle.  What I've settled on (for now) is injecting jQuery and selecting my elements inside a casper.evaluate().  I'm having pretty good success with this, much more consistent behavior.

Injecting jQuery probably isn't the best answer ... I've only been using Casper for about a month ... but it's solving the problem for me right now.  If anyone has a better solution that doesn't require jumping into an evaluate, I'd love to hear about it!  :)

[Durian Gray]

Has anyone solved this? 

I am running into the same issue.

Clicking a link that opens a new window to a page with jQuery that redirects using: 

 $(location).attr('href', 'http://www.yahoo.com'); 

The error i get: 

ReferenceError: Can't find variable: $

Not sure how to solve this.

[nikhil goyal]

Same issue and can't find an answer ! 

Issue: when some hidden form fields depend on inputs from prior fields to become visible via AJAX, casper just doesn't see the newly visible fields for some reason. I have taken snapshots with long time waits - the hidden fields never become visible those trying waitForSelector etc are doing in vain.it will simply time out as per wait timeout settings. 

Please help!!!! any url worth good advice will be much appreciated!

[Ken Soh]

Probably I will try with latest v1.1.4 casperjs, make an explicit wait using - http://docs.casperjs.org/en/latest/modules/casper.html#wait

then output the html dom to see if the element has appeared - http://docs.casperjs.org/en/latest/modules/casper.html#gethtml

if the element still has not appeared, it could be due to other reasons, such as website preventing access by automated browsers.

[Bear]

Greetings!

It seems that if a web page has dynamically-created DOM elements, then you cannot use casper to perform actions on those elements.

Does anyone have a decent workaround?  Some folks in this email thread mentioned things that worked for them, such as phantomjs with a custom waitFor.  I was wondering what they meant by that... if you're still online, could you possibly share a bit of code that actually works?

Thanks!

[Ken Soh]

For elements which have not appeared yet you can try something like below - 

"Probably I will try with latest v1.1.4 casperjs, make an explicit wait using - http://docs.casperjs.org/en/latest/modules/casper.html#wait

then output the html dom to see if the element has appeared - http://docs.casperjs.org/en/latest/modules/casper.html#gethtml

if the element still has not appeared, it could be due to other reasons, such as website preventing access by automated browsers."

For elements with unique randomly generated identifier IDs, one way is find an 'anchor' XPath element to reference to and then use a relative XPath to that anchor element.

There are other samples here for waiting for element to appear - http://docs.casperjs.org/en/latest/modules/casper.html#waitforselector

--
CasperJS homepage & documentation: http://casperjs.org/
CasperJS @github: https://github.com/n1k0/casperjs
 
You received this message because you are subscribed to the Google Groups "casperjs" group.
Visit this group at http://groups.google.com/group/casperjs?hl=en.
---
You received this message because you are subscribed to a topic in the Google Groups "CasperJS" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/casperjs/8yzM5gQbW-g/unsubscribe.
To unsubscribe from this group and all its topics, send an email to casperjs+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Bear]

Thanks.  I'll shed a little light...

Page is generated by PHP.  Using CasperJS, if you do this.echo(this.getHTML(), 'INFO'); you will get PHP.  You will not get the actual rendered HTML.  In fact, you can "wait" for 10 hours; you're still not going to have your element that you want to click on.

To get the actual rendered HTML, I imagine one would have to get hold of the HTML DOM "Document" (like you would do in, for example, Chrome's developer console.  Type "document", and you have it.)

But... how to do that with CasperJS?  Let's say I'm able to get that document.  Then what?  I probably can't use it to click one of those elements that CasperJS didn't "get" in the first place! 

That's what I mean.  I didn't think I was having a problem separate from everybody else.  And one person said they were able to "click" such an element using PhantomJS.

Maybe I'll try my luck going the PhantomJS route.

To unsubscribe from this group and all its topics, send an email to casperjs+u...@googlegroups.com.