waitForSelector unexpectedly failing

[Robert Poor]

I've been doing a lot of page scraping of late, and I thought I'd gotten pretty good at it (except for AJAX intensive pages, but I'll save that for another post.)  But I came across the following:

    casper.start(LOGIN_URL, function a01() {

        downloader.setStatus("contacting web site");

    });

    casper.then(function a02() {

        casper.waitForSelector('form#Enrollmentmyaccountlogin',

                               function() { downloader.setStatus('found login form'); },

                               function() { downloader.setStatus('failed to find login form');

                                            downloader.dumpHTML('/tmp/login-a02-err.html');

                                            casper.exit();

                                          },

                               15000); // fifteen second timeout

    });

This consistently takes the "failed to find login form" path.

What's odd about this is that, running under a web proxy, I can see the page get loaded and <form id="Enrollmentmyaccountlogin" ...> is clearly present.  As far as I can tell, there are no weird redirects or dynamic DOM manipulations going on.  And everything seems to load quickly; I bumped the timeout to fifteen seconds, but I can see the page load long before that.  I also tried 'form[id="Enrollmentmyaccountlogin"]' as the selector, but no joy.

Also note the call to dumpHTML() -- the page that gets dumped also clearly contains <form id="Enrollmentmyaccountlogin" ...> on it.

So why isn't waitForSelector finding the form?

Note:

$ casperjs --version

1.1.0-beta3

$ phantomjs --version

1.9.8

[Robert Poor]

Additional info running with logLevel: 'debug':

$ casperjs --ignore-ssl-errors=true --web-security=no lib/downloaders/downloader.js

[info] [phantom] Starting...

[info] [phantom] Running suite: 10 steps

[debug] [phantom] opening url: http://www.[redacted].com/en/login, HTTP GET

[debug] [phantom] Navigation requested: url=http://www.[redacted].com/en/login, type=Other, willNavigate=true, isMainFrame=true

[debug] [phantom] url changed to "http://www.[redacted].com/en/login"

[debug] [phantom] Navigation requested: url=about:blank, type=Other, willNavigate=true, isMainFrame=false

[debug] [phantom] Navigation requested: url=http://seg.sharethis.com/getSegment.php?purl=http://www.[redacted].com/en/login&jsref=&rnd=1436118064727, type=Other, willNavigate=true, isMainFrame=false

[debug] [phantom] Navigation requested: url=http://edge.sharethis.com/share5x/index.d6500e5f759595106980a9f8436a3906.html, type=Other, willNavigate=true, isMainFrame=false

[debug] [phantom] Successfully injected Casper client-side utilities

[debug] [phantom] Successfully injected Casper client-side utilities

[debug] [phantom] Successfully injected Casper client-side utilities

[debug] [phantom] Successfully injected Casper client-side utilities

[info] [phantom] Step a01 2/10 http://www.[redacted].com/en/login (HTTP 200)

status: contacting company web site

[info] [phantom] Step a01 2/10: done in 2130ms.

[debug] [phantom] Successfully injected Casper client-side utilities

[debug] [phantom] Successfully injected Casper client-side utilities

[info] [phantom] Step a02 3/10 http://www.[redacted].com/en/login (HTTP 200)

[info] [phantom] Step a02 3/10: done in 2153ms.

[debug] [phantom] Successfully injected Casper client-side utilities

[debug] [phantom] Successfully injected Casper client-side utilities

[info] [phantom] Step _step 4/11 http://www.[redacted].com/en/login (HTTP 200)

[info] [phantom] Step _step 4/11: done in 2176ms.

[debug] [phantom] Successfully injected Casper client-side utilities

[debug] [phantom] Successfully injected Casper client-side utilities

<repeated 686 times until...>

[debug] [phantom] Successfully injected Casper client-side utilities

[warning] [phantom] Casper.waitFor() timeout

status: failed to find login form

$

It's odd that it repeated the "injected Casper client-side utilities so many times.  Is that normal behavior, or is that a hint about what's going wrong in this case?

[Gabe Misura]

I don't think you want form#Enrollmentmyaccountlogin, you jus twant #Enrollmentmyaccountlogin (if that's the id).

I also don't think you need casper.then(), you can just 

casper.waitForSelector('#Enrollmentmyaccountlogin', function a02() { 

 // code here

});