Crawl URLs with client side scripting

[pa...@sparklinesolutions.com]

Here is a simplified version of what  have running:

var casper = require("casper").create({
    verbose: true,
    logLevel: 'error',
    ignoreSslErrors: true,
    loadPlugins: false,
    pageSettings: {
        userAgent: "Mozilla/5.0 (Windows NT 6.3; WOW64; rv:44.0) Gecko/20100101 Firefox/44.0"
    }
});
var fs = require('fs');
var url = 'https://www.reebonz.com/'
casper.start(url, function() {
    var html = this.getHTML('html', true);
    fs.write('dump.html', html, 'w');
});
casper.run();

CasperJS doesn't seem to redirect to the correct site as it only loads the javascript and ends after receiving the code 200

Any idea to force casperjs to accept the redirect?

[bruce]

Hey Paolo.

Saw your post and wanted to let you know (if you didn't already). It
appears that phantomjs/casperjs are no longer being actively
developed.

It also appears that chrome headless from google runs and in a number
of cases runs better.

As far as I can tell, running the python+selenium+chrome webdriver
stack gets you most of where you want to go.

My reason for posting, if you're going to be doing a bunch of long
term scraping, you might want to bite the bullet and make the change
now instead of later..

> --
> CasperJS homepage & documentation: http://casperjs.org/
> CasperJS @github: https://github.com/n1k0/casperjs
>
> You received this message because you are subscribed to the Google Groups
> "casperjs" group.
> Visit this group at http://groups.google.com/group/casperjs?hl=en.
> ---
> You received this message because you are subscribed to the Google Groups
> "CasperJS" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to casperjs+u...@googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.

[pa...@sparklinesolutions.com]

Thanks man.  Yeah I know I should be migrating, which is indeed in the pipeline.  Just working on a hotfix to keep things working for now