How to run faster a CasperJS script

[Jérémy Brunet]

Hello there,

I'm currently executing a CasperJS script to scrap infos from 100K links.

Right now, I've scrapped 3.7K pages in a bit less than 2 hours. At this speed, I have a few hours (or days) ahead of scrapping... 😂

Basically, I'm executing a loop to store every urls in an array and then open each link using thenOpen() before calling 2 custom functions to scrap the infos I'm looking for and store them in a json file.

Is there any tricks to execute such a script faster than I'm doing right now ?

I've tried to write LOC to fill the array with the first link at index 0 then immediately call my functions to scrap. But at the end, this process doesn't work as it first fill the array with the 100K links before scrapping each of them.

Thanks for your help!

[Miguel Poegel]

Hello, you can disable images and plugins, like this should be much faster, if its still slow because of your slow internet connection, you can use cloud system, please check "https://phantomjscloud.com".

Regards

[Ken Soh]

Cool trick Miguel!

Another idea you can try is having separate installations for phantomjs and casperjs. and split your automation into parts to run in separate environments. Provided CPU / memory is sufficient to support.