Setting proxy per PhantomJS session

48 views
Skip to first unread message

Remy Gwaramadze

unread,
Dec 13, 2017, 7:46:07 AM12/13/17
to phantomjs
Hello.
My use case is web scraping dynamic content (javascript). I'd like to run PhantomJS as a server and connect to it using Selenium Remote webdriver instances. Every webdriver instance creates a session within PhantomJS process. But even though capabilities contain "proxy" section, it is impossible to set an individual proxy per session.

It is possible to change the proxy for the whole PhantomJS instance: https://stackoverflow.com/a/41032182/1344854

setProxy is calling QNetworkProxy::setApplicationProxy(proxy) which sets it for the whole application:
Application-wide proxying can be enabled for all subsequent socket connections through the use of the QNetworkProxy::setApplicationProxy() function.

Now, is it possible to tweak PhantomJS in such a way that would open a possibility to set proxy per session? API seems to be already there - desired capabilities passed when instantiating webdriver.

As for the Qt framework, it looks like it is possible:
Proxying can be enabled on a per-socket basis or for all network communication in an application. A newly opened socket can be made to use a proxy by calling its QAbstractSocket::setProxy() function before it is connected.

I believe I could find a time to work on this improvement. For now I need advice and some guidance.

Related issue:
https://groups.google.com/forum/#!searchin/phantomjs/proxy|sort:date/phantomjs/ejDUqz2VYwQ/HvMtG9R3BwAJ

Extra question is about that comment:
Running phantomjs as webserver, will introduce memory leak. And you must monitor it and restart the process if the process hit certain limit.
I will not use phantomjs as webserver just for achieving better performance, as this will lead to a complicated memory  management.

What exactly is leaking? Was it fixed by now?

Thanks,
Remy
Reply all
Reply to author
Forward
0 new messages