python PhantomJS doesn't load interactive dom (instagram pages)
236 views
Skip to first unread message
Max Pierini
Sep 12, 2016, 6:18:03 AM9/12/16
to phantomjs
Hi :D
I'm using PhantomJS 2.1.1 in python scripts under Ubuntu Server 16.04.1
I start a Display from pyvirtualdisplay, visible=0,size=(800,600).
I'm not able to understand why it doesn't correctly load instagram interactive dom pages. The code of the page should be within but it remains empty.
<span id="react-root"></span>
It correctly loads other websites (twitter, tumblr etc).
With PhatomJS 2.1.1 in python on MacOS terminal, it correctly loads instagram pages too, so I guess there's some missing module in Ubuntu Server, but can't understand which one.
Could someone help?
Thank you
:D
This is the incomplete instagram page loaded in Ubuntu Server:
<!DOCTYPE html><!--[if lt IE 7]> <html lang="en" class="no-js lt-ie9 lt-ie8 lt-ie7 not-logged-in "> <![endif]--><!--[if IE 7]> <html lang="en" class="no-js lt-ie9 lt-ie8 not-logged-in "> <![endif]--><!--[if IE 8]> <html lang="en" class="no-js lt-ie9 not-logged-in "> <![endif]--><!--[if gt IE 8]><!--><html lang="en" class="js not-logged-in "><!--<![endif]--><head>
<script async="" src="//connect.facebook.net/en_US/fbevents.js"></script><script type="text/javascript">window._timings = {"domLoading": Date.now()}</script>
<meta charset="utf-8">
<meta http-equiv="X-UA-Compatible" content="IE=edge">
<title>
Instagram
</title>
<script type="text/javascript">
WebFontConfig = {
custom: {
families: ['proxima-nova:n3,n4,n6,n7'],
}
};
</script>
<script src="//instagramstatic-a.akamaihd.net/h1/scripts/webfont.js/c0456c81549b.js" type="text/javascript" async=""></script>
<style type="text/css">
/* @license
* MyFonts Webfont Build ID 2164953, 2012-03-23T23:06:30-0400
*
* The fonts listed in this notice are subject to the End User License
* Agreement(s) entered into by the website owner. All other parties are
* explicitly restricted from using the Licensed Webfonts(s).
*
* You may obtain a valid license at the URLs below.
*
*
* Webfont: Proxima Nova Light by Mark Simonson
* URL: http://www.myfonts.com/fonts/marksimonson/proxima-nova/light/
* Licensed pageviews: unlimited
*
* Webfont: Proxima Nova Regular by Mark Simonson
* URL: http://www.myfonts.com/fonts/marksimonson/proxima-nova/regular/
* Licensed pageviews: unlimited
*
* Webfont: Proxima Nova Semibold by Mark Simonson
* URL: http://www.myfonts.com/fonts/marksimonson/proxima-nova/semibold/
* Licensed pageviews: unlimited
*
* Webfont: Proxima Nova Bold Italic by Mark Simonson
* URL: http://www.myfonts.com/fonts/marksimonson/proxima-nova/bold-it/
* Licensed pageviews: unlimited
*
* Webfont: Proxima Nova Bold by Mark Simonson
* URL: http://www.myfonts.com/fonts/marksimonson/proxima-nova/bold/
* Licensed pageviews: unlimited
*
* Webfont: Proxima Nova Italic by Mark Simonson
* URL: http://www.myfonts.com/fonts/marksimonson/proxima-nova/regular-it/
* Licensed pageviews: unlimited
*
*
* License: http://www.myfonts.com/viewlicense?type=web&buildid=2164953
* Webfonts copyright: Copyright (c) Mark Simonson, 2005. All rights reserved.
*
* (c) 2012 Bitstream Inc
*/
@font-face {
font-family: 'proxima-nova';
src: url(' //instagramstatic-a.akamaihd.net/h1/webfonts/proximanova-bold-webfont.eot/115b1f7f9c04.eot');
src: url(' //instagramstatic-a.akamaihd.net/h1/webfonts/proximanova-bold-webfont.eot/115b1f7f9c04.eot?#iefix') format("embedded-opentype"),
url(' //instagramstatic-a.akamaihd.net/h1/webfonts/proximanova-bold-webfont.woff/618250d25a4d.woff') format("woff"),
url(' //instagramstatic-a.akamaihd.net/h1/webfonts/proximanova-bold-webfont.ttf/646346e03084.ttf') format("truetype"),
url(' //instagramstatic-a.akamaihd.net/h1/webfonts/proximanova-bold-webfont.svg/e55a9d6051e8.svg#ProximaNovaBold') format("svg");
font-weight: bold;
font-style: normal; }
@font-face {
font-family: 'proxima-nova';
src: url(' //instagramstatic-a.akamaihd.net/h1/webfonts/proximanova-boldit-webfont.eot/1cbb869da891.eot');
src: url(' //instagramstatic-a.akamaihd.net/h1/webfonts/proximanova-boldit-webfont.eot/1cbb869da891.eot?#iefix') format("embedded-opentype"),
url(' //instagramstatic-a.akamaihd.net/h1/webfonts/proximanova-boldit-webfont.woff/b1cf049474c9.woff') format("woff"),
url(' //instagramstatic-a.akamaihd.net/h1/webfonts/proximanova-boldit-webfont.ttf/3adb020ceae3.ttf') format("truetype"),
url(' //instagramstatic-a.akamaihd.net/h1/webfonts/proximanova-boldit-webfont.svg/29948a2d3c58.svg#ProximaNovaBoldItalic') format("svg");
font-weight: bold;
font-style: italic; }
@font-face {
font-family: 'proxima-nova';
src: url(' //instagramstatic-a.akamaihd.net/h1/webfonts/proximanova-reg-webfont.eot/12af77715cee.eot');
src: url(' //instagramstatic-a.akamaihd.net/h1/webfonts/proximanova-reg-webfont.eot/12af77715cee.eot?#iefix') format("embedded-opentype"),
url(' //instagramstatic-a.akamaihd.net/h1/webfonts/proximanova-reg-webfont.woff/a9a9773b8e29.woff') format("woff"),
url(' //instagramstatic-a.akamaihd.net/h1/webfonts/proximanova-reg-webfont.ttf/99e19808976a.ttf') format("truetype"),
url(' //instagramstatic-a.akamaihd.net/h1/webfonts/proximanova-reg-webfont.svg/c33d2fd56309.svg#ProximaNovaRegular') format("svg");
font-weight: normal;
font-style: normal; }
@font-face {
font-family: 'proxima-nova';
src: url(' //instagramstatic-a.akamaihd.net/h1/webfonts/proximanova-regit-webfont.eot/1bbbd1312b0d.eot');
src: url(' //instagramstatic-a.akamaihd.net/h1/webfonts/proximanova-regit-webfont.eot/1bbbd1312b0d.eot?#iefix') format("embedded-opentype"),
url(' //instagramstatic-a.akamaihd.net/h1/webfonts/proximanova-regit-webfont.woff/9e306befca91.woff') format("woff"),
url(' //instagramstatic-a.akamaihd.net/h1/webfonts/proximanova-regit-webfont.ttf/4a8663684135.ttf') format("truetype"),
url(' //instagramstatic-a.akamaihd.net/h1/webfonts/proximanova-regit-webfont.svg/876278d4b189.svg#ProximaNovaRegularItalic') format("svg");
font-weight: normal;
font-style: italic; }
@font-face {
font-family: 'proxima-nova';
src: url(' //instagramstatic-a.akamaihd.net/h1/webfonts/proximanova-sbold-webfont.eot/5016edf79e1d.eot');
src: url(' //instagramstatic-a.akamaihd.net/h1/webfonts/proximanova-sbold-webfont.eot/5016edf79e1d.eot?#iefix') format("embedded-opentype"),
url(' //instagramstatic-a.akamaihd.net/h1/webfonts/proximanova-sbold-webfont.woff/615c1b06d8fa.woff') format("woff"),
url(' //instagramstatic-a.akamaihd.net/h1/webfonts/proximanova-sbold-webfont.ttf/2973bd483f7a.ttf') format("truetype"),
url(' //instagramstatic-a.akamaihd.net/h1/webfonts/proximanova-sbold-webfont.svg/868597833e49.svg#ProximaNovaSemibold') format("svg");
font-weight: 600;
font-style: normal; }
</style>
<style type="text/css">
@font-face {
font-family: 'proxima-nova';
src: url(' //instagramstatic-a.akamaihd.net/h1/webfonts/proximanova-light-webfont.eot/63c84728610f.eot');
src: url(' //instagramstatic-a.akamaihd.net/h1/webfonts/proximanova-light-webfont.eot/63c84728610f.eot?#iefix') format("embedded-opentype"),
url(' //instagramstatic-a.akamaihd.net/h1/webfonts/proximanova-light-webfont.woff/66bbe029f180.woff') format("woff"),
url(' //instagramstatic-a.akamaihd.net/h1/webfonts/proximanova-light-webfont.ttf/eb408516399b.ttf') format("truetype"),
url(' //instagramstatic-a.akamaihd.net/h1/webfonts/proximanova-light-webfont.svg/858f6a9b7ef3.svg#ProximaNovaLight') format("svg");
font-weight: 300;
font-style: normal; }
</style>
<meta name="robots" content="noimageindex">
<meta name="mobile-web-app-capable" content="yes">
<meta name="apple-mobile-web-app-capable" content="yes">
<meta name="apple-mobile-web-app-status-bar-style" content="black">
<meta id="viewport" name="viewport" content="width=device-width, user-scalable=no, initial-scale=1, minimum-scale=1, maximum-scale=1">
<script type="text/javascript">
(function() {
var docElement = document.documentElement;
var classRE = new RegExp('(^|\\s)no-js(\\s|$)');
var className = docElement.className;
docElement.className = className.replace(classRE, '$1js$2');
})();
</script>
<link rel="apple-touch-icon-precomposed" sizes="76x76" href="//instagramstatic-a.akamaihd.net/h1/images/ico/apple-touch-icon-76x76-precomposed.png/932e4d9af891.png">
<link rel="apple-touch-icon-precomposed" sizes="120x120" href="//instagramstatic-a.akamaihd.net/h1/images/ico/apple-touch-icon-120x120-precomposed.png/004705c9353f.png">
<link rel="apple-touch-icon-precomposed" sizes="152x152" href="//instagramstatic-a.akamaihd.net/h1/images/ico/apple-touch-icon-152x152-precomposed.png/82467bc9bcce.png">
<link rel="apple-touch-icon-precomposed" sizes="167x167" href="//instagramstatic-a.akamaihd.net/h1/images/ico/apple-touch-icon-167x167-precomposed.png/515cb4eeeeee.png">
<link rel="apple-touch-icon-precomposed" sizes="180x180" href="//instagramstatic-a.akamaihd.net/h1/images/ico/apple-touch-icon-180x180-precomposed.png/94fd767f257b.png">
<link rel="icon" sizes="192x192" href="//instagramstatic-a.akamaihd.net/h1/images/ico/favicon-192.png/b407fa101800.png">
<link rel="mask-icon" href="//instagramstatic-a.akamaihd.net/h1/images/ico/favicon.svg/9d8680ab8a3c.svg" color="#262626"><link rel="shortcut icon" type="image/x-icon" href="//instagramstatic-a.akamaihd.net/h1/images/ico/favicon.ico/dfa85bb1fd63.ico">
<link href="https://www.instagram.com/accounts/login/" rel="alternate" hreflang="x-default">
<link href="https://www.instagram.com/accounts/login/?hl=el" rel="alternate" hreflang="el">
<link href="https://www.instagram.com/accounts/login/?hl=gu" rel="alternate" hreflang="gu">
<link href="https://www.instagram.com/accounts/login/?hl=en" rel="alternate" hreflang="en">
<link href="https://www.instagram.com/accounts/login/?hl=af" rel="alternate" hreflang="af">
<link href="https://www.instagram.com/accounts/login/?hl=vi" rel="alternate" hreflang="vi">
<link href="https://www.instagram.com/accounts/login/?hl=fr-ca" rel="alternate" hreflang="fr-ca">
<link href="https://www.instagram.com/accounts/login/?hl=it" rel="alternate" hreflang="it">
<link href="https://www.instagram.com/accounts/login/?hl=kn" rel="alternate" hreflang="kn">
<link href="https://www.instagram.com/accounts/login/?hl=zh-hk" rel="alternate" hreflang="zh-hk">
<link href="https://www.instagram.com/accounts/login/?hl=pt-br" rel="alternate" hreflang="pt-br">
<link href="https://www.instagram.com/accounts/login/?hl=cs" rel="alternate" hreflang="cs">
<link href="https://www.instagram.com/accounts/login/?hl=id" rel="alternate" hreflang="id">
<link href="https://www.instagram.com/accounts/login/?hl=es" rel="alternate" hreflang="es">
<link href="https://www.instagram.com/accounts/login/?hl=ru" rel="alternate" hreflang="ru">
<link href="https://www.instagram.com/accounts/login/?hl=sr" rel="alternate" hreflang="sr">
<link href="https://www.instagram.com/accounts/login/?hl=nl" rel="alternate" hreflang="nl">
<link href="https://www.instagram.com/accounts/login/?hl=pt" rel="alternate" hreflang="pt">
<link href="https://www.instagram.com/accounts/login/?hl=zh-tw" rel="alternate" hreflang="zh-tw">
<link href="https://www.instagram.com/accounts/login/?hl=nb" rel="alternate" hreflang="nb">
<link href="https://www.instagram.com/accounts/login/?hl=tr" rel="alternate" hreflang="tr">
<link href="https://www.instagram.com/accounts/login/?hl=ne" rel="alternate" hreflang="ne">
<link href="https://www.instagram.com/accounts/login/?hl=zh-cn" rel="alternate" hreflang="zh-cn">
<link href="https://www.instagram.com/accounts/login/?hl=tl" rel="alternate" hreflang="tl">
<link href="https://www.instagram.com/accounts/login/?hl=pa" rel="alternate" hreflang="pa">
<link href="https://www.instagram.com/accounts/login/?hl=th" rel="alternate" hreflang="th">
<link href="https://www.instagram.com/accounts/login/?hl=te" rel="alternate" hreflang="te">
<link href="https://www.instagram.com/accounts/login/?hl=pl" rel="alternate" hreflang="pl">
<link href="https://www.instagram.com/accounts/login/?hl=ta" rel="alternate" hreflang="ta">
<link href="https://www.instagram.com/accounts/login/?hl=fr" rel="alternate" hreflang="fr">
<link href="https://www.instagram.com/accounts/login/?hl=bg" rel="alternate" hreflang="bg">
<link href="https://www.instagram.com/accounts/login/?hl=uk" rel="alternate" hreflang="uk">
<link href="https://www.instagram.com/accounts/login/?hl=hr" rel="alternate" hreflang="hr">
<link href="https://www.instagram.com/accounts/login/?hl=bn" rel="alternate" hreflang="bn">
<link href="https://www.instagram.com/accounts/login/?hl=de" rel="alternate" hreflang="de">
<link href="https://www.instagram.com/accounts/login/?hl=da" rel="alternate" hreflang="da">
<link href="https://www.instagram.com/accounts/login/?hl=hi" rel="alternate" hreflang="hi">
<link href="https://www.instagram.com/accounts/login/?hl=fi" rel="alternate" hreflang="fi">
<link href="https://www.instagram.com/accounts/login/?hl=hu" rel="alternate" hreflang="hu">
<link href="https://www.instagram.com/accounts/login/?hl=ja" rel="alternate" hreflang="ja">
<link href="https://www.instagram.com/accounts/login/?hl=ro" rel="alternate" hreflang="ro">
<link href="https://www.instagram.com/accounts/login/?hl=ml" rel="alternate" hreflang="ml">
<link href="https://www.instagram.com/accounts/login/?hl=ko" rel="alternate" hreflang="ko">
<link href="https://www.instagram.com/accounts/login/?hl=sv" rel="alternate" hreflang="sv">
<link href="https://www.instagram.com/accounts/login/?hl=ur" rel="alternate" hreflang="ur">
<link href="https://www.instagram.com/accounts/login/?hl=sk" rel="alternate" hreflang="sk">
<link href="https://www.instagram.com/accounts/login/?hl=si" rel="alternate" hreflang="si">
<link href="https://www.instagram.com/accounts/login/?hl=ms" rel="alternate" hreflang="ms">
<link href="https://www.instagram.com/accounts/login/?hl=mr" rel="alternate" hreflang="mr">
</head>
<body class="">
<span id="react-root"></span>
<script src="//instagramstatic-a.akamaihd.net/h1/scripts/polyfills/es5-shim.min.js/6514c5e08593.js"></script>
<script src="//instagramstatic-a.akamaihd.net/h1/scripts/polyfills/es5-sham.min.js/fc3c22cf2d67.js"></script>
<script type="text/javascript">window._sharedData = {"country_code": "IT", "language_code": "en", "gatekeepers": {"sulgin": true}, "show_app_install": true, "static_root": "//instagramstatic-a.akamaihd.net/h1", "platform": "web", "hostname": "www.instagram.com", "entry_data": {"LoginPage": [{}]}, "qe": {"profile": {"p": {}, "g": ""}, "su_universe": {"p": {}, "g": ""}, "notif": {"p": {}, "g": ""}, "us": {"p": {"show_desktop_registration_upsell": "false"}, "g": "show_desktop_registration_upsell_05"}, "su": {"p": {"enabled": "true"}, "g": "rollout_20160325"}, "us_li": {"p": {}, "g": ""}, "discovery": {"p": {}, "g": ""}}, "display_properties_server_guess": {"viewport_width": 360, "pixel_ratio": 1.5}, "config": {"viewer": null, "csrf_token": "RwH4hiwxweK4xLfx3CXBAS12z8XBcROC"}, "environment_switcher_visible_server_guess": true};</script>
<script src="//instagramstatic-a.akamaihd.net/h1/bundles/en_US_Commons.js/738da0ab86b4.js" type="text/javascript" crossorigin="anonymous"></script>
<script src="//instagramstatic-a.akamaihd.net/h1/bundles/en_US_LoginPage.js/5a299fe05cb2.js" type="text/javascript" crossorigin="anonymous"></script>
<script>
!function(f,b,e,v,n,t,s){if(f.fbq)return;n=f.fbq=function(){n.callMethod?
n.callMethod.apply(n,arguments):n.queue.push(arguments)};if(!f._fbq)f._fbq=n;
n.push=n;n.loaded=!0;n.version='2.0';n.queue=[];t=b.createElement(e);t.async=!0;
t.src=v;s=b.getElementsByTagName(e)[0];s.parentNode.insertBefore(t,s)}(window,
document,'script','//connect.facebook.net/en_US/fbevents.js');
fbq('init', '1425767024389221');
fbq('track', 'PageView');
</script>
<noscript>
</noscript>
<script type="text/javascript">window._timings.domInteractive = Date.now()</script>
</body></html>Max Pierini
Sep 12, 2016, 12:11:46 PM9/12/16
to phantomjs
Here's the ghostdriver.log
[INFO - 2016-09-12T16:08:37.057Z] GhostDriver - Main - running on port 49739
[INFO - 2016-09-12T16:08:37.933Z] Session [2a14fc60-7903-11e6-a755-53e4799f55f3] - page.settings - {"XSSAuditingEnabled":false,"javascriptCanCloseWindows":true,"javascriptCanOpenWindows":true,"javascriptEnabled":true,"loadImages":true,"localToRemoteUrlAccessEnabled":false,"userAgent":"Mozilla/5.0 (Unknown; Linux x86_64) AppleWebKit/538.1 (KHTML, like Gecko) PhantomJS/2.1.1 Safari/538.1","webSecurityEnabled":true}
[INFO - 2016-09-12T16:08:37.933Z] Session [2a14fc60-7903-11e6-a755-53e4799f55f3] - page.customHeaders: - {}
[INFO - 2016-09-12T16:08:37.933Z] Session [2a14fc60-7903-11e6-a755-53e4799f55f3] - Session.negotiatedCapabilities - {"browserName":"phantomjs","version":"2.1.1","driverName":"ghostdriver","driverVersion":"1.2.0","platform":"linux-unknown-64bit","javascriptEnabled":true,"takesScreenshot":true,"handlesAlerts":false,"databaseEnabled":false,"locationContextEnabled":false,"applicationCacheEnabled":false,"browserConnectionEnabled":false,"cssSelectorsEnabled":true,"webStorageEnabled":false,"rotatable":false,"acceptSslCerts":false,"nativeEvents":true,"proxy":{"proxyType":"direct"}}
[INFO - 2016-09-12T16:08:37.934Z] SessionManagerReqHand - _postNewSessionCommand - New Session Created: 2a14fc60-7903-11e6-a755-53e4799f55f3
[ERROR - 2016-09-12T16:08:41.820Z] WebElementLocator - _handleLocateCommand - Element(s) NOT Found: GAVE UP. Search Stop Time: 1473696521775
phantomjs://platform/console++.js:263 in error
events.js:141
throw er; // Unhandled 'error' event
^
Error: read EIO
at exports._errnoException (util.js:870:11)
at TTY.onread (net.js:544:26)Max Pierini
Sep 12, 2016, 12:16:40 PM9/12/16
to phantomjs
The same error happens when trying to load https://www.pinterest.com/login/ but this page is correctly loaded using curl.
Max Pierini
Sep 13, 2016, 5:11:40 PM9/13/16
to phantomjs
Update:
installing phantomjs with
sudo apt-get install phantomjsit correctly loads the entire page. But this package is missing some important third-party dependencies (such as find_element Atom).
installing phantomjs with
npm install phantomjs-prebuiltit doesn't correctly load the page (even if it has got all third-party Atoms).
Is there a way to use the executable installed with apt-get and third-party Atoms installed by npm?
Message has been deleted
Max Pierini
Sep 25, 2016, 5:02:09 AM9/25/16
to phantomjs
SOLVED.
I solved compiling phantomjs on Ubuntu Server directly from git repository. So maybe the pre-compiled binaries are not complete.
Details here: http://phantomjs.org/build.html
Reply all
Reply to author
Forward
0 new messages