Re: Selenium PhantomJS and special characters

[David]

two thoughts:

• does your renewal site actually require a UI to render/work? You could simplify the solution with an HTTP POST to the server's back end faking the user agent as if it was a browser and avoid using Selenium altogether. Just simply HTTP calls and JSON/XML parsing.
• as last resort what about using Firefox (and maybe chrome) via xvfb on the Rpi? http://joejanuszk.com/blog/running-headless-selenium-on-a-raspberry-pi/

On Thursday, December 8, 2016 at 7:25:13 AM UTC-8, JayBee De wrote:

Hi everyone,

I've developped a selenium script that can automatically renew my no-ip DNS every 30 days. On my Windows Development machine, it works perfectly with Python and Chrome Webdriver. Then I wanted to make it runs by the way of the PhantomJS WebDriver on my Raspberry pi. The anthentication works since I can retrieve the number of days before expiration in the source page written on the file.txt (just to try to debug). Here is my code:

import time
import re
import os
import ctypes
import urllib
from selenium import webdriver
import logging
from selenium.webdriver.common.desired_capabilities import DesiredCapabilities

login="<no ip login>"
password="<no ip password>"
hostname="<hostname to renew/check>"
logFile='example2.log'
applicationName="Renew DNS"
debugMode=False

def renewDns(hostname,flag=None):
 global login
 global password
 global logger
 global browser

 url="https://my.noip.com/"
 browser.get(url)
 time.sleep(4)

 usernameField=browser.find_element_by_name('username')
 usernameField.send_keys(login)
 passwordField=browser.find_element_by_name('password')
 passwordField.send_keys(password)
 submitButton=browser.find_element_by_name('Login')
 submitButton.submit()

 time.sleep(2)
 inter=browser.find_element_by_xpath("//span[text()[contains(.,'Active')]]")
 inter.click()

 time.sleep(2)
 source=browser.page_source
 f=open('/var/www/blog/wp-content/file.txt','w')
 f.write(source.encode('latin-1'))
 f.close()
#till here it works because the source code written in the file is ok

 stt=browser.find_element_by_xpath("//td[@data-title='Expiration']//a").text
#this doesn't work (empty result and should be full)
 logger.error(stt)

initLogs()
#I have remove this function because useless for my issue
logger.info(applicationName+" has started!")
user_agent = (
    "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_8_4) " +
    "AppleWebKit/537.36 (KHTML, like Gecko) Chrome/29.0.1547.57 Safari/537.36"
)
dcap = dict(DesiredCapabilities.PHANTOMJS)
dcap["phantomjs.page.settings.userAgent"] = user_agent

browser = webdriver.PhantomJS(desired_capabilities=dcap)

renewDns(hostname,None)

browser.close()

So the xpath query should work, and I was wondering if the problem could come from the fact that:

-> The targeted webpage url contains special characters (https://my.noip.com/#!/dynamic-dns). But the source code can be written in the file.txt so it should not be that

-> The targeted td item by the non working Xpath query contains underscores and other strange characters. Here is the extract of the part of the source code written in the file.txt:

          <td class="hidden-xs" data-title="Expiration" _v-78c9c52b="">
            <span class="label ticket-label label-success" _v-78c9c52b="">
              <span _v-78c9c52b="">
                <div _v-78c9c52b=""><a target="_blank" class="no-link-style" _v-78c9c52b="" href="https://www.noip.com/remote-access">Expires in 9 days</a></div>
               
              </span>
            </span>
          </td>

What do you think? What should I do?

[JayBee De]

Hi everyone, 

Thank you David for your answer. Unfortunately I cannnot/don't want to use Firefox. As my question was not enough  focused on the problem I have removed it. Here is my new question with what I have found:

I have the following Python code:

import time
from selenium import webdriver
user_agent = (

    "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_8_4) " +
    "AppleWebKit/537.36 (KHTML, like Gecko) Chrome/29.0.1547.57 Safari/537.36"
)
dcap = dict(DesiredCapabilities.PHANTOMJS)
dcap["phantomjs.page.settings.userAgent"] = user_agent
browser = webdriver.PhantomJS(desired_capabilities=dcap)

browser.get(<urlToTheHTMLFile)
time.sleep(2)
print browser.page_source

stt=browser.find_element_by_xpath("//td[@data-title='Expiration']//a").text

print stt
browser.close()

The content of the HTML file is:

 <tr class="table-striped-row" _v-78c9c52b="">

  <td class="hidden-xs" data-title="Expiration" _v-78c9c52b="">
    <span class="label ticket-label label-success" _v-78c9c52b="">
      <span _v-78c9c52b="">
        <div _v-78c9c52b=""><a target="_blank" class="no-link-style" _v-78c9c52b="" href="https://www.noip.com/remote-access">Expires in 9 days</a></div>
      </span>
    </span>
  </td>

 </tr>

As you understand, I want the item in bold "Expires in 9 days". The result is empty and the line that prints the page source remove all td items. Nevertheless, I have noticed that if I replace the underlined bold item td by div, it works. 

Why? What should I do? I cannot change the content of the HTLM file, but I can change my Python code of course.

[David]

No you have the sample full HTML file to share for others to try and test against?