Selenium and Headless Chrome

[A Pwq]

Hi guys wondered if you could help here

I'm scrapping a site, in this case https://car.gocompare.com/vehicle

When using chrome it works fine, as in i can put a reg number and it finds the reg details, however when I use headless it doesnt find it as produce a cannot find vehicle error, I've tried it with the firefox and it works with headless and normal mode perfectly fine.

Does anyone have a clue as to why it would do this, I've using the latest chromedriver and chrome on windows, and I've been getting this error.

Is it their site not producing the data, or is it a setting in headless chrome which is different to the firefox's setting.

Any help would be appreciated.

[Akula Bhaskar]

will you be comfortable to share code 

--
You received this message because you are subscribed to the Google Groups "Selenium Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to selenium-user...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/selenium-users/d3d6f789-f2ef-4118-94b0-2569758f2b05%40googlegroups.com.

--

Best Regards

A Bhaskar Rao

07799870508

[Akula Bhaskar]

Hope you have tried 

following

FirefoxBinary firefoxBinary = new FirefoxBinary();
   firefoxBinary.addCommandLineOptions("--headless");
   System.setProperty("webdriver.gecko.driver", "/opt/geckodriver");
   FirefoxOptions firefoxOptions = new FirefoxOptions();
   firefoxOptions.setBinary(firefoxBinary);
   FirefoxDriver driver = new FirefoxDriver(firefoxOptions)

[A Pwq]

HI Akula, yes I've tried firefox and its works, but I need to use chrome for the time being 

I've pasted my code here, you can see I've got the switch to change to firefox so you can see it working, and I've got screenshots of before and after.

from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver.common.keys import Keys
from selenium.webdriver.common.action_chains import ActionChains
from selenium.webdriver.support.ui import Select
from selenium.common.exceptions import NoSuchElementException

# Ability to run headless
from selenium.webdriver.firefox.options import Options as f_Options
from selenium.webdriver.chrome.options import Options as c_Options
from selenium.webdriver.firefox.firefox_binary import FirefoxBinary

# This allows you to download the page
from parsel import Selector


import time
import datetime
import os



class headlessbypass:

    my_date_time = datetime.datetime.now().strftime('%Y%m%d%H%M%S')

    def my_set_up(self):
        """Executed before running, i.e. opening browser"""

        # This is required for running on the pipeline
        headless = os.getenv('HEADLESS_MODE')

        def firefox_headless_func():
            self.options = f_Options()
            self.options.headless = True
            binary = FirefoxBinary('c:/Users/Anish/AppData/Local/Mozilla Firefox/firefox.exe')
            self.driver = webdriver.Firefox(firefox_binary=binary, executable_path='bin/geckodriver.exe', options=self.options)#, options=self.options, executable_path='bin/geckodriver.exe')

        def chrome_headless_func():
            self.options = c_Options()
            #self.options.headless = True
            self.options.add_argument("--window-size=1920, 1080")
            #self.options.add_argument("--disable-extensions")
            #self.options.add_argument("--proxy-server='direct://'")
            #self.options.add_argument("--proxy-bypass-list=*")
            #self.options.add_argument("--start-maximized")
            self.options.add_argument('--headless')
            self.options.add_argument('--disable-gpu')
            #self.options.add_argument('--disable-dev-shm-usage')
            #self.options.add_argument('--no-sandbox')
            #self.options.add_argument('--ignore-certificate-errors')
            #self.options.add_argument("--allow-insecure-localhost")
            #self.options.add_argument("--allow-running-insecure-content")
            #self.options.add_argument('--disable-browser-side-navigation')
            self.options.add_argument("--enable-javascript")
            self.options.add_argument("--user-agent='Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:72.0) Gecko/20100101 Firefox/72.0'")
            #self.options.binary_location = "C:/Program Files (x86)/Google/Chrome/Application/chrome.exe"
            self.driver = webdriver.Chrome(options=self.options, executable_path='bin/chromedriver')




        # This is for running locally; select/toggle what you want to run
        headless_firefox = 0
        headless_chrome = 0
        chrome = 1
        safari = 0

        if headless:
            firefox_headless_func()
        else:
            if headless_firefox:
                firefox_headless_func()

            elif headless_chrome:
                chrome_headless_func()

            elif chrome:
                self.driver = webdriver.Chrome(executable_path='bin/chromedriver.exe')

            else:
                self.driver = webdriver.Firefox(executable_path='bin/geckodriver.exe')

        self.driver.implicitly_wait(30)
        self.driver.maximize_window()

        main_window = self.driver.current_window_handle
        self.driver.switch_to.window(main_window)

    def my_tear_down(self):
        """Executed after running, i.e. closing browser"""
        self.driver.quit()

    def my_decorator(func):
        """my_set_up and my_tear_down decorator, so that my_set_up is run before and my_tear_down is run after"""
        def wrapper(self, *args, **kwargs):
            self.my_set_up()
            func(self, *args, **kwargs)
            self.my_tear_down()
        return wrapper

    @my_decorator
    def visit_site(self):
        """Extract quotes"""
        self.driver.get("https://mygocompare.gocompare.com/newcustomer/")

        time.sleep(2)
        print(self.driver.page_source)

        # Enter registration number

        reg_field = self.driver.find_element(By.XPATH, "//fieldset[1]/div[2]/div[2]/div/input")
        reg_field.send_keys("AK47")
        time.sleep(5)
        print("Take screenshot")
        html = self.driver.find_element_by_tag_name('html')
        html.send_keys(Keys.PAGE_UP)
        self.driver.save_screenshot("csv_json_files/firstpagescreenshot.png")
        self.driver.find_element(By.XPATH, "//span[contains(text(), 'Find car')]").click()
        time.sleep(2)

        print("Take screenshot")
        html = self.driver.find_element_by_tag_name('html')
        html.send_keys(Keys.PAGE_UP)
        self.driver.save_screenshot("csv_json_files/firstpagescreenshot2.png")


if __name__ == '__main__':
    start_time = time.time()
    scrape = headlessbypass()
    scrape.visit_site()

[Sender]

Thank you for your email!

We are currently out of the office, but we will reply as soon as possible.

Best regards,
Joe Customer

"'A Pwq' via Selenium Users" <seleniu...@googlegroups.com> wrote:

> Hi guys wondered if you could help here
>
> I'm scrapping a site, in this case https://car.gocompare.com/vehicle
>
> When using chrome it works fine, as in i can put a reg number and it finds
> the reg details, however when I use headless it doesnt find it as produce a
> cannot find vehicle error, I've tried it with the firefox and it works with
> headless and normal mode perfectly fine.
>
> Does anyone have a clue as to why it would do this, I've using the latest
> chromedriver and chrome on windows, and I've been getting this error.
>
> Is it their site not producing the data, or is it a setting in headless
> chrome which is different to the firefox's setting.
>
> Any help would be appreciated.
>

[A Pwq]

Hi Akula

Have you had a chance to look at this.

Btw how big is the Firefox binary?

[Akula Bhaskar]

Hey Mate not able to understand where s d glitch

--

You received this message because you are subscribed to the Google Groups "Selenium Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to selenium-user...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/selenium-users/76331f0f-95f7-4012-b9c4-520687cf2764%40googlegroups.com.

[antonio....@gmail.com]

Please unsubscribe from this group

 

Thanks

 

Antonio

 

Da: seleniu...@googlegroups.com <seleniu...@googlegroups.com> Per conto di Akula Bhaskar
Inviato: martedì 19 maggio 2020 04:46
A: seleniu...@googlegroups.com
Oggetto: Re: [selenium-users] Re: Selenium and Headless Chrome

To view this discussion on the web visit https://groups.google.com/d/msgid/selenium-users/CACw8_OY412Rn5BfujJ-vE%2BPNFpb-_gxYp8G3QOD%2BmEX5ju-p4w%40mail.gmail.com.

---

Questa e-mail è stata controllata per individuare virus con Avast antivirus. www.avast.com

[A Pwq]

Hi 

When you put in a reg number normally in the text box, it provides reg details

When you use headless chrome, nothing appear back.

Is this a bug or the website doing that

To view this discussion on the web visit https://groups.google.com/d/msgid/selenium-users/CACw8_OY412Rn5BfujJ-vE%2BPNFpb-_gxYp8G3QOD%2BmEX5ju-p4w%40mail.gmail.com.

[Akula Bhaskar]

Who are you to tell some to unsubscribe?

To view this discussion on the web visit https://groups.google.com/d/msgid/selenium-users/00ae01d62dab%2491909ec0%24b4b1dc40%24%40gmail.com.

[antonio....@gmail.com]

I’ve sent an email because:

 

To unsubscribe from this group and stop receiving emails from it, send an email to selenium-user...@googlegroups.com.

 

Can you unsubscribe me?

 

Thanks

 

Antonio

 

Da: seleniu...@googlegroups.com <seleniu...@googlegroups.com> Per conto di Akula Bhaskar

Inviato: martedì 19 maggio 2020 16:17

To view this discussion on the web visit https://groups.google.com/d/msgid/selenium-users/CACw8_OZzefHNQ3PP6Esmo1Ef%2BdTLMnXy1YARiKQZtxkAj%2BP1aw%40mail.gmail.com.

[Joe Ward]

You didn’t send the email to selenium-user...@googlegroups.com but you just directly replied to a thread so therefore you have people from that thread responding to you. 🤣

To view this discussion on the web visit https://groups.google.com/d/msgid/selenium-users/020e01d62ded%24c619b180%24524d1480%24%40gmail.com.

[antonio....@gmail.com]

Ops 😝

 

Sorry

 

Da: seleniu...@googlegroups.com <seleniu...@googlegroups.com> Per conto di Joe Ward
Inviato: martedì 19 maggio 2020 19:53

To view this discussion on the web visit https://groups.google.com/d/msgid/selenium-users/CAJPyTs1cBsQpbYxbT9_N2QLShmpMzKVHfoAkcSocL3Kp_T1UOQ%40mail.gmail.com.

[A Pwq]

Hi Akula.

When you visit the site, you put in a reg number. The website produces the car details. In headless chrome it's says service is not available. This works in headless Firefox but not headless chrome.

Try it and you will see.

[A Pwq]

Also do you know what the size of Firefox binary is? And where can I get a version of it?

[Abhijit Mone]

Is your chrome version 83? Is your server remote server?

--