Web scraping help

[seleniumlover]

Hey Selenium users,

There is a website I'm trying to web scrape some data from using Selenium (Python), my problem is that I have to manually change some parameters and settings in the webpage before being able to get the data I want.

First I tried automating the process of applying the custom settings but there was some buttons and options on the poorly-designed site that were impossible to locate their HTML source so I can control using Selenium.

For now, I'm using Chrome in debugging mode so I can open a Chrome tab on my desktop, do whatever I want and then connect Selenium with Python to it. But this mandates me to keep the Chrome tab actively open on my desktop.

Is there a way I might be able to manually setup the site to my needs, somehow save the instance and then have Selenium with Python open the instance natively in the background without me needing to have it on my desktop?

I would really appreciate any help. Have a great day :)

[Scott Babcock]

The information in your post is a bit vague, but you can specify many different settings via ChromeOptions for your browser session. For myself, I disable Info Bars and the credentials service. I also turn on "headless" mode and disable GPU when I want to run without a UI.

For settings that get persisted with a profile, you can also save the profile to a file and specify this as the profile to use when launching a new session.

However, Selenuim API isn't a very efficient tool to use for screen scraping. I'm not a Python guy, but I strongly suspect that a bit of research will reveal better options.