Parsehub gets stuck on large list page

Roman Mueller

unread,

Oct 4, 2023, 6:06:05 AM10/4/23

to Web Scraping

Hello,

I have the problem that Parsehub gets stuck if scraping a list page with a few thousand people. The default number of persons on the page is 40. However there is a "Load More" button to get 6500 further results on the listing page. I click on it before I go to "get data" to load them all but then it gets stuck

Is there any way this can be handled better so Parsehub don't get stuck in the process of scraping all the data?

thanks in advance.

Roman

Andrew11

unread,

Oct 4, 2023, 9:49:50 AM10/4/23

to Web Scraping

If you click a button in the preview browser in the ParseHub app and go to Get Data, it doesn't save your progress but opens a brand new copy of the page. You need to put Select loadMore & Click inside the scrape template so that when you Get Data, the cloud scraper clicks on that button after the page loads.

Roman Mueller

unread,

Oct 5, 2023, 6:38:11 AM10/5/23

to Web Scraping

Hello Adrew,

thank you for your quick response. I'll try that again. Thanks for the tip. I have now solved it by entering the URLS of the individual detail pages directly as a starting value. Follow these instructions (https://www.parsehub.com/blog/web-scrape-multiple-urls/). I use the free version of Parsehub. So, I can only scrape 200 pages per run. I would have to divide the 6500 pages into about 30 runs and split up the URLs.

kind regards,

Roman

Sauban Abdullahi

unread,

Sep 8, 2025, 10:39:24 AMSep 8

to Web Scraping

Hi there!

I am a skilled Python developer offering custom automation, Telegram bots, web scraping, and Telegram scraping services tailored to your business needs. My Services Include: