Parsehub project not working for large number of loop size.

Sejan ahammed

unread,

Jun 8, 2024, 2:09:03 PM6/8/24

to Web Scraping

I want to scrape data from a website which uses show more button to load more data. There are 6000+ data but the show more button reveals only 5 after each click. So I had to use an array size of 1200 on the loop command. But the project file is only scraping 800 data points.

Andrew11

unread,

Jun 8, 2024, 4:14:27 PM6/8/24

to Web Scraping

Instead of a loop you can Select the button and put in a Click & repeat template infinite # of times. If the Select doesn't find a show more button, it'll stop repeating.

Sejan ahammed

unread,

Jun 9, 2024, 5:34:44 AM6/9/24

to Web Scraping

I've also tried that but the project file keeps failing.

Andrew11

unread,

Jun 9, 2024, 8:45:16 AM6/9/24

to Web Scraping

Can you post the address here?

Sejan ahammed

unread,

Jun 9, 2024, 1:07:23 PM6/9/24

to Web Scraping

Sorry for confidentiality I won't be able to post the address here. That website is live and I'm trying to scrape peoples comments from it. Currently It has more that 6500 comments. I have to use show more button to see more comments and each click reveals only 5 comments.

Andrew11

unread,

Jun 9, 2024, 1:12:16 PM6/9/24

to Web Scraping

In the URL in the address bar, is there a bunch of things like &size=5 ? Sometimes you can edit those in the URL before opening it and make it show 100 comments at a time or the like. If not, maybe the chat button in ParseHub or writing to their tech support @ https://www.parsehub.com/contact, you can give them the address of the page to work with (hard to figure it out without that info).

Sejan ahammed

unread,

Jun 9, 2024, 2:37:01 PM6/9/24

to Web Scraping

No there is no such thing like this in Url. This website loads all comments in one page.

Sejan ahammed

unread,

Jun 26, 2024, 5:51:47 AM6/26/24

to Web Scraping

Hi Andrew,
Here is the website I was trying to scrape and facing problem.
https://glaplanningapps.commonplace.is/planningapps/21-P2900
click on comments next to the comment number and comment section will load.

Andrew11

unread,

Jun 26, 2024, 5:23:43 PM6/26/24

to Web Scraping

I'm on the free plan, so unfortunately I have a page limit of 200 clicks and I can't verify that the number of comments matches the number in the output CSV at the end of running my attached sample project. Can you try importing & running this, and see if you can match them? Let me know what happens.

glaplanningapps.commonplace.is_Project.zip

Sauban Abdullahi

unread,

Sep 5, 2025, 12:06:55 PMSep 5

to Web Scraping

Hi there!

I am a skilled Python developer offering custom automation, Telegram bots, web scraping, and Telegram scraping services tailored to your business needs. My Services Include: