Scrape Elemente Click in Element Click

165 views
Skip to first unread message

Alex

unread,
Dec 14, 2017, 7:09:20 AM12/14/17
to Web Scraper
Hello,

I'm trying to scrape a list of items, which open up by clicking the title (javascript). I need to scrape shown content after clicking the title of each item. 

Furthermore there is a pagination with Javascript: 1, 2, 3, 4, 6, 7, 8, 9, 10, More. Each of the numbers and also "More" is a Javascript linking. 


I managed to do that either the items of the first page get scraped or that the pagination works fine. But not both at the same time/configuration.

Here is a simplified part of the code:

//the job list

<ul id="jobs" class="resultlist">
<li id="job-568945" class="job"><div class="row"><div class="col info"><a class="title" href="#" >Jobtitle</a></div></li>
<li>....</li>
<li>....</li>
.....
</ul>

//and the pagination

<ul class="pagination">
<li class="current">1</li>
<li><a href="#">2</a></li>
<li><a href="#">3</a></li>
<li><a href="#">4</a></li>
<li><a href="#">5</a></li>
<li><a href="#">6</a></li>
<li><a href="#">7</a></li>
<li><a href="#">8</a></li>
<li><a href="#">9</a></li>
<li><a href="#">10</a></li>
<li class="more"><a href="#">More»</a></li>
</ul>


By clicking on the linking "Jobtitle" more content is shown. I need to scrape this content. 

And my sitemap:

{"startUrl":"www.example.com","selectors":[{"parentSelectors":["_root","pagination"],"type":"SelectorElementClick","multiple":true,"id":"Job","selector":"#jobs li","delay":"3000","clickElementSelector":"#jobs li a.title","clickElementUniquenessType":"uniqueText","clickType":"clickOnce","discardInitialElements":false},{"parentSelectors":["Job"],"type":"SelectorText","multiple":false,"id":"description","selector":"div.description div","regex":"","delay":""},{"parentSelectors":["Job"],"type":"SelectorText","multiple":true,"id":"City","selector":"span.city","regex":"","delay":""},{"parentSelectors":["_root"],"type":"SelectorElementClick","multiple":true,"id":"pagination","selector":"#jobs","clickElementSelector":"li.more a","clickElementUniquenessType":"uniqueText","clickType":"clickMore","discardInitialElements":false,"delay":"10000"}],"_id":"joblist"}

Do you have any ideas? 

Many Thanks!

Alex


Reply all
Reply to author
Forward
0 new messages