Groups

Abot Web Crawler

Contact owners and managers

1–30 of 360

This is the google group for the Abot Web Crawler and AbotX Web Crawler. Please feel free to post questions or start discussions regarding the use of Abot.

0 selected

sjdirect, sbon...@gmail.com2

4/16/20

AbotX and Abot now target .NET Standard 2.0

Awesome !!! On Monday, 20 January 2020 02:05:51 UTC+1, sjdirect wrote: Exciting news!!! Abot and

unread,

AbotX and Abot now target .NET Standard 2.0

Awesome !!! On Monday, 20 January 2020 02:05:51 UTC+1, sjdirect wrote: Exciting news!!! Abot and

4/16/20

8/31/19

Abot now targets .NET Standard 2.0

Checkout the Github repo or Nuget package (>= 2.0)

unread,

Abot now targets .NET Standard 2.0

Checkout the Github repo or Nuget package (>= 2.0)

8/31/19

8/30/19

Getting ready to release Abot V2 (Dotnet Standard 2.0 support)

I have been working on Abot v2 for a while now. Getting ready to release the early version of the

unread,

Getting ready to release Abot V2 (Dotnet Standard 2.0 support)

I have been working on Abot v2 for a while now. Getting ready to release the early version of the

8/30/19

View more pinned conversations

Jeff Richmond, sjdi...@gmail.com2

8/20/24

There is a license in this demo project root repo that should work. There will likely be some fresh

unread,

There is a license in this demo project root repo that should work. There will likely be some fresh

8/20/24

7/5/24

How to only crawl new and updated pages of a website?

Let's say this example.com has been crawled today. How to only crawl the new and updated pages

unread,

How to only crawl new and updated pages of a website?

Let's say this example.com has been crawled today. How to only crawl the new and updated pages

7/5/24

ghislain borremans, … Chester Ayala5

6/20/24

Using scheduler to preload with Urls, but then, the links on those preloaded Urls are not scheduled

IsExternalPageLinksCrawlingEnabled works alone when I add the links into the scheduler. It's able

unread,

Using scheduler to preload with Urls, but then, the links on those preloaded Urls are not scheduled

IsExternalPageLinksCrawlingEnabled works alone when I add the links into the scheduler. It's able

6/20/24

rjone...@gmail.com, Sparklyn Services2

5/10/24

Crawling an HTML File

Instead of using HTTP requests to handle file input, you would usually override or enhance parts of a

unread,

Crawling an HTML File

Instead of using HTTP requests to handle file input, you would usually override or enhance parts of a

5/10/24

5/9/24

Uri Uniqueness with trailing slash

I have noticed that URIs with no trailing slash and the same URI with a trailing slash are not

unread,

Uri Uniqueness with trailing slash

I have noticed that URIs with no trailing slash and the same URI with a trailing slash are not

5/9/24

sjdirect, … alok jain5

4/29/24

Abot support for .NET Core

Please share me .net core repo. Thanks & Regards Alok On Monday, October 24, 2016 at 3:17:08 AM

unread,

Abot support for .NET Core

Please share me .net core repo. Thanks & Regards Alok On Monday, October 24, 2016 at 3:17:08 AM

4/29/24

rolf michielson, rolf michielson2

5/30/23

I am aware of your abilities

Hello I hope all is well with you and that your day is going well. I am an attorney and financial

unread,

I am aware of your abilities

Hello I hope all is well with you and that your day is going well. I am an attorney and financial

5/30/23

David Hansen, … Suresh Dayma8

7/14/22

Getting Binary data

In the configuration, set isHttpRequestAutomaticDecompressionEnabled="true", this will fix

unread,

Getting Binary data

In the configuration, set isHttpRequestAutomaticDecompressionEnabled="true", this will fix

7/14/22

4/28/22

AbotX Licensing

Hello, I have a question about the licensing. We agree that we need only 1 license if we use it for 2

unread,

AbotX Licensing

Hello, I have a question about the licensing. We agree that we need only 1 license if we use it for 2

4/28/22

Ethan, Steven Jones3

4/19/22

Hi, Thank you for your answer. I have an other question then. Is it possible to get details about the

unread,

Hi, Thank you for your answer. I have an other question then. Is it possible to get details about the

4/19/22

3/21/22

Cannot scan a particular site - every url throws timeout / no content

Hi, I'm using abot to crawl various sites and the vast majority work, however for one site, every

unread,

Cannot scan a particular site - every url throws timeout / no content

Hi, I'm using abot to crawl various sites and the vast majority work, however for one site, every

3/21/22

11/10/21

Saving state on shutdown

I'm using the AbotX parallel crawler engine, within the confines of a dotnet worker app (ie uses

unread,

Saving state on shutdown

I'm using the AbotX parallel crawler engine, within the confines of a dotnet worker app (ie uses

11/10/21

11/10/21

Seeding SiteToCrawl with multiple pages?

Also, if I take the cheat route and just add a new SiteToCrawl for every URL, is the parallel engine

unread,

Seeding SiteToCrawl with multiple pages?

Also, if I take the cheat route and just add a new SiteToCrawl for every URL, is the parallel engine

11/10/21

11/8/21

NullReferenceException at CrawlDecisionMaker.ShouldDownloadPageContent

Hello, I'm writing a basic WinForm which starts the web crawler based on a URL (checked and

unread,

NullReferenceException at CrawlDecisionMaker.ShouldDownloadPageContent

Hello, I'm writing a basic WinForm which starts the web crawler based on a URL (checked and

11/8/21

9/7/21

Hii, I bought abotx for a sister company and we didn't receive access to the source code can you

unread,

Hii, I bought abotx for a sister company and we didn't receive access to the source code can you

9/7/21

ghislain borremans, sjdi...@gmail.com3

7/1/21

Operation of Cookies

Hi, IsSendingCookiesEnabled = true tells Abot/AbotX to resend any cookies that are returned in the

unread,

Operation of Cookies

Hi, IsSendingCookiesEnabled = true tells Abot/AbotX to resend any cookies that are returned in the

7/1/21

rjone...@gmail.com, sjdi...@gmail.com2

6/23/21

Getting Page Data when Parallel Crawling

Yes, There is a clear example in the docs here under ParallelCrawlerEngine. See the code snippet with

unread,

Getting Page Data when Parallel Crawling

Yes, There is a clear example in the docs here under ParallelCrawlerEngine. See the code snippet with

6/23/21

ghislain borremans, sjdi...@gmail.com3

6/23/21

AbotX: screenshot from crawled page

Replied to the cookie issue thread instead of here. On Sat, Jun 19, 2021 at 8:42 AM ghislain

unread,

AbotX: screenshot from crawled page

Replied to the cookie issue thread instead of here. On Sat, Jun 19, 2021 at 8:42 AM ghislain

6/23/21

Saeid Babaei, sjdi...@gmail.com2

5/24/21

Yes, Abot has no limitations that would prevent you from doing this. If you need custom consulting on

unread,

Yes, Abot has no limitations that would prevent you from doing this. If you need custom consulting on

5/24/21

VLADIMIR KOZLOV

4/17/21

getting empy e.CrawledPage.Content.Text - don't undestand why

Hello, i'm getting always null or empty e.CrawledPage.Content.Text in PageCrawlCompleted(object

unread,

getting empy e.CrawledPage.Content.Text - don't undestand why

Hello, i'm getting always null or empty e.CrawledPage.Content.Text in PageCrawlCompleted(object

4/17/21

rjone...@gmail.com, sjdirect4

4/9/21

Net 5.x Support

You can follow the status of this in the github issue. On Friday, April 2, 2021 at 3:30:12 PM UTC-7

unread,

Net 5.x Support

You can follow the status of this in the github issue. On Friday, April 2, 2021 at 3:30:12 PM UTC-7

4/9/21

Ridhima Shukla, sjdi...@gmail.com2

3/18/21

Single Page Application

Hi, As your last request the same forum post answers this question as well. You can see this thread

unread,

Single Page Application

Hi, As your last request the same forum post answers this question as well. You can see this thread

3/18/21

Ridhima Shukla, sjdi...@gmail.com2

3/18/21

Infinite Scrolling Pages

Hi, You can see this thread for an answer that also applies to your use case. Abot/AbotX cannot

unread,

Infinite Scrolling Pages

Hi, You can see this thread for an answer that also applies to your use case. Abot/AbotX cannot

3/18/21

Sajjad Mortazavi, sjdi...@gmail.com2

3/16/21

No plans at this time. There is a very clean abstraction in place that allows another provider to

unread,

No plans at this time. There is a very clean abstraction in place that allows another provider to

3/16/21

Ridhima Shukla, sjdirect2

3/15/21

Add Keywords to URL

Would the crawl bag be what you are looking for? On Monday, March 15, 2021 at 4:50:33 AM UTC-7 rsh...

unread,

Add Keywords to URL

Would the crawl bag be what you are looking for? On Monday, March 15, 2021 at 4:50:33 AM UTC-7 rsh...

3/15/21

Anton Kheistver, … CS7

3/7/21

Best way to do page-by-page crawling

Thank-you! On Monday, January 25, 2021 at 10:45:56 PM UTC+2 sjdirect wrote: Hi, You can just create

unread,

Best way to do page-by-page crawling

Thank-you! On Monday, January 25, 2021 at 10:45:56 PM UTC+2 sjdirect wrote: Hi, You can just create

3/7/21

John Ligtenberg, sjdi...@gmail.com5

2/23/21

IIS crashes on cancellation

Anyone that cancels a task using a cancellationtoken (nothing to do with Abot/Abotx) will have to

unread,

IIS crashes on cancellation

Anyone that cancels a task using a cancellationtoken (nothing to do with Abot/Abotx) will have to

2/23/21

ghislain borremans, sjdi...@gmail.com3

2/22/21

Using Abot2Demo:crawledPage.HttpRequestException. on https://aanhangwagenspattyn.be/ while ok in browser

Discussed in github issue On Sat, Feb 20, 2021 at 4:16 AM ghislain borremans <ghislainborremans@

unread,

Using Abot2Demo:crawledPage.HttpRequestException. on https://aanhangwagenspattyn.be/ while ok in browser

Discussed in github issue On Sat, Feb 20, 2021 at 4:16 AM ghislain borremans <ghislainborremans@

2/22/21

sjdi...@gmail.com, John Ligtenberg2

2/16/21

Re: CrawlResult not returned if MaxPagesToCrawl is set ?

This problem has been solved, it was caused by a locked session. I needed to add [SessionState(

unread,

Re: CrawlResult not returned if MaxPagesToCrawl is set ?

This problem has been solved, it was caused by a locked session. I needed to add [SessionState(

2/16/21

ghislain borremans3

2/15/21

Overriding method is not called SchedulePageLinks(CrawledPage crawledPage)

Solved: i was mistaken by the class and method name. I renamed the class to "MyPoliteWebCrawler

unread,

Overriding method is not called SchedulePageLinks(CrawledPage crawledPage)

Solved: i was mistaken by the class and method name. I renamed the class to "MyPoliteWebCrawler

2/15/21

Search

Clear search

Close search

Google apps

Main menu