How do you tell your crawler service that all RPCs are complete?

[Shawn Johnson]

When trying to use the out-of-the-box Crawler Service, we were not seeing the performance and consistency needed - mostly because of threading and memory issues related to HTMLUnit.  We are currently working on a different implementation.  I was wondering how others out there have implemented a solution to tell the rendering service (headless browser) that all RPC requests are complete?

[Christian Goudreau]

I'm interested with that, we mostly used the crawler service for websites and never really had to revisit this yet. I'm pretty sure we will end up tackling that kind of problems really soon as well.

On Tue, Jan 21, 2014 at 11:46 AM, Shawn Johnson <shawnjo...@gmail.com> wrote:

When trying to use the out-of-the-box Crawler Service, we were not seeing the performance and consistency needed - mostly because of threading and memory issues related to HTMLUnit.  We are currently working on a different implementation.  I was wondering how others out there have implemented a solution to tell the rendering service (headless browser) that all RPC requests are complete?

--
You received this message because you are subscribed to the Google Groups "GWTP" group.
To unsubscribe from this group and stop receiving emails from it, send an email to gwt-platform...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

--

Christian Goudreau | CEO - Président

M: 1.877.635.1585 | S: christian.goudreau

ArcBees | Blog | Facebook | LinkedIn

[Shawn Johnson]

Right now we are calling a custom method of what we call the "DispatchDelegate" - serves the same role as the DispatchAsync class, with an extra method to take the "caller".  Then in our Base Presenter, we count each 'started' RPC and each ended RPC until we get to zero.  Well, that's the simplest explanation for now.  Having some issues upgrading from 0.6 to 1.0 that may be related to this customization - that's part of why I reached out to see if there might be an different/better way that someone else has tried.

For rendering we are using a PhantomJS based solution - though this is still a work in progress.