Do I need to dispose() of an IScheduledFuture if I don't care about the result?

[sc...@indosoft.com]

I'm submitting many fire-and-forget tasks that are scheduled to "fire" after a specified delay.  From reading the documentation, it looks like I need to unconditionally call IScheduledFuture.dispose() according to an example in 10.3.2 of https://docs.hazelcast.org/docs/3.12.2/manual/html-single/index.html#scheduled-exec-srv-examples:

future.dispose(); // Always dispose futures that are not in use any more, to release resources

My problem is that the only thing that knows about the task (hence the "forget" part) is the task itself, which ends up being responsible for its own disposal.  The most convenient time to dispose of it ends up being within a transaction.  If I call IScheduledFuture.dispose() from within the running task thread, I invariably end up with a ThreadInterruptedException at commit time.  This forces me to pass the Future back along to the eventual parent caller, outside the transaction, to dispose of it only immediately before my Runnable.run() or Callable.call() method returns. This is invariably outside the transaction, which can be difficult to implement.  Is there a simpler solution to these fire-and-forget delayed tasks?

[Thomas Kountis]

Hi Scott, you are absolutely right on your findings. A Future must be disposed to release resources, and this /still/ is the responsibility of the caller.
The reason this happens is the API itself which returns a Future object, and due to the distributed nature of the application, its not easy to track

the reference of the ScheduledFuture and whether its in use or not. This means that if you indeed hold a reference to that, and we auto-dispose, you

will end up with a "dead" Future object.

Having said that, I have seen the same thing been asked 2-3 times, so it seems like a useful feature.

I will have another look at it, to see whether we can include this with minimal effort, but feel free to contribute a solution, we appreciate them :-)

At the moment your only option would be to create another repeatable task, ie. 'task-cleaner' which periodically checks, which Futures are completed, and dispose them.

This means that you have to keep track of the futures in a list, and read-and-remove entries from that list within your task-cleaner. 
An alternative approach would be to wrap all your Runnables/Callables in an AutoDisposable class of yours, which can use the DurableExecutorService to execute a dispose task.
This is a bit more complex for my taste, so its up to you.

--
You received this message because you are subscribed to the Google Groups "Hazelcast" group.
To unsubscribe from this group and stop receiving emails from it, send an email to hazelcast+...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/hazelcast/67d1a325-8993-4404-a0a5-b1f51eb483b9%40googlegroups.com.

This message contains confidential information and is intended only for the individuals named. If you are not the named addressee you should not disseminate, distribute or copy this e-mail. Please notify the sender immediately by e-mail if you have received this e-mail by mistake and delete this e-mail from your system. E-mail transmission cannot be guaranteed to be secure or error-free as information could be intercepted, corrupted, lost, destroyed, arrive late or incomplete, or contain viruses. The sender therefore does not accept liability for any errors or omissions in the contents of this message, which arise as a result of e-mail transmission. If verification is required, please request a hard-copy version. -Hazelcast

[Belen Ibanez]

Hi Thomas,

We are having the same issue  and case scenario than Scott, so this feature would be very useful for us too.

We were no disposing the tasks either but until recently we didn't get the error: "Maximum capacity (100) of tasks reached, for scheduled executor (scheduler). Reminder that tasks must be disposed if not needed."

I am wondering if some kind of tasks are clean up somehow. We were using the default capacity for tasks (100) but I saw in the logs that we had even 1000 different tasks scheduled on one day so if the tasks were no disposed somehow it is clear that we would had had errors saying that we were over the limit but we didn't get them. These task are not named. We started having issues with the limit when we added other named tasks. Could it be related? Maybe tasks with name are not dispose while tasks without are... I am puzzle about this. Any insight you could share is highly appreciate it :)

Thanks,

Belen

On Sunday, 17 May 2020 09:53:23 UTC+1, Thomas Kountis wrote:

Hi Scott, you are absolutely right on your findings. A Future must be disposed to release resources, and this /still/ is the responsibility of the caller.
The reason this happens is the API itself which returns a Future object, and due to the distributed nature of the application, its not easy to track

the reference of the ScheduledFuture and whether its in use or not. This means that if you indeed hold a reference to that, and we auto-dispose, you

will end up with a "dead" Future object.

Having said that, I have seen the same thing been asked 2-3 times, so it seems like a useful feature.

I will have another look at it, to see whether we can include this with minimal effort, but feel free to contribute a solution, we appreciate them :-)

At the moment your only option would be to create another repeatable task, ie. 'task-cleaner' which periodically checks, which Futures are completed, and dispose them.

This means that you have to keep track of the futures in a list, and read-and-remove entries from that list within your task-cleaner. 
An alternative approach would be to wrap all your Runnables/Callables in an AutoDisposable class of yours, which can use the DurableExecutorService to execute a dispose task.
This is a bit more complex for my taste, so its up to you.

On Sat, May 16, 2020 at 1:04 PM <sc...@indosoft.com> wrote:

I'm submitting many fire-and-forget tasks that are scheduled to "fire" after a specified delay.  From reading the documentation, it looks like I need to unconditionally call IScheduledFuture.dispose() according to an example in 10.3.2 of https://docs.hazelcast.org/docs/3.12.2/manual/html-single/index.html#scheduled-exec-srv-examples:

future.dispose(); // Always dispose futures that are not in use any more, to release resources

My problem is that the only thing that knows about the task (hence the "forget" part) is the task itself, which ends up being responsible for its own disposal.  The most convenient time to dispose of it ends up being within a transaction.  If I call IScheduledFuture.dispose() from within the running task thread, I invariably end up with a ThreadInterruptedException at commit time.  This forces me to pass the Future back along to the eventual parent caller, outside the transaction, to dispose of it only immediately before my Runnable.run() or Callable.call() method returns. This is invariably outside the transaction, which can be difficult to implement.  Is there a simpler solution to these fire-and-forget delayed tasks?

--
You received this message because you are subscribed to the Google Groups "Hazelcast" group.

To unsubscribe from this group and stop receiving emails from it, send an email to haze...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/hazelcast/67d1a325-8993-4404-a0a5-b1f51eb483b9%40googlegroups.com.

[Thomas Kountis]

Hi Belen,

Thanks for your comment here.
The naming of the tasks certainly plays a role in the behavior you are seeing but not due to disposal.
The maximum capacity of 100, is per partition, so if you have 271 (default partition count) partitions, then you can have thousands of tasks without seeing this exception.
When you start naming tasks, you are forcing them to reside on a particular partition (the one that the name hashes to), thus its easier to hit that error. 

FWIT In Hazelcast 4.0 we introduced capacity setting per-node as well, which makes this restriction much more predictable.

To recap, disposal of the tasks is never automatic, since the API returns a Future, the caller is responsible for disposing it.
I created a feature request (https://github.com/hazelcast/hazelcast/issues/16994), feel free to upvote it, or even contribute to it :-)

Hope this helps

To unsubscribe from this group and stop receiving emails from it, send an email to hazelcast+...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/hazelcast/55293058-1e90-45c2-a1d2-f9d9be001f0b%40googlegroups.com.

[sc...@indosoft.com]

Hi Thomas,

Thanks very much for your reply.  It really clarifies things.  I've always wondered why IScheduledFuture required a dispose() whereas DurableExecutorServiceFuture doesn't.  It looks like durable executors use a ring buffer (completely different from the Ringbuffer distributed object).  They also run tasks ASAP. So if you don't acknowledge that you're finished with a durable executor task's result, it'll get overwritten anyways. I'm just not 100% sure what happens if I try to call Future.get() from a durable executor task that's no longer there.  And it has to go away eventually, all my durable executors have the default maximum capacity.

Whereas the scheduled executor task needs to stick around for at least as long as it takes for the task to complete.  So now Hazelcast would have to somehow decide which yet-to-be-acknowledged results need to be overwritten, but the tasks will all execute at arbitrary times depending on what you specified for "delay" (or never even completely finish, depending on what you gave scheduleAtFixedRate()).  A scheduled task could complete in one moment, get overwritten in the next millisecond, and then a client could do IScheduledFuture.get() the next millisecond and get an error.

Ultimately what I'd be looking for is a way to express to Hazelcast that the only result I care about is the guarantee that the task will run (which Hazelcast provides), with the understanding that I do NOT want the result if it runs successfully and I also do NOT want the result if it terminates with a Throwable. So once my run() or call() returns, that should be the end of it. Many of my own tasks are wrapped with try/catch that lead to a new task being submitted--so I can control on what partition the success/error logic runs.  If the task can't run because nodes keep failing and it keeps having to retry, there's nothing I want to do to respond to that, I just trust that Hazelcast will continue to retry as long as that run()/call() method doesn't return.

Your insight about NamedTask and the partition each one resides on is interesting.  Most of my tasks implement NamedTask but also implement PartitionAware (or are scheduled using scheduleOnKeyOwner()). Though I've also been making sure to dispose() of all the futures one way or another :).  The trickiest one is if one of my scheduled tasks needs to re-schedule itself.  I like to cancel it non-transactionally and then re-schedule it as part of a transaction synchronization.  That doesn't work in order because a scheduled task cancelling itself within a transaction will result in an InterruptedException if it tries to grab a transactional resource during commit (like a DB connection.  That was a fun day)

I'm hoping that if I both cancel the task and reschedule it as part of Synchronization.afterCompletion() I'll avoid the complexity of bubbling the IScheduledFuture up to the topmost caller and still avoid InterruptedExceptions.  I'm also hoping I can figure out how to check out and build Hazelcast on my own so I can start contributing with more than just lengthy forum posts.

I'm also looking forward to TransactionalDurableExecutorService and TransactionalScheduledExecutorService with two-phase commit.  A guy can dream.

Thanks again for your insight.

- Scott

On Tuesday, May 19, 2020 at 7:43:38 AM UTC-3, Thomas Kountis wrote:

Hi Belen,

Thanks for your comment here.
The naming of the tasks certainly plays a role in the behavior you are seeing but not due to disposal.
The maximum capacity of 100, is per partition, so if you have 271 (default partition count) partitions, then you can have thousands of tasks without seeing this exception.
When you start naming tasks, you are forcing them to reside on a particular partition (the one that the name hashes to), thus its easier to hit that error. 

FWIT In Hazelcast 4.0 we introduced capacity setting per-node as well, which makes this restriction much more predictable.

To recap, disposal of the tasks is never automatic, since the API returns a Future, the caller is responsible for disposing it.
I created a feature request (https://github.com/hazelcast/hazelcast/issues/16994), feel free to upvote it, or even contribute to it :-)

Hope this helps

To view this discussion on the web visit https://groups.google.com/d/msgid/hazelcast/55293058-1e90-45c2-a1d2-f9d9be001f0b%40googlegroups.com.

[Thomas Kountis]

You seem to have a solid understanding of the design now, good job, I believe that's a good motive to start contributing.

 

I'm hoping that if I both cancel the task and reschedule it as part of Synchronization.afterCompletion() I'll avoid the complexity of bubbling the IScheduledFuture up to the topmost caller and still avoid InterruptedExceptions.  

I'm also hoping I can figure out how to check out and build Hazelcast on my own so I can start contributing with more than just lengthy forum posts.

It's pretty straight forward really, once you fork & clone the project from Github, you can import it on your favourite IDE. It relies on very few external dependencies, so you should have no problem building it with maven.
Once you open a PR, I will gladly help you on any issues or design decisions. Looking forward to it, already :-)

I'm also looking forward to TransactionalDurableExecutorService and TransactionalScheduledExecutorService with two-phase commit.  A guy can dream.

Not just a guy, all of us can dream :-D  

To unsubscribe from this group and stop receiving emails from it, send an email to hazelcast+...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/hazelcast/5bfd803d-6158-45fa-ada8-c36e1c759b78%40googlegroups.com.