Access to variable in a UntypedActor causes a permanent stall on the 55th onReceive

[Olddave]

Hi,

I have a subclass of an UntypedActor that I use a Future to access. It works for hundreds of accesses then on the same nth message stalls at the same line in a method inside the Actor each time. I get the following stack trace, not an exception

Thread [actor-system-akka.actor.default-dispatcher-2] (Suspended)
 ActorCell.invoke(Envelope) line: 490
 Dispatcher$$anon$1(Mailbox).processMailbox(int, long) line: 238
 Dispatcher$$anon$1(Mailbox).run() line: 220
 ForkJoinExecutorConfigurator$AkkaForkJoinTask.exec() line: 397
 ForkJoinExecutorConfigurator$AkkaForkJoinTask(ForkJoinTask<V>).doExec() line: 260
 ForkJoinPool$WorkQueue.runTask(ForkJoinTask<?>) line: 1339
 ForkJoinExecutorConfigurator$AkkaForkJoinPool(ForkJoinPool).runWorker(ForkJoinPool$WorkQueue) line: 1979
 MonitorableThreadFactory$AkkaForkJoinWorkerThread(ForkJoinWorkerThread).run() line: 107

The relevant parts of the code are 

...
 private HashMap<Integer, List<String>> PA72UIds = new HashMap<Integer, List<String>>();
...
...
 private UIdUsed getUIds(UIdsRequired msg) {
 UIdUsed result = new UIdUsed();


 if (msg.getMpeBuild().getSource() == SOURCE.OMC) {
 result.setuIds24(populateUIds(msg, PA24UIds, 1, OMCLookback));
 result.setuIds48(populateUIds(msg, PA48UIds, 2, OMCLookback));
 result.setuIds72(populateUIds(msg, PA72UIds, 3, OMCLookback));             //2
...
...
 private List<String> populateUIds(UIdsRequired msg,
 HashMap<Integer, List<String>> idHolder, int range, int lookback) {


 int numReq = msg.getNumRequired();
 // We split the MPEr across the 5 scan sources equally then in
 // proportion to product duration type
 numReq = numReq/5;
 List<String> avail = null;
 List<String> remaining = null;
 ArrayList<String> availUIds = new ArrayList<String>(0);


 int i = 0;
 while (availUIds.size() < numReq) {
 // Look one slot closer, after i increments > 0
 int lookbackSlot = msg.getMpeBuild().getSlot() - lookback + i;
 if (lookbackSlot >= 0) {
 avail = idHolder.get(lookbackSlot);
 if (avail != null && avail.size() >= numReq) {                //3
...
...
 @Override
 public void onReceive(Object msg) throws Exception {
 if (msg instanceof UIdUsed) {
 addToStore((UIdUsed) msg);
 } else if (msg instanceof UIdsRequired) {
 UIdUsed ids = getUIds((UIdsRequired) msg); //1
...

The call count of the Actor is 55 each time when the avail variable causes this issue. You can see in the comments I added //1 calls getUIds, in getUIds //2 calls populateUIds, the previous 2 calls you see both succeed and hit the same line where I later get the error. Then at //3 I get the stack trace you see above, it will stall there forever.

I really have no clue how to debug this as there is no exception, it just stops there and the timeout on the Future that sent the message triggers.

Can someone explain how you debug this type of issue?

Thx

David

[Akka Team]

Hi David,

This looks weird, but from this I cannot really see what the issue. What is clear that something very wrong happens in internal message processing. Is there something among your calls that can cause an InterrputedException? Are you using blocking calls? Are you using a non-default mailbox?

Does anything in that code using assert?

-Endre

 

Can someone explain how you debug this type of issue?

Thx

David

--
>>>>>>>>>> Read the docs: http://akka.io/docs/
>>>>>>>>>> Check the FAQ: http://doc.akka.io/docs/akka/current/additional/faq.html
>>>>>>>>>> Search the archives: https://groups.google.com/group/akka-user
---
You received this message because you are subscribed to the Google Groups "Akka User List" group.
To unsubscribe from this group and stop receiving emails from it, send an email to akka-user+...@googlegroups.com.
To post to this group, send email to akka...@googlegroups.com.
Visit this group at http://groups.google.com/group/akka-user.
For more options, visit https://groups.google.com/d/optout.

--

Akka Team

Typesafe - Reactive apps on the JVM

Blog: letitcrash.com
Twitter: @akkateam

[Olddave]

Hi,

I was testing other stuff and change a bunch of code, not related to this but altering the overall message passing sequences, and bingo I get a real exception

[ERROR] [10/02/2015 13:45:46.318] [actor-system-akka.actor.default-dispatcher-6] [akka://actor-system/user/metadata-actor/create-messages/create-PA/liveUIDS] null

java.util.ConcurrentModificationException

at java.util.ArrayList$SubList.checkForComodification(ArrayList.java:1169)

at java.util.ArrayList$SubList.size(ArrayList.java:998)

at com.******.*******.***tester.actors.LiveUIDs.populateUIds(LiveUIDs.java:208)

line 208 equates to the comment //3

I find this confusing as there is only a single LiveUIds Actor in the runtime. You can see there are 2 message types in the onReceive method and these both later the content of the ArrayList<String> that is 'avail'. MAybe I misunderstand the way messages are processed by an actor. I thought that they were taken off the mailbox quque in the order they arrived? 

[Olddave]

To specifically answer you questiomns. I use the default mailbox, I have not seen an InterrputedException at all. I am switching between using Futures and tell at various points as I work out the dynamic behaviour I need,

David

On Friday, 2 October 2015 12:28:38 UTC+1, Akka Team wrote:

[Viktor Klang]

The problem resides in the code you redacted.

Cheers,

√

[Akka Team]

Hi,

On Fri, Oct 2, 2015 at 3:26 PM, Olddave <stp...@gmail.com> wrote:

Hi,

I was testing other stuff and change a bunch of code, not related to this but altering the overall message passing sequences, and bingo I get a real exception

[ERROR] [10/02/2015 13:45:46.318] [actor-system-akka.actor.default-dispatcher-6] [akka://actor-system/user/metadata-actor/create-messages/create-PA/liveUIDS] null

java.util.ConcurrentModificationException

at java.util.ArrayList$SubList.checkForComodification(ArrayList.java:1169)

at java.util.ArrayList$SubList.size(ArrayList.java:998)

at com.******.*******.***tester.actors.LiveUIDs.populateUIds(LiveUIDs.java:208)

line 208 equates to the comment //3

I find this confusing as there is only a single LiveUIds Actor in the runtime. You can see there are 2 message types in the onReceive method and these both later the content of the ArrayList<String> that is 'avail'. MAybe I misunderstand the way messages are processed by an actor. I thought that they were taken off the mailbox quque in the order they arrived? 

Yes, they do, that is the whole point. Are you modifying anything from a Future? Please try to assemble a reproducer for the issue. From the code you pasted I cannot conclude anything particular. By providing a reproducer you can save us a lot of guesswork.

-Endre

[Olddave]

Hi,

I have created a cut down version of what I am trying to achieve. The app will simulate String messages that are created and go through 5 stages over a period of time, in the example l attached it is 96 time slots. So an id in a message is created in an initial pool before being used in Stage 1 then stored as a "LiveId" and then sometime later will move to Stage 2. In fact Stage 2 will look back a number of slots for an available "LiveId" and if there are not enough available in that lookback slot and either side of that slot for the number of messages required at that slot it will take (*) the remaining ids from the unused pool. The number of message to generate in a given slot varies, I have simulated a ramp up and ramp down in the number of messages required.

The problem I have is that using ActorRef.tell it skips whole sets of slot output for each stage. 

But with the example I have an additional problem, when it used a Future to get unused Ids as described above (*) it is always a DeadLetter, no idea why. I think I must misunderstand something as I expected the Actor queue to process in order and I make sure all stateful info is in the message passed and is used from the received actor message. If this was the case it would not skip messages in the mailbox, or whatever it is that causes it to miss outputting for a given slot and stage. 

Any explanation of why it does not behave as I have understood the documentation is welcome.

[Olddave]

I expected DeadLetters to be rare, they happen all of the time. I log them but there is no visibility of why they occur and thus no hint about how we prevent them. I changed the design in the example I have given to overcome most but not all issues. The LiveIds and UnuserIds classes are no longer actors but fields in CreateContent. I find the semantics of guaranteed delivery way too complex. Might have a go at the Kontraktor library as it has much simpler syntax for guaranteed delivery where you need it.