Akka | Work - Pull pattern running into problem !

[Kunal Ghosh]

Hi,

I am using Akka with work pulling pattern (producer-consumer) in my application. (Ref - https://github.com/MartinKanters/java-akka-workpulling-example)

I have single producer and multiple consumer. My machine has 16 cores and 32 threads cpu. The number of consumer is configurable.

When I give more than 8 consumers then everthing works fine. But as soon as I lower that number to 4 then a issue comes in

Issue - 

 Java HotSpot(TM) 64-Bit Server VM warning: INFO: os::commit_memory(0x00007f2136580000, 514326528, 0) failed; error='Cannot allocate memory' (errno=12)

#

# There is insufficient memory for the Java Runtime Environment to continue.

# Native memory allocation (mmap) failed to map 514326528 bytes for committing reserved memory.

# An error report file with more information is saved as:

# /opt/app/akka/tomcat/hs_err_pid3244.log

application.config file ----

akka{

actor {

deployment{

  default {

dispatcher = "worker-dispatcher"

  }

}

}

}

  worker-dispatcher {

   type = Dispatcher

  executor = "fork-join-executor"

  fork-join-executor {

        parallelism-min = 2

       parallelism-factor = 2.0

   parallelism-max = 64

   }

   throughput = 5

   } 

    

    

Can you tell me what can be the cause of this issue? Please Help !!

[Bwmat]

Sounds like your producers are out running your consumers?

Have you looked at akka streams?

[Kunal Ghosh]

yes producer is fast and consumers can not keep up with it that is why i used work-pulling pattern.

Does akka stream has solution for producer out running the consumer??? or is there any other option???

[Justin du coeur]

There's basically no solution to the producer outrunning the consumers.  The point of Akka Streams is to signal upstream that things are fully loaded, so that the producer knows to slow down; that makes the whole system more robust, and puts the problem where it belongs, at the head end.

But if there is no way to slow down production, you *have* to build in enough capacity to handle it, or sooner or later you're going to run out of buffer space -- there's no magical solution to that...

--
>>>>>>>>>> Read the docs: http://akka.io/docs/
>>>>>>>>>> Check the FAQ: http://doc.akka.io/docs/akka/current/additional/faq.html
>>>>>>>>>> Search the archives: https://groups.google.com/group/akka-user
---
You received this message because you are subscribed to the Google Groups "Akka User List" group.
To unsubscribe from this group and stop receiving emails from it, send an email to akka-user+unsubscribe@googlegroups.com.
To post to this group, send email to akka...@googlegroups.com.
Visit this group at https://groups.google.com/group/akka-user.
For more options, visit https://groups.google.com/d/optout.

[Bwmat]

If, on average, your producers Don't outrun your producers, and only do so in limited bursts, you could try storing some of the buffered data on disk, until the burst is over?

Hard to imagine it would be reliable though

[Justin du coeur]

Yeah -- this approach (which is fairly common) tends to mask the underlying problem until things get a little more pathological, and then you just get a bigger explosion...

On Thu, Oct 12, 2017 at 4:20 PM, Bwmat <bwmat.r...@gmail.com> wrote:

If, on average, your producers Don't outrun your producers, and only do so in limited bursts, you could try storing some of the buffered data on disk, until the burst is over?

Hard to imagine it would be reliable though