Can't get backpressure to work...

[Roy Russo]

Scenario: Read from one database using an ActorPublisher, write to another database using a subscriber.

I expect the reads to be much faster than the writes, so we need to slow down the reads at some threshold. Growing an unbounded queue of data, will simply OOM. The below works for small datasets. With large datasets, the gap between read-write becomes enormous and so OOM. 

My ActorPublisher:

class ScrollPublisher(clientFrom: ElasticClient, config: Config) extends ActorPublisher[SearchHits] {

  val logger = Logger(LoggerFactory.getLogger(this.getClass))
  var readCount = 0
  var processing = false

  import akka.stream.actor.ActorPublisherMessage._

  @volatile var executeQuery = () => clientFrom.execute {
    search in config.indexFrom / config.mapping scroll "30m" limit config.scrollSize
  }

  def nextHits(): Unit = {
    if (!processing) {
      processing = true
      val future = executeQuery()
      future.foreach {
        response =>
          processing = false
          if (response.getHits.hits.nonEmpty) {
            logger.info("Fetched: \t" + response.getHits.getHits.length + " documents in\t" + response.getTookInMillis + "ms.")
            readCount += response.getHits.getHits.length
            logger.info("Total Fetched:\t" + readCount)
            if (isActive && totalDemand > 0) {
              executeQuery = () => clientFrom.execute {
                searchScroll(response.getScrollId).keepAlive("30m")
              }
              nextHits()
              onNext(response.getHits) // sends elements to the stream
            }
          } else {
            onComplete()
          }
      }
      future.onFailure {
        case t =>
          processing = false
          throw t
      }
    }
  }

  def receive = {
    case Request(cnt) =>
      logger.info("ActorPublisher Received: \t" + cnt)
      if (isActive && totalDemand > 0) {
        nextHits()
      }
    case Cancel =>
      context.stop(self)
    case _ =>
  }
}

Enter code here...

Source declaration:

// SearchHits Akka Stream Source
val documentSource = Source.actorPublisher[SearchHits](Props(new ScrollPublisher(clientFrom, config))).map {
  case searchHits =>
    searchHits.getHits
}

My Sink, which performs an asynch write to the new database: 

documentSource.buffer(16, OverflowStrategy.backpressure).runWith(Sink.foreach {
  searchHits =>
    Thread.sleep(1000)
    totalRec += searchHits.size
    logger.info("\t\t\tRECEIVED: " + searchHits.size + " \t\t\t TOTAL RECEIVED: "+ totalRec)
    val bulkIndexes = searchHits.map(hit => (hit.`type`, hit.id, hit.sourceAsString())).collect {
      case (typ, _id, source) =>
        index into config.indexTo / config.mapping id _id -> typ doc JsonDocumentSource(source)
    }
    val future = clientTo.execute {
      bulk(
        bulkIndexes
      )
    }

The sleep is put in there to simulate lag for local development. I've tried changing values for the buffer, and the max/initial values for the materializer, and still it seems to ignore back pressure. 

Is there a logic flaw in this code?

[Endre Varga]

Hi,

Your actor code is faulty, you are closing over the actor's state in Future callbacks, namely future.foreach and future.onFailure. This is not thread safe and will fail in various interesting ways. 

There is not much reason to use an ActorPublisher for this to be honest. There are built-in combinators to achieve similar things with less chance for mistakes:

http://doc.akka.io/api/akka/2.4/index.html#akka.stream.scaladsl.Source$@unfoldResource[T,S](create:()=>S,read:S=>Option[T],close:S=>Unit):akka.stream.scaladsl.Source[T,akka.NotUsed]

http://doc.akka.io/api/akka/2.4/index.html#akka.stream.scaladsl.Source$@unfoldResourceAsync[T,S](create:()=>scala.concurrent.Future[S],read:S=>scala.concurrent.Future[Option[T]],close:S=>scala.concurrent.Future[akka.Done]):akka.stream.scaladsl.Source[T,akka.NotUsed]

-Endre

--
>>>>>>>>>> Read the docs: http://akka.io/docs/
>>>>>>>>>> Check the FAQ: http://doc.akka.io/docs/akka/current/additional/faq.html
>>>>>>>>>> Search the archives: https://groups.google.com/group/akka-user
---
You received this message because you are subscribed to the Google Groups "Akka User List" group.
To unsubscribe from this group and stop receiving emails from it, send an email to akka-user+...@googlegroups.com.
To post to this group, send email to akka...@googlegroups.com.
Visit this group at https://groups.google.com/group/akka-user.
For more options, visit https://groups.google.com/d/optout.

[Roy Russo]

Thanks Endre,

I have indeed found it failing in interesting ways, and impossible to debug. 

I'll look at the unfold APIs... http://doc.akka.io/docs/akka/snapshot/scala/stream/stages-overview.html#unfold