akka persistence - silent failure for non serializing messages - is that the best we can do?

[Tim Pigden]

I've just spent a few hours tracking down a mysterious failure in previously working code that used akka persistence.

Eventually I tracked it down to the fact that akka persistence silently swallows non-serializable messages with no warning.

Now I appreciate that

serialize-message = on

is a documented setting and it does the job.

But it's kinda hidden way down the serialization page in the docs.

And since at least 2 serializers (default java and contributed kyro) need no configuration, and there's a big red warning not to use it in production, it is quite likely that people will not see it or not think to turn it on.

So firstly, is it that hard to efficiently report on a failed attempt at serialization at run time, for normal use? After all - it's got to be a programming error hasn't it? At least in akka-persistence.

Secondly, perhaps users of akka persistence should get a strong recommendation in the docs to turn it on in their tests.

------------------------------------

“But the plans were on display…”
“On display? I eventually had to go down to the cellar to find them.”
“That’s the display department.”
“With a flashlight.”
“Ah, well, the lights had probably gone.”
“So had the stairs.”
“But look, you found the notice, didn’t you?”
“Yes,” said Arthur, “yes I did. It was on display in the bottom of a locked filing cabinet stuck in a disused lavatory with a sign on the door saying ‘Beware of the Leopard.” 

The Hitchhiker's Guide to the Galaxy

[Patrik Nordwall]

What do you mean with silently swallows? Either it will be logged as warning "Rejected to persist event type..." or as error "Failed to persist event type..." depending on if your journal plugin treats serialization issues as rejections or failures.

/Patrik

--
>>>>>>>>>> Read the docs: http://akka.io/docs/
>>>>>>>>>> Check the FAQ: http://doc.akka.io/docs/akka/current/additional/faq.html
>>>>>>>>>> Search the archives: https://groups.google.com/group/akka-user
---
You received this message because you are subscribed to the Google Groups "Akka User List" group.
To unsubscribe from this group and stop receiving emails from it, send an email to akka-user+...@googlegroups.com.
To post to this group, send email to akka...@googlegroups.com.
Visit this group at https://groups.google.com/group/akka-user.
For more options, visit https://groups.google.com/d/optout.

--

Patrik Nordwall
Typesafe -  Reactive apps on the JVM
Twitter: @patriknw

[Tim Pigden]

No warning logged that I could see. 

I "wrote" the thing out and when I ran recovery it wasn't there.

But investigations are still on-going as turning the messages on causes exceptions prior to this particular item as I'm apparently attempting to serialize things that I didn't think I was.

Is some form of warning part of the tck?

You received this message because you are subscribed to a topic in the Google Groups "Akka User List" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/akka-user/437oIzqVhTk/unsubscribe.
To unsubscribe from this group and all its topics, send an email to akka-user+...@googlegroups.com.

To post to this group, send email to akka...@googlegroups.com.
Visit this group at https://groups.google.com/group/akka-user.
For more options, visit https://groups.google.com/d/optout.

--

Tim Pigden
Optrak Distribution Software Limited
+44 (0)1992 517100
http://www.linkedin.com/in/timpigden
http://optrak.com
Optrak Distribution Software Ltd is a limited company registered in England and Wales.
Company Registration No. 2327613 Registered Offices: Suite 6,The Maltings, Hoe Lane, Ware, SG12 9LR England 
This email and any attachments to it may be confidential and are intended solely for the use of the individual to whom it is addressed. Any views or opinions expressed are solely those of the author and do not necessarily represent those of Optrak Distribution Software Ltd. If you are not the intended recipient of this email, you must neither take any action based upon its contents, nor copy or show it to anyone. Please contact the sender if you believe you have received this email in error.

[Tim Pigden]

Ok, light at the end of the tunnel, this is what fails.

I'd call that silently swallows.

Essentially the only indication of anything wrong is that the persistence stores nothing.

I think this is a plugin bug in the in-memory plugin and a thus by implication TCK issue. Should I raise the TCK issue?

Switching to postgres aysnc and I do get a proper error message. It doesn't tell me why but it does tell me it went wrong.

package com.optrak.vrp.ddd

import akka.actor.{ActorLogging, PoisonPill, Props}
import akka.persistence.PersistentActor
import com.optrak.opkakka.test.TestSupport.AkkaTestkitContext
import com.optrak.vrp.ddd.SimplePersistor.PersistMe
import org.specs2.mutable.Specification

/**
  * Created by tim on 21/02/16.
  */

object SimplePersistor {
  case class PersistMe(k: AnyRef)

  case class GotBack(kOpt: Option[AnyRef])

  case object Request

  def props[K] = Props(new SimplePersistor)
}
import SimplePersistor._

class SimplePersistor extends PersistentActor with ActorLogging {

  var local: Option[AnyRef] = None

  def handler(msg: PersistMe) = {
    local = Some(msg.k)
  }
  override def receiveRecover: Receive = {
    case pm: PersistMe =>
      local = Some(pm.k)
  }

  override def receiveCommand: Receive = {
    case pm: PersistMe =>
      persist(pm)(handler)
    case Request =>
      sender ! GotBack(local)

  }

  override def persistenceId: String = "simplePeristor"
}

class TestAkkaSerializability extends Specification {
  sequential //our set always has the same persistence id, so have to

  trait Checkit extends AkkaTestkitContext {

    def checkItOut(k: AnyRef) = {
      val persy1 = system.actorOf(SimplePersistor.props)
      persy1 ! Request
      expectMsg(GotBack(None))

      persy1 ! PersistMe(k)
      persy1 ! Request
      expectMsg(GotBack(Some(k)))

      persy1 ! PoisonPill
      Thread.sleep(200)

      val persy2 = system.actorOf(SimplePersistor.props)
      persy2 ! Request
      expectMsg(GotBack(Some(k)))

    }
  }

  "persistent set" should {
    "work with String" in new Checkit()  {
      checkItOut("ho")
    }

    "fails with unserializable" in new Checkit() {
      val x = 1
      case class Thing(y: Int)
      val tricky = new Thing(2) {
        def xMe = x
      }

      checkItOut(tricky)

    }

  }

}

assertion failed: expected GotBack(Some(Thing(2))), found GotBack(None)

java.lang.AssertionError: assertion failed: expected GotBack(Some(Thing(2))), found GotBack(None)

my log file

2016-02-21 12:08:42,871 - INFO - from akka.event.slf4j.Slf4jLogger Slf4jLogger started 
2016-02-21 12:08:43,160 - WARN - from akka.serialization.Serialization(akka://default) Using the default Java serializer for class [com.optrak.vrp.ddd.SimplePersistor$PersistMe] which is not recommended because of performance implications. Use another serializer or disable this warning using the setting 'akka.actor.warn-about-java-serializer-usage' 
2016-02-21 12:08:43,572 - INFO - from akka.event.slf4j.Slf4jLogger Slf4jLogger started 
2016-02-21 12:08:43,604 - WARN - from akka.serialization.Serialization(akka://default) Using the default Java serializer for class [com.optrak.vrp.ddd.SimplePersistor$PersistMe] which is not recommended because of performance implications. Use another serializer or disable this warning using the setting 'akka.actor.warn-about-java-serializer-usage' 

relevant bit of config

akka {
  loglevel = DEBUG
  logger-startup-timeout = 30s
  logging-filter = "akka.event.slf4j.Slf4jLoggingFilter"
  actor.debug.fsm = true
  loggers = ["akka.event.slf4j.Slf4jLogger"]

  extensions = ["com.romix.akka.serialization.kryo.KryoSerializationExtension$"]

  persistence {
    //    journal.plugin = "akka-persistence-sql-async.journal"
    //    snapshot-store.plugin = "akka-persistence-sql-async.snapshot-store"
    journal.plugin = "inmemory-journal"
    snapshot-store.plugin = "inmemory-snapshot-store"
    journal-plugin-fallback {
      replay-filter {
        mode = fail
      }
    }
  }

  //----------------- Kryo config ----------------------

  actor {
    serialize-messages = off

    serializers {
      java = "akka.serialization.JavaSerializer"
      # Define kryo serializer
      kryo = "com.romix.akka.serialization.kryo.KryoSerializer"
    }

    serialization-bindings {
      "java.io.Serializable" = java
    }

    kryo {
      type = "nograph"
      idstrategy = "default"
      kyro-trace = true
    }
  }
}

[Akka Team]

Hi Tim,

On Sun, Feb 21, 2016 at 1:23 PM, Tim Pigden <tim.p...@optrak.com> wrote:

Ok, light at the end of the tunnel, this is what fails.

I'd call that silently swallows.

Essentially the only indication of anything wrong is that the persistence stores nothing.

Ah, so the in-memory journal does not log an issue if serialization fails? Can y

 

I think this is a plugin bug in the in-memory plugin and a thus by implication TCK issue. Should I raise the TCK issue?

I don't know about the TCK part, but if the in-mem journal doesn't signal such failures properly then please open a ticket with a small reproducer.

Thanks

-Endre

 

--
>>>>>>>>>> Read the docs: http://akka.io/docs/
>>>>>>>>>> Check the FAQ: http://doc.akka.io/docs/akka/current/additional/faq.html
>>>>>>>>>> Search the archives: https://groups.google.com/group/akka-user
---
You received this message because you are subscribed to the Google Groups "Akka User List" group.
To unsubscribe from this group and stop receiving emails from it, send an email to akka-user+...@googlegroups.com.
To post to this group, send email to akka...@googlegroups.com.
Visit this group at https://groups.google.com/group/akka-user.
For more options, visit https://groups.google.com/d/optout.

--

Akka Team

Typesafe - Reactive apps on the JVM

Blog: letitcrash.com
Twitter: @akkateam

[Tim Pigden]

Hi - it's not your in-mem it's a third party one and I've logged an issue with them.

The point about the TCK is that it's there to ensure that we can trust the plugin providers to do a good job. So if there's an error in a plug-in that has an clearly identifiable cause then I would think that the error case ought to be added to the TCK so that you can help them make their solutions more robust.

You received this message because you are subscribed to a topic in the Google Groups "Akka User List" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/akka-user/437oIzqVhTk/unsubscribe.
To unsubscribe from this group and all its topics, send an email to akka-user+...@googlegroups.com.

To post to this group, send email to akka...@googlegroups.com.
Visit this group at https://groups.google.com/group/akka-user.
For more options, visit https://groups.google.com/d/optout.

--

[Akka Team]

I am not really involved with persistence plugins and TCK, so this is why I said I have no clue. Please file a ticket and then the whole team can discuss, others know way more about that part than me.

[Patrik Nordwall]

On Fri, Feb 26, 2016 at 12:47 PM, Tim Pigden <tim.p...@optrak.com> wrote:

Hi - it's not your in-mem it's a third party one and I've logged an issue with them.

The point about the TCK is that it's there to ensure that we can trust the plugin providers to do a good job. So if there's an error in a plug-in that has an clearly identifiable cause then I would think that the error case ought to be added to the TCK so that you can help them make their solutions more robust.

There is a test for rejections of non-serializable events, but it is possible to opt out from that feature. I don't know what the in-memory journal does? Should we then require that if they don't support rejections they must fail the persist if it's not serializable? I would imagine that an in-memory journal does not care about serialization.

--

Patrik Nordwall
Lightbend -  Reactive apps on the JVM
Twitter: @patriknw

[Tim Pigden]

Well in this case they evidently do care about serialization - since the behaviour changes when serialization doesn't work - it's just the reporting that's wrong. 

I will add the issue (it's a nice simple test case - once I figured out the problem)

[Patrik Nordwall]

On Fri, Feb 26, 2016 at 1:35 PM, Tim Pigden <tim.p...@optrak.com> wrote:

Well in this case they evidently do care about serialization - since the behaviour changes when serialization doesn't work - it's just the reporting that's wrong. 

I will add the issue (it's a nice simple test case - once I figured out the problem)

Good, contributing to the TCK would be great.