Actor supervised by BackoffSupervisor loses stashed messages after restart

Question

I have an actor with stash usage. Sometimes, when it crashes, it loses all stashed messages. I found that it depends on what supervision logic I use.

I wrote a simple example.

An actor with the stash:

case object WrongMessage
case object TestMessage
case object InitialMessage

class TestActor extends Actor with Stash {
  override def receive: Receive = uninitializedReceive

  def uninitializedReceive: Receive = {
    case TestMessage =>
      println(s"stash test message")
      stash()

    case WrongMessage =>
      println(s"wrong message")
      throw new Throwable("wrong message")

    case InitialMessage =>
      println(s"initial message")
      context.become(initializedReceive)
      unstashAll()
  }

  def initializedReceive: Receive = {
    case TestMessage =>
      println(s"test message")
  }
}

In the following code, TestActor never receives stashed TestMessage:

object Test1 extends App {
  implicit val system: ActorSystem = ActorSystem()
  val actorRef = system.actorOf(BackoffSupervisor.props(Backoff.onFailure(
      Props[TestActor], "TestActor", 1 seconds, 1 seconds, 0
  ).withSupervisorStrategy(OneForOneStrategy()({
    case _ => SupervisorStrategy.Restart
  }))))
  actorRef ! TestMessage
  Thread.sleep(5000L)
  actorRef ! WrongMessage
  Thread.sleep(5000L)
  actorRef ! InitialMessage
}

But this code works well:

class SupervisionActor extends Actor {
  val testActorRef: ActorRef = context.actorOf(Props[TestActor])

  override def supervisorStrategy: SupervisorStrategy = OneForOneStrategy()({
    case _ => SupervisorStrategy.Restart
  })

  override def receive: Receive = {
    case message => testActorRef forward message
  }
}

object Test2 extends App {
  implicit val system: ActorSystem = ActorSystem()
  val actorRef = system.actorOf(Props(classOf[SupervisionActor]))
  actorRef ! TestMessage
  Thread.sleep(5000L)
  actorRef ! WrongMessage
  Thread.sleep(5000L)
  actorRef ! InitialMessage
}

I looked into sources and found that actor supervision uses LocalActorRef.restart method which backed by system dispatcher logic, but BackoffSupervisor simply creates a new actor after termination of the old one. Is there any way to work around it?

Answer 1

I'm not sure one can make restart under BackoffSupervisor properly send stashed messages without some custom re-implementation effort.

As you've already pointed out that BackoffSupervisor does its own restart that bypasses the standard actor lifecycle. In fact, it's explicitly noted in the BackoffOnRestartSupervisor source code:

Whatever the final Directive is, we will translate all Restarts to our own Restarts, which involves stopping the child.

In case you haven't read about this reported issue, it has a relevant discussion re: problem with Backoff.onFailure.

Backoff.onStop would also give the wanted BackoffSupervisor feature, but unfortunately it has its own use cases and won't be triggered by an exception.