Akka.Cluster.Sharding unexpected messages in HashCodeMessageExtractor

[Binelli]

Hello,

I have an application using Akka Cluster Sharding. When I enable remember-entities and start some actors, I get some unexpceted messages going through my HashCodeMessageExtractor implementation - public override string EntityId(object message). I observed messages of type Akka.DistributedData.UpdateSuccess, Akka.Actor.Terminated and Akka.Cluster.Sharding.Passivate. Is this expected? How should I handle these type of messages? I'm using state-store-mode = ddata

[Aaronontheweb]

Hi @Binelli - I've never seen that happen before and those messages should not be going through the message extractor. The only explanation I can think of is that the ShardRegion is doing something improper and routing messages it should be handling itself through the extractor. Which version of Akka.NET are you using?

[Binelli]

Hey @Aaronontheweb

I caught this behaviour in an application using Akka.NET version 1.4.35 but I also created this little example program to isolate the "issue" using the latest version 1.4.37:

https://github.com/Binelli/AkkaTests/tree/master/Sharding

[Aaronontheweb]

Thanks, I'll take a look at this!

[Aaronontheweb]

Found the problem - these pattern matches in the switch statement aren't correct:

akka.net/src/contrib/cluster/Akka.Cluster.Sharding/DDataShard.cs

Lines 297 to 302 in 5233233

	case UpdateSuccess success when Equals((((Shard.StateChange, int))success.Request).Item1, e):
	Log.Debug("The DDataShard state was successfully updated with {0}", e);
	Context.UnbecomeStacked();
	afterUpdateCallback(e);
	Stash.UnstashAll();
	break;

So the messages that should be handled there are instead going to

akka.net/src/contrib/cluster/Akka.Cluster.Sharding/DDataShard.cs

Lines 327 to 334 in 5233233

	case var _ when ExtractEntityId(message).HasValue:
	this.DeliverMessage(message, Context.Sender);
	break;
	default:
	Log.Debug("Stashing unexpected message [{0}] while waiting for DDataShard update of {0}",
	message.GetType(), e);
	Stash.Stash();
	break;

That's why this traffic is appearing in your message extractor. Let me check and see if this is an issue on the dev branch now that we've just merged in a very large change to the design of the sharding system.

[Aaronontheweb]

This issue is for sure fixed by #5857 - just tested it locally. That's going into the 1.5 nightlies and I'm not sure I want to backport it to Akka.NET V1.4 since it involves a migration of sharding state data that, while harmless and robust, is irreversible once completed. That's why I felt it was important to put that into v1.5 - to set user expectations correctly.

I wonder if I can do something small in the short run to fix this inside the v1.4 branch and get a fix out.

[Aaronontheweb]

I took a stab at fixing this for v1.4, but unfortunately the issue is more complex there - the DDataShard's use of a behavior stack makes this issue somewhat difficult to debug performantly and requires re-architecting the way this class is designed, which is exactly what #5857 does. I think the right fix for this is to upgrade to the 1.5 nightlies and the public betas we'll push on NuGet once available. I can commit to getting the first v1.5 beta up on NuGet next week.

[Binelli]

That's fine. I'll try to upgrade when it's available. Thank you very much for the quick reply.