PersistentShardCoordinator: Persistence failure when replaying events for persistenceId

[RyanGhd]

Akka.Cluster.Sharding.PersistentShardCoordinator throws exception when replaying the events.
It can connect to the database and create all tables related to persistence. It also sends the following query to the database :

SELECT DISTINCT e.PersistenceId as PersistenceId 
                FROM dbo.ShardingJournal e;

and after that fails with the following exception:

Error Akka.Cluster.Sharding.PersistentShardCoordinator: 
Persistence failure when replaying events for persistenceId [/user/sharding/FirmBillActorCoordinator/singleton/coordinator]. 
Last known sequence number [0] System.AggregateException: One or more errors occurred. 
---> System.MissingFieldException: Field not found: /Akka.Persistence.SnapshotSelectionCriteria.MaxSequenceNr/.
   at Akka.Persistence.Sql.Common.Snapshot.SqlSnapshotStore.<LoadAsync>d__20.MoveNext()
   at System.Runtime.CompilerServices.AsyncTaskMethodBuilder'1.Start[TStateMachine](TStateMachine& stateMachine)
   at Akka.Persistence.Sql.Common.Snapshot.SqlSnapshotStore.LoadAsync(String persistenceId, SnapshotSelectionCriteria criteria)
   at Akka.Persistence.Snapshot.SnapshotStore.<>c__DisplayClass5_0.<ReceiveSnapshotStore>b__0()
   at Akka.Util.Internal.AtomicState.<CallThrough>d__7'1.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at Akka.Util.Internal.AtomicState.<CallThrough>d__7'1.MoveNext()
   --- End of inner exception stack trace ---
---> (Inner Exception #0) System.MissingFieldException: Field not found: /Akka.Persistence.SnapshotSelectionCriteria.MaxSequenceNr/.
   at Akka.Persistence.Sql.Common.Snapshot.SqlSnapshotStore.<LoadAsync>d__20.MoveNext()
   at System.Runtime.CompilerServices.AsyncTaskMethodBuilder'1.Start[TStateMachine](TStateMachine& stateMachine)
   at Akka.Persistence.Sql.Common.Snapshot.SqlSnapshotStore.LoadAsync(String persistenceId, SnapshotSelectionCriteria criteria)
   at Akka.Persistence.Snapshot.SnapshotStore.<>c__DisplayClass5_0.<ReceiveSnapshotStore>b__0()
   at Akka.Util.Internal.AtomicState.<CallThrough>d__7'1.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at Akka.Util.Internal.AtomicState.<CallThrough>d__7'1.MoveNext()<---

Akka.net version: 1.2.3
platform windows: 10
db: sqlserver 2016

To reproduce the exception please run following sample app :
AkkaClusterTest.zip

logs:

Info AkkaClusterTest.Program: Starting the Main app
Info AkkaClusterTest.DaemonRunner: Run - finished. Billing Deamon activated.
Info Topshelf.HostFactory: Configuration Result:
[Success] Name Akkaclustertest
[Success] DisplayName AkkaClusterTest
[Success] Description AkkaClusterTest
[Success] ServiceName Akkaclustertest
Info Topshelf.HostConfigurators.HostConfiguratorImpl: Topshelf v4.0.0.0, .NET Framework v4.0.30319.42000
Debug Akka.Event.EventStream: subscribing [akka://BillingActorSystem/system/log1-NLogLogger#379829244] to channel Akka.Event.Debug
Debug Akka.Event.EventStream: subscribing [akka://BillingActorSystem/system/log1-NLogLogger#379829244] to channel Akka.Event.Info
Debug Akka.Event.EventStream: subscribing [akka://BillingActorSystem/system/log1-NLogLogger#379829244] to channel Akka.Event.Warning
Debug Akka.Event.EventStream: subscribing [akka://BillingActorSystem/system/log1-NLogLogger#379829244] to channel Akka.Event.Error
Debug Akka.Event.EventStream: Logger log1-NLogLogger [NLogLogger] started
Debug Akka.Event.EventStream: subscribing [akka://BillingActorSystem/system/UnhandledMessageForwarder#82248996] to channel Akka.Event.UnhandledMessage
Debug Akka.Event.EventStream: StandardOutLogger being removed
Debug Akka.Actor.SystemGuardianActor: now supervising akka://BillingActorSystem/system/UnhandledMessageForwarder
Debug Akka.Event.LoggingBus+UnhandledMessageForwarder: Started (Akka.Event.LoggingBus+UnhandledMessageForwarder)
Debug Akka.Event.EventStream: unsubscribing [akka://all-systems/] from all channels
Debug Akka.Event.EventStream: Default Loggers started
Info Akka.Event.DummyClassForStringSources: Starting remoting
Debug Akka.Actor.SystemGuardianActor: now supervising akka://BillingActorSystem/system/remoting-terminator
Debug Akka.Actor.SystemGuardianActor: now supervising akka://BillingActorSystem/system/transports
Debug Akka.Actor.SystemGuardianActor: now supervising akka://BillingActorSystem/system/endpointManager
Debug Akka.Remote.TransportSupervisor: Started (Akka.Remote.TransportSupervisor)
Debug Akka.Remote.RemoteActorRefProvider+RemotingTerminator: Started (Akka.Remote.RemoteActorRefProvider+RemotingTerminator)
Debug Akka.Event.DummyClassForStringSources: Starting prune timer for endpoint manager...
Debug Akka.Remote.EndpointManager: Started (Akka.Remote.EndpointManager)
Debug Akka.Remote.RemoteActorRefProvider+RemotingTerminator: now watched by [akka://BillingActorSystem/system]
Debug Akka.Remote.Transport.AkkaProtocolManager: Started (Akka.Remote.Transport.AkkaProtocolManager)
Debug Akka.Remote.TransportSupervisor: now supervising akka://BillingActorSystem/system/transports/akkaprotocolmanager.tcp.0
Info Akka.Event.DummyClassForStringSources: Remoting started; listening on addresses : [akka.tcp://BillingActorSystem@127.0.0.1:5053]
Info Akka.Event.DummyClassForStringSources: Remoting now listens on addresses: [akka.tcp://BillingActorSystem@127.0.0.1:5053]
Info Akka.Event.DummyClassForStringSources: Cluster Node [akka.tcp://BillingActorSystem@127.0.0.1:5053] - Starting up...
Debug Akka.Actor.SystemGuardianActor: now supervising akka://BillingActorSystem/system/cluster
Debug Akka.Actor.SystemGuardianActor: now supervising akka://BillingActorSystem/system/clusterEventBusListener
Debug Akka.Cluster.ClusterDaemon: Started (Akka.Cluster.ClusterDaemon)
Debug Akka.Cluster.ClusterDaemon: now supervising akka://BillingActorSystem/system/cluster/core
Debug Akka.Cluster.ClusterDaemon: now supervising akka://BillingActorSystem/system/cluster/heartbeatReceiver
Debug Akka.Cluster.ClusterCoreSupervisor: Started (Akka.Cluster.ClusterCoreSupervisor)
Debug Akka.Cluster.ClusterHeartbeatReceiver: Started (Akka.Cluster.ClusterHeartbeatReceiver)
Info Akka.Event.DummyClassForStringSources: Cluster Node [akka.tcp://BillingActorSystem@127.0.0.1:5053] - Started up successfully
Debug Akka.Cluster.ClusterCoreSupervisor: now supervising akka://BillingActorSystem/system/cluster/core/publisher
Debug Akka.Cluster.ClusterCoreSupervisor: now supervising akka://BillingActorSystem/system/cluster/core/daemon
Debug Akka.Cluster.ClusterReadView+EventBusListener: Started (Akka.Cluster.ClusterReadView+EventBusListener)
Debug Akka.Actor.SystemGuardianActor: now supervising akka://BillingActorSystem/system/remote-watcher
Debug Akka.Actor.SystemGuardianActor: now supervising akka://BillingActorSystem/system/remote-deployment-watcher
Debug Akka.Remote.RemoteDeploymentWatcher: Started (Akka.Remote.RemoteDeploymentWatcher)
Debug Akka.Actor.SystemGuardianActor: now supervising akka://BillingActorSystem/system/deadLetterListener
Debug Akka.Event.EventStream: subscribing [akka://BillingActorSystem/system/deadLetterListener#1347638266] to channel Akka.Event.DeadLetter
Debug Akka.Event.DeadLetterListener: Started (Akka.Event.DeadLetterListener)
Debug Akka.Actor.SystemGuardianActor: now supervising akka://BillingActorSystem/system/EventStreamUnsubscriber-1
Debug Akka.Event.EventStreamUnsubscriber: registering unsubscriber with Akka.Event.EventStream
Debug Akka.Event.EventStreamUnsubscriber: Started (Akka.Event.EventStreamUnsubscriber)
Debug Akka.Cluster.ClusterRemoteWatcher: Started (Akka.Cluster.ClusterRemoteWatcher)
Info Akka.Actor.Internal.ActorSystemImpl: {
  akka : {
    actor : {
      provider : "Akka.Cluster.ClusterActorRefProvider, Akka.Cluster"
      serializers : {
        hyperion : "Akka.Serialization.HyperionSerializer, Akka.Serialization.Hyperion"
      }
      serialization-bindings : {
        System.Object : hyperion
      }
      debug : {
        receive : on
        autoreceive : on
        lifecycle : on
        event-stream : on
        unhandled : on
      }
    }
    persistence : {
      journal : {
        plugin : akka.persistence.journal.sql-server
        sql-server : {
          class : "Akka.Persistence.SqlServer.Journal.SqlServerJournal, Akka.Persistence.SqlServer"
          plugin-dispatcher : akka.actor.default-dispatcher
          connection-string : "server=localhost; database=AkkaClusterTest; user=sa; password=123456; Application Name=AkkaClusterTest;"
          connection-timeout : 30s
          schema-name : dbo
          table-name : AkkaEventJournal
          auto-initialize : on
          timestamp-provider : "Akka.Persistence.Sql.Common.Journal.DefaultTimestampProvider, Akka.Persistence.Sql.Common"
          metadata-table-name : AkkaMetadata
        }
        sharding : {
          connection-string : "server=localhost; database=AkkaClusterTest; user=sa; password=123456; Application Name=AkkaClusterTest;"
          auto-initialize : on
          plugin-dispatcher : akka.actor.default-dispatcher
          class : "Akka.Persistence.SqlServer.Journal.SqlServerJournal, Akka.Persistence.SqlServer"
          connection-timeout : 30s
          schema-name : dbo
          table-name : ShardingJournal
          timestamp-provider : "Akka.Persistence.Sql.Common.Journal.DefaultTimestampProvider,
                                Akka.Persistence.Sql.Common"
          metadata-table-name : ShardingMetadata
        }
      }
      snapshot-store : {
        plugin : akka.persistence.snapshot-store.sql-server
        sql-server : {
          class : "Akka.Persistence.SqlServer.Snapshot.SqlServerSnapshotStore, Akka.Persistence.SqlServer"
          plugin-dispatcher : akka.actor.default-dispatcher
          connection-string : "server=localhost; database=AkkaClusterTest; user=sa; password=123456; Application Name=AkkaClusterTest;"
          connection-timeout : 30s
          schema-name : dbo
          table-name : AkkaSnapshotStore
          auto-initialize : on
        }
        sharding : {
          class : "Akka.Persistence.SqlServer.Snapshot.SqlServerSnapshotStore, Akka.Persistence.SqlServer"
          plugin-dispatcher : akka.actor.default-dispatcher
          connection-string : "server=localhost; database=AkkaClusterTest; user=sa; password=123456; Application Name=AkkaClusterTest;"
          connection-timeout : 30s
          schema-name : dbo
          table-name : ShardingSnapshotStore
          auto-initialize : on
        }
      }
    }
    loggers : ["Akka.Logger.NLog.NLogLogger, Akka.Logger.NLog"]
    stdout-loglevel : Error
    loglevel : DEBUG
    log-config-on-start : on
    remote : {
      log-remote-lifecycle-events : DEBUG
      log-received-messages : on
      log-sent-messages : on
      helios : {
        tcp : {
          transport-class : "Akka.Remote.Transport.Helios.HeliosTcpTransport, Akka.Remote"
          applied-adapters : []
          transport-protocol : tcp
          hostname : 127.0.0.1
          port : 5053
        }
      }
    }
    cluster : {
      seed-nodes : [akka.tcp://BillingActorSystem@127.0.0.1:5053]
      auto-down-unreachable-after : 5s
      roles : [billing]
      sharding : {
        least-shard-allocation-strategy : {
          rebalance-threshold : 3
        }
        journal-plugin-id : akka.persistence.journal.sharding
        snapshot-plugin-id : akka.persistence.snapshot-store.sharding
      }
    }
  }
}
Debug Akka.Actor.GuardianActor: now supervising akka://BillingActorSystem/user/EventJournalActor
Debug Akka.Cluster.ClusterDaemon: now supervising akka://BillingActorSystem/system/cluster/$a
Debug Akka.Cluster.OnMemberStatusChangedListener: Started (Akka.Cluster.OnMemberStatusChangedListener)
Debug Akka.Cluster.ClusterDomainEventPublisher: Started (Akka.Cluster.ClusterDomainEventPublisher)
Debug Akka.Cluster.Tools.Singleton.ClusterSingletonManager: Started (Akka.Cluster.Tools.Singleton.ClusterSingletonManager)
Debug Akka.Cluster.ClusterDaemon: now supervising akka://BillingActorSystem/system/cluster/$b
Debug Akka.Cluster.OnMemberStatusChangedListener: Started (Akka.Cluster.OnMemberStatusChangedListener)
Debug Akka.Event.EventStream: subscribing [akka://BillingActorSystem/system/cluster/core/daemon#1972584045] to channel Akka.Remote.QuarantinedEvent
Debug Akka.Cluster.ClusterCoreDaemon: Started (Akka.Cluster.ClusterCoreDaemon)
Debug Akka.Cluster.ClusterCoreDaemon: now watched by [akka://BillingActorSystem/system/cluster/core#476145235]
Debug Akka.Cluster.ClusterCoreDaemon: now supervising akka://BillingActorSystem/system/cluster/core/daemon/downingProvider
Debug Akka.Cluster.ClusterCoreDaemon: [Uninitialized] Received Akka.Cluster.InternalClusterAction+Subscribe
Debug Akka.Cluster.ClusterCoreDaemon: [Uninitialized] Received Akka.Cluster.InternalClusterAction+Subscribe
Debug Akka.Cluster.ClusterCoreDaemon: [Uninitialized] Received Akka.Cluster.InternalClusterAction+Subscribe
Debug Akka.Cluster.ClusterCoreDaemon: [Uninitialized] Received Akka.Cluster.InternalClusterAction+Subscribe
Debug Akka.Cluster.ClusterCoreDaemon: [Uninitialized] Received Akka.Cluster.InternalClusterAction+Subscribe
Debug Topshelf.Builders.RunBuilder: Running as a console application, creating the console host.
Debug Akka.Cluster.ClusterCoreDaemon: [Uninitialized] Received Akka.Cluster.InternalClusterAction+JoinSeedNodes
Debug Akka.Cluster.AutoDown: Started (Akka.Cluster.AutoDown)
Debug Akka.Cluster.ClusterCoreDaemon: [Uninitialized] Received Akka.Cluster.InternalClusterAction+Subscribe
Debug Akka.Cluster.ClusterCoreDaemon: [Uninitialized] Received Akka.Cluster.ClusterUserAction+JoinTo
Debug Topshelf.Hosts.ConsoleRunHost: Starting up as a console application
Info Topshelf.Hosts.ConsoleRunHost: The Akkaclustertest service is now running, press Control+C to exit.
Debug Akka.Event.EventStream: subscribing [akka://BillingActorSystem/system/clusterEventBusListener#1496646962] to channel Akka.Cluster.ClusterEvent+IClusterDomainEvent
Debug Akka.Event.EventStream: subscribing [akka://BillingActorSystem/system/remote-watcher#1969304559] to channel Akka.Cluster.ClusterEvent+IMemberEvent
Debug Akka.Event.EventStream: subscribing [akka://BillingActorSystem/system/cluster/$a#572461226] to channel Akka.Cluster.ClusterEvent+MemberUp
Debug Akka.Cluster.ClusterHeartbeatSender: Started (Akka.Cluster.ClusterHeartbeatSender)
Info Akka.Cluster.ClusterCoreDaemon: Node [akka.tcp://BillingActorSystem@127.0.0.1:5053] is JOINING, roles [billing]
Debug Akka.Event.EventStream: subscribing [akka://BillingActorSystem/user/EventJournalActor#1162070063] to channel Akka.Cluster.ClusterEvent+MemberExited
Debug Akka.Event.EventStream: subscribing [akka://BillingActorSystem/user/EventJournalActor#1162070063] to channel Akka.Cluster.ClusterEvent+MemberRemoved
Debug Akka.Event.EventStream: subscribing [akka://BillingActorSystem/system/cluster/$b#455212081] to channel Akka.Cluster.ClusterEvent+MemberUp
Debug Akka.Event.EventStream: subscribing [akka://BillingActorSystem/system/cluster/core/daemon/downingProvider#997615947] to channel Akka.Cluster.ClusterEvent+IClusterDomainEvent
Info Akka.Cluster.ClusterCoreDaemon: Leader is moving node [akka.tcp://BillingActorSystem@127.0.0.1:5053] to [Up]
Debug Akka.Cluster.ClusterCoreDaemon: now supervising akka://BillingActorSystem/system/cluster/core/daemon/heartbeatSender
Debug Akka.Cluster.ClusterCoreDaemon: [Initialized] Received Akka.Cluster.InternalClusterAction+Subscribe
Debug Akka.Event.EventStream: subscribing [akka://BillingActorSystem/system/cluster/core/daemon/heartbeatSender#67265529] to channel Akka.Cluster.ClusterEvent+IMemberEvent
Debug Akka.Event.EventStream: subscribing [akka://BillingActorSystem/system/cluster/core/daemon/heartbeatSender#67265529] to channel Akka.Cluster.ClusterEvent+IReachabilityEvent
Debug Akka.Cluster.Tools.Singleton.OldestChangedBuffer: Started (Akka.Cluster.Tools.Singleton.OldestChangedBuffer)
Debug Akka.Cluster.ClusterCoreDaemon: [Initialized] Received Akka.Cluster.InternalClusterAction+Subscribe
Debug Akka.Event.EventStream: subscribing [akka://BillingActorSystem/user/EventJournalActor/$a#1852262858] to channel Akka.Cluster.ClusterEvent+IMemberEvent
Debug Akka.Cluster.ClusterCoreDaemon: [Initialized] Received Akka.Cluster.InternalClusterAction+Unsubscribe
Debug Akka.Event.EventStream: unsubscribing [akka://BillingActorSystem/system/cluster/$b#455212081] from all channels
Debug Akka.Cluster.Tools.Singleton.ClusterSingletonManager: now supervising akka://BillingActorSystem/user/EventJournalActor/$a
Debug Akka.Actor.GuardianActor: now supervising akka://BillingActorSystem/user/sharding
Debug Akka.Cluster.OnMemberStatusChangedListener: Stopped
Debug Akka.Cluster.Sharding.ClusterShardingGuardian: Started (Akka.Cluster.Sharding.ClusterShardingGuardian)
Info Akka.Cluster.Tools.Singleton.ClusterSingletonManager: Singleton manager [akka.tcp://BillingActorSystem@127.0.0.1:5053] starting singleton actor
Info Akka.Cluster.Tools.Singleton.ClusterSingletonManager: ClusterSingletonManager state change [Start -> Oldest] Akka.Cluster.Tools.Singleton.Uninitialized
Debug Akka.Cluster.Tools.Singleton.ClusterSingletonManager: now supervising akka://BillingActorSystem/user/EventJournalActor/singleton
Debug AkkaClusterTest.Actors.EventJournalActor: Started (AkkaClusterTest.Actors.EventJournalActor)
Debug AkkaClusterTest.Actors.EventJournalActor: now watched by [akka://BillingActorSystem/user/EventJournalActor#1162070063]
Debug Akka.Cluster.ClusterCoreDaemon: [Initialized] Received Akka.Cluster.InternalClusterAction+Subscribe
Debug Akka.Cluster.Tools.Singleton.ClusterSingletonManager: Started (Akka.Cluster.Tools.Singleton.ClusterSingletonManager)
Debug Akka.Cluster.ClusterDaemon: now supervising akka://BillingActorSystem/system/cluster/$c
Debug Akka.Cluster.ClusterCoreDaemon: [Initialized] Received Akka.Cluster.InternalClusterAction+Subscribe
Debug Akka.Cluster.OnMemberStatusChangedListener: Started (Akka.Cluster.OnMemberStatusChangedListener)
Debug Akka.Event.EventStream: subscribing [akka://BillingActorSystem/user/sharding/FirmBillActorCoordinator#351197579] to channel Akka.Cluster.ClusterEvent+MemberExited
Debug Akka.Event.EventStream: subscribing [akka://BillingActorSystem/user/sharding/FirmBillActorCoordinator#351197579] to channel Akka.Cluster.ClusterEvent+MemberRemoved
Debug Akka.Event.EventStream: subscribing [akka://BillingActorSystem/system/cluster/$c#712360679] to channel Akka.Cluster.ClusterEvent+MemberUp
Debug Akka.Cluster.ClusterCoreDaemon: [Initialized] Received Akka.Cluster.InternalClusterAction+Unsubscribe
Debug Akka.Event.EventStream: unsubscribing [akka://BillingActorSystem/system/cluster/$c#712360679] from all channels
Debug Akka.Cluster.OnMemberStatusChangedListener: Stopped
Debug Akka.Cluster.Tools.Singleton.ClusterSingletonManager: now supervising akka://BillingActorSystem/user/sharding/FirmBillActorCoordinator/$a
Debug Akka.Cluster.Tools.Singleton.OldestChangedBuffer: Started (Akka.Cluster.Tools.Singleton.OldestChangedBuffer)
Debug Akka.Cluster.ClusterCoreDaemon: [Initialized] Received Akka.Cluster.InternalClusterAction+Subscribe
Debug Akka.Event.EventStream: subscribing [akka://BillingActorSystem/user/sharding/FirmBillActorCoordinator/$a#668777217] to channel Akka.Cluster.ClusterEvent+IMemberEvent
Info Akka.Cluster.Tools.Singleton.ClusterSingletonManager: Singleton manager [akka.tcp://BillingActorSystem@127.0.0.1:5053] starting singleton actor
Info Akka.Cluster.Tools.Singleton.ClusterSingletonManager: ClusterSingletonManager state change [Start -> Oldest] Akka.Cluster.Tools.Singleton.Uninitialized
Debug Akka.Cluster.Tools.Singleton.ClusterSingletonManager: now supervising akka://BillingActorSystem/user/sharding/FirmBillActorCoordinator/singleton
Debug Akka.Cluster.Sharding.ClusterShardingGuardian: now supervising akka://BillingActorSystem/user/sharding/FirmBillActorCoordinator
Debug Akka.Cluster.Sharding.ClusterShardingGuardian: now supervising akka://BillingActorSystem/user/sharding/FirmBillActor
Debug Akka.Cluster.ClusterCoreDaemon: [Initialized] Received Akka.Cluster.InternalClusterAction+Unsubscribe
Debug Akka.Event.EventStream: unsubscribing [akka://BillingActorSystem/system/cluster/$a#572461226] from all channels
Debug Akka.Cluster.OnMemberStatusChangedListener: Stopped
Debug Akka.Pattern.BackoffSupervisor: Started (Akka.Pattern.BackoffSupervisor)
Debug Akka.Pattern.BackoffSupervisor: now watched by [akka://BillingActorSystem/user/sharding/FirmBillActorCoordinator#351197579]
Debug Akka.Pattern.BackoffSupervisor: now supervising akka://BillingActorSystem/user/sharding/FirmBillActorCoordinator/singleton/coordinator
Debug Akka.Cluster.Sharding.ShardRegion: Started (Akka.Cluster.Sharding.ShardRegion)
Debug Akka.Cluster.ClusterCoreDaemon: [Initialized] Received Akka.Cluster.InternalClusterAction+Subscribe
Debug Akka.Event.EventStream: subscribing [akka://BillingActorSystem/user/sharding/FirmBillActor#525630732] to channel Akka.Cluster.ClusterEvent+IMemberEvent
Debug Akka.Cluster.Sharding.ShardRegion: Coordinator moved from [] to [akka.tcp://BillingActorSystem@127.0.0.1:5053]
Debug Akka.Cluster.ClusterCoreDaemon: [Initialized] Received Akka.Cluster.InternalClusterAction+Subscribe
Debug Akka.Event.EventStream: subscribing [akka://BillingActorSystem/user/sharding/FirmBillActorCoordinator/singleton/coordinator#1790677036] to channel Akka.Cluster.ClusterEvent+ClusterShuttingDown
Debug Akka.Actor.SystemGuardianActor: now supervising akka://BillingActorSystem/system/akka.persistence.journal.sharding
Debug Akka.Actor.SystemGuardianActor: now supervising akka://BillingActorSystem/system/akka.persistence.snapshot-store.sharding
Debug Akka.Actor.GuardianActor: now supervising akka://BillingActorSystem/user/$a
Debug Akka.Persistence.Journal.AsyncWriteJournal+Resequencer: Started (Akka.Persistence.Journal.AsyncWriteJournal+Resequencer)
Debug Akka.Cluster.Sharding.PersistentShardCoordinator: Started (Akka.Cluster.Sharding.PersistentShardCoordinator)
Debug Akka.Cluster.Sharding.PersistentShardCoordinator: now watched by [akka://BillingActorSystem/user/sharding/FirmBillActorCoordinator/singleton#983995935]
Debug Akka.Persistence.SqlServer.Journal.SqlServerJournal: Started (Akka.Persistence.SqlServer.Journal.SqlServerJournal)
Debug Akka.Persistence.SqlServer.Snapshot.SqlServerSnapshotStore: Started (Akka.Persistence.SqlServer.Snapshot.SqlServerSnapshotStore)
Error Akka.Cluster.Sharding.PersistentShardCoordinator: Persistence failure when replaying events for persistenceId [/user/sharding/FirmBillActorCoordinator/singleton/coordinator]. Last known sequence number [0] System.AggregateException: One or more errors occurred. ---> System.MissingFieldException: Field not found: 'Akka.Persistence.SnapshotSelectionCriteria.MaxSequenceNr'.
   at Akka.Persistence.Sql.Common.Snapshot.SqlSnapshotStore.<LoadAsync>d__20.MoveNext()
   at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1.Start[TStateMachine](TStateMachine& stateMachine)
   at Akka.Persistence.Sql.Common.Snapshot.SqlSnapshotStore.LoadAsync(String persistenceId, SnapshotSelectionCriteria criteria)
   at Akka.Persistence.Snapshot.SnapshotStore.<>c__DisplayClass5_0.<ReceiveSnapshotStore>b__0()
   at Akka.Util.Internal.AtomicState.<CallThrough>d__7`1.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at Akka.Util.Internal.AtomicState.<CallThrough>d__7`1.MoveNext()
   --- End of inner exception stack trace ---
---> (Inner Exception #0) System.MissingFieldException: Field not found: 'Akka.Persistence.SnapshotSelectionCriteria.MaxSequenceNr'.
   at Akka.Persistence.Sql.Common.Snapshot.SqlSnapshotStore.<LoadAsync>d__20.MoveNext()
   at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1.Start[TStateMachine](TStateMachine& stateMachine)
   at Akka.Persistence.Sql.Common.Snapshot.SqlSnapshotStore.LoadAsync(String persistenceId, SnapshotSelectionCriteria criteria)
   at Akka.Persistence.Snapshot.SnapshotStore.<>c__DisplayClass5_0.<ReceiveSnapshotStore>b__0()
   at Akka.Util.Internal.AtomicState.<CallThrough>d__7`1.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at Akka.Util.Internal.AtomicState.<CallThrough>d__7`1.MoveNext()<---

[Horusiath]

Stack trace suggests that the SnapshotSelectionCriteria.MaxSequenceNr field is missing (in terms of System.MissingFieldException). Something that is rather impossible to occur at compile time, so I guess you may have some problem with dependencies/assemblies loaded. Working version of akka.cluster.sharding has prerelease flag on it and it also depends on the akka.persistence, which currently has prerelease flag too.

[RyanGhd]

@Horusiath thank you for your quick reply.
tried different versions and the following packages are the latest versions that work properly in my case:

  Akka                         1.2.3        
  Akka.Cluster                 1.2.3        
  Akka.Cluster.Sharding        1.2.0.36-beta
  Akka.Cluster.Tools           1.2.3        
  Akka.Logger.NLog             1.2.0        
  Akka.Persistence             1.2.0.36-beta
  Akka.Persistence.Sql.Common  1.2.0.36-beta
  Akka.Persistence.SqlServer   1.1.1.7-beta 
  Akka.Remote                  1.2.3        
  Akka.Remote.Transport.Helios 1.2.3        
  Akka.Serialization.Hyperion  1.2.3.43-beta

Also tried packages released after the following versions and each version has some issues:

  Akka.Cluster.Sharding        1.2.0.36-beta
  Akka.Persistence             1.2.0.36-beta
  Akka.Persistence.Sql.Common  1.2.0.36-beta

[ia64mail]

I have the similar cause

System.MissingFieldException: Field not found: 'Akka.Persistence.SnapshotSelectionCriteria.MaxSequenceNr'.

but from the different stack trace

[ERROR][16/08/2017 22:17:06][Thread 0012][[akka://test/user/pra#590350025]] Persistence failure when replaying events for persistenceId [tracked-xxxs]. Last known sequence number [0]
Cause: System.MissingFieldException: Field not found: 'Akka.Persistence.SnapshotSelectionCriteria.MaxSequenceNr'.
   at Akka.Persistence.Redis.Snapshot.RedisSnapshotStore.<LoadAsync>d__6.MoveNext()
   at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1.Start[TStateMachine](TStateMachine& stateMachine)
   at Akka.Persistence.Redis.Snapshot.RedisSnapshotStore.LoadAsync(String persistenceId, SnapshotSelectionCriteria criteria)
   at Akka.Persistence.Snapshot.SnapshotStore.<>c__DisplayClass5_0.<ReceiveSnapshotStore>b__0()
   at Akka.Util.Internal.AtomicState.<CallThrough>d__7`1.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at Akka.Util.Internal.AtomicState.<CallThrough>d__7`1.MoveNext()

I've got it today after updating from Akka 1.2 to 1.3. I've created a test application to check this issue and found the same behavior. In may case, the Redis server does not report about any successful connection at all.

  <package id="Akka" version="1.3.0" targetFramework="net46" />
  <package id="Akka.Persistence" version="1.3.0" targetFramework="net46" />
  <package id="Akka.Persistence.Redis" version="0.2.6-beta" targetFramework="net46" />
  <package id="Akka.Serialization.Hyperion" version="1.2.3.43-beta" targetFramework="net46" />
  <package id="DotNetty.Buffers" version="0.4.6" targetFramework="net46" />
  <package id="DotNetty.Codecs" version="0.4.6" targetFramework="net46" />
  <package id="DotNetty.Common" version="0.4.6" targetFramework="net46" />
  <package id="DotNetty.Handlers" version="0.4.6" targetFramework="net46" />
  <package id="DotNetty.Transport" version="0.4.6" targetFramework="net46" />
  <package id="Google.Protobuf" version="3.3.0" targetFramework="net46" />
  <package id="Google.ProtocolBuffers" version="2.4.1.555" targetFramework="net46" />
  <package id="Hyperion" version="0.9.2" targetFramework="net46" />
  <package id="Microsoft.Extensions.DependencyInjection.Abstractions" version="1.1.0" targetFramework="net46" />
  <package id="Microsoft.Extensions.Logging" version="1.1.1" targetFramework="net46" />
  <package id="Microsoft.Extensions.Logging.Abstractions" version="1.1.1" targetFramework="net46" />
  <package id="Microsoft.NETCore.Platforms" version="1.1.0" targetFramework="net46" />
  <package id="Microsoft.Win32.Primitives" version="4.3.0" targetFramework="net46" />
  <package id="NETStandard.Library" version="1.6.1" targetFramework="net46" />
  <package id="Newtonsoft.Json" version="9.0.1" targetFramework="net46" />

[Danthar]

@ia64mail Persistence.Redis is not yet updated for the 1.3 Release. I know work is being done, but i cannot give you an estimate right now.

[ia64mail]

@Danthar Thank you, will look forward for updates.