I am facing issues in activating grains. I am successfully able to up the silos but the grain activation is throwing me TaskCanceledException.
Below are the logs might help understanding the problem:
[Error] Orleans.Grain: Error activating grain [Activation: S192.168.1.33:10025:58975009/Placement=RandomPlacement State=Activating]
System.Threading.Tasks.TaskCanceledException: Timed out waiting for OnActivateAsync to complete
---> System.Threading.Tasks.TaskCanceledException: A task was canceled.
at Orleans.Internal.OrleansTaskExtentions.MakeCancellable(Task task, CancellationToken cancellationToken) in /_/src/Orleans.Core/Async/TaskExtensions.cs:line 188
at Orleans.Internal.OrleansTaskExtentions.WithCancellation(Task taskToComplete, String message, CancellationToken cancellationToken) in /_/src/Orleans.Core/Async/TaskExtensions.cs:line 145
[Error] Orleans.Runtime.GrainDirectory.ClientDirectory: Exception publishing client routing table to silo
Orleans.Runtime.OrleansMessageRejectionException: The target silo is no longer active: target was ......(silo details).. but this silo is ......(silo details)... The rejected message is Request [......(silo details).. Orleans.Runtime.GrainDirectory.IRemoteClientDirectory.OnUpdateClientRoutes(System.Collections.Immutable.ImmutableDictionary`2[Orleans.Runtime.SiloAddress,System.ValueTuple`2[System.Collections.Immutable.ImmutableHashSet`1[Orleans.Runtime.GrainId],System.Int64]]) #2.
at Orleans.Serialization.Invocation.ResponseCompletionSource.System.Threading.Tasks.Sources.IValueTaskSource.GetResult(Int16 token) in /_/src/Orleans.Serialization/Invocation/ResponseCompletionSource.cs:line 98
at System.Threading.Tasks.ValueTask.ValueTaskSourceAsTask.<>c.<.cctor>b__4_0(Object state)
--- End of stack trace from previous location ---
My guess is my grain is not able to communicate with the silos.
Tried Scenarios:
1. Changed the ActivationTimeout for grains to 3 minutes (default from 2 minutes) when configuring the Host Builder.
2. Checked the OnActivateAsync Methods for long running applications.
Kindly help with this.