Stateless Worker service in Service Fabric restarted in the same process

Question

I have a stateless service that pulls messages from an Azure queue and processes them. The service also starts some threads in charge of cleanup operations. We recently ran into an issue where these threads which ideally should have been killed when the service shuts down continue to remain active (definitely a bug in our service shutdown process).

Further looking at logs, it seemed that, the RunAsync methods cancellation token received a cancellation request, and later within the same process a new instance of the stateless service that was registered in ServiceRuntime.RegisterServiceAsync was created.

Is this expected behavior that service fabric can re-use the same process to start a new instance of the stateless service after shutting down the current instance.

The service fabric documentation https://learn.microsoft.com/en-us/azure/service-fabric/service-fabric-hosting-model does seem to suggest that different services can be hosted on the same process, but I could not find the above behavior being mentioned there.

Answer 1

In the shared process model, there's one process per ServicePackage instance on every node. Adding or replacing services will reuse the existing process.

This is done to save some (process-level) overhead on the node that runs the service, so hosting is more cost-efficient. Also, it enables port sharing for services.

You can change this (default) behavior, by configuring the 'exclusive process' mode in the application manifest, so every replica/instance will run in a separate process.

<Service Name="VotingWeb" ServicePackageActivationMode="ExclusiveProcess">

As you mentioned, you can monitor the CancellationToken to abort the separate worker threads, so they can stop when the service stops.

More info about the app model here and configuring process activation mode.