How do I cleanly shutdown a distributed ActivePivot setup?

Question

We have an ActivePivot cube that is a polymorphic cube (2 nodes) where 1 node is itself a horisontally distributed cube (8 nodes). Running in Tomcat using JGroup TCP for distribution. It is restarted on a daily basis, but every time it is shut down (node services are stopped in sequence), various errors show up in the logs. This is harmless, but anoying from a monitoring perspective.

Example from one day (all same node):

19:04:43.100 ERROR [Pool-LongPollin][streaming] A listener dropped (5f587379-ac67-4645-8554-2e02ed739924). The number of listeners is now 1
19:04:45.767 ERROR [Pool-LongPollin][streaming] Publishing global failure
19:05:16.313 ERROR [localhost-start][core] Failed to stop feed type MDXFEED with id A1C1D8D92CF7D867F09DCB7E65077B18.0.PT0

Example from another day (same error from multiple different nodes):

19:00:17.353 ERROR [pivot-remote-0-][distribution] A safe broadcasting task could not be performed
com.quartetfs.fwk.QuartetRuntimeException: [<node name>] Cannot run a broadcasting task with a STOPPED messenger

Does anyone know of a clean way to shut down a setup like this?

Answer 1

Those errors appear because on application shutdown the ActivePivotManager is aggressively stopping the distribution, without waiting for each distributed ActivePivot to be notified that other cubes have been stopped.

To smoothly stop distribution you can use the methods from the DistributionUtil class. For instance:

public class DistributionStopper {

protected final IActivePivotManager manager;

public DistributionStopper (IActivePivotManager manager){
    this.manager = manager;
}

public void stop(){
    // Get all the schemas from the manager
    final Collection<IActivePivotSchema> schemas = manager.getSchemas().values();

    // To store all the available messengers
    final List<IDistributedMessenger<?>> availableMessengers = new LinkedList<>();

    // Find all the messengers
    for(IActivePivotSchema schema : schemas){
        for(String pivotId : schema.getPivotIds()){
            // Retrieve the activePivot matching this id
            final IMultiVersionActivePivot pivot = schema.retrieveActivePivot(pivotId);

            if(pivot instanceof IMultiVersionDistributedActivePivot){
                IDistributedMessenger<IActivePivotSession> messenger = ((IMultiVersionDistributedActivePivot) pivot).getMessenger();
                if(messenger != null){
                    availableMessengers.add(messenger);
                }
            }
        }
    }

    // Smoothly stop the messengers
    DistributionUtil.stopMessengers(availableMessengers);
}

}

Then register this custom class as a Spring bean depending on the activePivotManager singleton bean, in order to have its destroyMethod called before the one of the manager.

@Bean(destroyMethod="stop")
@DependsOn("activePivotManager")
public DistributionStopper distributionStopper(IActivePivotManager manager){
    return new DistributionStopper(manager);
}