Cannot update Heron topology using SchedulerStateManagerAdaptor

Question

When I attempted to create custom scheduler of Heron, and used SchedulerStateManagerApaptor object to update topology, this Exception happened:

Exception in thread java.util.ConcurrentModificationException: The update lock can not be obtained for topology AuroraMonitorSentenceWordCountTopology. Another actor is performing an update on it. Failing this request, try again once current update is complete
    at com.twitter.heron.scheduler.UpdateTopologyManager.updateTopology(UpdateTopologyManager.java:117)
    at zyt.custom.my.scheduler.aurora.AuroraHotEdgeSchedulerWithTxtLog.triggerSchedule(AuroraHotEdgeSchedulerWithTxtLog.java:323)
    at zyt.custom.my.scheduler.aurora.AuroraHotEdgeSchedulerWithTxtLog.access$400(AuroraHotEdgeSchedulerWithTxtLog.java:55)
    at zyt.custom.my.scheduler.aurora.AuroraHotEdgeSchedulerWithTxtLog$2.run(AuroraHotEdgeSchedulerWithTxtLog.java:258)
    at java.lang.Thread.run(Thread.java:748)

And the code of instantiating SchedulerStateManagerAdaptor object as follows:

Config config = Config.toClusterMode(this.config);
String stateMgrClass = Context.stateManagerClass(config); // get state manager instance
IStateManager stateMgr = null;
try {
    stateMgr = ReflectionUtils.newInstance(stateMgrClass);
} catch (ClassNotFoundException | InstantiationException | IllegalAccessException e) {
    e.printStackTrace();
}
stateMgr.initialize(config);
return stateManagerAdaptor = new SchedulerStateManagerAdaptor(stateMgr, 1000); // 5000: timeout

How to fix this exception? Thanks for your help!

Answer 1

Yitan - Are you trying to create a new Aurora scheduler?

Answer 2

Based on the exception message, there's one update operation going on when a second update operation is scheduled. The first on-going operation prevents the second one proceeding.

So could you try the update operation cleanly? And also please check if your implementation correctly releases locks once an update is done?