I have the following simplified code. I want to achieve that the manager's TryHandle function is only executed when the mutex is not locked, otherwise skip the execution. TryHandle can be accessed from multiple go routines:
var (
mu sync.Mutex
)
// can be accessed by multiple go routines
func (m *Manager) TryHandle(work *Work) {
if mu.TryLock() {
defer mu.Unlock()
defer func() {
if r := recover(); r != nil {
fmt.Printf("Recovered from panic. Error: %s\n", r)
fmt.Println("stacktrace from panic: \n" + string(debug.Stack()))
}
}()
m.HandleWork(work)
} else {
fmt.Println("mutex is busy.")
}
}
At first it seemed to work, the program was running for some hours quite well. TryHandle was accessed frequently up to 5-10 times per second, and a very few times the mutex was actually busy.
But this morning I saw in the logs that "mutex is busy" every time TryHandle was called, and one cpu was running at 100%.
Of course, the most obvious reason could be that one call to m.HandleWork() never returned and the mutex was never unlocked. Unfortunately the log buffer is not so big so I don't have the proof.
But apart from that, and since I am (almost!) sure m.HandleWork can not be the culprit, is it possible that there was a panic and my defer recover function somehow prevented the defer mu.Unlock()?
Or is it maybe a problem that I define mu standalone instead of making it a field on my managers struct?