What's the best way to resume goroutines work

Question

I have to update thousands of structs with datas available on some remote servers. So, I have to deal with thousands of Goroutines querying these remote servers (http requests or db requests) to update the struct with the responses. But the update (or not) of the struct is depending of the results of other structs.

So I imagined a simple code in which the goroutines are running, each of them performs its own request, put the result in a global struct that contain any information retrieved by the goroutines and informs the main func that the first part of the job is done (waiting for the signal that every goroutines do the same before deciding of updating or not its struct.)

The goroutine should now waits for a signal of the main thread that all goroutines are done before deciding updating or not

the simplified code looks like that :

type struct StructToUpdate {
  //some properties
}

type struct GlobalWatcher {
  //some userful informations for all structs to update (or not)
}

func main() {
  //retrieving all structs to update
  structsToUpdates := foo() 

  //launching goroutines
  c := make(chan bool)
  for _,structToUpdate := range structsToUpdates {
    go updateStruct(&structToUpdate,c)
  }

  //waiting for all goroutines do their first part of job
  for i:=0; i<len(structsToUpdates); i++ {
    <- c
  }

  //HERE IS THE CODE TO INFORM GOROUTINES THEY CAN RESUME


}

func updateStruct(s *StructToUpdate, c chan bool) {
  result := performSomeRequest(s)
  informGlobalWatcherOfResult(result)
  c <- true //I did my job

  //HERE IS THE CODE TO WAIT FOR THE SIGNAL TO RESUME
}

The question is : What the more performant / idiomatic / elegant wait to :

• send a signal from the main script ?
• wait for this signal from the goroutine ?

I can imagine 3 ways to do this

• in a first way, I can imagine a global bool var resume := false that will be turned to true by the main func when all goroutines do the first part of job. In this case, each goroutine can use an ugly for !resume { continue }....
• in a more idiomatic code, I can imagine do the same thing but instead of using a bool, I can use a context.WithValue(ctx, "resume", false) and pass it to goroutine, but I still have a for !ctx.Value("resume")
• in a last more elegant code, I can imagine using another resume := make(chan bool) passed to the goroutine. The main func could inform goroutines to resume closing this chan with a simple close(resume). The goroutine will wait the signal with something than :

for {
  _, more := <-resume
  if !more {
    break
  }
}
//update or not

Is there any other good idea to do this ? One of the above solutions is better than others ?

Answer 1

I'm not sure if I understand your question completely, but a simple solution to blocking the main thread is with an OS signal ie.:

done := make(chan os.Signal, 1)
signal.Notify(done, os.Interrupt, syscall.SIGTERM)

// Blocks main thread
<-done

This doesn't have to be an OS signal, this could be a channel that receives a struct{}{} value from somewhere else in your runtime.

If you need to do this as a result of multiple go routines working, I'd look into using sync.WaitGroup.

Answer 2

OK. Thanks to @blixenkrone I found an elegant solution using WaitGroup !

Usually, we use a sync.WaitGroup to synchronize goroutines and be sure that they finish their job : you add "1" to the WaitGroup and then wait... Goroutines, when they finish their job, use the Done() function to decrement the Waitgroup counter. In fact, the Wait() function of the WaitGroup must act like a for loop, looking for its internal counter to become "0" to resume.

What I do is exactly the opposite !!! I create a WaitGroup and increment it to 1 in the main program before launching the goroutines, and pass it to them. The goroutine do their first past of job, notify the main program and then wait for the WaitGroup. When the main program get every notification of first part job, it release the waitgroup, then each goroutine can finish the job. Here is a squeleton of what I did :

func main() {
  structsToUpdate := getStructs() //some func to get my structs

  communication := make(chan bool)

  //the waitgroup used to block the goroutine
  var resume sync.WaitGroup
  resume.Add(1) //The waitgroup will be blocked !!!

  success := 0 

  for _,structToUpdate := range structsToUpdates {
    go updateStruct(&structToUpdate,communication,&resume)
  }

  //waiting for all goroutines do their first part of job
  for i:=0; i<len(structsToUpdates); i++ {
    res := <- communication
    if res {
      success++
    }
  }

  //all structs should have done with first part now... Resume their job !
  resume.Done()

  //receive second part job
  for success != 0 {
    <-communication
    success--
  }
}

func updateStruct(s *StructToUpdate, c chan bool, resume *sync.WaitGroup) {
  //First part of the job
  result, err := performSomeRequest(s)
  if err != nil  {
    //something wrong happened
    c <- false
    return
  }

  informGlobalWatcherOfResult(result)
  c <- true //I did my job

  //wait for all goroutines do their first part job
  resume.Wait()

  //here the main program 'Done()' the WaitGroup so it resumes the execution of the goroutine

  updateOrNotStruct() //Some operations...

  c <- true
}

Even if I think the wg.Wait() func should call a for loop itself, I think this code is cleaner than the solutions a proposed above...