Retrieve object from cluster environment after parLapply in SNOW

161 Views Asked by At

I'm applying a function to a list in parallel using R's SNOW package.

I've exported objects from the global environment into the cluster environment, and the function has generated new objects. Is there a way to bring objects from the cluster environment back into the global environment? (ie. an opposite of clusterExport()?)

Or must one write a file in order to work with the output of a parallel processed function?

Note: SNOW (Simple Network Of Workstations) provides a high-level interface for using a workstation cluster for parallel computations in R

Here's some example code:

    example.list <- c("region1","region2","region3") #list of regions in study area


    eco.fn<-function(eco){

    eco.rast <- mask(base.raster,subset(hexagons,REGION_NAME==eco)) #clip raster to grid of hexagons covering region of interest

  ## reclassify raster based on a table (table generation omitted for simplicity)
    eco.hab.pix <- reclassify(eco.rast,eco.matrix) # new raster with 

  ## now merge new rasters together one by one as they're created
    if (exists("merge.eco.hab.pix")) {
        merge.eco.hab.pix <<- merge(merge.eco.hab.pix, eco.hab.pix)  #on first run, create the dataset, export it to the globabl environment so it's not forgotten between runs 
        } else {                                                       
        merge.eco.hab.pix <<- eco.hab.pix                            #consecutive iterations add new areas, then export it to the globabl environment           

        }

    }

    lapply(eexample.list,eco.fn) #works great, but takes a terribly long time


#in paralell:


   library(doSNOW) 
   library(foreach)

   cl <- makeCluster(Ncores) #make cluster
   registerDoSNOW(cl) #registers cluster on parallel backend
  #export variables and packages to the cluster
   clusterExport(cl, c("base.raster", "hexagons"))
   clusterEvalQ(cl, library(c("raster","rgdal","sf"))



   parLapply(cl,example.list,eco.fn) #still runs but the output is just the last subset of the raster generated, rather than the merged subsets

Edit: I did find a workaround, but I'm still curious as to whether there's an parallel processing equivalent of using the double arrows (<<-) in functions.

Thanks!

0

There are 0 best solutions below