I'm applying a function to a list in parallel using R's SNOW package.
I've exported objects from the global environment into the cluster environment, and the function has generated new objects. Is there a way to bring objects from the cluster environment back into the global environment? (ie. an opposite of clusterExport()?)
Or must one write a file in order to work with the output of a parallel processed function?
Note: SNOW (Simple Network Of Workstations) provides a high-level interface for using a workstation cluster for parallel computations in R
Here's some example code:
example.list <- c("region1","region2","region3") #list of regions in study area
eco.fn<-function(eco){
eco.rast <- mask(base.raster,subset(hexagons,REGION_NAME==eco)) #clip raster to grid of hexagons covering region of interest
## reclassify raster based on a table (table generation omitted for simplicity)
eco.hab.pix <- reclassify(eco.rast,eco.matrix) # new raster with
## now merge new rasters together one by one as they're created
if (exists("merge.eco.hab.pix")) {
merge.eco.hab.pix <<- merge(merge.eco.hab.pix, eco.hab.pix) #on first run, create the dataset, export it to the globabl environment so it's not forgotten between runs
} else {
merge.eco.hab.pix <<- eco.hab.pix #consecutive iterations add new areas, then export it to the globabl environment
}
}
lapply(eexample.list,eco.fn) #works great, but takes a terribly long time
#in paralell:
library(doSNOW)
library(foreach)
cl <- makeCluster(Ncores) #make cluster
registerDoSNOW(cl) #registers cluster on parallel backend
#export variables and packages to the cluster
clusterExport(cl, c("base.raster", "hexagons"))
clusterEvalQ(cl, library(c("raster","rgdal","sf"))
parLapply(cl,example.list,eco.fn) #still runs but the output is just the last subset of the raster generated, rather than the merged subsets
Edit: I did find a workaround, but I'm still curious as to whether there's an parallel processing equivalent of using the double arrows (<<-) in functions.
Thanks!