Parallel package for windows 10 in R

555 Views Asked by At

I have this dataset that I'm trying to parse in R. The data from HMDB and the dataset name is Serum Metabolites (in a format of xml file). The xml file contains about 25K metabolites nodes, each I want to parse to sub-nodes

I have a code that parses the XML file to a list object in R. Since the XML file is quite big and since for each metabolite there are about 12 sub-nodes I want, It takes a long time to parse the file. about 3 hours to 1,000 metabolites. I'm trying to use the package parallel but receive and error.

The packages:

library("XML")
library("xml2")
library( "magrittr" )  #for pipe operator %>%
library("pbapply") # to track on progress  
library("parallel") 

The function:

# The function receives an XML file (its location) and returns a list of nodes
 
Short_Parser_HMDB <- function(xml.file_location){
  start.time<- Sys.time()
  # Read as xml file
  doc <- read_xml( xml.file_location )
  #get metabolite nodes (only first three used in this sample)
  
  met.nodes <- xml_find_all( doc, ".//d1:metabolite" )  [1:1000] # [(i*1000+1):(1000*i+1000)]  # [1:3]  
  #list of data.frame
  xpath_child.v <- c( "./d1:accession",
                      "./d1:name"  ,
                      "./d1:description",
                      "./d1:synonyms/d1:synonym"  ,
                      "./d1:chemical_formula"   ,
                      "./d1:smiles" ,
                      "./d1:inchikey"    ,
                      "./d1:biological_properties/d1:pathways/d1:pathway/d1:name"   ,
                      "./d1:diseases/d1:disease/d1:name"   ,
                      "./d1:diseases/d1:disease/d1:references",
                      
                      "./d1:kegg_id"   ,                
                      "./d1:meta_cyc_id"
  )
  
  child.names.v <- c( "accession",
                      "name" ,  
                      "description" ,
                      "synonyms"  ,
                      "chemical_formula" , 
                      "smiles" ,
                      "inchikey"  , 
                      "pathways_names" ,
                      "diseases_name",
                      "references",
                      
                      "kegg_id" , 
                      "meta_cyc_id"
  ) 
  #first, loop over the met.nodes
  L.sec_acc <- parLapply(cl, met.nodes, function(x) {   # pblapply to track progress or lapply but slows down dramticlly the function  and parLapply fo parallel 
    #second, loop over the xpath desired child-nodes
    temp <-  parLapply(cl, xpath_child.v, function(y) { 
      xml_find_all(x, y ) %>% xml_text(trim = T) %>% data.frame( value = .)
    })
    #set their names
    names(temp) = child.names.v
    return(temp)
  }) 
  end.time<- Sys.time()
  total.time<- end.time-start.time
  print(total.time)
  return(L.sec_acc )
    
}

Now create the enviroment :

# select the location where the XML file is 
location= "D:/path/to/file//HMDB/DataSets/serum_metabolites/serum_metabolites.xml"


cl <-makeCluster(detectCores(), type="PSOCK")
clusterExport(cl, c("Short_Parser_HMDB", "cl"))
clusterEvalQ(cl,{library("parallel") 
                library("magrittr")
                library("XML")
                library("xml2")
  })

And execute :

Short_outp<-Short_Parser_HMDB(location)
stopCluster(cl)

The error received:

> Short_outp<-Short_Parser_HMDB(location)
Error in checkForRemoteErrors(val) : 
  one node produced an error: invalid connection

base on those links, Tried to implement the parallel :

  1. Parallel Processing in R
  2. How to call global function from the parLapply function?
  3. Error in R parallel:Error in checkForRemoteErrors(val) : 2 nodes produced errors; first error: cannot open the connection

but couldn't find invalid connection as an error

I'm using windows 10 the latest R version 4.0.2 (not sure if it's enough information)

Any hint or idea will be appreciated

0

There are 0 best solutions below