limit number of processors used by R with write.table

60 Views Asked by At

I have very large matrices of genomic data (counts of nucleotides across dozens of individuals for millions of sites) that we process with an R script, and then write to file with 'write.table'. As our matrices got larger (more individuals, more sites), I had to modify the write.table code below to loop over each row of the matrix and append-write to the file, as doing it as a single command would cause the script to crash (as I recall it was a memory error, but don't quote me on that). It's slower, but it doesn't crash! For some reason, as R writes to file, it's using all the threads available, which in and of itself is a good thing, but there are multiple users on this server and I need to be able to limit the number of processors being used over such a long time to write this large of files. I'll probably end up switching to fwrite, but just for my edification can anyone give me an idea why it's doing this and how I could limit it?

  print("R ALERT: Writing output files")
  #  write.table(Freqz, file=outname1,row.names=FALSE, col.names=TRUE, quote=FALSE)
  write.table(Freqz[1,], file=outname1,row.names=FALSE, col.names=TRUE, quote=FALSE)
  for(r in 2:nrow(Freqz)){
    write.table(Freqz[r,], file=outname1,row.names=FALSE, col.names=FALSE, quote=FALSE,append=TRUE)
  }
0

There are 0 best solutions below