How to merge large PDF files without running of memory on node js

77 Views Asked by At

I am using node js and pupeeter to generate a lot of pdf files with high resolution images on them. After that all generated pdf files i store in array. Then i merge them one by one using pdf-merger-js library. It does merge for the few first pdf files but then runs out memory so node js kills the process and exits. Here what my merging code looks like:

const PDFs = [PDF1, PDF2, ...34+] // arrayBuffers or Blobs
for (const PDF of PDFs) {
    await merger.add(PDF);
}
await merger.save("saved.pdf");

PDFs array at the moment size is about 2 GBs. While handling 30 files. But user may need to merge way more files in much better resolution, so i am trying to find the solution.

I tried to end merging and save after merging every file, but the problem still remains

let mergedSum;
    for (const PDF of PDFs) {
      const merger = new PDFmerger();
      if (mergedSum) {
        await merger.add(mergedSum);
      }
      await merger.add(PDF);
      mergedSum = await merger.saveAsBuffer();
}

I also used a function to monitor memory usage. The function is:

function getMemory() {
  return Object.entries(process.memoryUsage()).reduce((carry, [key, value]) => {
    return `${carry}${key}:${Math.round((value / 1024 / 1024) * 100) / 100}MB;`;
  }, "");
}

I get something like that in logs: rss:158.8MB;heapTotal:44.79MB;heapUsed:40.88MB;external:52.25MB;arrayBuffers:65.39MB And it says that on every iteration rss, arrayBuffers and external double on every iteration. HeapUserd and HeapTotal seem not to change at all.

So how to merge large files without running of memory?

0

There are 0 best solutions below