How to perform operations on memory map without loading whole file into memory?

73 Views Asked by At

An approximately 4.3 GB memory map I want to take the log of without loading the whole thing into memory. Is there a way to assign cl that minimizes amount of memory used?

import numpy
import psutil

print(psutil.virtual_memory().available)  #32156053504
cc = numpy.memmap(r'data//cc.dat',mode='r',dtype=float)
print(psutil.virtual_memory().available)  #32156049408  #very little memory used
cl = numpy.log(cc) + 1
print(psutil.virtual_memory().available)  #23610658816  #8.6 GB used

I want to take slices of cc and cl which normally wouldn't require loading all of cc.dat into memory.

0

There are 0 best solutions below