Faster zgrep for Python 3.8.2

191 Views Asked by At

I have a large directory of *.gz files that I need to find all files that contain the required ID on a given date. My code works as expected, but it is very slow. I was curious if anybody knew of a faster was to process these files with python. 3.8.2 Any help would be appreciated.

snippet of code

def proc_data(id, date):
    os.chdir(dir)
    date = '*_' + date + '_*'
    data = subprocess.getoutput('zgrep %s %s' % (id, date,))
    return data

Sample of File names in Dir

RADT_App_20201022_0002.dat.gz

amof_app_order_20201012_4.dat.gz

0

There are 0 best solutions below