I have some py scripts to read data from db and write it to a feather file (with pandas to_feather() method). But somehow the written file cannot be read and it throws error as below:
Traceback (most recent call last):
File "/opt/***/***pipeline.py", line 303, in <module>
__main__()
File "/opt/***/***pipeline.py", line 124, in __main__
imt_preprocess.preprocess_data(config, thread_id=None, filtered_durations=durations, module=module, output_directory=output_directory, valid_from=valid_from, valid_until=valid_until)
File "/opt/***/***modules/imt_preprocess.py", line 27, in preprocess_data
df = commons.read_data(source_path)
File "/opt/***/***commons.py", line 62, in read_data
df = pd.read_feather(path)
File "/home/***/.local/lib/python3.7/site-packages/pandas/io/feather_format.py", line 131, in read_feather
handles.handle, columns=columns, use_threads=bool(use_threads)
File "/home/***/.local/lib/python3.7/site-packages/pyarrow/feather.py", line 220, in read_feather
return (read_table(source, columns=columns, memory_map=memory_map)
File "/home/***/.local/lib/python3.7/site-packages/pyarrow/feather.py", line 241, in read_table
reader = _feather.FeatherReader(source, use_memory_map=memory_map)
File "pyarrow/_feather.pyx", line 75, in pyarrow._feather.FeatherReader.__cinit__
File "pyarrow/error.pxi", line 143, in pyarrow.lib.pyarrow_internal_check_status
File "pyarrow/error.pxi", line 114, in pyarrow.lib.check_status
OSError: Verification of flatbuffer-encoded Footer failed.
Any idea what is the reason of this?