I am using cat command to merge multiple files using %sh command in Azure Databricks Notebook. I have around 1200 csv files in the data_files folder and the total size of files are around 300 GB. When I run the below code, sometimes it is merging the files without any errors, but sometimes it throws the error cat: write error: Resource temporarily unavailable and the output.txt file is created with no data.
err=$(cat /dbfs/mnt/devl/header_file/*.csv /dbfs/mnt/devl/data_files/*.csv 2>&1 > /dbfs/mnt/devl/output.txt)
RC=$?
if [ $RC -ne 0 ]; then
echo "Error code : $RC"
echo "Error msg : $err"
fi
Can anyone tell what is the rootcause of the error cat: write error: Resource temporarily unavailable and how to resolve this issue?