Github Action Writing Large TIFF file to AWS S3 with GDAL via RCLONE: file not created

Question

Situation Trying to merge 4 files (each 700MB) with GDAL which are stored on S3. created a mosaic.vrt which is stored on S3 bucket and accessible. Dat can be accessed with rclone mount from github action with python module subprocess. For smale files (700KB) the process below works for the big 700MB files not

TH GH Workflow

name: Run merger

on:
  schedule:
  - cron:  '4 0 * * *'
  # Allows you to run this workflow manually from the Actions tab
  workflow_dispatch:
  
jobs:
  build:
    # cf. https://docs.github.com/en/github/setting-up-and-managing-billing-and-payments-on-github/about-billing-for-github-actions
    runs-on: ubuntu-latest

    steps:
    - name: Checkout the repository
      uses: actions/checkout@v3
    
    - name: Increase git buffer size
      run: |
        git config --global http.postBuffer 1048576000
    
    - name: Set up Python 3.9
      uses: actions/setup-python@v4
      with:
        python-version: 3.9
    
    - name: Install dependencies
      run: |
        python -m pip install --upgrade pip

        
    - name: Install rclone
      run: |
        curl https://rclone.org/install.sh | sudo bash

    
    - name: install gdal
      run: |
        sudo apt-get install gdal-bin 

    #- name: Create rclone config file
    #  run: echo "${{ secrets.RCLONE_CONFIG }}" > rclone.conf
    
    - name: Run Python script
      run: |
        python myscript.py
      env: # Set the secrets as env var
        GOOGLE_CLIENT_SECRET: ${{ secrets.GOOGLE_CLIENT_SECRET }}
        RCONF_SECRET: ${{ secrets.RCONF_SECRET }}

The Python code myscript.py

# -*- coding: utf-8 -*-
import configuration as config
import subprocess
import os
import time


# Write rclone config to a file
rclone_config = os.environ.get('RCONF_SECRET')
rclone_config_file = "rclone.conf"
with open(rclone_config_file, "w") as f:
    f.write(rclone_config)

mountpoint="locals3"

command = ["mkdir", mountpoint]
print(command)
result=subprocess.run(command, check=True)
   
# Run rclone command to move files

command = ["rclone", "mount", "--config", "rclone.conf", "--s3-no-check-bucket",
            os.path.join(config.S3_DESTINATION),mountpoint,"--vfs-cache-mode", "full"]

print(command)
subprocess.Popen(command)

# Wait for 10 seconds
time.sleep(10)

# Set the CPL_DEBUG environment variable to enable verbose output
os.environ["CPL_DEBUG"] = "ON"
      
#run gdal
command = ["gdal_translate", os.path.join(mountpoint , "NDVI-MAX","mosaic.vrt"), os.path.join(mountpoint ,"NDVI-MAX","mosaicBIG.tif"), "-of", "COG",
            "-co", "NUM_THREADS=ALL_CPUS",
            "-co", "COMPRESS=LZW",
            "-co", "BIGTIFF=YES"
            "--config", "GDAL_CACHEMAX", "9999",
            "--config", "GDAL_NUM_THREADS", "ALL_CPUS"
            ]
print(command)
result=subprocess.run(command, check=True, capture_output=True, text=True)

if result.returncode == 0:
    print("GDAL_TRANLSATE Command executed successfully!")
    print("Output:")
    print(result.stdout)
    print(result.stderr)
else:
    print("Command failed with a non-zero exit status!")
    print("Error message:")
    print(result.stderr)

I get

['gdal_translate', 'locals3/NDVI-MAX/mosaic.vrt', 'locals3/NDVI-MAX/mosaicBIG.tif', '-of', 'COG', '-co', 'NUM_THREADS=ALL_CPUS', '-co', 'COMPRESS=LZW', '-co', 'BIGTIFF=YES', '--config', 'GDAL_CACHEMAX', '9999', '--config', 'GDAL_NUM_THREADS', 'ALL_CPUS']
GDAL_TRANLSATE Command executed successfully!
Output:
Input file size is 43773, 29858
0...10...20...30...40...50...60...70...80...90...100 - done.


GDAL: GDALOpen(locals3/NDVI-MAX/mosaic.vrt, this=0x55cff997c1b0) succeeds as VRT.
GDAL: GDALDefaultOverviews::OverviewScan()
COG: Generating overviews of the imagery: start
GTiff: File being created as a BigTIFF.
GTiff: Using 2 threads for compression
GDAL: GDALOpen(mosaicBIG.tif.ovr.tmp, this=0x55cff998ba40) succeeds as GTiff.
GTiff: ScanDirectories()
GTiff: Opened 10943x7464 overview.
GTiff: Opened 5471x3732 overview.
GTiff: Opened 2735x1866 overview.
GTiff: Opened 1367x933 overview.
GTiff: Opened 683x466 overview.
GTiff: Opened 341x233 overview.
GDAL: GDAL_CACHEMAX = 9999 MB
GDAL: GDALOpen(locals3/NDVI-MAX/Sentinel_NDVI-MAX_SR_CloudFree_crop20230507-20230605_run20230606quadrant4.tif, this=0x55cffa83b000) succeeds as GTiff.
GDAL: GDALOpen(locals3/NDVI-MAX/Sentinel_NDVI-MAX_SR_CloudFree_crop20230507-20230605_run20230606quadrant3.tif, this=0x55cffdb30c00) succeeds as GTiff.
GDAL: GDALOpen(locals3/NDVI-MAX/Sentinel_NDVI-MAX_SR_CloudFree_crop20230507-20230605_run20230606quadrant2.tif, this=0x55cffd9eaa60) succeeds as GTiff.
GDAL: GDALOpen(locals3/NDVI-MAX/Sentinel_NDVI-MAX_SR_CloudFree_crop20230507-20230605_run20230606quadrant1.tif, this=0x55cffd9ed6f0) succeeds as GTiff.
GTIFF: Waiting for worker job to finish handling block 172
...
...
GTIFF: Waiting for worker job to finish handling block 4993
GTIFF: Waiting for worker job to finish handling block 4996
GTIFF: Waiting for worker job to finish handling block 4997
GTIFF: Waiting for worker job to finish handling block 4999
GDAL: GDALClose(locals3/NDVI-MAX/mosaicBIG.tif.ovr.tmp, this=0x55cff998ba40)
COG: Generating final product: end
GDAL: GDALClose(locals3/NDVI-MAX/mosaicBIG.tif, this=0x55cffa91a8b0)
GDAL: GDALClose(locals3/NDVI-MAX/mosaic.vrt, this=0x55cff997c1b0)
GDAL: GDALClose(locals3/NDVI-MAX/Sentinel_NDVI-MAX_SR_CloudFree_crop20230507-20230605_run20230606quadrant1.tif, this=0x55cffd9ed6f0)
GDAL: GDALClose(locals3/NDVI-MAX/Sentinel_NDVI-MAX_SR_CloudFree_crop20230507-20230605_run20230606quadrant2.tif, this=0x55cffd9eaa60)
GDAL: GDALClose(locals3/NDVI-MAX/Sentinel_NDVI-MAX_SR_CloudFree_crop20230507-20230605_run20230606quadrant3.tif, this=0x55cffdb30c00)
GDAL: GDALClose(locals3/NDVI-MAX/Sentinel_NDVI-MAX_SR_CloudFree_crop20230507-20230605_run20230606quadrant4.tif, this=0x55cffa83b000)
GDAL: In GDALDestroy - unloading GDAL shared library.

Although during the run mosaicBIG.tif.ovr.tmp is in the bucket, after the process finished there is no mosaicBIG.tif

Any hints?