Controlling Concurrency in tfenv Installations for Terraform Versions

Question

Description:

I am currently facing an issue with my Terraform infrastructure setup using Atlantis and Terragrunt. The problem arises when Terragrunt attempts to create multiple workspaces, each requiring different versions of Terraform. To handle this, Terragrunt utilizes tfenv install to install the necessary Terraform versions in parallel.

However, this concurrent execution is causing permission-related errors during the installation process, leading to issues like:

/home/user/.tfenv/lib/tfenv-exec.sh: line 43: /home/user/.tfenv/versions/1.6.2/terraform: Permission denied
/home/user/.tfenv/lib/tfenv-exec.sh: line 43: exec: /home/user/.tfenv/versions/1.6.2/terraform: cannot execute: Permission denied

I have already created an issue in the tfenv GitHub repository here.

To mitigate this problem, I am exploring the option of creating a shell wrapper script for tfenv install that ensures the serialized installation of Terraform versions.

The script should guarantee that even when multiple workspaces are executed in parallel, any installation of Terraform versions is done sequentially. This is to prevent race conditions that could occur if multiple installations were attempted simultaneously.

How I see the flow:

1. Atlantis calls "tfenv wrapper."

2. The wrapper checks for the presence of a lock:

   a. If there is no lock, this invocation acquires the lock and proceeds to step 3.

   b. If there is a lock, this invocation waits for the lock to become available and proceeds to step 4.

3. The invocation with the lock installs the desired version of tfenv and releases the lock.

4. The invocation without the lock checks the version being installed by the active lock. If it matches the desired version, it can exit safely. If not, it waits for the lock to become available and becomes step 3.

I have drafted a script, but it doesn't seem to be addressing the issue correctly. Here's the script I've created:

#!/bin/bash

LOCK_FILE="/tmp/tfenv-wrapper.lock"
PROCESS_COUNTER_FILE="/tmp/tfenv-process-counter"
MAX_CONCURRENT_PROCESSES=2

# Function to acquire a lock
function acquire_lock() {
    exec 100>"$LOCK_FILE"
    flock -x 100
}

# Function to release the lock
function release_lock() {
    flock -u 100
}

# Function to get the current process count
function get_process_count() {
    pgrep -f "tfenv install" | grep -v $$ | wc -l

}

# Function to update the process count
function update_process_count() {
    get_process_count > "$PROCESS_COUNTER_FILE"
}

# Set up the EXIT trap to release the lock and remove the lock file on script exit
trap 'release_lock' EXIT

# Infinite loop to keep the script running
while true; do
    # Acquire the lock
    acquire_lock
    echo "Lock acquired."

    # Read the number of running processes from the counter file
    num_processes=$(get_process_count)
    echo "Number of tfenv processes: $num_processes"

    # If the number of running processes exceeds the limit, wait
    while [ "$num_processes" -ge "$MAX_CONCURRENT_PROCESSES" ]; do
        echo "Maximum number of concurrent 'tfenv' processes reached. Waiting for processes to complete."
        sleep 2
        num_processes=$(get_process_count)
    done

    # Increment the process counter
    ((num_processes++))
    update_process_count

    # Release the lock
    release_lock
    echo "Released lock."

    # Wait for any 'tfenv' process launched outside this script to complete
    while [ "$(get_process_count)" -gt 0 ]; do
        sleep 2
    done

    # Acquire the lock again before decrementing the counter
    acquire_lock
    echo "Lock acquired."

    # Decrement the process counter
    ((num_processes--))
    update_process_count

    # Release the lock
    release_lock
    echo "Released lock."
done

So this script should keep running as long as there is a "lock" and wait for that "lock" to become free, then install the version it needs (if it hasn't already by the process that has the lock "previously"

I would appreciate any insights, suggestions, or alternative solutions to control the concurrency of tfenv install commands and avoid the permission-related errors. Thank you!