Laravel queue runs for a while and stops without any exception

Question

Problem

I have a job that pushes emails into a Laravel queue that sends them to SendGrid. The mail queue has 15k jobs but as soon as it arrives at more or less 3.8k it stops, without any error or warnings. It just stops as it arrives at the end of the queue, leaving a failed job behind.

Code

Here is the code I used to add the job to the queue:

$message = (new SongChallengeEmail(
    $this->getUsername($emailRawData->name, $emailRawData->username, $emailRawData->email),
    $emailRawData->email,
    $emailRawData->artists
))
    ->onConnection('database')
    ->onQueue('emails');

Mail::queue($message);

What I tried

Initially, I thought it was a problem with php's memory limit, so I tried to allocate more memory than the default (128MB). I tried both configurations:

php artisan queue:work database --queue=emails --memory=8000

and

php -d memory_limit=1G database --queue=emails artisan queue:work

but neither of them worked, so I'm assuming there's something more I'm not grasping about this.

I'm thinking about implementing supervisor has Laravel's docs suggest but first I would like to understand why this happens.

Has anyone ever managed to solve this same problem?

Answer 1

Queue Workers intentionally have runtime limits. This helps the OS manage potential memory leaks, locks on system resources, and cruft that accumulates the longer a thread runs.

Easy to fix. Any one of these can get you back on track:

• You can schedule a task every minute to run a queue worker 'php artisan queue:work --stop-when-empty'
• Or install Horizon to automate Queue Worker Management