I've got a Laravel 10 project running PHP 8.1, I'm using the latest version of Supervisor and have some 40 or so processes allocated to Supervisor as I process hundreds of jobs per minute via my database on various queues.
Time and time again in Laravel projects I find myself seeing this regular occurrence with the same kind of failed job popping up which doesn't account to any code errors.
Illuminate\Queue\MaxAttemptsExceededException
So much so that I've now got a cron running hourly to prune the failed jobs table for the past 12 hours, to put it into context, I have over 12,000 failed jobs for this error in this window.
Some would say to simply increase the number of tries, but this in turn would simply slow down my processing and my Jobs table would build up, and yes, I have my retry_after set slightly longer than my maximum timeout.
Here's one particular error, caused by my certificates queue:
Illuminate\Queue\MaxAttemptsExceededException: App\Jobs\CertificateChecker has been attempted too many times. in /var/www/domain-monitor-api/vendor/laravel/framework/src/Illuminate/Queue/Worker.php:785
Stack trace:
#0 /var/www/domain-monitor-api/vendor/laravel/framework/src/Illuminate/Queue/Worker.php(519): Illuminate\Queue\Worker->maxAttemptsExceededException()
#1 /var/www/domain-monitor-api/vendor/laravel/framework/src/Illuminate/Queue/Worker.php(429): Illuminate\Queue\Worker->markJobAsFailedIfAlreadyExceedsMaxAttempts()
#2 /var/www/domain-monitor-api/vendor/laravel/framework/src/Illuminate/Queue/Worker.php(389): Illuminate\Queue\Worker->process()
#3 /var/www/domain-monitor-api/vendor/laravel/framework/src/Illuminate/Queue/Worker.php(176): Illuminate\Queue\Worker->runJob()
#4 /var/www/domain-monitor-api/vendor/laravel/framework/src/Illuminate/Queue/Console/WorkCommand.php(138): Illuminate\Queue\Worker->daemon()
#5 /var/www/domain-monitor-api/vendor/laravel/framework/src/Illuminate/Queue/Console/WorkCommand.php(121): Illuminate\Queue\Console\WorkCommand->runWorker()
#6 /var/www/domain-monitor-api/vendor/laravel/framework/src/Illuminate/Container/BoundMethod.php(36): Illuminate\Queue\Console\WorkCommand->handle()
#7 /var/www/domain-monitor-api/vendor/laravel/framework/src/Illuminate/Container/Util.php(41): Illuminate\Container\BoundMethod::Illuminate\Container\{closure}()
#8 /var/www/domain-monitor-api/vendor/laravel/framework/src/Illuminate/Container/BoundMethod.php(93): Illuminate\Container\Util::unwrapIfClosure()
#9 /var/www/domain-monitor-api/vendor/laravel/framework/src/Illuminate/Container/BoundMethod.php(37): Illuminate\Container\BoundMethod::callBoundMethod()
#10 /var/www/domain-monitor-api/vendor/laravel/framework/src/Illuminate/Container/Container.php(662): Illuminate\Container\BoundMethod::call()
#11 /var/www/domain-monitor-api/vendor/laravel/framework/src/Illuminate/Console/Command.php(211): Illuminate\Container\Container->call()
#12 /var/www/domain-monitor-api/vendor/symfony/console/Command/Command.php(326): Illuminate\Console\Command->execute()
#13 /var/www/domain-monitor-api/vendor/laravel/framework/src/Illuminate/Console/Command.php(181): Symfony\Component\Console\Command\Command->run()
#14 /var/www/domain-monitor-api/vendor/symfony/console/Application.php(1081): Illuminate\Console\Command->run()
#15 /var/www/domain-monitor-api/vendor/symfony/console/Application.php(320): Symfony\Component\Console\Application->doRunCommand()
#16 /var/www/domain-monitor-api/vendor/symfony/console/Application.php(174): Symfony\Component\Console\Application->doRun()
#17 /var/www/domain-monitor-api/vendor/laravel/framework/src/Illuminate/Foundation/Console/Kernel.php(201): Symfony\Component\Console\Application->run()
#18 /var/www/domain-monitor-api/artisan(37): Illuminate\Foundation\Console\Kernel->handle()
#19 {main}
Here's my queue.php config file:
<?php
return [
/*
|--------------------------------------------------------------------------
| Default Queue Connection Name
|--------------------------------------------------------------------------
|
| Laravel's queue API supports an assortment of back-ends via a single
| API, giving you convenient access to each back-end using the same
| syntax for every one. Here you may define a default connection.
|
*/
'default' => env('QUEUE_CONNECTION', 'sync'),
/*
|--------------------------------------------------------------------------
| Queue Connections
|--------------------------------------------------------------------------
|
| Here you may configure the connection information for each server that
| is used by your application. A default configuration has been added
| for each back-end shipped with Laravel. You are free to add more.
|
| Drivers: "sync", "database", "beanstalkd", "sqs", "redis", "null"
|
*/
'connections' => [
'sync' => [
'driver' => 'sync',
],
// for short running queues
'database-short-running' => [
'driver' => 'database',
'table' => 'jobs',
'queue' => 'short-running',
'retry_after' => 80,
'after_commit' => false,
],
'database' => [
'driver' => 'database',
'table' => 'jobs',
'queue' => 'default',
'retry_after' => 90,
'after_commit' => false,
],
// for long running queues
'database-long-running' => [
'driver' => 'database',
'table' => 'jobs',
'queue' => 'long-running',
'retry_after' => 300,
'after_commit' => false,
],
'beanstalkd' => [
'driver' => 'beanstalkd',
'host' => 'localhost',
'queue' => 'default',
'retry_after' => 90,
'block_for' => 0,
'after_commit' => false,
],
'sqs' => [
'driver' => 'sqs',
'key' => env('AWS_ACCESS_KEY_ID'),
'secret' => env('AWS_SECRET_ACCESS_KEY'),
'prefix' => env('SQS_PREFIX', 'https://sqs.us-east-1.amazonaws.com/your-account-id'),
'queue' => env('SQS_QUEUE', 'default'),
'suffix' => env('SQS_SUFFIX'),
'region' => env('AWS_DEFAULT_REGION', 'us-east-1'),
'after_commit' => false,
],
'redis' => [
'driver' => 'redis',
'connection' => 'default',
'queue' => env('REDIS_QUEUE', 'default'),
'retry_after' => 90,
'block_for' => null,
'after_commit' => false,
],
],
/*
|--------------------------------------------------------------------------
| Failed Queue Jobs
|--------------------------------------------------------------------------
|
| These options configure the behavior of failed queue job logging so you
| can control which database and table are used to store the jobs that
| have failed. You may change them to any database / table you wish.
|
*/
'failed' => [
'driver' => env('QUEUE_FAILED_DRIVER', 'database'),
'database' => env('DB_CONNECTION', 'mysql'),
'table' => 'failed_jobs',
],
];
My CertificateChecker job file:
<?php
namespace App\Jobs;
use App\Models\Certificate;
use App\Models\CertificateCheck;
use App\Models\Domain;
use Carbon\Carbon;
use Illuminate\Bus\Queueable;
use Illuminate\Contracts\Queue\ShouldQueue;
use Illuminate\Contracts\Queue\ShouldBeUniqueUntilProcessing;
use Illuminate\Foundation\Bus\Dispatchable;
use Illuminate\Queue\InteractsWithQueue;
use Illuminate\Queue\SerializesModels;
use Illuminate\Support\Facades\Log;
class CertificateChecker implements ShouldQueue, ShouldBeUniqueUntilProcessing
{
use Dispatchable, InteractsWithQueue, Queueable, SerializesModels;
/**
* The number of times the job may be attempted.
*
* @var int
*/
public $tries = 1;
/**
* The number of seconds the job can run before timing out.
*
* @var int
*/
public $timeout = 60;
/**
* The number of seconds after which the job's unique lock will be released.
*
* @var int
*/
public $uniqueFor = 300;
/**
* The certificate id
*/
protected $certificate;
/**
* When we checked this SSL
*/
protected $checkedAt;
/**
* Create a new job instance.
*
* @return void
*/
public function __construct($certificate, $checkedAt)
{
$this->onConnection('database-short-running');
$this->certificate = $certificate;
$this->checkedAt = $checkedAt;
}
/**
* The unique ID of the job.
*/
public function uniqueId(): string
{
return 'certificate_check_id_'.$this->certificate;
}
/**
* Get SSL data from domain
*/
public function obtainCertificateData($domain)
{
try {
$streamContext = stream_context_create([
'ssl' => [
'capture_peer_cert' => true,
],
]);
$domainToCheck = 'ssl://'.$domain.':443';
$client = @stream_socket_client($domainToCheck,
$errorNumber,
$errorDescription,
5,
STREAM_CLIENT_CONNECT,
$streamContext
);
// 110 - connection timed out (might be too short connection)
// 111 - connection refused (could be wrong domain)
// 113 - no route to host (wrong domain?)
if (in_array($errorNumber, [110, 111, 113])) {
return null;
}
// stream_socket_client returned false, likely an issue with checking
// the ssl certificate and could be unsupported by our platform, or
// incorrect input provided.
if (! $client) {
return null;
}
$response = stream_context_get_params($client);
$response = openssl_x509_parse($response['options']['ssl']['peer_certificate'], false);
return $response;
} catch (\Exception $e) {
}
// invalid ssl, or ssl doesn't exist
return null;
}
/**
* Get certificate
*/
public function getCertificate($certificate)
{
$certificate = Certificate::where('id', $certificate)
->whereNull('disabled_at')
->first();
return $certificate;
}
/**
* Update certificate
*/
public function updateCertificate($certificate, $fields)
{
$certificate->is_valid = $fields['is_valid'];
$certificate->expires_on = $fields['expires_on'];
$certificate->last_checked_at = $this->checkedAt;
$certificate->save();
}
/**
* Create a certificate check entry
*/
public function createCertificateCheckEntry($certificate, $ssl)
{
$certificateCheck = new CertificateCheck;
// meta
$certificateCheck->certificate_id = $certificate->id;
$certificateCheck->user_id = $certificate->user_id;
// SSL subject
if (isset($ssl['subject']) && ! empty($ssl['subject'])) {
$certificateCheck->subject_country_name = (isset($ssl['subject']['countryName']) && ! empty($ssl['subject']['countryName'])) ? $ssl['subject']['countryName'] : null;
$certificateCheck->subject_state_or_province_name = (isset($ssl['subject']['stateOrProvinceName']) && ! empty($ssl['subject']['stateOrProvinceName'])) ? $ssl['subject']['stateOrProvinceName'] : null;
$certificateCheck->subject_locality_name = (isset($ssl['subject']['localityName']) && ! empty($ssl['subject']['localityName'])) ? $ssl['subject']['localityName'] : null;
$certificateCheck->subject_organisation_name = (isset($ssl['subject']['organizationName']) && ! empty($ssl['subject']['organizationName'])) ? $ssl['subject']['organizationName'] : null;
$certificateCheck->subject_common_name = (isset($ssl['subject']['commonName']) && ! empty($ssl['subject']['commonName'])) ? $ssl['subject']['commonName'] : null;
}
// SSL issuer
if (isset($ssl['issuer']) && ! empty($ssl['issuer'])) {
$certificateCheck->issuer_country_name = (isset($ssl['issuer']['countryName']) && ! empty($ssl['issuer']['countryName'])) ? $ssl['issuer']['countryName'] : null;
$certificateCheck->issuer_state_or_province_name = (isset($ssl['issuer']['stateOrProvinceName']) && ! empty($ssl['issuer']['stateOrProvinceName'])) ? $ssl['issuer']['stateOrProvinceName'] : null;
$certificateCheck->issuer_locality_name = (isset($ssl['issuer']['localityName']) && ! empty($ssl['issuer']['localityName'])) ? $ssl['issuer']['localityName'] : null;
$certificateCheck->issuer_organisation_name = (isset($ssl['issuer']['organizationName']) && ! empty($ssl['issuer']['organizationName'])) ? $ssl['issuer']['organizationName'] : null;
$certificateCheck->issuer_common_name = (isset($ssl['issuer']['commonName']) && ! empty($ssl['issuer']['commonName'])) ? $ssl['issuer']['commonName'] : null;
}
// SSL version number
$certificateCheck->version = (isset($ssl['version']) && ! empty($ssl['version'])) ? intval($ssl['version']) : 1;
// SSL serial numbers
$certificateCheck->serial_number = (isset($ssl['serialNumber']) && ! empty($ssl['serialNumber'])) ? $ssl['serialNumber'] : null;
$certificateCheck->serial_number_hex = (isset($ssl['serialNumberHex']) && ! empty($ssl['serialNumberHex'])) ? $ssl['serialNumberHex'] : null;
// SSL signatures
$certificateCheck->signature_type_sn = (isset($ssl['signatureTypeSN']) && ! empty($ssl['signatureTypeSN'])) ? $ssl['signatureTypeSN'] : null;
$certificateCheck->signature_type_ln = (isset($ssl['signatureTypeLN']) && ! empty($ssl['signatureTypeLN'])) ? $ssl['signatureTypeLN'] : null;
$certificateCheck->signature_type_nid = (isset($ssl['signatureTypeNID']) && ! empty($ssl['signatureTypeNID'])) ? $ssl['signatureTypeNID'] : null;
// SSL valid from
if (isset($ssl['validFrom_time_t']) && ! empty($ssl['validFrom_time_t'])) {
try {
$certificateCheck->valid_from = Carbon::createFromTimestamp($ssl['validFrom_time_t'])->toDateTimeString();
} catch (\Exception $e) {
}
}
// SSL valid to
if (isset($ssl['validTo_time_t']) && ! empty($ssl['validTo_time_t'])) {
try {
$certificateCheck->valid_to = Carbon::createFromTimestamp($ssl['validTo_time_t'])->toDateTimeString();
} catch (\Exception $e) {
}
}
$certificateCheck->checked_at = $this->checkedAt;
$certificateCheck->save();
// return the saved check
return $certificateCheck;
}
/**
* Delete old records
*/
public function deleteOldRecords($certificate)
{
$checks = CertificateCheck::select('id')
->where('certificate_id', $certificate->id)
->where('user_id', $certificate->user_id)
->orderBy('id', 'desc')
->skip(3)
->take(10000)
->pluck('id')
->toArray();
if (isset($checks) && count($checks) > 0) {
CertificateCheck::whereIn('id', $checks)->chunk(500, function ($checks) {
foreach ($checks as $check) {
$check->delete();
}
});
}
}
/**
* Execute the job.
*/
public function handle(): void
{
$certificate = $this->getCertificate($this->certificate);
// the certificate
if (! $certificate) {
return;
}
// the domain name
if (! isset($certificate->domain)) {
return;
}
// get certificate data for domain
$sslData = $this->obtainCertificateData($certificate->domain->domain);
$savedCheck = $this->createCertificateCheckEntry($certificate, $sslData);
// no ssl data
if (! $sslData) {
$this->updateCertificate($certificate, [
'is_valid' => false,
'expires_on' => null,
]);
if ($savedCheck) {
$this->deleteOldRecords($certificate);
}
return;
}
// no ssl expiry
if (! isset($sslData['validTo_time_t']) || empty($sslData['validTo_time_t'])) {
$this->updateCertificate($certificate, [
'is_valid' => false,
'expires_on' => null,
]);
if ($savedCheck) {
$this->deleteOldRecords($certificate);
}
return;
}
// valid
if ($savedCheck && $savedCheck->valid_to) {
$this->updateCertificate($certificate, [
'is_valid' => true,
'expires_on' => $savedCheck->valid_to,
]);
if ($savedCheck) {
$this->deleteOldRecords($certificate);
}
return;
} else {
$this->updateCertificate($certificate, [
'is_valid' => false,
'expires_on' => null,
]);
if ($savedCheck) {
$this->deleteOldRecords($certificate);
}
}
}
}
Running it on supervisor:
php artisan queue:work --queue=on-demand-runs-now,cron,uptime,notifications,statistics-hourly,domains,dns,certificates,domains-expired,subscriptions,blacklists,listeners,statistics-daily,short-running,long-running,default --tries=2 --max-time=900
Is anyone able to defeat this error once and for all?