qbo3 Import Running Multiple Times

Question

I have queued a qbo3 API call importfile/import to bulk import a file with 80M records. The job completes successfully, but continues to run. The logging shows:

• Queued @ 2:39:42 AM
• Processing @ 2:39:54 AM
• Processing @ 3:39:57 AM
• Success @ 3:54:29 AM
• Processing @ 4:40:04 AM
• Success @ 5:11:16 AM
• Processing @ 5:40:09 AM
• ... and so on ...

How do I prevent repeated executions.

Answer 1

The root cause of this issue is Amazon's SQS visibility timeout, which defaults to 1 hour (3600 seconds). The clue to this answer is in the log's Processing times being almost exactly 1 hour apart:

• Processing @ 2:39:54 AM
• Processing @ 3:39:57 AM
• Processing @ 4:40:04 AM
• Processing @ 5:40:09 AM

What's happening is:

• qbo3 QueueService pops the message from SQS @ 2:39
  • starts processing the import
• at 3:39, SQS makes that message available again, assuming the consumer has timed out
• qbo3 QueueService pops the message (again) from SQS @ 3:39
  • starts processing the import - again
  • note that the first thread has not completed the import, so it's eligible to start again
• at 3:54 the the first thread completes the job
  • but the second thread is already processing the import again, based on the SQS message
• rinse and repeat

The solution is to extend the SQS visibility timeout to be greater than the maximum time you expect any job to take to be successful.

The easiest way to do this is set the application setting qbo.Queue.Amazon.Properties.Settings.SQSVisibilityTimeout via a SystemDefault. In this case, the job took about 90 minutes; consider making the timeout 14400 seconds (4 hours).

From Home > Import > Test Harness, import the following:

<SystemDefaultCollection>
  <SystemDefaultItem>
    <SystemDefault>qbo.Queue.Amazon.Properties.Settings.SQSVisibilityTimeout</SystemDefault>
    <Value>14400</Value>
  </SystemDefaultItem>
</SystemDefaultCollection>