I am having an issue where data is being published to a topic within AWS using an IoT Thing, and a Lambda function designed to upload this data to an RDS MySQL table uploads the same row twice. Data is published by the IoT device at most once every 10 seconds, so my initial solution was to use the timestamp as an idempotency check. This works about half the time, but sometimes the Lambda function is invoked for the same event twice almost exactly at the same time. The Lambda function takes 8-10ms to run, so when the Lambda function is invoked twice less than 8ms apart, the timestamp has not yet been added to the table and both invocations add the same row of data from the same MQTT event. It seems that invocations where this method of idempotency work are logged in the same logstream, but instances that occur very close together are in different logstreams (perhaps because a new RDS connection must be created?).
Many of the issues I have read online stem from errors causing the Lambda to run twice about a second apart. If this were the case, the idempotency check should work (and it does work when using a test event or when Lambda functions are not run at the exact same time). Has anyone experienced this issue?
Any advice or suggestions would be greatly appreciated.
What I tried:
Implementing an idempotency check using the following code to check if a Time_Stamp exists:
idempodence_str = f"select count(if(Time_Stamp = '{Time_Stamp}', 1, NULL)) from myTable"
with conn.cursor() as cur:
cur.execute(idempodence_str)
if (cur.fetchone()[0] < 1):
cur.execute(sql_string) #sql_string is a simple insert into myTable command
conn.commit()
return "Added row to RDS MySQL table"
else:
logger.info("Time_Stamp already exists")
return "Duplicate Lambda Invocation"