remove duplicates from the for loop from QueueTrigger python script

Question

I have written a queueTrigger - Azure functions

def main(msg: func.QueueMessage) -> None:
    logging.info('Python queue trigger function processed a queue item: %s',
                 msg.get_body().decode('utf-8'))
    
    mk_client = MyKiosk()
    storage = StorageAccount(os.environ["AzureWebJobsStorage"])

    data = mk_client.get_allgroupdata()


    for i in data:
        main_id = i["HauptgruppeId"]
        for j in i["Untergruppe"]:
            sub_id = j["UntergruppeId"]
            outbound_msg = {"mainid_val": main_id,"subid_val": sub_id,"n_records": 12}
              
              <code to remove duplicates>

            storage.write_queue_message("pricequeue", json.dumps(outbound_msg))

This is my QueueTrigger script and I want to make sure there are no duplicates in 'outbound_msg' and unique messages can be written in the 'pricequeue'

Answer 1

You can remove the duplicates from queue trigger python script by using Popleft or collections dequeue

Below is the example script for using popleft

>>> from collections import deque
>>> queue = deque(["Eric", "John", "Michael"])
>>> queue.append("Terry")           # Terry arrives
>>> queue.append("Graham")          # Graham arrives
>>> queue.popleft()                 # The first to arrive now leaves
'Eric'
>>> queue.popleft()                 # The second to arrive now leaves
'John'
>>> queue                           # Remaining queue in order of arrival
deque(['Michael', 'Terry', 'Graham'])

Also here are few other links which will give you complete information about removing duplicates and which are having related discussions.

• Fastest way in uniqify a list in python.
• Remove Duplicates from a python list using for loops.