What is the difference in Python between "dictionary changed size during iteration" and "dictionary keys changed during iteration" exception

Question

I am preparing material about how iterators work in Python and I am currently working on a part about what happens if you change the dictionary you are iterating over. I know and added to my material that it is bad practice, but I still wanted to try to see what happens.

The code below printed the key 5 and gave me a "dictionary changed size during iteration" exception. So I got the exception the moment I had a change and wanted to go to the next one.

d = {5:5, 10:10, 15:15, 20:20}
for elem in d:
    print(elem)
    d[elem+1] = elem+1

This is OK. But then it occurred to me, what if I do both an add and a remove, so that the size technically, does not change? So I wrote this:

d = {5:5, 10:10, 15:15, 20:20, 25:25, 30:30, 35:35}
for elem in d:
    print(elem)
    del d[elem]
    d[elem+1] = elem+1  

It prints as many keys as elements in the dictionary (7), some of them being already newly added values, but after printing 7 values it throws a "dictionary keys changed during iteration" exception. And now I am really curious what happens internally, that leads to this. I still know it is bad practice, I just want to understand it. Thanks!

Answer 1

Yeah that's quite a "pythonic" thing: Python's dictionaries and iterators are not designed to handle changes dynamically, hence the exceptions you observed.

The reason is because when you iterate over a dictionary, it creates an iterator object that keeps track of the state of the iteration by traversing the dictionary's keys one by one. However, if you modify the dictionary while iterating over it, Python can't guarantee the integrity of the iteration state. This is because modifying the dictionary can change its internal structure, potentially invalidating the current iteration state.

In your second code, you are deleting the current key elem from the dictionary and adding a new key elem+1 in each iteration, but when you delete a key, it alters the internal structure of the dictionary. Even though you are adding a new key immediately after, Python's iterator doesn't handle this scenario well because it's not designed to track dynamic changes like these.

When Python tries to fetch the next key during iteration, it discovers that the dictionary's internal structure has changed, leading to the "dictionary keys changed during iteration" exception.

Why the first code immediately raises the RunTimeError:

In the first code,dictionary changed size during iteration. Python cannot dynamically handle changes to the size of the dictionary while iterating over it. When you add a new element during the iteration, Python detects that the size of the dictionary has changed from its initial state, violating the assumption made by the iterator about the total number of elements in the dictionary.

Why it does not happen in the second:

In the second code, you are deleting and adding elements to the dictionary while iterating over it. Python doesn't immediately detect changes to the size of the dictionary. It continues iterating through the dictionary and making changes until it reaches the end of the iteration. Only at the end of the for loop does Python check if the size of the dictionary has been modified during the iteration.

So, in the second code, the exception is raised after a complete iteration of the dictionary because Python performs the check on the dictionary size only at the end of the iteration.

Answer 2

You should not try to add or delete entries from Python list types while iterating their conent as it can confuse the iterator. You should iterate a shallow copy instead, so the original list can be modified. Although you may still have a problem if you add or remove items. In the case above you can do it by getting a list of the keys thus:

for elem in list(d):

See Mapping types - dict for further details. You may also find that the Python documentation does a very good job itself of explaining how iterators work.