What is the maximum tuple array size in Python 3?

Question

I am building a web scraper that stores data retrieved from four different websites into a tuple array. I later iterate through the tuple and save the entire lot as both CSV and Excel. Are tuple arrays or arrays in general, limited to the processor's RAM/disc-space?

Thanks

Answer 1

According to the doc, this is given by sys.maxsize

sys.maxsize

An integer giving the maximum value a variable of type Py_ssize_t can take. It’s usually 2**31 - 1 on a 32-bit platform and 2**63 - 1 on a 64-bit platform.

And interestingly enough, this the Python3 doc about data model gives more implementation details under object.__len__.

CPython implementation detail: In CPython, the length is required to be at most sys.maxsize. If the length is larger than sys.maxsize some features (such as len()) may raise OverflowError.

Answer 2

I believe tuples and lists are limited by the size of the machine's virtual memory, unless you're on a 32 bit system in which case you're limited by the small word size. Also, lists are dynamically resized by... I believe about 12% each time they grow too small, so there's a little overhead there as well.

If you're concerned you're going to run out of virtual memory, it might be a good idea to write to a file or files instead.