AttributeError caused by existing attribute

Question

I sometimes get the following errors when my django app tries to get from or store something in the cache:

    c = cache.get(pk)
  File "/opt/python3/lib/python3.4/site-packages/django/core/cache/__init__.py", line 131, in __getattr__
    return getattr(caches[DEFAULT_CACHE_ALIAS], name)
  File "/opt/python3/lib/python3.4/site-packages/django/core/cache/__init__.py", line 113, in __getitem__
    cache = _create_cache(alias)
  File "/opt/python3/lib/python3.4/site-packages/django/core/cache/__init__.py", line 88, in _create_cache
    return backend_cls(location, params)
  File "/opt/python3/lib/python3.4/site-packages/django/core/cache/backends/memcached.py", line 185, in __init__
    value_not_found_exception=pylibmc.NotFound)
AttributeError: 'module' object has no attribute 'NotFound'

but why? the module has the attribute, and most of the time it works, there is no file with the same name that can break this, where to look for the cause of this?

>>> import pylibmc
>>> pylibmc.NotFound
<class '_pylibmc.NotFound'>
>>>

Answer 1

tl;dr: try importing pylibmc at an application startup location, such as the uwsgi or manage.py file.

My guess is that it's a multithreading issue while importing pylibmc in a request thread inside PyLibMCCache.__init__ as opposed to at application startup. (IMO Django does the import there because not all Django installations use pylibmc, and thus they shouldn't force it on each app as a dependency)

While I'm not familiar enough with the internals of how importing works, I suspect what happens is something like the below

1. thread #1 tries to import pylibmc
2. thread #1 places a placeholder in sys.modules for pylibmc
3. thread #2 tries to import pylibmc -> the AttributeError is raised
4. thread #1 finished updating sys.modules and now pylibmc.NotFound is available

In general, Python seem to discourage runtime loading of modules as opposed to startup time loading.

emphasis is mine

Note: For projects where startup time is critical, this class [importlib.util.LazyLoader] allows for potentially minimizing the cost of loading a module if it is never used. For projects where startup time is not essential then use of this class is heavily discouraged due to error messages created during loading being postponed and thus occurring out of context.