I was debugging a Python import issue where I can't load jinja2 properly while running my code using a pex file.
Setup a breakpoint in my code, and found myself in a very interesting situation
(Pdb) import jinja2
(Pdb) import jinja2.utils
(Pdb) jinja2.utils
*** AttributeError: 'module' object has no attribute 'utils'
(Pdb) sys.modules.get("jinja2")
<module 'jinja2' from '/home/xxx/.pex/install/Jinja2-2.10.3-py2.py3-none-any.whl.02f611acfe90225e024bd496a640e01844bf5a32/Jinja2-2.10.3-py2.py3-none-any.whl/jinja2/__init__.py'>
(Pdb) sys.modules.get("jinja2.utils")
<module 'jinja2.utils' from '/usr/local/lib/python2.7/dist-packages/jinja2/utils.pyc'>
(Pdb) [i for i in sys.path if "usr/local/lib/python2.7" in i]
[]
Before the breakpoint, jinja2 was already partially loaded because I set up the breakpoint inside jinja2's __init__.py, here is the stacktrace:
-> exec code in run_globals
/home/xxx/.pex/code/e7fe7e585c258eb7224ec5030b9d53b74cdfb06f/application/app.py(2)<module>()
-> from flask import Flask, abort, request
/home/xxx/.pex/install/Flask-1.0-py2.py3-none-any.whl.abe8dd5f49600e54576b3360b7622e1cc97e86fc/Flask-1.0-py2.py3-none-any.whl/flask/__init__.py(19)<module>()
-> from jinja2 import Markup, escape
> /home/xxx/.pex/install/Jinja2-2.10.3-py2.py3-none-any.whl.02f611acfe90225e024bd496a640e01844bf5a32/Jinja2-2.10.3-py2.py3-none-any.whl/jinja2/__init__.py(58)<module>()
-> print("aaaaaaaaa")
Questions are:
- How does python even load the jinja2.utils from
/usr/local/lib/python2.7/dist-packageswhen it is not even not part of the sys.path? - Why python found jinja2's
__init__.pypath correctly but failed to find theutils.pywhich is under the same directory?
~$ ls /home/xxx/.pex/install/Jinja2-2.10.3-py2.py3-none-any.whl.02f611acfe90225e024bd496a640e01844bf5a32/Jinja2-2.10.3-py2.py3-none-any.whl/jinja2/
asyncfilters.py bccache.py compiler.py debug.py environment.py ext.py _identifier.py __init__.py loaders.py nativetypes.py optimizer.py runtime.py tests.py visitor.py
asyncsupport.py _compat.py constants.py defaults.py exceptions.py filters.py idtracking.py lexer.py meta.py nodes.py parser.py sandbox.py utils.py
Confirmed it is a pex issue.
Pex loaded
jinja2in the beginning (which also includedjinja2.utils), then it removed thejinaj2module and the path but failed to removejinja2.utils.The issue had been fixed with latest version of pex:https://github.com/pantsbuild/pex/commit/0f9bf622e1565dcdd27c3b79b788571c9608c272