Unable to compile lxml using pip and NetBSD

Question

I'm running NetBSD 7.1 and I'm trying to install Python's lxml package, using pip.

However I get this compilation error:

# pip install lxml
Collecting lxml
  Using cached lxml-3.8.0.tar.gz
Installing collected packages: lxml
  Running setup.py install for lxml ... error
    Complete output from command /usr/pkg/bin/python2.7 -u -c "import setuptools, tokenize;__file__='/tmp/pip-build-Xb5RT3/lxml/setup.py';f=getattr(tokenize, 'open', open)(__file__);code=f.read().replace('\r\n', '\n');f.close();exec(compile(code, __file__, 'exec'))" install --record /tmp/pip-N6HUOc-record/install-record.txt --single-version-externally-managed --compile:
    Building lxml version 3.8.0.
    Building without Cython.
    Using build configuration of libxslt 1.1.29
    Building against libxml2/libxslt in one of the following directories:
      /usr/pkg/lib
      /usr/lib
    running install
    running build
    running build_py
    creating build
    creating build/lib.netbsd-7.1-amd64-2.7
    creating build/lib.netbsd-7.1-amd64-2.7/lxml
    copying src/lxml/sax.py -> build/lib.netbsd-7.1-amd64-2.7/lxml
    copying src/lxml/_elementpath.py -> build/lib.netbsd-7.1-amd64-2.7/lxml
    copying src/lxml/builder.py -> build/lib.netbsd-7.1-amd64-2.7/lxml
    copying src/lxml/ElementInclude.py -> build/lib.netbsd-7.1-amd64-2.7/lxml
    copying src/lxml/__init__.py -> build/lib.netbsd-7.1-amd64-2.7/lxml
    copying src/lxml/usedoctest.py -> build/lib.netbsd-7.1-amd64-2.7/lxml
    copying src/lxml/cssselect.py -> build/lib.netbsd-7.1-amd64-2.7/lxml
    copying src/lxml/pyclasslookup.py -> build/lib.netbsd-7.1-amd64-2.7/lxml
    copying src/lxml/doctestcompare.py -> build/lib.netbsd-7.1-amd64-2.7/lxml
    creating build/lib.netbsd-7.1-amd64-2.7/lxml/includes
    copying src/lxml/includes/__init__.py -> build/lib.netbsd-7.1-amd64-2.7/lxml/includes
    creating build/lib.netbsd-7.1-amd64-2.7/lxml/html
    copying src/lxml/html/diff.py -> build/lib.netbsd-7.1-amd64-2.7/lxml/html
    copying src/lxml/html/ElementSoup.py -> build/lib.netbsd-7.1-amd64-2.7/lxml/html
    copying src/lxml/html/html5parser.py -> build/lib.netbsd-7.1-amd64-2.7/lxml/html
    copying src/lxml/html/builder.py -> build/lib.netbsd-7.1-amd64-2.7/lxml/html
    copying src/lxml/html/__init__.py -> build/lib.netbsd-7.1-amd64-2.7/lxml/html
    copying src/lxml/html/usedoctest.py -> build/lib.netbsd-7.1-amd64-2.7/lxml/html
    copying src/lxml/html/_setmixin.py -> build/lib.netbsd-7.1-amd64-2.7/lxml/html
    copying src/lxml/html/clean.py -> build/lib.netbsd-7.1-amd64-2.7/lxml/html
    copying src/lxml/html/soupparser.py -> build/lib.netbsd-7.1-amd64-2.7/lxml/html
    copying src/lxml/html/_html5builder.py -> build/lib.netbsd-7.1-amd64-2.7/lxml/html
    copying src/lxml/html/_diffcommand.py -> build/lib.netbsd-7.1-amd64-2.7/lxml/html
    copying src/lxml/html/formfill.py -> build/lib.netbsd-7.1-amd64-2.7/lxml/html
    copying src/lxml/html/defs.py -> build/lib.netbsd-7.1-amd64-2.7/lxml/html
    creating build/lib.netbsd-7.1-amd64-2.7/lxml/isoschematron
    copying src/lxml/isoschematron/__init__.py -> build/lib.netbsd-7.1-amd64-2.7/lxml/isoschematron
    copying src/lxml/lxml.etree.h -> build/lib.netbsd-7.1-amd64-2.7/lxml
    copying src/lxml/lxml.etree_api.h -> build/lib.netbsd-7.1-amd64-2.7/lxml
    copying src/lxml/includes/etreepublic.pxd -> build/lib.netbsd-7.1-amd64-2.7/lxml/includes
    copying src/lxml/includes/xinclude.pxd -> build/lib.netbsd-7.1-amd64-2.7/lxml/includes
    copying src/lxml/includes/xmlerror.pxd -> build/lib.netbsd-7.1-amd64-2.7/lxml/includes
    copying src/lxml/includes/htmlparser.pxd -> build/lib.netbsd-7.1-amd64-2.7/lxml/includes
    copying src/lxml/includes/config.pxd -> build/lib.netbsd-7.1-amd64-2.7/lxml/includes
    copying src/lxml/includes/xmlschema.pxd -> build/lib.netbsd-7.1-amd64-2.7/lxml/includes
    copying src/lxml/includes/uri.pxd -> build/lib.netbsd-7.1-amd64-2.7/lxml/includes
    copying src/lxml/includes/xslt.pxd -> build/lib.netbsd-7.1-amd64-2.7/lxml/includes
    copying src/lxml/includes/tree.pxd -> build/lib.netbsd-7.1-amd64-2.7/lxml/includes
    copying src/lxml/includes/xmlparser.pxd -> build/lib.netbsd-7.1-amd64-2.7/lxml/includes
    copying src/lxml/includes/dtdvalid.pxd -> build/lib.netbsd-7.1-amd64-2.7/lxml/includes
    copying src/lxml/includes/xpath.pxd -> build/lib.netbsd-7.1-amd64-2.7/lxml/includes
    copying src/lxml/includes/relaxng.pxd -> build/lib.netbsd-7.1-amd64-2.7/lxml/includes
    copying src/lxml/includes/c14n.pxd -> build/lib.netbsd-7.1-amd64-2.7/lxml/includes
    copying src/lxml/includes/schematron.pxd -> build/lib.netbsd-7.1-amd64-2.7/lxml/includes
    copying src/lxml/includes/etree_defs.h -> build/lib.netbsd-7.1-amd64-2.7/lxml/includes
    copying src/lxml/includes/lxml-version.h -> build/lib.netbsd-7.1-amd64-2.7/lxml/includes
    creating build/lib.netbsd-7.1-amd64-2.7/lxml/isoschematron/resources
    creating build/lib.netbsd-7.1-amd64-2.7/lxml/isoschematron/resources/rng
    copying src/lxml/isoschematron/resources/rng/iso-schematron.rng -> build/lib.netbsd-7.1-amd64-2.7/lxml/isoschematron/resources/rng
    creating build/lib.netbsd-7.1-amd64-2.7/lxml/isoschematron/resources/xsl
    copying src/lxml/isoschematron/resources/xsl/RNG2Schtrn.xsl -> build/lib.netbsd-7.1-amd64-2.7/lxml/isoschematron/resources/xsl
    copying src/lxml/isoschematron/resources/xsl/XSD2Schtrn.xsl -> build/lib.netbsd-7.1-amd64-2.7/lxml/isoschematron/resources/xsl
    creating build/lib.netbsd-7.1-amd64-2.7/lxml/isoschematron/resources/xsl/iso-schematron-xslt1
    copying src/lxml/isoschematron/resources/xsl/iso-schematron-xslt1/iso_dsdl_include.xsl -> build/lib.netbsd-7.1-amd64-2.7/lxml/isoschematron/resources/xsl/iso-schematron-xslt1
    copying src/lxml/isoschematron/resources/xsl/iso-schematron-xslt1/iso_abstract_expand.xsl -> build/lib.netbsd-7.1-amd64-2.7/lxml/isoschematron/resources/xsl/iso-schematron-xslt1
    copying src/lxml/isoschematron/resources/xsl/iso-schematron-xslt1/iso_schematron_skeleton_for_xslt1.xsl -> build/lib.netbsd-7.1-amd64-2.7/lxml/isoschematron/resources/xsl/iso-schematron-xslt1
    copying src/lxml/isoschematron/resources/xsl/iso-schematron-xslt1/iso_schematron_message.xsl -> build/lib.netbsd-7.1-amd64-2.7/lxml/isoschematron/resources/xsl/iso-schematron-xslt1
    copying src/lxml/isoschematron/resources/xsl/iso-schematron-xslt1/iso_svrl_for_xslt1.xsl -> build/lib.netbsd-7.1-amd64-2.7/lxml/isoschematron/resources/xsl/iso-schematron-xslt1
    copying src/lxml/isoschematron/resources/xsl/iso-schematron-xslt1/readme.txt -> build/lib.netbsd-7.1-amd64-2.7/lxml/isoschematron/resources/xsl/iso-schematron-xslt1
    running build_ext
    building 'lxml.etree' extension
    creating build/temp.netbsd-7.1-amd64-2.7
    creating build/temp.netbsd-7.1-amd64-2.7/src
    creating build/temp.netbsd-7.1-amd64-2.7/src/lxml
    gcc -fno-strict-aliasing -O2 -pthread -I/usr/include -I/usr/pkg/include -DNDEBUG -O2 -pthread -I/usr/include -I/usr/pkg/include -fPIC -I/usr/pkg/include -I/usr/pkg/include/libxml2 -I/usr/include -Isrc/lxml/includes -I/usr/pkg/include/python2.7 -c src/lxml/lxml.etree.c -o build/temp.netbsd-7.1-amd64-2.7/src/lxml/lxml.etree.o -w
    In file included from /usr/pkg/include/libxml2/limits.h:168:0,
                     from /usr/pkg/include/libxml2/syslimits.h:7,
                     from /usr/pkg/include/libxml2/limits.h:34,
                     from /usr/pkg/include/python2.7/Python.h:19,
                     from src/lxml/lxml.etree.c:84:
    /usr/pkg/gcc5/lib/gcc/x86_64--netbsd/5.4.0/include-fixed/limits.h:168:61: fatal error: limits.h: No such file or directory
    compilation terminated.
    Compile failed: command 'gcc' failed with exit status 1
    creating tmp
    cc -I/usr/pkg/include -I/usr/pkg/include/libxml2 -I/usr/include -I/usr/include/libxml2 -c /tmp/xmlXPathInit_nXifU.c -o tmp/xmlXPathInit_nXifU.o
    unable to execute 'cc': No such file or directory
    *********************************************************************************
    Could not find function xmlCheckVersion in library libxml2. Is libxml2 installed?
    *********************************************************************************
    error: command 'gcc' failed with exit status 1

    ----------------------------------------
Command "/usr/pkg/bin/python2.7 -u -c "import setuptools, tokenize;__file__='/tmp/pip-build-Xb5RT3/lxml/setup.py';f=getattr(tokenize, 'open', open)(__file__);code=f.read().replace('\r\n', '\n');f.close();exec(compile(code, __file__, 'exec'))" install --record /tmp/pip-N6HUOc-record/install-record.txt --single-version-externally-managed --compile" failed with error code 1 in /tmp/pip-build-Xb5RT3/lxml/

I'm on a clean NetBSD 7.1 minimal installation, but have added the following using pkgsrc:

# pkg_info
xmlcatmgr-2.2nb1    XML and SGML catalog manager
libxml2-2.9.4nb3    XML parser library from the GNOME project
libgpg-error-1.27   Definitions of common error values for all GnuPG components
libgcrypt-1.7.6     GNU cryptographic library
libxslt-1.1.29nb2   XSLT parser library from the GNOME project
mozilla-rootcerts-1.0.20170121nb5 Root CA certificates from the Mozilla Project
libffi-3.2.1nb2     Foreign function interface
python27-2.7.13nb1  Interpreted, interactive, object-oriented programming language
py27-expat-2.7.13   Python interface to expat
py27-appdirs-1.4.0  Module for determining appropriate, platform-specific dirs
py27-pyparsing-2.1.10 Pyparsing module for Python
py27-six-1.10.0     Python 2 and 3 compatibility utilities
py27-packaging-16.8 Core utilities for Python packages
py27-setuptools-34.3.2 New Python packaging system
py27-pip-9.0.1      Installs Python packages as an easy_install replacement
libxml-1.8.17nb4    XML parser (version 1), mainly used by the GNOME project
py27-libxml2-2.9.4nb2 Python wrapper for libxml2
py27-libxslt-1.1.29 Python wrapper for libxslt
gccmakedep-1.0.3    Create dependencies in Makefiles using gcc
gcc5-5.4.0nb2       The GNU Compiler Collection (GCC) - 5 Release Series
boost-headers-1.63.0 Free, peer-reviewed portable C++ source libraries (build-time headers)
compat_headers-0.2  Compatibility headers

Answer 1

This was due to the compiler set not being installed, as it's not part of the "minimal installation" package.

To keep things at minimum, one can choose "Custom installation" option instead and set Compiler Tools to "Yes".

Another way is to download and install the comp.tgz manually:

# cd /
# ftp ftp://ftp.netbsd.org (as an anonymous user)
> cd pub/NetBSD/NetBSD-7.1/amd64/binary/sets
> get comp.tgz
> exit
# tar --unlink -zxvpf comp.tgz

More details available on NetBSD's FAQ.