Unable to install OpenFST Python extension

Question

I was trying to install OpenFST Python extension according to this guide. There were two options for doing this.
The 1st option one was to issue --enable-python during configuration of OpenFst, which I tried and failed.
So I went for the 2nd option. I successfully installed OpenFST with these commands:

./configure --enable-far
make
sudo make install

Then I tried to install PyPi package openfst with pip:
pip install openfst
and got the following error:

Collecting openfst
  Using cached https://files.pythonhosted.org/packages/cc/6b/cc05392480e2232e176be895b9ca2b9f4a5f153f99ab276b37d440b3bace/openfst-1.6.6.tar.gz
Building wheels for collected packages: openfst
  Running setup.py bdist_wheel for openfst ... error
  Complete output from command /home/arif/anaconda3/bin/python -u -c "import setuptools, tokenize;__file__='/tmp/pip-install-7y6dl6o2/openfst/setup.py';f=getattr(tokenize, 'open', open)(__file__);code=f.read().replace('\r\n', '\n');f.close();exec(compile(code, __file__, 'exec'))" bdist_wheel -d /tmp/pip-wheel-zadciiwk --python-tag cp36:
  running bdist_wheel
  running build
  running build_ext
  building 'pywrapfst' extension
  creating build
  creating build/temp.linux-x86_64-3.6
  gcc -pthread -B /home/arif/anaconda3/compiler_compat -Wl,--sysroot=/ -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -fPIC -I/home/arif/anaconda3/include/python3.6m -c pywrapfst.cc -o build/temp.linux-x86_64-3.6/pywrapfst.o -std=c++11 -Wno-unneeded-internal-declaration -Wno-unused-function
  cc1plus: warning: command line option ‘-Wstrict-prototypes’ is valid for C/ObjC but not for C++
  pywrapfst.cc:557:40: fatal error: fst/extensions/far/getters.h: No such file or directory
  compilation terminated.
  error: command 'gcc' failed with exit status 1

  ----------------------------------------
  Failed building wheel for openfst
  Running setup.py clean for openfst
Failed to build openfst
Installing collected packages: openfst
  Running setup.py install for openfst ... error
    Complete output from command /home/arif/anaconda3/bin/python -u -c "import setuptools, tokenize;__file__='/tmp/pip-install-7y6dl6o2/openfst/setup.py';f=getattr(tokenize, 'open', open)(__file__);code=f.read().replace('\r\n', '\n');f.close();exec(compile(code, __file__, 'exec'))" install --record /tmp/pip-record-3niypfz6/install-record.txt --single-version-externally-managed --compile:
    running install
    running build
    running build_ext
    building 'pywrapfst' extension
    creating build
    creating build/temp.linux-x86_64-3.6
    gcc -pthread -B /home/arif/anaconda3/compiler_compat -Wl,--sysroot=/ -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -fPIC -I/home/arif/anaconda3/include/python3.6m -c pywrapfst.cc -o build/temp.linux-x86_64-3.6/pywrapfst.o -std=c++11 -Wno-unneeded-internal-declaration -Wno-unused-function
    cc1plus: warning: command line option ‘-Wstrict-prototypes’ is valid for C/ObjC but not for C++
    pywrapfst.cc:557:40: fatal error: fst/extensions/far/getters.h: No such file or directory
    compilation terminated.
    error: command 'gcc' failed with exit status 1

    ----------------------------------------
Command "/home/arif/anaconda3/bin/python -u -c "import setuptools, tokenize;__file__='/tmp/pip-install-7y6dl6o2/openfst/setup.py';f=getattr(tokenize, 'open', open)(__file__);code=f.read().replace('\r\n', '\n');f.close();exec(compile(code, __file__, 'exec'))" install --record /tmp/pip-record-3niypfz6/install-record.txt --single-version-externally-managed --compile" failed with error code 1 in /tmp/pip-install-7y6dl6o2/openfst/

Can anyone help me solve this problem?

I am using python 3.6 (anaconda) and OpenFST-1.5.4 in Linux Mint 18.3.

Answer 1

There are two problems here.

---

First, as far as I can tell, despite being documented, the pip install method really isn't supported, or even expected to work. Kyle Gorman (who I assume is one of the main authors) has commented on multiple forum threads with replies like:

"Using pip for this doesn't make much sense, and that's out of date anyways."

"There's no reason to call pip. The version up there is not compatible with arbitrary OpenFst versions. If you do --enable-python then make and make install, you have already installed pywrapfst for whatever version of OpenFst you're building."

---

Second, despite saying that it works with "any Python 2.7 or later", it actually only works only with Python 2.7 exactly:

"Before submitting you may want to test Python 3 with the Python extension. We still develop on Python 2.7, exclusively, unfortunately, and it's not tested on Python 3."

And, in fact, that's exactly why your attempt at using --enable-python failed:

checking for a version of Python >= '2.1.0'...   File "<string>", line 1
    import sys, string;         ver = string.split(sys.version)[0];     print ver >= '2.1.0'
                                                                        ^
SyntaxError: invalid syntax
no

Their autoconf test for Python 2.1 or later is using syntax that's illegal in Python 3. And it doesn't do very good error handling, so it takes that SyntaxError to mean that your Python is 2.0 or earlier, and therefore it aborts the configure.

---

If you look further down the same thread, a user, NurL, posted:

After a lot of tweaks to the configuration file and Makefile I managed to successfully install the Python extension for Python 3. See this Dockerfile: https://gist.github.com/0xnurl/6f97eb39409ea48db31fe315fd1e208f

I obviously can't vouch for whether this works.

And, unless you're using exactly the same target as NurL, you'll have to read through that giant RUN wget command line, break it into steps, and do the equivalent steps yourself.

And there may be issues that NurL didn't run into that you will. (I'd strongly suggest at least running 2to3 on the resulting code before or after installing to make sure it doesn't find anything.)

But this is as close as you're likely to get to something that works out of the box, given that what you're trying to do isn't supported. If you can't get it working for yourself from there, you probably just can't use this library.