How can I create a standalone PEX (PEX that includes its dependencies) file with BUCK?

Question

Suppose I have the following project structure:

├── BUCK
├── main.py
└── setup.py

Where:

main.py

from markdown import markdown

def joke():
    return markdown(u'Wenn ist das Nunst\u00fcck git und Slotermeyer?'
                    u'Ja! ... **Beiherhund** das Oder die Flipperwaldt '
                    u'gersput.')

print joke()

setup.py

import setuptools

setuptools.setup(
    name="packaging-hello-world",
    version="0.0.1",
    description="A hello world that attempts to package a Python project with its dependencies",
    packages=setuptools.find_packages(),
    include_package_data=True,
    install_requires=[
        'markdown',
    ],
)

and

BUCK

python_binary(
    name = 'bin_main',
    main_module = 'main',
    deps = [
        ':src_main',
    ],
    package_style = 'standalone',
    visibility = [
        'PUBLIC',
    ],
)

python_library(
    name = 'src_main',
    srcs = glob([
        '*.py',
    ]),
    visibility = [
        'PUBLIC',
    ],
)

markdown is installed in the virtualenvironment.

Using the standard pex tool, I can do: pex . markdown -c main.py -o joke.pex where the generated joke.pex contains the required dependencies (markdown). Example: unzip joke.pex -d tmp ; tree -a -I .bootstrap -L 2 tmp leads to:

tmp
├── .bootstrap
│   ├── _pex
│   └── pex
├── .deps
│   ├── Markdown-3.1-py2.py3-none-any.whl
│   ├── packaging_hello_world-0.0.1-py2-none-any.whl
│   └── setuptools-41.0.0-py2.py3-none-any.whl
├── PEX-INFO
├── __main__.py
└── __main__.pyc

How can I make BUCK's generated PEX (buck build :bin_main) also contains markdown? I've tried to use prebuilt_python_library pointing to a wheel file generated from setup.py and add it as dep of bin_main, however, the generate PEX still doesn't contain the required deps.

Answer 1

You should be able to add it with the prebuilt_python_library but don't forget to add it as a deps inside the python_library.

remote_file(
    name='markdown-download',
    url='https://files.pythonhosted.org/packages/c0/4e/fd492e91abdc2d2fcb70ef453064d980688762079397f779758e055f6575/Markdown-3.1.1-py2.py3-none-any.whl',
    sha1='b80598369cacd1f28b9746dd5469f2573e545178',
    out='Markdown-3.1.1-py2.py3-none-any.whl',
)

prebuilt_python_library(
    name='markdown',
    binary_src=':markdown-download',
    visibility=['PUBLIC'],
)

The issue with this approach is that you have to specify all the deps by hand. Markdown is using setuptools so you need to download it with a remote_file/prebuilt_python_library.

I don't know if this is the best solution but it worked for me.