How can I separate a rust library and the pyo3 exported python extensions which wrap it

Question

I have a rust library which provides useful functionality for use in other rust programs. Additionally I would like to provide this functionality as a python extension (using pyo3 and setuptools-rust, although most of this is tool agnostic)

The documentation examples all show a single library. This means that anyone building the rust library and using it in the rust ecosystem needs the python headers and gets the exported python module as well.

How can I separate this as an independent rust library offering the functionality in rust and a python package offering the functionality in python?

Answer 1

After some experimentation I have arrived on the following setup which makes use of rust workspaces:

Directory structure:

FizzBuzz
- fizzbuzz
  - src
    - lib.rs
  - tests
    - test_....rs
    ...
  - Cargo.toml
- fizzbuzzo3
  - src
    - lib.rs
  - Cargo.toml
- fizzbuzzpy
  - fizzbuzzpy
    - __init__.py
    - fizzbuzzer.py
  - tests
    - test_fizzbuzzo3.py
    - test_fizzbuzzpy.py
- Cargo.toml
- pyproject.toml

Where:

• fizzbuzz contains the pure rust implementation
• fizzbuzzo3 contains the pyo3 wrapped version for python
• fizzbuzzpy contains additional native python functionality and pytest tests

Config files:

fizzbuzz/Cargo.toml

[package]
name = "fizzbuzz"
version = "0.2.1"
edition = "2021"

[lib]
name = "fizzbuzz"
path = "src/lib.rs"
crate-type = ["rlib"]  # cdylib required for python import, rlib required for rust tests.

fizzbuzzo3/Cargo.toml

[package]
name = "fizzbuzzo3"
version = "0.1.0"
edition = "2021"

[lib]
name = "fizzbuzzo3"
path = "src/lib.rs"
crate-type = ["cdylib"]  # cdylib required for python import, rlib required for rust tests.

[dependencies]
pyo3 = { version = "0.21.0-beta.0" }
fizzbuzz = { path = "../fizzbuzz" }

./Cargo.toml

[workspace]

members = [
    "fizzbuzz",
    "fizzbuzzo3"
]

resolver = "2"

pyproject.toml

[build-system]
requires = ["setuptools", "setuptools-rust"]
build-backend = "setuptools.build_meta"

[project]
name = "fizzbuzz"
description = "An implementation of the traditional fizzbuzz kata"
...
[tool.setuptools.packages.find]
where = ["fizzbuzzpy"]

[[tool.setuptools-rust.ext-modules]]
# Private Rust extension module to be nested into the Python package
target = "fizzbuzzo3"           # The last part of the name (e.g. "_lib") has to match lib.name in Cargo.toml,
                                # but you can add a prefix to nest it inside of a Python package.
path = "fizzbuzzo3/Cargo.toml"  # Default value, can be omitted
binding = "PyO3"                # Default value, can be omitted

[project.optional-dependencies]
dev = [
    "black",
    "pytest",
    "pytest-doctest-mkdocstrings",
    "ruff",
    ]

How To:

• build and test the pure rust implementation: ~/Fizzbuzz/fizzbuzz$ cargo test -v
• build and unit test the python bindings: ~/Fizzbuzz/fizzbuzzo3$ cargo test -v
• run all the rust tests: ~/Fizzbuzz$ cargo test -v
• build, install and test the rust bindings in python: (.venv) ~/Fizzbuzz$ pip install -e .[dev] && pytest

The full codebase is at: MusicalNinjaDad/FizzBuzz if you want to look in more detail.