How to compose functions through purely using Python's standard library?

Question

Python's standard library is vast, and my intuition tells that there must be a way in it to accomplish this, but I just can't figure it out. This is purely for curiosity and learning purposes:

I have two simple functions:

def increment(x):
    return x + 1

def double(x):
    return x * 2

and I want to compose them into a new function double_and_increment. I could of course simply do that as such:

double_and_increment = lambda x: increment(double(x))

but I could also do it in a more convoluted but perhaps more "ergonomically scalable" way:

import functools

double_and_increment = functools.partial(functools.reduce, lambda acc, f: f(acc), [double, increment])

Both of the above work fine:

>>> double_and_increment(1)
3

Now, the question is, is there tooling in the standard library that would allow achieving the composition without any user-defined lambdas, regular functions, or classes.

The first intuition is to replace the lambda acc, f: f(acc) definition in the functools.reduce call with operator.call, but that unfortunately takes the arguments in the reverse order:

>>> (lambda acc, f: f(acc))(1, str)  # What we want to replace.
>>> '1'
>>> import operator
>>> operator.call(str, 1)  # Incorrect argument order.
>>> '1'

I have a hunch that using functools.reduce is still the way to accomplish the composition, but for the life of me I can't figure out a way to get rid of the user-defined lambda.

Few out-of-the-box methods that got me close:

import functools, operator

# Curried form, can't figure out how to uncurry.
functools.partial(operator.methodcaller, '__call__')(1)(str)

# The arguments needs to be in the middle of the expression, which does not work.
operator.call(*reversed(operator.attrgetter('args')(functools.partial(functools.partial, operator.call)(1, str))))

Have looked through all the existing questions, but they are completely different and rely on using user-defined functions and/or lambdas.

Answer 1

As mentioned in the other answer of mine I don't agree that the test suite discovered by @AKX should be considered as part of the standard library per the OP's rules.

As it turns out, while researching for an existing function to modify for my other answer, I found that there is this helper function _int_to_enum in the signal module that perfectly implements operator.call for a callable with a single argument, but with parameters reversed, exactly how the OP wants it, and is available since Python 3.5:

def _int_to_enum(value, enum_klass):
    """Convert a numeric value to an IntEnum member.
    If it's not a known member, return the numeric value itself.
    """
    try:
        return enum_klass(value)
    except ValueError:
        return value

So we can simply repurpose/abuse it:

from signal import _int_to_enum as rcall
from functools import reduce, partial

def increment(x):
    return x + 1

def double(x):
    return x * 2

double_and_increment = partial(reduce, rcall, [double, increment])
print(double_and_increment(1))

This outputs:

3

Demo: here

Answer 2

Well, since you're saying

I want to "abuse" the language and only use existing definitions from the standard library

starting with Python 3.12, the test suite happens to contain the gadget you want:

import functools
import operator
from test.test_zipfile._path._functools import compose

increment = functools.partial(operator.add, 1)
double = functools.partial(operator.mul, 2)
increment_and_double = compose(increment, double)
print(increment_and_double(10))

(I found this by way of a strategic ag compose in my local CPython checkout.)

Answer 3

While it is cool that @AKX found the function test.test_zipfile._path._functools.compose in the CPython code tree that perfectly implements the OP's desired functionality of function composition, it does not actually belong to the standard library as required by the rules of the question, for reasons that:

• It belongs to a helper module in a test suite of the CPython implementation of the Python language.
• A test suite is not part of the standard library of the language; it is just code that validates a particular implementation of the language and its standard library.
• A test suite, let alone any helper functions within the test suite, may be removed at any time without any normal due process of advanced deprecation warnings.
• Other implementations of Python does not need to include any of CPython's test suite in order to conform to Python's specifications.

So, without the helper function in the test suite of CPython 3.12 that is not part of the standard library, I believe the OP is indeed correct in the assessment that there is no out-of-the-box tooling in Python's standard library that can implement function composition.

BUT, that doesn't mean we can't achieve it by modifying existing tooling, since the OP's rules are simply to use "tooling in the standard library that would allow achieving the composition without any user-defined lambdas, regular functions, or classes".

Since the OP almost got it already with:

double_and_increment = partial(reduce, lambda acc, f: f(acc), [double, increment])

and:

>>> (lambda acc, f: f(acc))(1, str)  # What we want to replace.
>>> '1'
>>> import operator
>>> operator.call(str, 1)  # Incorrect argument order.
>>> '1'

The real question here is then how we can modify an existing function in the standard library such that it becomes:

def rcall(value, obj):
    return obj(value)

To do that, let's take a look at the bytecode of the above function, as well as relevant attributes of the code object that defines the parameters:

>>> import dis
>>> def call(value, obj):
...     return obj(value)
...
>>> dis.dis(call)
  1           0 RESUME                   0

  2           2 PUSH_NULL
              4 LOAD_FAST                1 (obj)
              6 LOAD_FAST                0 (value)
              8 PRECALL                  1
             12 CALL                     1
             22 RETURN_VALUE
>>> c = call.__code__
>>> c.co_varnames
('value', 'obj')
>>> c.co_argcount
2
>>> c.co_nlocals
2
>>>

No surprise there. A simple function body that loads the second argument (obj) and the first argument (value) onto the stack, then make a call with the callable and the argument in the stack, and finally returns the value at the top of the stack to the caller.

Now, let's find a similarly simple function in the standard library that takes an argument or two and make a call with it/them, so it can more easily be modified into our desired function. As it turns out, operator.abs is one such function, which takes one argument and makes a wrapper call to the built-in _abs function:

def abs(a):
    "Same as abs(a)."
    return _abs(a)

We'd want to disassemble it for comparison, and yet unfortunately, if we try accessing operator.abs.__code__, you would get an error:

>>> import operator
>>> operator.abs.__code__
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
AttributeError: 'builtin_function_or_method' object has no attribute '__code__'. Did you mean: '__call__'?
>>>

This is because CPython's implementation of the operator module includes an _operator module, which overrides all of the operator.py's pure-Python functions with ones implemented in C, wit a try block in operator.py:

try:
    from _operator import *
except ImportError:
    pass

Functions implemented in C do not have __code__ objects and therefore cannot be modified. What we need is the pure Python version of operator.call, before it's overridden by _operator.call. But how do we avoid the override? Well, we can import _operator module ourselves first and and delete the call attribute from it so that the modified module is cached in sys.modules such that when operator.py imports _operator, it is our modified version that it gets, without call in it:

>>> try: # other Python implementations may not have _operator.py
...     import _operator
...     del _operator.call
... except ImportError:
...     pass
...
>>> import operator
>>> operator.call.__code__
<code object call at 0x000001F68F4FADB0, file "C:\python311\Lib\operator.py", line 226>

Great! Now we can finally get to look at the bytecode and relevant attributes of the code object of operator.abs:

>>> dis.dis(operator.abs)
 71           0 RESUME                   0

 73           2 LOAD_GLOBAL              1 (NULL + _abs)
             14 LOAD_FAST                0 (a)
             16 PRECALL                  1
             20 CALL                     1
             30 RETURN_VALUE
 71           0 RESUME                   0
>>> c = operator.abs.__code__
>>> c.co_varnames
('a',)
>>> c.co_argcount
1
>>> c.co_nlocals
1
>>>

As can be seen, all we need to modify to turn operator.abs into our desired function object is to replace the LOAD_GLOBAL instruction into PUSH_NULL (to indicate a regular function call for CALL) and LOAD_FAST 1 (to load the second argument, the callable), as well as co_varnames, co_argcount and co_nlocals to add a second parameter obj.

To obtain a modified code object from the existing code object of operator.abs we can call its replace method:

try:
    import _operator
    del _operator.abs
except ImportError:
    pass
from operator import abs as rcall
from opcode import opmap
from functools import partial, reduce

code = bytearray(rcall.__code__.co_code)
code[code.find(opmap['LOAD_GLOBAL']):code.find(opmap['LOAD_FAST'])] = \
    opmap['PUSH_NULL'], 0, opmap['LOAD_FAST'], 1
rcall.__code__ = rcall.__code__.replace(
    co_code=bytes(code),
    co_varnames=('value', 'obj'),
    co_argcount=2,
    co_nlocals=2
)
print(rcall(1, str))

This correctly outputs:

1

So it then becomes trivial to implement the composite function that the OP wants, by plugging in the modified operator.call into the OP's close attempt:

def increment(x):
    return x + 1

def double(x):
    return x * 2

double_and_increment = partial(reduce, rcall, [double, increment])
print(double_and_increment(1))

This outputs:

3

Demo: here