Is there a way to use python argparse with nargs='*', choices, AND default?

Question

My use case is multiple optional positional arguments, taken from a constrained set of choices, with a default value that is a list containing two of those choices. I can't change the interface, due to backwards compatibility issues. I also have to maintain compatibility with Python 3.4.

Here is my code. You can see that I want my default to be a list of two values from the set of choices.

parser = argparse.ArgumentParser()
parser.add_argument('tests', nargs='*', choices=['a', 'b', 'c', 'd'],
                    default=['a', 'd'])
args = parser.parse_args()
print(args.tests)

All of this is correct:

$ ./test.py a
['a']
$ ./test.py a d
['a', 'd']
$ ./test.py a e
usage: test.py [-h] [{a,b,c,d} ...]
test.py: error: argument tests: invalid choice: 'e' (choose from 'a', 'b', 'c', 'd')

This is incorrect:

$ ./test.py
usage: test.py [-h] [{a,b,c,d} ...]
test.py: error: argument tests: invalid choice: ['a', 'd'] (choose from 'a', 'b', 'c', 'd')

I've found a LOT of similar questions but none that address this particular use case. The most promising suggestion I've found (in a different context) is to write a custom action and use that instead of choices:

• https://stackoverflow.com/a/8624107/7660197

That's not ideal. I'm hoping someone can point me to an option I've missed.

Here's the workaround I plan to use if not:

parser.add_argument('tests', nargs='*',
                    choices=['a', 'b', 'c', 'd', 'default'],
                    default='default')

I'm allowed to add arguments as long as I maintain backwards compatibility.

Thanks!

---

Update: I ended up going with a custom action. I was resistant because this doesn't feel like a use case that should require custom anything. However, it seems like more or less the intended use case of subclassing argparse.Action, and it makes the intent very explicit and gives the cleanest user-facing result I've found.

class TestsArgAction(argparse.Action):
    def __call__(self, parser, namespace, values, option_string=None):
        all_tests = ['a', 'b', 'c', 'd']
        default_tests = ['a', 'd']

        if not values:
            setattr(namespace, self.dest, default_tests)
            return

        # If no argument is specified, the default gets passed as a
        # string 'default' instead of as a list ['default']. Probably
        # a bug in argparse. The below gives us a list.
        if not isinstance(values, list):
            values = [values]

        tests = set(values)

        # If 'all', is found, replace it with the tests it represents.
        # For reasons of compatibility, 'all' does not actually include
        # one of the tests (let's call it 'e'). So we can't just do
        # tests = all_tests.
        try:
            tests.remove('all')
            tests.update(set(all_tests))
        except KeyError:
            pass

        # Same for 'default'
        try:
            tests.remove('default')
            tests.update(set(default_tests))
        except KeyError:
            pass

        setattr(namespace, self.dest, sorted(list(tests)))

Answer 1

The behavior noted as incorrect is caused by the fact that the raw default value ['a', 'd'] is not inside the specified choices (see: relevant code as found in Python 3.4.10; this check method is effectively unchanged as of Python 3.10.3). I will reproduce the code from the Python argparse.py source code:

    def _check_value(self, action, value):
        # converted value must be one of the choices (if specified)
        if action.choices is not None and value not in action.choices:
            args = {'value': value,
                    'choices': ', '.join(map(repr, action.choices))}
            msg = _('invalid choice: %(value)r (choose from %(choices)s)')
            raise ArgumentError(action, msg % args)

When a default value is specified as a list, that entire value is passed to that _check_value method and thus it will fail (as any given list will not match any strings inside another list). You can actually verify that by setting a breakpoint with pdb in that method and trace through the values by stepping through each line, or alternatively test and verify the stated limitations with the following code:

import argparse
DEFAULT = ['a', 'd']
parser = argparse.ArgumentParser()
parser.add_argument('tests', nargs='*', choices=['a', 'b', 'c', 'd', DEFAULT],
                    default=DEFAULT)
args = parser.parse_args()
print(args.tests)

Then run python test.py

$ python test.py
['a', 'd']

This clearly passed because that very same DEFAULT value is present in the list of choices.

However, calling -h or passing any unsupported value will result in:

$ python test.py z
usage: test.py [-h] [{a,b,c,d,['a', 'd']} ...]
test.py: error: argument tests: invalid choice: 'z' (choose from 'a', 'b', 'c', 'd', ['a', 'd'])
$ python test.py -h
usage: test.py [-h] [{a,b,c,d,['a', 'd']} ...]

positional arguments:
  {a,b,c,d,['a', 'd']}
...

Which may or may not be ideal depending on use case as the output looks weird if not confusing. If this output is going to be user-facing it's probably not ideal, but if this is to maintain some internal system call emulation that won't leak out to users, the messages are probably not visible so this may be an acceptable workaround. Hence, I do not recommend this approach if the clarity of the choice message being generated is vital (which is >99% of typical use cases).

However, given that custom action is considered not ideal, I will assume overriding the ArgumentParser class may be a possible choice, and given that _check_value has not changed between 3.4 and 3.10, this might represent the most minimum additional code to nip out the incompatible check (with the specified use case as per the question):

class ArgumentParser(argparse.ArgumentParser):
    def _check_value(self, action, value):
        if value is action.default:
            return
        return super()._check_value(action, value)

This would ensure that the default value be considered a valid choice (return None if the value is the action's default, otherwise return the default check) before using the default implementation that is unsuitable for the requirement as outlined in the question; do please note that this prevents deeper inspection of what that action.default provides being a valid one (if that's necessary, custom Action class is most certainly the way to go).

Might as well show the example usage with the custom class (i.e. copy/pasted the original code, remove the argparse. to use the new custom class):

parser = ArgumentParser()
parser.add_argument('tests', nargs='*', choices=['a', 'b', 'c', 'd'],
                    default=['a', 'd'])
args = parser.parse_args()
print(args.tests)

Usage:

$ python test.py
['a', 'd']
$ python test.py a z
usage: test.py [-h] [{a,b,c,d} ...]
test.py: error: argument tests: invalid choice: 'z' (choose from 'a', 'b', 'c', 'd')
$ python test.py -h
usage: test.py [-h] [{a,b,c,d} ...]

positional arguments:
  {a,b,c,d}

optional arguments:
  -h, --help  show this help message and exit