Creating a dynamic partial callable object in Python

Rather than trying a partial or methodcaller, why not call the function directly, using only kwargs, but use the configuration to drive most of the kwargs/args contents? I use a closure for that, where the prepped "remembers" the configuration.

Notice that my final call does not care that string is the keyword for re.match. I found that your example has a fair bit of coupling to regex specific stuff, some of which like re.X could not be stored in a YAML without further manipulation.

Likewise, the partial/methodcaller way to call the function should not have to care which line number in a file the value comes from, that is too much coupling. If you must, add something else in the config, not under kwargs, that deals with runtime parameter acquisition.

So I changed things around a bit. I believe, but you may disagree, that when calling a parse rule, the calling function should not have to know how the argument is called. Well, that is, unless you rules are only regex in style, in which case you don't need a kind in the config.

This is a quick, imperfect, sketch of an alternative approach. Details will depend on how exactly you want to use this.

I also punted on the *args handling, though it could probably carried out the same way if you had to.

import importlib

rule_1 = {
    'kind': 're',
    'method': 'match',
    'args': None,
    "positional_mapper" : ["string"],
    'kwargs': {
        'pattern': "[a-z_]+",
        # I don't know how this would be stored in a YAML
        # 'flags': re.X,
        'string': 's_test.log',
    }
}

rule_2 = {
    'kind': 're',
    'method': 'match',
    'args': None,
    "positional_mapper" : ["string"],
    'kwargs': {
        'pattern': "[a-z_]+",
    }
}


def prep(config):

    mod = app_urls = importlib.import_module(config["kind"])
    f = getattr(mod, config["method"])

    pre_args = config.get("args") or []
    pre_kwargs = config.get("kwargs") or {}
    positional_mapper = config["positional_mapper"]

    def prepped(*args, **kwargs):

        kwargs2 = pre_kwargs.copy()

        for value, argname in zip(args, positional_mapper):
            kwargs2[argname] = value
        kwargs2.update(**kwargs)

        return f(**kwargs2)

    return prepped


parsed_rule1 = prep(rule_1)

print ("#1", parsed_rule1("second line"))
print ("#2", parsed_rule1())

parsed_rule2 = prep(rule_2)
print ("#3", parsed_rule2("second line"))
print ("#3.5", parsed_rule2(string="second line"))
print ("#4", parsed_rule2())

As expected, call #4 chokes as it is missing an argument to put into string.

#1 <re.Match object; span=(0, 6), match='second'>
#2 <re.Match object; span=(0, 6), match='s_test'>
#3 <re.Match object; span=(0, 6), match='second'>
#3.5 <re.Match object; span=(0, 6), match='second'>
Traceback (most recent call last):
  File "test_299_dyn.py:57", in <module>
    print ("#4", parsed_rule2())
  File "test_299_dyn.py:44", in prepped
    return f(**kwargs2)
TypeError: match() missing 1 required positional argument: 'string'

You don't have to create a partial function. You can compile the pattern first then call the desired method with that:

rule_2 = {
    'kind': 'regex',
    'method': 'match',
    'args': None,
    'kwargs': {
        'pattern': "[a-z_]+",
        'flags': re.X,
        # 'line_number': 2, commented out this line
    }
}

content = ['', 'first line', 'second line', 'third line']

pattern = re.compile(**rule_2['kwargs'])
method = getattr(pattern, rule_2['method'])
>>> method(content[2])
<re.Match object; span=(0, 6), match='second'>

If you want to keep the line number, you can do something like this:

rule_2 = {
    'kind': 'regex',
    'method': 'match',
    'args': None,
    'kwargs': {
        'pattern': "[a-z_]+",
        'flags': re.X,
        'line_number': 2,
    }
}

content = ['', 'first line', 'second line', 'third line']
def rule_parser_re(*, kind, method, args=None, kwargs=None):
    copied_kwargs = kwargs.copy()
    line_number = copied_kwargs.pop('line_number')
    pattern = re.compile(**copied_kwargs)
    method = getattr(pattern, method)
    return method, line_number
    
parser, line_number = rule_parser_re(**rule_2)
>>> parser(content[line_number])
<re.Match object; span=(0, 6), match='second'>