6
\$\begingroup\$

The code below is a Python module to flatten JSON-like structure (nested dictionaries and lists) to a single string. It also provides partial result in the form of flat list of strings. This flattening is intended to be used to generate commands for the command line variant of the Check Point Management API: https://sc1.checkpoint.com/documents/latest/APIs/

The details of how it works and examples are documented in the code which uses docstrings made to be able to generate documentation using Sphinx.

I am concerned mainly about:

  • The style
    • Identifier naming
    • Docstrings - aren't they excessive (for the private functions)?
    • Later I noticed I can replace _is_convertible_to_str(value) by much simpler isinstance(value, _CONVERTIBLE_TO_STR) and define the constant tuple instead of the function. Which way should be preferred?
  • The way of defining the default behaviour while allowing some flexibility
  • The module API design (public functions, parameters)
""" JSON-like structure flattening

The module provides functions to flatten nested structures of dictionaries
and lists to a flat list of strings or a single string.

Functions:
    * `flatten_to_list()`: Convert structure to a flat list of strings.
    * `flist_to_str()`: Convert list of strings to a single string.

Examples:

    >>> flat_list1 = flatten_to_list({'name': 'John', 'surname': 'Doe'})
    >>> flist_to_str(flat_list1)
    'name John surname Doe'

    >>> flat_list2 = flatten_to_list({
    ...         'add': 'access-rule',
    ...         'layer': 'policy1 Network',
    ...         'position': {'bottom': 'RADIUS rules'},
    ...         'source': ['web_serers'],
    ...         'destination': ['internet'],
    ...         'action': 'Accept',
    ...         'track': {'type': 'Log'}},
    ...     key_order=('add', 'layer', 'position'))
    >>> flat_list2
    ['add', 'access-rule', 'layer', 'policy1 Network', 'position.bottom',\
 'RADIUS rules', 'source.1', 'web_serers', 'destination.1', 'internet',\
 'action', 'Accept', 'track.type', 'Log']

    >>> flist_to_str(flat_list2)
    'add access-rule layer "policy1 Network" position.bottom "RADIUS rules"\
 source.1 web_serers destination.1 internet action Accept track.type Log'
"""

from __future__ import annotations

import string

from typing import (
    Any, Hashable, ItemsView, Iterator, Union, Callable, Iterable, Generator)


# --- private constants

# whitespace characters which cause a string to require quoting
_WHITESPACE = set(string.whitespace)


# --- private helper functions

def _is_convertible_to_str(value: Any) -> bool:
    """Decide if we want to convert the value using `str(value)`.

    Return `False` for container types. (`dict`, `list`...) The function
    decides if we are willing to convert the `value` in a JSON-like
    structure to a string.

    Args:
        value: the value to test the convertibility of

    Returns:
        `True` if we want to convert the value using `str(value)`

    Examples:

        >>> _is_convertible_to_str(1)
        True

        >>> _is_convertible_to_str([])
        False
    """
    return (isinstance(value, str)
            or isinstance(value, int)
            or isinstance(value, float))


def _ordered_dict_items(
        dictionary: dict[Hashable, Any], key_order: Iterable[Hashable] = ()
        ) -> Generator[tuple[Hashable, Any], None, None]:
    """Iterate dictionary like `dict.items()`_ with optional key order.

    Dictionary keys listed in `key_order` are iterated first in the order
    as listed. The rest is iterated in unspecified order.

    Args:
        dictionary: dictionary to iterate
        key_order: these keys will be iterated first in the given order

    Yields:
        `(key, value)` tuples as standard `dict.items()`_ does

    Examples:

        >>> list(_ordered_dict_items({'key': 42}))
        [('key', 42)]

        >>> list(_ordered_dict_items(
        ...         {'key': 42, 'id': 8569, 'name': 'Marc'}, ['name', 'id']))
        [('name', 'Marc'), ('id', 8569), ('key', 42)]

    .. _dict.items():
        https://docs.python.org/3/library/stdtypes.html#dict.items
    """
    dictionary = dictionary.copy()  # we will remove processed keys
    for ordered_key in key_order:
        if ordered_key in dictionary:
            yield ordered_key, dictionary[ordered_key]
            del dictionary[ordered_key]
    yield from dictionary.items()   # yield the rest in unspecified order


def _contains_any(set1: Iterable[Hashable], set2: Iterable[Hashable]) -> bool:
    r"""Test if `set1` contains any elements of `set2` or vice versa.

    The function tests if the intersection of the sets is not empty.
    Unlike the plain `&` operator the function operates on any iterables
    of `Hashable`. For example the function is useful to test if one string
    contains any character from the other string (or any iterable of
    characters).

    Args:
        set1: an iterable for the intersection test
        set2: another iterable for the intersection test

    Returns:
        `True` if the intersection is not empty

    Examples:

        >>> _contains_any('good morning', ' \t')
        True

        >>> _contains_any('hello John', 'xXyY')
        False
    """
    if not isinstance(set1, set):
        set1 = set(set1)
    if not isinstance(set2, set):
        set2 = set(set2)
    return bool(set1 & set2)


# --- public functions

def flatten_to_list(
        json_struct: Union[dict[str, Any], list[Any]], /, *,
        parent: str = '', startindex: int = 1, parent_sep: str = '.',
        key_order: Iterable[str] = (),
        value_converter: Callable = str,
        key_converter: Callable = str) -> list[str]:
    """Flatten JSON-like structure to a list of strings.

    The JSON-like structure consists of dictionaries, lists and simple values.
    The resulting list consists of pairs: `[key1, value1, key2, value2 ...]`.
    Key produced for a JSON list item is an ordinal number of the position
    in the list: `1, 2, 3, ...` Key from a nested container is preceded
    by the parent container key: *parent_key.key*.

    Args:
        json_struct: the JSON-like structure to flatten
        parent: parent key name
        startindex: first number for indexing list items
        parent_sep: parent key or index separator string
        key_order: list of keys needing defined order
        value_converter: function converting values to strings
        key_converter: function converting keys to strings

    Returns:
        flat list of key, value pairs: `[key1, value1, key2, value2 ...]`

    Examples:

        >>> flatten_to_list({'name': 'John', 'surname': 'Doe'})
        ['name', 'John', 'surname', 'Doe']

        >>> flatten_to_list({'name': 'Alice', 'siblings': ['Jeff', 'Anna']})
        ['name', 'Alice', 'siblings.1', 'Jeff', 'siblings.2', 'Anna']

        >>> flatten_to_list({
        ...         'name': 'Zip',
        ...         'eye': {'left': 'red', 'right': 'black'}})
        ['name', 'Zip', 'eye.left', 'red', 'eye.right', 'black']

        >>> flatten_to_list(['red', 'green', 'blue'],
        ...                 parent='color', startindex=0)
        ['color.0', 'red', 'color.1', 'green', 'color.2', 'blue']

        >>> flatten_to_list({'name': 'John', 'surname': 'Doe'},\
 key_order=['surname'])
        ['surname', 'Doe', 'name', 'John']
    """
    result: list[str] = []
    if parent:
        parent = parent + parent_sep
    struct_iterator: Union[ItemsView, Iterator]     # will yield (key, value)
    if isinstance(json_struct, dict):
        struct_iterator = _ordered_dict_items(json_struct, key_order)
    elif isinstance(json_struct, list):
        struct_iterator = enumerate(json_struct, startindex)
    else:
        raise TypeError(
                f"Unexpected data type {type(json_struct)} of the structure.")
    for key, value in struct_iterator:
        ext_key = parent + key_converter(key)
        if isinstance(value, (list, dict)):
            result.extend(flatten_to_list(
                    value, parent=ext_key, startindex=startindex,
                    parent_sep=parent_sep, key_order=key_order,
                    value_converter=value_converter,
                    key_converter=key_converter))
        elif _is_convertible_to_str(value):
            result.extend([ext_key, value_converter(value)])
        else:
            raise TypeError(
                    f"Unexpected data type {type(value)} inside structure.")
    return result


def flist_to_str(
        flist: list[str], /, *, separator: str = ' ',
        quote_str: str = '"', quote_always: bool = False) -> str:
    """Convert flat list of strings to a string with quoting.

    The function is useful to convert the resulting list from
    :py:func:`flatten_to_list()` to a single string.

    Args:
        flist: flat list of strings to be converted to a single string
        separator: separator between list items
        quote_str: character or string to quote list items if needed
        quote_always: if list items should be quoted even if not necessary

    Examples:

        >>> flist_to_str(['good', 'morning'])
        'good morning'

        >>> flist_to_str(['good morning'])
        '"good morning"'

    Todo:
         * No escaping implemented for quote characters. We need to find out
           which way of escaping does Check Point CLI API support.
    """
    def quote(string1: str) -> str:
        """Quote string1 as needed"""
        if quote_always or _contains_any(string1, _WHITESPACE) or not string1:
            return quote_str + string1 + quote_str
        return string1
    return separator.join(quote(item) for item in flist)
\$\endgroup\$

2 Answers 2

Reset to default
2
\$\begingroup\$

A couple comments:

_is_convertible_to_str()

As you discovered, the second argument to isinstance() can be a tuple of classes:

def _is_convertible_to_str(value: Any) -> bool:
    return isinstance(value, (str, int, float))

You can call str() on just about anything, so _is_convertable_to_str() isn't very descriptive. I'd drop the _is_convertible_to_str() function and just use isinstance() directly. You already use isinstance(value, (list, dict)) in the same if statement.

_ordered_dict_items()

dict.keys() acts like a set, so you can use set difference to determine the unordered keys, instead of making a copy of all the keys and deleting the ordered keys. For small dicts, they are about the same speed, so use which ever is clearer to you.

def _ordered_dict_items(
        dictionary: dict[Hashable, Any],
        key_order: Iterable[Hashable] = ()
        ) -> Generator[tuple[Hashable, Any], None, None]:

    yield from ((key, dictionary[key] for key in key_order if key in dictionary)
    yield from ((key, dictionary[key] for key in dictionary.keys() - key_order)

_contains_any()

The method form of & takes an iterable, so it isn't necessary to convert set2 to an actual set:

def _contains_any(set1: Iterable[Hashable], set2: Iterable[Hashable]) -> bool:
    if not isinstance(set1, set):
        set1 = set(set1)
    return bool(set1.intersection(set2))

flatten_to_list()

flatten_to_list() has a lot of parameters to "configure" the function. They don't change when making a recursive call to handle a nested list or dict. To me, this suggests this might be a good use for a class. The "configuration" parameters can be passed to __init__(). A method then does the actual flattening. json_struct and parent change during recursion, so they can be method arguments.

Lastly, functools in the standard library contains decorators for overloading functions or methods based on the types of their first argument. @singledispatchmethod is used to decorate the "default" method to use. .register(type) is used to register a version of the method to handle the specified type. These decorators don't seem to get used a lot and there is some overhead, but I find they can be cleaner than a set of if isinstance(...):... elif isinstance(...):... statements.

from functools import singledispatchmethod

class Flattener:
    def __init__(self,
        startindex=1,
        parent_sep='.',
        key_order=(),
        value_converter=str,
        key_converter=str
    ):
        self.startindex = startindex
        self.parent_sep = parent_sep
        self.key_order = key_order
        self.value_converter = value_converter
        self.key_converter = key_converter

        
    @singledispatchmethod
    def flatten(self, thing, parent=''):
        """Base 'flatten' method when a more specific method isn't
        defined for the type of 'thing'.
        
        Methods for specific types are registered below.
        """
        yield from (parent, self.value_converter(thing))

    @flatten.register(dict)
    def flatten_dict(self, thing, parent=''):
        for key, value in _ordered_dict_items(thing, self.key_order):
            if parent:
                ext_key = f"{parent}{self.parent_sep}{self.key_converter(key)}"
            else:
                ext_key = self.key_converter(key)
            yield from self.flatten(value, ext_key)
    
    @flatten.register(list)
    def flatten_list(self, thing, parent=''):
        for key, value in enumerate(thing, self.startindex):
            if parent:
                ext_key = f"{parent}{self.parent_sep}{self.key_converter(key)}"
            else:
                ext_key = self.key_converter(key)
            yield from self.flatten(value, ext_key)

Here, the decorator @singledispatchmethod sets the flatten method up as the default version. The decorator @flatten.register(dict) registers a version that gets called when the first argument (after self) is a dict (the name of the registered method doesn't matter, the docs use _). Similarly, @flatten.register(list) registers a version to use for lists. This makes the bodies of the flatten methods much simpler and easier to understand.

\$\endgroup\$
1
\$\begingroup\$
  • I agree with your assessment here; isinstance(value, _CONVERTIBLE_TO_STR) (where _CONVERTIBLE_TO_STR = (str, int, float)) is simpler and better than _is_convertible_to_str(value). It is also a little bit faster. I determined this by measuring execution times with timeit.
  • _contains_any looks like it was written in a way so it can be useful in a general sense, e.g. it could be part of a utility library. But right now the only way it is used is to check if a string contains any whitespace. For simplicity, I would go with something like this instead: 
    def contains_whitespace(text: str) -> bool:
    return any(character in _WHITESPACE for character in text)
    Based on measurements with timeit and randomly-generated strings, I found that this was also generally faster than _contains_any(text, _WHITESPACE).
  • You can get a small performance boost by using string interpolation with f-strings (f"{quote_str}{string1}{quote_str}") instead of string concatenation via the + operator (quote_str + string1 + quote_str). Nothing huge, but improving performance without having to sacrifice readability is pretty nice.
  • For flist_to_str, I would make some naming changes.
    • If I pretended that I didn't know the context of the larger problem statement, seeing the names flist_to_str and flist: list[str] and the documentation "flat list of strings", I would probably wonder, "Why call it a flat list of strings? How is that any different from a list of strings?" And if we look at the type hints there really isn't a difference, so I would change the naming to make that more clear:
      • flist_to_str -> list_to_str
      • flist: list[str] -> items: list[str]
      • "flat list of strings" -> "list of strings" (in the documentation)
    • quote_str: str -> quote_mark: str
      • I think this is a little more precise.
    • string1: str -> text: str
      • I totally understand choosing the name string1 to avoid shadowing the string module, but text is probably a better name here.
      • To be fair, coming up with a good generic name for a string in Python is hard because the two best candidates (str and string) are already taken by a built-in function and a standard library module, respectively. I always end up using something like s or text as a result.

In summary, there were some small nits pointed out here and there but overall I found the code and the public API it exposes very well-documented and easy to read. The docstrings were useful and had illuminating examples; I didn't think they were excessive at all.

\$\endgroup\$

You must log in to answer this question.

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.