1

I have been working with a project for which I give the choice of using two backends (backends 1 and 2, let's say) following what they do in this project. However, that project counts on having already defined environment variables for deciding which backend to use before executing the code. It would not be the case for the code I'm writing.

I would like to know if there is any alternative to use environment variables in this scenario so that, on execution time, I can load one backend or another depending on the value of a variable. The general structure of my project is as follows:

Project

I thought of directly setting the environment variable directly in the python code (os.environ['NAME_OF_ENV_VARIABLE'] = 'BACKEND 1'), but that feels potentially insecure and I really dislike the idea, even if the variable name is... unique. Given this need, I would like to know if it is possible to have some kind of variable spanning different files so that, when I do import a module, the __init__.py file can disambiguate between the backends.

PS: Maybe what I'm doing makes no sense whatsoever.


[UPDATE] Some more information about the problem, reduced to its minimal extension. My main file processes some data and is the following:

from argparse import ArgumentParser
from utils.loader import load_data
from utils.do import do_stuff

def main(config):
    data = load_data(config)
    do_stuff(config, data)

if __name__ == '__main__':
    # Retrieve input data
    parser = ArgumentParser()
    parser.add_argument('--backend', type=str, default='backend 1', help='backend to use')
    inputs = parser.parse_args()

    config = "backend 1" if inputs.backend == "1" else "backend 2"

    # Call main function
    main(config)

The data loader load_data(config) I guess is not important for this. Then, the file containing do_stuff(data) is the following:

import backend

def do_stuff(config, data):
    # Do some really important stuff that is coded in backend 1 and backend 2
    a = backend.do_something(data)
    print(a)

It simply loads the backend (!!!) and does something. The do_stuff(data) function itself does something coded in either backend 1 or backend 2:

def do_something(data):
    data.values = "Value obtained through functions of 'BACKEND 1' (same function names and inputs, different backends used)"

and

def do_something(data):
    data.values = "Value obtained through functions of 'BACKEND 2' (same function names and inputs, different backends used)"

Finally, the backend module has in itself the following __init__.py file:

from .load_backend import do_something

Which loads from the load_backend.py file, which simply disambiguates the backend given an environmental variable:

from __future__ import absolute_import
from __future__ import print_function
import os
import sys

# Default backend: backend 1
if 'ENVIRONMENT_VARIABLE' in os.environ:
    _BACKEND = os.environ['ENVIRONMENT_VARIABLE']
else:
    _BACKEND = 'backend 1'

# Import backend functions.
if _BACKEND == "backend 1":
    sys.stderr.write('Using backend 1\n')
    from .backend_1 import *
elif _BACKEND == "backend 2":
    sys.stderr.write('Using backend 2\n')
    from .backend_2 import *
else:
    raise ValueError('Unable to import backend : ' + str(_BACKEND))


def backend():
    """Publicly accessible method
    for determining the current backend.
    # Returns
        String, the name of the backend
    # Example
    ```python
        >>> backend.backend()
        'backend 1'
    ```
    """
    return _BACKEND

What I want is to reduce this last environment variable with anything else, but I don't know what can I use.

10
  • 1
    I'm not really sure why you're hung up on environment variables here. That project just uses them to determine the path to load but why do you think you couldn't use a normal variable in exactly the same way? Commented Jul 7, 2019 at 11:29
  • Hi, Daniel, thanks for taking interest. Maybe I'm not knowledgeable enough to see it but, since the disambiguation would be done in ProjectX/backend/__init__.py and I would be defining the variable you mention in ProjectX/main.py, I'm not sure on how to pass that variable from main.py to __init__.py, although it might be really obvious and I'm not seeing it. The idea is that I want to be able to use import backend and forget about it, but how would I be able to disamiguate during an import? Commented Jul 7, 2019 at 11:33
  • Can you give an example of what you want to have in main.py? Commented Jul 7, 2019 at 11:37
  • @DanielRoseman I just updated the question for improved context. Thanks! Commented Jul 7, 2019 at 17:00
  • 1
    (Yes, environment variables are also copied to child processes -- but that too is common to all memory that hasn't explicitly been flagged to not be mapped on fork; really, the only thing that makes them unusual is not being discarded on an execv-family call, and having a dedicated region for storage rather than being on the stack or heap). Commented Jul 7, 2019 at 18:21

1 Answer 1

1

Like @DanielRoseman asked, I would just pass the backend argument around. For example in load_backend, while changing your code as litte as possible:

from __future__ import absolute_import
from __future__ import print_function
import os
import sys

def backend(backend):
    """Returns the wanted backend module"""
    # Import backend functions.
    if backend == "backend 1":
        sys.stderr.write('Using backend 1\n')
        from . import backend_1 as backend_module
    elif backend == "backend 2":
        sys.stderr.write('Using backend 2\n')
        from . import backend_2 as backend_module
    else:
        raise ValueError('Unable to import backend : ' + str(_BACKEND))

    return backend_module

An improvement could be to use importlib to dynamically import the backend and move the magic strings to a constant:

...
import importlib

BACKENDS = {
    "backend 1": "backend_1",
    "backend 2": "backend_2"
}

def load_backend(backend):
    try:
        module = importlib.import_module(
            BACKENDS[backend]
        )
    except KeyError:
        raise ImportError('Unable to import backend : %s' % backend)

    sys.stderr.write('Using %s\n' % backend)
    return module

So you can do this in the do_stuff file:

import load_backend

def do_stuff(config, data):
    # Do some really important stuff that is coded in backend 1 and backend 2
    backend = load_backend.backend(config)
    a = backend.do_something(data)
    print(a)

Another way to go about this is to use a singleton pattern, where you set the backend variable once (and other settings you want broadly available):

in a settings.py or whereever:

class SettingSingleton(object):
    _backend = None

    def __new__(cls, backend=None, *args, **kwargs):
        cls._backend = cls._backend or backend
        return super(SettingsSingleton, cls).__new__(cls, *args, **kwargs)

    @property
    def backend(self):
        return self._backend

You can initialize that in the main.

from argparse import ArgumentParser
from utils.loader import load_data
from utils.do import do_stuff
from settings import SettingSingleton


def main(config):
    SettingsSingleton(backend=config)
    data = load_data(config)
    do_stuff(config, data)

...

Now you can do something like:

from __future__ import absolute_import
from __future__ import print_function
import os
import sys

from settings import SettingsSingleton

_BACKEND = SettingsSingleton().backend

# Import backend functions.
if _BACKEND == "backend 1":
    sys.stderr.write('Using backend 1\n')
    from .backend_1 import *
elif _BACKEND == "backend 2":
    sys.stderr.write('Using backend 2\n')
    from .backend_2 import *
else:
    raise ValueError('Unable to import backend : ' + str(_BACKEND))

Downside to this is that it is somewhat implicit.

Sign up to request clarification or add additional context in comments.

Comments

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.