Import arbitrary python source file. (Python 3.3+)

Question

How can I import an arbitrary python source file (whose filename could contain any characters, and does not always ends with .py) in Python 3.3+?

I used imp.load_module as follows:

>>> import imp
>>> path = '/tmp/a-b.txt'
>>> with open(path, 'U') as f:
...     mod = imp.load_module('a_b', f, path, ('.py', 'U', imp.PY_SOURCE))
...
>>> mod
<module 'a_b' from '/tmp/a-b.txt'>

It still works in Python 3.3, but according to imp.load_module documentation, it is deprecated:

Deprecated since version 3.3: Unneeded as loaders should be used to load modules and find_module() is deprecated.

and imp module documentation recommends to use importlib:

Note New programs should use importlib rather than this module.

What is the proper way to load an arbitrary python source file in Python 3.3+ without using the deprecated imp.load_module function?

Can I ask why you are doing this? I'm the maintainer of importlib and I have been trying to get answers from folks as to why they use imp.load_module() over a straight import statement. Do you expect to import the module by name later (e.g. import a_b)? Do you care that any custom importers won't be used in this approach? Do you expect the module to be full-featured (e.g. define __name__ and __loader__)? — Brett Cannon
– Brett Cannon, Commented Mar 31, 2014 at 15:49
@BrettCannon, A third-party program regularly (once a hour) modify a text file that contains python statements (mainly THIS='blah' like lines). The name of the file is not ended with .py. My program read that file. — falsetru
– falsetru, Commented Mar 31, 2014 at 15:58
@BrettCannon, I'm not aware of custom importers. I don't care the module to be full-featured. — falsetru
– falsetru, Commented Mar 31, 2014 at 15:59
IOW using Python as a really simple data structure format. Thanks for the info! — Brett Cannon
– Brett Cannon, Commented Mar 31, 2014 at 16:33
@BrettCannon — I just ran into a case where I needed to import some Python code from within a directory which was named as a version number (e.g., "v1.0.2"). While possible, it would be highly undesirable to rename the directory. I wound up using stefan-scherfke's solution below. — Andrew Miner
– Andrew Miner, Commented Feb 27, 2018 at 20:16

falsetru · Accepted Answer · 2020-01-17 03:08:47Z

123

Found a solution from importlib test code.

Using importlib.machinery.SourceFileLoader:

>>> import importlib.machinery
>>> loader = importlib.machinery.SourceFileLoader('a_b', '/tmp/a-b.txt')
>>> mod = loader.load_module()
>>> mod
<module 'a_b' from '/tmp/a-b.txt'>

NOTE: only works in Python 3.3+.

UPDATE Loader.load_module is deprecated since Python 3.4. Use Loader.exec_module instead:

>>> import types
>>> import importlib.machinery
>>> loader = importlib.machinery.SourceFileLoader('a_b', '/tmp/a-b.txt')
>>> mod = types.ModuleType(loader.name)
>>> loader.exec_module(mod)
>>> mod
<module 'a_b'>

>>> import importlib.machinery
>>> import importlib.util
>>> loader = importlib.machinery.SourceFileLoader('a_b', '/tmp/a-b.txt')
>>> spec = importlib.util.spec_from_loader(loader.name, loader)
>>> mod = importlib.util.module_from_spec(spec)
>>> loader.exec_module(mod)
>>> mod
<module 'a_b' from '/tmp/a-b.txt'>

edited Jan 17, 2020 at 3:08

answered Sep 25, 2013 at 17:17

falsetru

371k69 gold badges769 silver badges659 bronze badges

Sign up to request clarification or add additional context in comments.

17 Comments

falsetru Over a year ago

Downvoter: How can I improve the answer? If you have a better way to accomplish what I want, please let me know.

Eryk Sun Over a year ago

There's a helpful warning that load_module ignores via warnings.catch_warnings. If you instead use mod = imp.load_source('a_b', '/tmp/a-b.txt'), it raises the following warning (use -Wall):

DeprecationWarning: imp.load_source() is deprecated; use importlib.machinery.SourceFileLoader(name, pathname).load_module() instead

.

falsetru Over a year ago

@eryksun, You're right. Thank you for the comment. BTW, Python 3.4(rc1) does not display the alternative usage unlike Python 3.3.x.

Matthew D. Scholefield Over a year ago

What's the difference between the first and the second example at the bottom?

falsetru Over a year ago

@ihavenoidea, Please post a separated question, so that others can answer you, and other users can also read answers.

|

Stefan Scherfke · Accepted Answer · 2022-02-09 07:51:07Z

50

Updated for Python >= 3.8:

Short version:

>>> # https://docs.python.org/3/library/importlib.html#importing-a-source-file-directly
>>> import importlib.util, sys
>>> spec = importlib.util.spec_from_file_location(modname, fname)
>>> module = importlib.util.module_from_spec(spec)
>>> sys.modules[modname] = module
>>> spec.loader.exec_module(module)

Full version:

>>> import importlib.util
>>> import sys
>>> from pathlib import Path
>>> from typing import TYPE_CHECKING
>>> 
>>> 
>>> if TYPE_CHECKING:
...     import types
...
...
>>> def import_source_file(fname: str | Path, modname: str) -> "types.ModuleType":
...     """
...     Import a Python source file and return the loaded module.

...     Args:
...         fname: The full path to the source file.  It may container characters like `.`
...             or `-`.
...         modname: The name for the loaded module.  It may contain `.` and even characters
...             that would normally not be allowed (e.g., `-`).
...     Return:
...         The imported module

...     Raises:
...         ImportError: If the file cannot be imported (e.g, if it's not a `.py` file or if
...             it does not exist).
...         Exception: Any exception that is raised while executing the module (e.g.,
...             :exc:`SyntaxError).  These are errors made by the author of the module!
...     """
...     # https://docs.python.org/3/library/importlib.html#importing-a-source-file-directly
...     spec = importlib.util.spec_from_file_location(modname, fname)
...     if spec is None:
...         raise ImportError(f"Could not load spec for module '{modname}' at: {fname}")
...     module = importlib.util.module_from_spec(spec)
...     sys.modules[modname] = module
...     try:
...         spec.loader.exec_module(module)
...     except FileNotFoundError as e:
...         raise ImportError(f"{e.strerror}: {fname}") from e
...     return module
...
>>> import_source_file(Path("/tmp/my_mod.py"), "my_mod")
<module 'my_mod' from '/tmp/my_mod.py'>

Original answer for Python 3.5 and 3.6

Shorter version of @falsetru 's solution:

>>> import importlib.util
>>> spec = importlib.util.spec_from_file_location('a_b', '/tmp/a-b.py')
>>> mod = importlib.util.module_from_spec(spec)
>>> spec.loader.exec_module(mod)
>>> mod
<module 'a_b' from '/tmp/a-b.txt'>

I tested it with Python 3.5 and 3.6.

According to the comments, it does not work with arbitrary file extensions.

edited Feb 9, 2022 at 7:51

answered Jan 11, 2017 at 16:09

Stefan Scherfke

3,2521 gold badge21 silver badges30 bronze badges

3 Comments

falsetru Over a year ago

importlib.util.spec_from_file_location(..) returns None for me; causing an exception for the following importlib.util.module_from_spec(..) call. (See i.imgur.com/ZjyFhif.png)

falsetru Over a year ago

importlib.util.spec_from_file_location works for known file name extensions (.py, .so, ..), but not for others (.txt...)

Stefan Scherfke Over a year ago

Oh, I’m using it only with Python files but modified my example to look like the one above and did not test it … I updated it.

Alex Walczak · Accepted Answer · 2018-07-28 23:14:49Z

16

Similar to @falsetru but for Python 3.5+ and accounting for what the importlib doc states on using importlib.util.module_from_spec over types.ModuleType:

This function [importlib.util.module_from_spec] is preferred over using types.ModuleType to create a new module as spec is used to set as many import-controlled attributes on the module as possible.

We are able to import any file with importlib alone by modifying the importlib.machinery.SOURCE_SUFFIXES list.

import importlib

importlib.machinery.SOURCE_SUFFIXES.append('') # empty string to allow any file
spec = importlib.util.spec_from_file_location(module_name, file_path)
module = importlib.util.module_from_spec(spec)
spec.loader.exec_module(module)
# if desired: importlib.machinery.SOURCE_SUFFIXES.pop()

answered Jul 28, 2018 at 23:14

Alex Walczak

1,4061 gold badge13 silver badges28 bronze badges

3 Comments

mxxk Over a year ago

Interestingly enough, while this hack of appending the empty string to the list of source suffixes works great for importing renamed Python source modules, the equivalent for importing renamed extension modules does not work... That is, using importlib.machinery.EXTENSION_SUFFIXES.append('') still makes importlib.util.spec_from_file_location return None.

Alex Walczak Over a year ago

presumably, importlib.util.spec_from_file_location should still work with extensions if you specify a loader

Jon Over a year ago

@mxxk that's because, if you look at the source, it's only checking if the suffix is present for files named __init__

Ciro Santilli OurBigBook.com · Accepted Answer · 2024-08-15 11:24:07Z

importlib helper function tested on Python 3.10

Here is a convenient, ready-to-use helper to replace imp, with an example. The technique is the same as that of https://stackoverflow.com/a/19011259/895245 , this is just providing a more convenient function.

main.py

#!/usr/bin/env python3

import os
from importlib import util, machinery

def import_path(path):
    module_name = os.path.basename(path).replace('-', '_')
    spec = util.spec_from_loader(
        module_name,
        importlib.machinery.SourceFileLoader(module_name, path)
    )
    module = util.module_from_spec(spec)
    spec.loader.exec_module(module)
    sys.modules[module_name] = module
    return module

notmain = import_path('not-main')
print(notmain)
print(notmain.x)

not-main

x = 1

Run:

python3 main.py

Output:

<module 'not_main' from 'not-main'>
1

I replace - with _ because my importable Python executables without extension have hyphens as in my-cmd. This is not mandatory, but produces better module names like my_cmd.

How can I import a module dynamically given the full path?

Python 3.7

In that version I was instead using the following invocations for the helper, but they do not seem to work on Python 3.10

import importlib    
    spec = importlib.util.spec_from_loader(
    module = importlib.util.module_from_spec(

This pattern is also mentioned in the docs at: https://docs.python.org/3.7/library/importlib.html#importing-a-source-file-directly

I ended up moving to it because after updating to Python 3.7, import imp prints:

DeprecationWarning: the imp module is deprecated in favour of importlib; see the module's documentation for alternative uses

and I don't know how to turn that off, this was asked at:

urek mazino · Accepted Answer · 2022-03-18 18:23:53Z

1

after many failure solutions this one works for me

def _import(func,*args):
    import os
    from importlib import util
    module_name = "my_module"
    BASE_DIR = "wanted module directory path"
    path =  os.path.join(BASE_DIR,module_name)
    spec = util.spec_from_file_location(func, path)
    mod = util.module_from_spec(spec)
    spec.loader.exec_module(mod)
    return getattr(mod,func)(*args)

and to call it just write the function name and it's parameters _import("function",*args)

answered Mar 18, 2022 at 18:23

urek mazino

3073 silver badges17 bronze badges

Collectives™ on Stack Overflow

Import arbitrary python source file. (Python 3.3+)

5 Answers 5

17 Comments

Updated for Python >= 3.8:

Original answer for Python 3.5 and 3.6

3 Comments

3 Comments

Comments

Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

5 Answers 5

17 Comments

Updated for Python >= 3.8:

Original answer for Python 3.5 and 3.6

3 Comments

3 Comments

Comments

Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related