How to access code in a top-level file in a python package

Question

I have a Python 2.7 package laid out like this:

hdl/
    __init__.py
    run_job.py
    other_stuff/
        __init__.py
        other_files/
setup.py
scripts/
    client.py

run_job.py contains:

def run_job():
    pass

There is no class. Actually, there are no classes anywhere in this codebase; I inherited it.

How do I execute the run_job() function from clients of this package?

(Right now, the run_job() function lives in __init__.py, which I don't like at all.)

I put these in __init__.py:

import hdl.run_job

import run_job

Either way, I get No module named run_job. If I leave off the import entirely, the client code says either 'module' object has no attribute 'run_job' or name 'run_job' is not defined, depending on which of these I do:

import hdl
hdl.run_job

import hdl
run_job

My setup.py:

from setuptools import setup
from setuptools import find_packages
from os import walk, path

install_requires = [
    "ecdsa==0.11",
    "importlib==1.0.3",
    "paramiko==1.13.0",
    "pyasn1==0.1.7",
    "pycrypto==2.6.1",
    "wsgiref==0.1.2",
]

script_base = "scripts"
my_path = path.dirname(path.abspath(__file__))
walk_path = path.join(my_path, script_base)
for(_, _, filenames) in walk(walk_path):
    scripts = [("%s/%s"  % (script_base, f)) for f in filenames]
    break

setup(
    name="hdl",
    version="dev",
    description="HDL",
    url="http://example.com",
    long_description="HDLA",
    author="heh",
    author_email=".com",
    maintainer="foo",
    maintainer_email=".com",
    license="Proprietary",
    packages=find_packages(exclude=("tests",)),
    test_suite="nose.collector",
    include_package_data=True,
    package_data = {'': [ '*.csv', '*.hql', '*.pig', '*.sh', '*.sql' ]},
    install_requires=install_requires,
    zip_safe=False,
    classifiers=[
        "Development Status :: 2 - Alpha",
        "Environment :: Command Line",
        "Framework :: None",
        "Intended Audience :: Developers",
        "Intended Audience :: System Administrators",
        "License :: Other/Proprietary License",
        "Natural Language :: English",
        "Operating System :: MacOS :: MacOS X",
        "Operating System :: POSIX :: Linux",
        "Programming Language :: Python :: 2.7",
        "Topic :: Software Development",
        "Topic :: ETL"
    ],
    scripts=scripts,
)

If I stick the run_job function in __init__.py it works fine: import hdl hdl.run_job()

which init.py are you editing? The one in hd1 or other_stuff? — Bryan
– Bryan, Commented Jun 25, 2014 at 4:33
The one in hdl. The one in other_stuff is irrelevant, and if its relevant then there's something very weird about Python. — Satya
– Satya, Commented Jun 25, 2014 at 4:34
What do you mean by "clients of this package"? You mean programs that import hdl? Have you tried using import hdl.run_job inside the client code? — BrenBarn
– BrenBarn, Commented Jun 25, 2014 at 4:44

BrenBarn · Accepted Answer · 2014-06-25 04:46:48Z

3

Your client code needs to do from hdl import run_job and then do run_job.run_job(), or (probably more readable) from hdl.run_job import run_job.

If you want to provide "top level" access to run_job (that is, not require users to import the run_job module to get the run_job function), then in your top-level __init__.py do from .run_job import run_job. Then clients can do import hdl and later use hdl.run_job.

answered Jun 25, 2014 at 4:46

BrenBarn

253k39 gold badges421 silver badges392 bronze badges

Sign up to request clarification or add additional context in comments.

5 Comments

Satya Over a year ago

Says cannot import name run_job, and in your 2nd suggestion, No module named run_job (on the from .run_job import run_job line)

BrenBarn Over a year ago

@Satya: It works for me using the file layout you've described. Check that your files are where you think they are and are named what you think they are. Is is possible you have another file names hdl.py somewhere else that is being imported instead of your package?

Satya Over a year ago

I'm installing them to the virtualenv by using python setup.py install, is that correct? The package directory structure is as I stated in my question. I don't have another hdl.py, as my changes are resulting in slightly different error messages as I go along.

BrenBarn Over a year ago

@Satya: What is the name of the "client code" file from which you are trying the imports, and where is it located?

Satya Over a year ago

It is outside the package hierarchy. It is in the scripts directory and is called client.py (question updated after I post this comment). Since it is in scripts, I am running it as: source virtualenv/bin/activate && client.py (the source && is in my history and I've been hitting up-arrow)

Oleg Gryb · Accepted Answer · 2014-06-25 06:10:53Z

2

Put the following in __init__.py:

__all__ = ["run_job"]

In your client:

from hdl import run_job
run_job.run_job()

in your setup.py "scripts" variable is not defined, so I had to remove the following string:

scripts=scripts,

After that I ran "sudo python setup.py install" and tested it with the client:

$ python
Python 2.7.6 (default, Mar 22 2014, 17:40:27) 
[GCC 4.8.2] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> from hdl import run_job
>>> run_job.run_job()
>>>

Everything works. Try yet another thing:

pydoc hdl

Check if the path provided in the generated doc refers to the right location.

Help on package hdl:

NAME
    hdl

FILE
    <path-to-hdl>/hdl/__init__.py

PACKAGE CONTENTS
    run_job

So, I have exactly the same layout and setup.py file as yours. Below is the content of files and it works on my comp. Try to carefully compare with what you have and make sure that "setup.py install" doesn't have any errors. If pydoc doesn't show anything, hdl was not installed.

==========
run_job.py
=========== 
def run_job():
    pass

============ client.py ===========

from hdl import run_job
run_job.run_job()

============ setup.py ===========

exactly as yours

edited Jun 25, 2014 at 6:10

answered Jun 25, 2014 at 4:43

Oleg Gryb

5,3091 gold badge35 silver badges46 bronze badges

14 Comments

BrenBarn Over a year ago

The __all__ is not necessary and may cause confusion later if he tries to add other things to __init__.py.

Oleg Gryb Over a year ago

If he has more modules under hd1 and wants to use "from hd1 import *", he'll need all. It's just a common pattern for exportable modules and I'm not sure why it's confusing.

Satya Over a year ago

cannot import name run_job in the client code's import line

Oleg Gryb Over a year ago

I stay corrected, but I think, it's still a good pattern to explicitly control what you want to export and I still don't understand why you're saying that it's confusing.

Oleg Gryb Over a year ago

@Satya - you need to publish your setup.py script then

|

Satya · Accepted Answer · 2014-06-25 06:12:07Z

0

The file wasn't named run_job.py it was named something else. Sigh. Fixing the name causes everything to work.

The other answers are correct; leave the __init__.py empty and in client.py do this:

from hdl import run_job
run_job.run_job()

or

from hdl.run_job import run_job
run_job()

or put this in __init__.py:

from .run_job import run_job

and this in client.py:

import hdl
hdl.run_job() # won't work

edited Jun 25, 2014 at 6:12

answered Jun 25, 2014 at 6:00

Satya

4,49824 silver badges30 bronze badges

1 Comment

Oleg Gryb Over a year ago

OK, I'm glad that the mystery is solved - don't forget to vote :)

Collectives™ on Stack Overflow

How to access code in a top-level file in a python package

3 Answers 3

5 Comments

14 Comments

1 Comment

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

3 Answers 3

5 Comments

14 Comments

1 Comment

Your Answer

Sign up or log in

Post as a guest

Related