Organising packages/modules - importing functions from a common.py in a separate directory?

[Victor Hooi]

Hi,

I have a collection of Python scripts I'm using to load various bits of data into a database.

I'd like to move some of the common functions (e.g. to setup loggers, reading in configuration etc.) into a common file, and import them from there.

I've created empty __init__.py files, and my current directory structure looks something like this:

foo_loading/
__init__.py
common/
common_foo.py
em_load/
__init__.py
config.yaml
sync_em.py
pg_load/
__init__.py
config.yaml
sync_pg.py

So from within the sync_em.py script, I'm trying to import a function from foo_loading/common/common_foo.py.

from ..common.common_foo import setup_foo_logging

I get the error:

ValueError: Attempted relative import in non-package

If I change directories to the parent of "foo_loading", then run

python -m foo_loading.em_load.sync_em sync_em.py

it works. However, this seems a bit roundabout, and I suspect I'm not doing things correctly.

Ideally, I want a user to be able to just run sync_em.py from it's own directory, and have it correctly import the logging/config modules from common_foo.py, and just work.

What is the correct way to achieve this?

Secondly, if I want to move all of the config.yaml files to a common foo_loading/config.yaml, or even foo_loading/config/config.yaml, what is the correct way to access this from within the scripts? Should I just be using "../", or is there a better way?

Cheers,
Victor

[Jean-Michel Pichavant]

Long story short : use absolute imports.

name properly your module with a distinct name and import that way, even inside your package:

import foo_loading.common.common_foo

Names like common, lib, setup are farely prone to collision with other badly referenced import from other modules. One way to solve this is to use a distinct namespace, in other words, prefix every import with the module name.

cheers,

JM

-- IMPORTANT NOTICE:

The contents of this email and any attachments are confidential and may also be privileged. If you are not the intended recipient, please notify the sender immediately and do not disclose the contents to any other person, use it for any purpose, or store or copy the information in any medium. Thank you.

[Victor Hooi]

Hi,

Ok, so I should be using absolute imports, not relative imports.

Hmm, I just tried to use absolute imports, and it can't seem to locate the modules:

In the file "foo_loading/em_load/sync_em.py", I have:

from common.common_bex import setup_foo_logging

When I try to run that script:

python sync_em.py

I get:

ImportError: No module named common.common_foo

I've also tried adding "foo_loading" (the package name):

from foo_loading.common.common_bex import setup_foo_logging

Same error:

ImportError: No module named foo_loading.common.bex_common

Any thoughts?

Cheers,
Victor

[Ben Finney]

Victor Hooi <victo...@gmail.com> writes:

> Ok, so I should be using absolute imports, not relative imports.

I'd say it is fine to use relative imports, so long as they are
explicit. (In Python 3, the default for an import is to be absolute, and
the *only* way to do a relative import is to make it explicitly
relative. So you may as well start doing so now.)

> Hmm, I just tried to use absolute imports, and it can't seem to locate
> the modules:
>
> In the file "foo_loading/em_load/sync_em.py", I have:
>
> from common.common_bex import setup_foo_logging

So I'd recommend this be done with an explicit relative import:

from .common.common_bex import setup_foo_logging

or, better, import a module:

from .common import common_bex

or a whole package:

from . import common

--
\ “I went over to the neighbor's and asked to borrow a cup of |
`\ salt. ‘What are you making?’ ‘A salt lick.’” —Steven Wright |
_o__) |
Ben Finney

[Victor Hooi]

Hi,

If I try to use:

from .common.common_foo import setup_foo_logging

I get:

ValueError: Attempted relative import in non-package

And the absolute imports don't seem to be able to find the right modules.

Is it something to do with the fact I'm running the sync_em.py script from the "foo_loading/em_load" directory?

I thought I could just refer to the full path, and it'd find it, but evidently not...hmm.

Cheers,
Victor

[Victor Hooi]

Hi,

Hmm, this post on SO seems to suggest that importing from another sibling directory in a package ins't actually possibly in Python without some ugly hacks?

http://stackoverflow.com/questions/6323860/sibling-package-imports

Did I read the above correctly?

Is there another way I can structure my code so that I can run the sync_em.py and sync_pg.py scripts, and they can pull common functions from somewhere?

Cheers,
Victor

[Peter Otten]

Victor Hooi wrote:

> Hi,
>
> Hmm, this post on SO seems to suggest that importing from another sibling
> directory in a package ins't actually possibly in Python without some ugly
> hacks?
>
> http://stackoverflow.com/questions/6323860/sibling-package-imports
>
> Did I read the above correctly?

Yes.

> Is there another way I can structure my code so that I can run the
> sync_em.py and sync_pg.py scripts, and they can pull common functions from
> somewhere?

The packages you are trying to access in your original post

> foo_loading/
> __init__.py
> common/
> common_foo.py
> em_load/
> __init__.py
> config.yaml
> sync_em.py
> pg_load/
> __init__.py
> config.yaml
> sync_pg.py

aren't actually siblings in the sense of the stackoverflow topic above, they
are subpackages of foo_loading, and as you already found out

> So from within the sync_em.py script, I'm trying to import a function from
foo_loading/common/common_foo.py.
>
> from ..common.common_foo import setup_foo_logging
>
> I get the error:
>

> ValueError: Attempted relative import in non-package
>

> If I change directories to the parent of "foo_loading", then run
>
> python -m foo_loading.em_load.sync_em sync_em.py
>
> it works. However, this seems a bit roundabout, and I suspect I'm not
doing things correctly.
>
> Ideally, I want a user to be able to just run sync_em.py from it's own
directory, and have it correctly import the logging/config modules from
common_foo.py, and just work.
>
> What is the correct way to achieve this?

you can access them as long as the *parent* directory of foo_loading is in
sys.path through PYTHONPATH, or as the working directory, or any other
means. However, if you step into the package, e. g.

$ cd foo_loading
$ python -c 'import common'

then from Python's point of view 'common' is a toplevel package rather than
the intended 'foo_loading.common', and intra-package imports will break.

To preserve your sanity I therefore recommend that you

(1) avoid to put package directories into sys.path
(1a) avoid to cd into a package
(2) put scripts you plan to invoke directly rather than import outside the
package.

[Jean-Michel Pichavant]

----- Original Message -----
> Hi,
>
> If I try to use:
>
> from .common.common_foo import setup_foo_logging
>
> I get:
>
> ValueError: Attempted relative import in non-package

If you're using python 3, forget what I said about not using relative imports. I think they've implemented the necessary grammar to make it work.

Cheers,

[Victor Hooi]

Hi,

Wait - err, subpackage != module, right? Do you think you could explain what a sub-package is please? I tried Googling, and couldn't seem to find the term in this context.

Also, so you're saying to put the actual script that I want to invoke *outside* the Python package.

Do you mean something like this:

> sync_em.py
> sync_pg.py

> foo_loading/
> __init__.py
> common/
> common_foo.py
> em_load/
> __init__.py
> config.yaml

> em.py
> pg_load/
> __init__.py
> config.yaml
> pg.py

and the sync_em.py and sync_pg.py would just be thin wrappers pulling in things from em.py and pg.py? Is that a recommended approach to organise the code?

Would it make any difference if I actually packaged it up so you could install it in site-packages? Could I then call modules from other modules within the package?

Cheers,
Victor

[Peter Otten]

Victor Hooi wrote:

> Wait - err, subpackage != module, right? Do you think you could explain
> what a sub-package is please? I tried Googling, and couldn't seem to find
> the term in this context.

In analogy to subdirectory I em_load and pg_load -- and common if you add an
__init__.py would be sub-packages, provided only the parent of foo_loading
is in sys.path and you import them with

import foo_loading.pg_load

etc.

> Also, so you're saying to put the actual script that I want to invoke
> *outside* the Python package.
>
> Do you mean something like this:
>
>> sync_em.py
>> sync_pg.py
>> foo_loading/
>> __init__.py
>> common/

__init__.py

>> common_foo.py
>> em_load/
>> __init__.py
>> config.yaml
>> em.py
>> pg_load/
>> __init__.py
>> config.yaml
>> pg.py
>
> and the sync_em.py and sync_pg.py would just be thin wrappers pulling in
> things from em.py and pg.py? Is that a recommended approach to organise
> the code?

I don't know. I prefer it that way.

> Would it make any difference if I actually packaged it up so you could
> install it in site-packages? Could I then call modules from other modules
> within the package?

If you mean "import", yes, installing is one way to get it into sys.path.

[Peter Otten]

Victor Hooi wrote:

> Wait - err, subpackage != module, right? Do you think you could explain
> what a sub-package is please? I tried Googling, and couldn't seem to find
> the term in this context.

[second attempt]

In analogy to the term "subdirectory" em_load and pg_load -- and common if
you add an __init__.py -- would be sub-packages, provided only the parent of