I Finally Figured Out Python’s Module and Package System

I Finally Figured Out Python's Module and Package System Featured Image

Python is a language I recommend to anyone who wants to learn to code, but there’s one aspect of the language that has frustrated me for some time: the package system. Dealing with directory structures, build files, and dependencies are always a source of difficulty for any programming language, but they seem more pronounced with Python. As a result, I thought I’d share my approach to dealing with Python’s module and package system. Hope you enjoy it!

Table of Contents

History

As many folks in the community know, I started using Python in 2017. Despite having been formally taught Java, I would consider myself fully self-taught as far as Python is concerned. As a result, there are a few fundamentals that I’m still uncomfortable with to this day.

One of those fundamentals is Python’s package and module system which has given me headaches for the last 5 years. As you may know, I maintain a variety of Python packages that you can install casually using pip:

Because of the work I put in to make those packages easy to use, you might not know how much trouble I had along the way. That said, it wasn’t until very recently that I was able to put together a complex package with multiple levels of nesting. As a result, I figured I’d spend some time today talking about the things I’ve learned about the package system and how you can have ease of mind as well.

The Problem

The project that officially got me to understand the module system in Python was a little 2D game that I’m still fleshing out called Mage GameOpens in a new tab.. It seems that a lot of folks who use PyGame to make games in Python merge all of there data and game code in one place. I didn’t want that, so I opted for a design that follows MVCOpens in a new tab. (i.e., model-view-controller).

Because MVC design focuses on the separation of code structures, you end up with several files. As anyone who has tried to make a Python package knows, challenges arise when you split your package into multiple files (i.e., multiple modules). For instance, it’s common for circular import errors to occur:

For me, I’m a big fan of type hints. As a result, any time a function in one module references a type from another module, I like to include that type hint. This tends to causes various problems with the import system.

To make matters worse, I like to use folders to organize my code into hierarchies. For example, in an MVC design, I like to have my model code in a model folder, the controller code in the controller folder, and the view code in the view folder. As you can imagine, importing those local modules becomes a nightmare.

After enough messing around, you can probably get your imports to work. However, once you’ve decided to actually run your code, a new problem will arise: the imports don’t work at runtime. This is singlehandedly the biggest nightmare fuel in Python for me. How is it that my IDE can recognize the imports correctly but Python itself can’t? And don’t get me started on trying to test your code. Needless to say, here were some of the Stack Overflow threads I relied on for a time:

And as I kept reading, I kept seeing more and more cursed solutions—many of which involving the patching of the system path through Python. So today, I want to share with you some strategies for getting your complex Python projects working without any hacks. Let’s do it!

How to Setup a Basic Python Package

Note: there are resources that are often referenced for directory structures and whatnot for Python packages that I don’t particularly like. One of those references is from Jean-Paul Calderone. You can check it out hereOpens in a new tab., if you’d like. It’s quite old at this point (written in 2007), but you may find it useful. Just know that I don’t adhere to basically any of the points made in it. If you want a reference that I find a lot better, check out the work of Ionel Cristian Mărieș hereOpens in a new tab. (written in 2014). I follow their recommendations more closely. Also, I should note that any mention of “source-code” below refers to the package name directly. A lot of folks advocate for packages to be nested in an “src” folder—like Hynek SchlawackOpens in a new tab. (written in 2015)—and I think that’s generally a good idea. I don’t do that here because I find the “src” folder setup to be slightly more annoying. That said, the choice is up to you!

To start, let’s talk about the big picture. How should you lay out your package directory? To me, there are a few main items that we care about:

  • Source code
  • Test code
  • Scripts (i.e., main.py)
  • Packaging (i.e., setup.py)

Projects might also include assets or other data files, but we’ll focus on these four main areas.

When I setup a project, I like to have the four key areas represented at the root of my project directory. As a result, I might have a folder structure that mirrors the list above (i.e., a folder for source code, a folder for test code, a main script, and a setup script). If you prefer a folder style diagram, here’s what I would expect to see:

project-directory
  |-- source-code
  |-- test-code
  |-- main.py
  |-- setup.py

Basic setup.py

Now, there are two main routes you can take with a project like this: make a pip package or make an executable. Because I’m basing this guide off of game design, I’m going to show you how to make an executable. That said, a traditional setup file could work as well. Regardless, your setup file should look something like this:

import cx_Freeze

executables = [cx_Freeze.Executable("main.py")]

cx_Freeze.setup(
    name="Package Name",
    version="0.1.0",
    options={
        "build_exe": {
            "packages":[],
            "include_files":[]
        }
    },
    executables = executables,
)

Right now, this file would probably work as-is to create an executable that did nothing. That said, having a file setup like this is absolutely critical to having a good time. To use this script to make an executable, use the following command:

python setup.py bdist_msi

It should create an MSI file for Windows that you can use to install your script.

Basic main.py

As far as Python scripts go, you don’t need a lot of code to get running. I might make a script that does nothing but follows good conventions:

if __name__ == "__main__":
  pass

Eventually, you’ll want to fill this file with the proper imports from your source code to actually do something, but we’ll get to that later.

Basic Tests Folder

While we don’t currently have any code, we’ll want to follow good design principles by putting some testing in place. I like to use pytest for its simplicity, so my test folder might look like this:

project-directory
  |-- test-code/
    |-- __init__.py
    |-- test_example.py

I always make sure to put the `__init__.py` file in the testing folder, so Python knows to treat it as a package. You may have read that the `__init__.py` file can cause problems with pytestOpens in a new tab. but that’s only if you include the test folder inside of your source code folder. Because we’ve separated the two out, we shouldn’t run into that problem. That said, the pytest docs showOpens in a new tab. that separating the test files out into their own directory without a `__init__.py` file should also be fine.

Basic Source Code Folder

As mentioned above, there is a lot of disagreement about the appropriate design of the source code folder. Should this folder include tests? Should this folder be included in an “src” folder? According to the setuptools docsOpens in a new tab., the design I’m sharing with you is known as the “flat layout”. In other words, we are not nesting our source code in an “src” folder; the package will be in our root directory.

I prefer this layout because I tend to include scripts at the root structure of the repo. For example, in game development, I can have the main script sitting in the project root with direct access to the local package. Otherwise, I’d have to do a little bit of work to get the script to find the appropriate packages. Similar arguments can be made for testing, but there are also counterarguments mentioned in the resource above as well. Every choice has a tradeoff.

At any rate, here’s what the inside of a source code folder might look like:

project-directory
  |-- source-code
    |-- __init__.py
    |-- example.py

Again, the `__init__.py` file indicates a package. In this case, our package is called “source-code”, but you would call it whatever your project was called (i.e., mage_game).

Handling Imports In a Basic Project Setup

Assuming you have a project setup like the one above, you can start to create some files. For example, here’s what our `example.py` module might look like:

import random

def attack(damage: int, crit_rate: float) -> int:
  if random.random() < crit_rate:
    return damage * 2
  else:
    return damage

In the spirit of gaming, I wrote a silly attack function that returns the damage we do on an attack while factoring in the crit rate.

Now, this isn’t easily testable, but we could imagine that we want to make sure this function returns one of two possible values: damage or damage * 2. We can do that easily in our `test_example.py` file:

from source-code.example import *

def test_attack():
  damage = attack(5, .5)
  assert damage in (5, 10)

If all is well, you can run pytest in your root directory and get back a passing test. The key here is that we can import the function we want to test basically directly from the package name. This comes with a caveat of course. If `source-code` is installed on your system, the tests will run against that—not your code. Because we’re creating an executable and not a PyPI package, there should be no issue in this case.

Making a Mess

Now, in my experience, there is never just one file like this in a package. If there was, we wouldn’t need to worry about folder structure at all. Just have all four files sitting in the root of your repository. To make matters worse, no one really talks about how to scale a project. So, let’s now imagine that our `example.py` file has grown out of control, and we want to split it into multiple files. A great example of this is my `model` package in Mage Game:

model
  |-- __init__.py
  |-- bindings.py
  |-- character.py
  |-- magic.py
  |-- state.py
  |-- world.py

In this module, we have several files that all reference each other in some capacity. To make sure this works, we use relative imports (as opposed to absolute imports). For example, here’s the top of the `character.py` file:

from __future__ import annotations

from dataclasses import dataclass, field

from .magic import *
from .world import Entity, WorldPoint

The bottom two imports are the interesting ones. We import everything from the `magic.py` module and a couple of things from the `world.py` module. To ensure these are imported from the files in the same folder, we place a period in front of the module names. Otherwise, we might run into a nasty ImportError.

Now, let’s say that our model package used to be a single module (like `example.py`). There’s a good chance that someone was depending on that module. No worries, we can pretend it still exists with a simple trick. Edit the `__init__.py` to import all of the submodules:

from .bindings import *
from .character import *
from .magic import *
from .state import *
from .world import *

To the outside world, `model.py` still exists. We’ve just converted it into a folder (i.e., package). Obviously, there are risks with this type of design—namely clashes of symbols. That said, if the code is appropriated separated, I think it’s unlikely that you’ll end up with functions of objects with the same name.

What’s really cool is you can then test the model code as if it were just a single file, not a complicated folder structure. Here’s an example that tests some of the magic:

from magegame.model import *

def test_palette_item_default():
    default = PaletteItem()
    assert default.can_use() == True, "Default palette items should be usable from the start"
    assert default.get_spell() == Projectile(), "Default palette items should store default projectiles"
    
def test_palette_item_reset_cooldown():
    default = PaletteItem()
    default.reset_cooldown()
    assert default.get_cooldown() == SpellAttribute.COOLDOWN.base_value, "Default palette items should store the default cooldown after reset"

Notice how the import just pulls in everything from `model`. You could make them more specific if you’d like, but this gets the job done.

Finally, we can actually make up our main script. Here’s the one I currently use in my game:

import argparse
import logging
import os
import sys
from logging.handlers import RotatingFileHandler

from magegame import controller, eventmanager, model, view

logger = logging.getLogger(__name__)


def _init_logger(args: argparse.Namespace) -> None:
    ...


def _run_with_crash_handling(game_model: model.GameEngine, log_path: str):
    ...


def _process_arguments() -> argparse.Namespace:
    ...


def run():
    """
    The main function that launches the game.
    """
    args = _process_arguments()
    _init_logger(args)
    event_manager = eventmanager.EventManager()
    game_model = model.GameEngine(event_manager)
    keyboard = controller.MouseAndKeyboard(event_manager, game_model)
    graphics = view.GraphicalView(event_manager, game_model)
    _run_with_crash_handling(game_model, args.log_path)


if __name__ == '__main__':
    run()

Obviously, this one is a bit more complicated than the previous script, but it gives an excellent overview of how things are imported and used. To keep things simple, I used ellipses to hide some of the unimportant details. The important bits are the imports—specifically, the parts the import the model, view, and controller, and use them in our run function. Notice how we just have to import the `magegame` package to get what we want.

To prove that this works, I’ll point you to one of the game releases. Under each release, you should see an MSI file that you can use to install the game on Windows. Note that the setup file (shown in the next section) is barely different than the default one I mentioned above. In other words, you can build out all the source code you want without much risk involved in building the game. That type of convenience makes me very happy.

Complex Directory Structure; Simple Code

If you’ve looked at the Mage GameOpens in a new tab. directory structure, you’ll find that the folder structure is kind of intense. As mentioned previously, I have the top-level package, `magegame`, which is filled with the MVC code. The model and view are then in their own folders as there are many pieces to track and maintain. Meanwhile, the controller code can fit in a single file. No need to bother with nasty folder structures in that case.

Oddly enough, despite how messy the source code gets, the testing and packaging code can stay largely unchanged. In fact, here’s my setup.py file:

import cx_Freeze

executables = [cx_Freeze.Executable("main.py")]

cx_Freeze.setup(
    name="Mage Game",
    version="0.4.0",
    options={
        "build_exe": {
            "packages":["pygame"],
            "include_files":["assets/"]
        }
    },
    executables = executables,
)

It doesn’t really get simpler than that.

Bonus Tips

While I think working with Python packaging can be a huge pain, there are things you can do to make your life easier:

  • Use a version control system like Git to work through variations on directory structure
  • Use a continuous integration tool like GitHub Actions to ensure tests continue to pass as you mess with the structure (and to deploy to PyPI as needed)
  • Use the releases feature of GitHub to deploy working executables on a regular basis
  • Use a requirements.txt file in conjunction with virtualenv to keep a consistent environment from machine to machine
  • Use a GitHub template (like this oneOpens in a new tab.) to start a repo with the structure described in this article

And as mentioned already, I wouldn’t consider myself an expert on this topic. However, I do feel a lot better about it every since I was able to get the Mage Game code packaging smoothly. I’ve never felt this good about shipping Python code, so I couldn’t help but share these tips. Feel free to make use of some of those other references throughout as you develop your preferences. Otherwise, that’s all I got!

In the meantime, why don’t you check out one of these related articles:

If you really enjoyed this work, feel free to head over to my list of ways to grow the site to show your support. Otherwise, take care!

Jeremy Grifski

Jeremy grew up in a small town where he enjoyed playing soccer and video games, practicing taekwondo, and trading Pokémon cards. Once out of the nest, he pursued a Bachelors in Computer Engineering with a minor in Game Design. After college, he spent about two years writing software for a major engineering company. Then, he earned a master's in Computer Science and Engineering. Today, he pursues a PhD in Engineering Education in order to ultimately land a teaching gig. In his spare time, Jeremy enjoys spending time with his wife, playing Overwatch and Phantasy Star Online 2, practicing trombone, watching Penguins hockey, and traveling the world.

Recent Posts