Software Development – Page 11

Python is a language I recommend to anyone who wants to learn to code, but there's one aspect of the language that has frustrated me for some time: the package system. Dealing with directory structures, build files, and dependencies are always a source of difficulty for any programming language, but they seem more pronounced with Python. As a result, I thought I'd share my approach to dealing with Python's module and package system. Hope you enjoy it! History As many folks in the community know, I started using Python in 2017. Despite having been formally taught Java, I would consider myself fully self-taught as far as Python is concerned. As a result, there are a few fundamentals that I'm still uncomfortable with to this day. One of those fundamentals is Python's package and module system which has given me headaches for the last 5 years. As you may know, I maintain a variety of Python packages that you can install casually using pip: SnakeMD Subete Image Titler Because of the work I put in to make those packages easy to use, you might not know how much trouble I had along the way. That said, it wasn't until very recently that I was able to put together a complex package with multiple levels of nesting. As a result, I figured I'd spend some time today talking about the things I've learned about the package system and how you can have ease of mind as well. The Problem The project that officially got me to understand the module system in Python was a little 2D game that I'm still fleshing out called Mage Game. It seems that a lot of folks who use PyGame to make games in Python merge all of there data and game code in one place. I didn't want that, so I opted for a design that follows MVC (i.e., model-view-controller). Because MVC design focuses on the separation of code structures, you end up with several files. As anyone who has tried to make a Python package knows, challenges arise when you split your package into multiple files (i.e., multiple modules). For instance, it's common for circular import errors to occur: Why do circular imports seemingly work further up in the call stack but then raise an ImportError further down? What happens when using mutual or circular (cyclic) imports in Python? For me, I'm a big fan of type hints. As a result, any time a function in one module references a type from another module, I like to include that type hint. This tends to causes various problems with the import system. To make matters worse, I like to use folders to organize my code into hierarchies. For example, in an MVC design, I like to have my model code in a model folder, the controller code in the controller folder, and the view code in the view folder. As you can imagine, importing those local modules becomes a nightmare. After enough messing around, you can probably get your imports to work. However, once you've decided to actually run your code, a new problem will arise: the imports don't work at runtime. This is singlehandedly the biggest nightmare fuel in Python for me. How is it that my IDE can recognize the imports correctly but Python itself can't? And don't get me started on trying to test your code. Needless to say, here were some of the Stack Overflow threads I relied on for a time: Relative imports in Python 3 How to import python class file from same directory? Python error "ImportError: No module named" How to import the class within the same directory or sub directory? Why pytest always says " ImportError: attempted relative import with no known parent package" And as I kept reading, I kept seeing more and more cursed solutions—many of which involving the patching of the system path through Python. So today, I want to share with you some strategies for getting your complex Python projects working without any hacks. Let's do it! How to Setup a Basic Python Package Note: there are resources that are often referenced for directory structures and whatnot for Python packages that I don't particularly like. One of those references is from Jean-Paul Calderone. You can check it out here, if you'd like. It's quite old at this point (written in 2007), but you may find it useful. Just know that I don't adhere to basically any of the points made in it. If you want a reference that I find a lot better, check out the work of Ionel Cristian Mărieș here (written in 2014). I follow their recommendations more closely. Also, I should note that any mention of "source-code" below refers to the package name directly. A lot of folks advocate for packages to be nested in an "src" folder—like Hynek Schlawack (written in 2015)—and I think that's generally a good idea. I don't do that here because I find the "src" folder setup to be slightly more annoying. That said, the choice is up to you! To start, let's talk about the big picture. How should you lay out your package directory? To me, there are a few main items that we care about: Source code Test code Scripts (i.e., main.py) Packaging (i.e., setup.py) Projects might also include assets or other data files, but we'll focus on these four main areas. When I setup a project, I like to have the four key areas represented at the root of my project directory. As a result, I might have a folder structure that mirrors the list above (i.e., a folder for source code, a folder for test code, a main script, and a setup script). If you prefer a folder style diagram, here's what I would expect to see: project-directory |-- source-code |-- test-code |-- main.py |-- setup.py Basic setup.py Now, there are two main routes you can take with a project like this: make a pip package or make an executable. Because I'm basing this guide off of game design, I'm going to show you how to make an executable. That said, a traditional setup file could work as well. Regardless, your setup file should look something like this: import cx_Freeze executables = [cx_Freeze.Executable("main.py")] cx_Freeze.setup( name="Package Name", version="0.1.0", options={ "build_exe": { "packages":[], "include_files":[] } }, executables = executables, ) Right now, this file would probably work as-is to create an executable that did nothing. That said, having a file setup like this is absolutely critical to having a good time. To use this script to make an executable, use the following command: python setup.py bdist_msi It should create an MSI file for Windows that you can use to install your script. Basic main.py As far as Python scripts go, you don't need a lot of code to get running. I might make a script that does nothing but follows good conventions: if __name__ == "__main__": pass Eventually, you'll want to fill this file with the proper imports from your source code to actually do something, but we'll get to that later. Basic Tests Folder While we don't currently have any code, we'll want to follow good design principles by putting some testing in place. I like to use pytest for its simplicity, so my test folder might look like this: project-directory |-- test-code/ |-- __init__.py |-- test_example.py I always make sure to put the __init__.py file in the testing folder, so Python knows to treat it as a package. You may have read that the __init__.py file can cause problems with pytest but that's only if you include the test folder inside of your source code folder. Because we've separated the two out, we shouldn't run into that problem. That said, the pytest docs show that separating the test files out into their own directory without a __init__.py file should also be fine. Basic Source Code Folder As mentioned above, there is a lot of disagreement about the appropriate design of the source code folder. Should this folder include tests? Should this folder be included in an "src" folder? According to the setuptools docs, the design I'm sharing with you is known as the "flat layout". In other words, we are not nesting our source code in an "src" folder; the package will be in our root directory. I prefer this layout because I tend to include scripts at the root structure of the repo. For example, in game development, I can have the main script sitting in the project root with direct access to the local package. Otherwise, I'd have to do a little bit of work to get the script to find the appropriate packages. Similar arguments can be made for testing, but there are also counterarguments mentioned in the resource above as well. Every choice has a tradeoff. At any rate, here's what the inside of a source code folder might look like: project-directory |-- source-code |-- __init__.py |-- example.py Again, the __init__.py file indicates a package. In this case, our package is called "source-code", but you would call it whatever your project was called (i.e., mage_game). Handling Imports In a Basic Project Setup Assuming you have a project setup like the one above, you can start to create some files. For example, here's what our example.py module might look like: import random def attack(damage: int, crit_rate: float) -> int: if random.random() < crit_rate: return damage * 2 else: return damage In the spirit of gaming, I wrote a silly attack function that returns the damage we do on an attack while factoring in the crit rate. Now, this isn't easily testable, but we could imagine that we want to make sure this function returns one of two possible values: damage or damage * 2. We can do that easily in our test_example.py file: from source-code.example import * def test_attack(): damage = attack(5, .5) assert damage in (5, 10) If all is well, you can run pytest in your root directory and get back a passing test. The key here is that we can import the function we want to test basically directly from the package name. This comes with a caveat of course. If source-code is installed on your system, the tests will run against that—not your code. Because we're creating an executable and not a PyPI package, there should be no issue in this case. Making a Mess Now, in my experience, there is never just one file like this in a package. If there was, we wouldn't need to worry about folder structure at all. Just have all four files sitting in the root of your repository. To make matters worse, no one really talks about how to scale a project. So, let's now imagine that our example.py file has grown out of control, and we want to split it into multiple files. A great example of this is my model package in Mage Game: model |-- __init__.py |-- bindings.py |-- character.py |-- magic.py |-- state.py |-- world.py In this module, we have several files that all reference each other in some capacity. To make sure this works, we use relative imports (as opposed to absolute imports). For example, here's the top of the character.py file: from __future__ import annotations from dataclasses import dataclass, field from .magic import * from .world import Entity, WorldPoint The bottom two imports are the interesting ones. We import everything from the magic.py module and a couple of things from the world.py module. To ensure these are imported from the files in the same folder, we place a period in front of the module names. Otherwise, we might run into a nasty ImportError. Now, let's say that our model package used to be a single module (like example.py). There's a good chance that someone was depending on that module. No worries, we can pretend it still exists with a simple trick. Edit the __init__.py to import all of the submodules: from .bindings import * from .character import * from .magic import * from .state import * from .world import * To the outside world, model.py still exists. We've just converted it into a folder (i.e., package). Obviously, there are risks with this type of design—namely clashes of symbols. That said, if the code is appropriated separated, I think it's unlikely that you'll end up with functions of objects with the same name. What's really cool is you can then test the model code as if it were just a single file, not a complicated folder structure. Here's an example that tests some of the magic: from magegame.model import * def test_palette_item_default(): default = PaletteItem() assert default.can_use() == True, "Default palette items should be usable from the start" assert default.get_spell() == Projectile(), "Default palette items should store default projectiles" def test_palette_item_reset_cooldown(): default = PaletteItem() default.reset_cooldown() assert default.get_cooldown() == SpellAttribute.COOLDOWN.base_value, "Default palette items should store the default cooldown after reset" Notice how the import just pulls in everything from model. You could make them more specific if you'd like, but this gets the job done. Finally, we can actually make up our main script. Here's the one I currently use in my game: import argparse import logging import os import sys from logging.handlers import RotatingFileHandler from magegame import controller, eventmanager, model, view logger = logging.getLogger(__name__) def _init_logger(args: argparse.Namespace) -> None: ... def _run_with_crash_handling(game_model: model.GameEngine, log_path: str): ... def _process_arguments() -> argparse.Namespace: ... def run(): """ The main function that launches the game. """ args = _process_arguments() _init_logger(args) event_manager = eventmanager.EventManager() game_model = model.GameEngine(event_manager) keyboard = controller.MouseAndKeyboard(event_manager, game_model) graphics = view.GraphicalView(event_manager, game_model) _run_with_crash_handling(game_model, args.log_path) if __name__ == '__main__': run() Obviously, this one is a bit more complicated than the previous script, but it gives an excellent overview of how things are imported and used. To keep things simple, I used ellipses to hide some of the unimportant details. The important bits are the imports—specifically, the parts the import the model, view, and controller, and use them in our run function. Notice how we just have to import the magegame package to get what we want. To prove that this works, I'll point you to one of the game releases. Under each release, you should see an MSI file that you can use to install the game on Windows. Note that the setup file (shown in the next section) is barely different than the default one I mentioned above. In other words, you can build out all the source code you want without much risk involved in building the game. That type of convenience makes me very happy. Complex Directory Structure; Simple Code If you've looked at the Mage Game directory structure, you'll find that the folder structure is kind of intense. As mentioned previously, I have the top-level package, magegame, which is filled with the MVC code. The model and view are then in their own folders as there are many pieces to track and maintain. Meanwhile, the controller code can fit in a single file. No need to bother with nasty folder structures in that case. Oddly enough, despite how messy the source code gets, the testing and packaging code can stay largely unchanged. In fact, here's my setup.py file: import cx_Freeze executables = [cx_Freeze.Executable("main.py")] cx_Freeze.setup( name="Mage Game", version="0.4.0", options={ "build_exe": { "packages":["pygame"], "include_files":["assets/"] } }, executables = executables, ) It doesn't really get simpler than that. Bonus Tips While I think working with Python packaging can be a huge pain, there are things you can do to make your life easier: Use a version control system like Git to work through variations on directory structure Use a continuous integration tool like GitHub Actions to ensure tests continue to pass as you mess with the structure (and to deploy to PyPI as needed) Use the releases feature of GitHub to deploy working executables on a regular basis Use a requirements.txt file in conjunction with virtualenv to keep a consistent environment from machine to machine Use a GitHub template (like this one) to start a repo with the structure described in this article And as mentioned already, I wouldn't consider myself an expert on this topic. However, I do feel a lot better about it every since I was able to get the Mage Game code packaging smoothly. I've never felt this good about shipping Python code, so I couldn't help but share these tips. Feel free to make use of some of those other references throughout as you develop your preferences. Otherwise, that's all I got! In the meantime, why don't you check out one of these related articles: How to Get Better at Programming: Lessons from Competitive Shooter Games So, How Bad Is Artifact Farming in Genshin Impact Really? Phantasy Star Online: A Beautiful Mess If you really enjoyed this work, feel free to head over to my list of ways to grow the site to show your support. Otherwise, take care!