How to Python – Page 7 – The Renegade Coder

Once again, welcome to the How to Python series. In this collection, we explore programming problems that have quick solutions in Python. In this edition, we explore a few ways to check if a file exists in Python, so let's dive in! Problem Introduction Recently, I was looking for a way to persist some user settings of an app to a file. For instance, I wanted to save options like font size, background color, etc. Naturally, I settled on a simple mapping format like YAML, but I could have just as easily used a CSV which we already know how to parse. Unfortunately, the configuration file wouldn't be backwards compatible without some support in the software. In other words, if the configuration file didn't exist, I would need to assume some original preset values. Otherwise, I would pull the data from the configuration file. Fortunately, I did my research and came up with a solution. The plan was to check to see if the configuration file existed. If it did, the program would read from it and populate the necessary fields. Otherwise, the program would lean back on some arbitrary preset values. To do that though, I had to find a way to verify the existence of a file. As it turns out, there are plenty of ways to do that in Python. Solutions If we're looking to check if a file exists, there are a few solutions: Check if a file exists with a try/except block (Python 2+)Check if a file exists using os.path (Python 2+)Check if a file exists using the Path object (Python 3.4+) Of course, it's up to us to determine which solution is the best for us! Check if a File Exists with a Try Block Up first on the list is a simple try-except block. In this scenario, we would attempt to open our file in the try block. If the file fails to open, we run the preset values. For example: try: with open('/path/to/file', 'r') as fh: # Load configuration file values except FileNotFoundError: # Keep preset values In this example, we try to open the file using a with statement. If successful, we can load our configuration file values. Otherwise, the open command will throw an error that we catch in the except block. At that point, we keep our preset values and move on. To be honest, this solution is perhaps the simplest and most robust, but the FileNotFoundError is an update from Python 3. You'll have more trouble with catching an IOError in Python 2. Also, it's worth noting that we use a with statement in this example. As an alternative, we can open the file directly. We just have to remember to close it when we're done: Check if a File Exists with OS Path Another option is to skip error handling altogether and directly verify that the path exists. For example: import os exists = os.path.isfile('/path/to/file') if exists: # Load configuration file values else: # Keep presets Of course, the drawback here is the race condition from line 2 to line 4. If for some reason the configuration file gets deleted between line 2 and line 4, then the script will crash. If that's not a risk in your application, then this solution is great. That said, I'll emphasize this issue again because I did get a bit of pushback on it in the past: Checking if a file exists and opening that file are two different operations in this example. They would need to be one atomic operation to avoid a race condition. Check if a File Exists with a Path Object If you're obsessed with object-oriented programming like me, then maybe this solution is for you. As of Python 3.4, we can wrap our file reference in an object which brings along a host of new functionality. For example: from pathlib import Path config = Path('/path/to/file') if config.is_file(): # Load configuration file values else: # Keep presets In addition, this new object representation allows us to use our original try-except block: try: absolute_path = config.resolve() # Load configuration file values except FileNotFoundError: # Keep presets Of course, we may not need all this functionality. After all, if reading the contents is the goal, then the first option is probably the best. Performance While we've already shared all the solutions, it may be important for our application to look at performance as well. To do that, we'll leverage the timeit library. First, however, we need to generate a few strings: setup = """ import os from pathlib import Path """ try_except = """ try: with open('/path/to/file', 'r') as fh: pass except FileNotFoundError: pass """ os_isfile = """ exists = os.path.isfile('/path/to/file') """ path_lib = """ config = Path('/path/to/file') if config.is_file(): pass """ With the strings ready to go, we'll run this test twice: once where the file exists and again when it doesn't, respectively. >>> timeit.timeit(stmt=try_except, setup=setup) 25.758140300000036 >>> timeit.timeit(stmt=os_isfile, setup=setup) 23.302945200000067 >>> timeit.timeit(stmt=path_lib, setup=setup) 36.851380800000015 Normally, we would use the repeat function to try to calculate some sort of lower bound for each function, but it was just way too slow. Feel free to try it and share the results. For the existing file tests, we'll have to change the paths in each of the strings above, so they include an existing file. As a result, some of these solutions are significantly slower: timeit.timeit(stmt=try_except, setup=setup) 220.5547474 >>> timeit.timeit(stmt=os_isfile, setup=setup) 194.13558469999975 >>> timeit.timeit(stmt=path_lib, setup=setup) 208.86859360000017 Here, we can see all of the solutions are quite a bit slower when dealing with an existing file. That said, it seems the os solution is the fastest in both circumstances. Of course, it does have the race condition drawback, so be sure to take that into account when choosing one of these methods. For reference, all tests were completed using Windows 10 and Python 3.7.3. If you want to learn more about Python performance testing, check out my article titled How to Performance Test Python Code. A Little Recap Using the methods above, we have several options to check if a file exists in Python: # Brute force with a try-except block try: with open('/path/to/file', 'r') as fh: pass except FileNotFoundError: pass # Leverage the OS package import os exists = os.path.isfile('/path/to/file') # Wrap the path in an object for enhanced functionality from pathlib import Path config = Path('/path/to/file') if config.is_file(): pass For the purposes of this tutorial, we were only interested in files. However, these solutions can be adapted to verify the existences of directories and symbolic links, so don't be afraid to play around. That's the beauty of Python! At any rate, thanks for taking the time to check out this article. If it's your first time here and you found this article helpful, why not subscribe to The Renegade Coder? Alternatively, you can always hop on the mailing list and decide to become a member at a later time. In the meantime, you may benefit from my list of Python Code Snippets for Everyday Problems. If you're not convinced, check out some the following related posts: How to Parse a Spreadsheet in PythonRock Paper Scissors Using Modular ArithmeticMake Featured Images Just Like The Renegade Coder Finally, feel free to browse the following related books: Learn Python 3 The Hard Way by Zed A. ShawPython for Kids: A Playful Introduction to Programming by Jason R. Briggs See you next time!