Once again, we’re back with yet another Sample Programs update! If you were following along during Hacktoberfest 2020, then you might know that we hit one of our latest milestones: 25 project tests. Don’t worry if you don’t know what that means! We’ll talk all about it.
Table of Contents
What Are Project Tests?
For the uninitiated, project tests are basically Python unit tests written with the assumption that every project will handle input and output on the command line. Currently, this is all managed through Glotter, a testing framework using Docker.
While future iterations of Glotter may eliminate the dependency on Python altogether, we’re stuck writing our own tests for the time being. A typical test file looks as follows:
from runner import ProjectType
from glotter import project_test, project_fixture
@project_fixture(ProjectType.HelloWorld.key)
def hello_world(request):
request.param.build()
yield request.param
request.param.cleanup()
@project_test(ProjectType.HelloWorld.key)
def test_hello_world(hello_world):
actual = hello_world.run()
assert actual.strip() == 'Hello, World!'
Then, when the project is being built, the test will be executed for all languages that include a Glotter configuration file. For example, Python’s Glotter config file looks like this:
folder:
extension: ".py"
naming: "underscore"
container:
image: "python"
tag: "3.7-alpine"
cmd: "python {{ source.name }}{{ source.extension }}"
In the Sample Programs repo, we actually launch Glotter during our Travis builds which look like this:
$ python runner.py test ============================= test session starts ============================== platform linux -- Python 3.7.1, pytest-5.2.4, py-1.8.0, pluggy-0.12.0 -- /home/travis/virtualenv/python3.7.1/bin/python cachedir: .pytest_cache rootdir: /home/travis/build/TheRenegadeCoder/sample-programs collecting ... collected 1853 items test/projects/test_baklava.py::test_baklava[baklava.go] PASSED [ 0%] test/projects/test_baklava.py::test_baklava[Baklava.kt] PASSED [ 0%] test/projects/test_baklava.py::test_baklava[Baklava.fs] PASSED [ 0%] test/projects/test_baklava.py::test_baklava[baklava.f95] PASSED [ 0%] test/projects/test_baklava.py::test_baklava[baklava.swift] PASSED [ 0%] test/projects/test_baklava.py::test_baklava[baklava.ex] PASSED [ 0%] ... test/projects/sorting/test_sleep_sort.py::test_sleep_sort_valid[sleep-sort.dart-sample input: already sorted] PASSED [ 99%] test/projects/sorting/test_sleep_sort.py::test_sleep_sort_valid[sleep-sort.dart-sample input: reverse sorted] PASSED [ 99%] test/projects/sorting/test_sleep_sort.py::test_sleep_sort_invalid[sleep-sort.dart-no input] PASSED [ 99%] test/projects/sorting/test_sleep_sort.py::test_sleep_sort_invalid[sleep-sort.dart-empty input] PASSED [ 99%] test/projects/sorting/test_sleep_sort.py::test_sleep_sort_invalid[sleep-sort.dart-invalid input: not a list] PASSED [ 99%] test/projects/sorting/test_sleep_sort.py::test_sleep_sort_invalid[sleep-sort.dart-invalid input: wrong format] PASSED [100%] ======================= 1853 passed in 832.41s (0:13:52) ======================= The command "python runner.py test" exited with 0.
Naturally, putting together tests for 25 different projects is quite the feat, so we wanted to celebrate it with a little milestone.
Project Testing Challenges
When it comes to writing tests for each project, there are few challenges.
First, before we can begin to write tests, we must write documentation. During the documentation process, our authors are expected to also write up a testing table which is ultimately converted into testing. This process is long and iterative as we need to be very careful that testing is both complete and correct—it’s not easy to make changes once testing is official.
Second, we must write the tests. For the most part, testing is a matter of defining inputs and outputs, but creating the files is a bit of a process. In particular, there are three files that need to be updated:
- .glotter.yml
- test_project.py
- runner.py
Assuming everything is done correctly, the next challenge is making sure all existing scripts adhere to testing. Ideally, we would implement testing before we let folks submit code, but we’ve largely been adding tests retroactively. As a result, a lot of code snippets usually need to be updated. This is a long and challenging process; it’s why it’s taken over a year just to cover 25 projects.
Finally, the biggest challenge has been enforcing our requirements. Generally, folks like adding new projects to their favorite languages, and the more popular the language, the more often this happens. However, we don’t want to have folks creating unmanageable technical debt for us, so we ask that they both document and test their code. Unfortunately, this turns a lot of folks away, so it’s been tough.
That said, I’m just happy to have hit our milestone of 25 project tests. Speaking of that, let’s talk about it!
25 Project Tests
As the title of this article suggests, we finally hit our project testing milestone during Hacktoberfest 2020. Here’s the official list of projects that are now tested:
- Baklava
- BinarySearch
- BubbleSort
- Capitalize
- ConvexHull
- EvenOdd
- Factorial
- Fibonacci
- FileIO
- FizzBuzz
- HelloWorld
- InsertionSort
- JobSequencing
- LCS
- LinearSearch
- MergeSort
- MST
- Prime
- QuickSort
- Quine
- ROT13
- ReverseString
- RomanNumeral
- SelectionSort
- SleepSort
For a complete and current list of projects, check out the official projects page
.
Next Stop: 50 Project Tests
As you know, the grind never stops! Now that we’ve covered 25 of our projects, what’s stopping us from hitting 50? These days I’m a bit too busy to add the tests myself, but I’m totally open to support from the community. Anyone want to help out?
Conveniently, there are more ways to help out than submitting code. You can always throw the repo a star. Or even better, check out my list of ways to grow the site. These days you can find my Discord, Patreon, and Newsletter there.
Otherwise, you can read all about other Sample Programs updates below:
- The Sample Programs Website Automatically Generates Featured Images
- The Sample Programs Repo Squashes Commits
As always, thanks for stopping by! I appreciate it.
Recent Posts
Recently, I was thinking about the old Pavlov's dog story and how we hardly treat our students any different. While reflecting on this idea, I decided to write the whole thing up for others to read....
In the world of programming languages, expressions are an interesting concept that folks tend to implicitly understand but might not be able to define. As a result, I figured I'd take a crack at...