Meredith Johnson
Many of you recently noticed that PooShooter: Toilet Invaders is newly playable on Proton Experimental! While obviously having a fix for a game with such a compelling storyline is worthy of a blog post in and of itself, we wanted to share with you why getting this game running was especially interesting.
Our Proton quality assurance (QA) team spends a lot of time trying to reproduce reported issues with running various games on the Steam Deck and on Linux desktops. One such recent issue was PooShooter: Toilet Invaders launching to a black screen with Proton. The report mentioned that the game worked on earlier Proton versions, so we busted out our standard regression test cases. That’s when things started to get weird.
When we hear of possible regressions, we start by testing various Proton versions until we figure out which Proton version is the first one to have the bad behavior. Most of the time, this is pretty straightforward for our QA team. Once we have this information, we start working with our developers to narrow down which code change caused the problem so that a solution can be found.
But sometimes, it’s not that easy: either we don’t see the bug on any Proton version or we see it on all Proton versions we test. This is obviously a confusing state to be in when we have a report that contradicts our test results! We need to dig deep and find the confounding factor. One thing we look into is controlling for the Proton version that the game is first run on. In a perfect world, this wouldn’t make a difference: if your first run of a game is on an older Proton version and you subsequently upgrade to a newer Proton version, we would want and expect the same behavior as if your first run was on the newer Proton version. But we don’t always get what we want, and so we have to test first launch behavior too if we have an extra sneaky regression. Fully uninstalling and reinstalling games is thrilling work, but since it’s inefficient, we have our tricks to simulate a first run without deleting game data. Instead of a full reinstall, we just delete the Proton-specific files created on the first run. Our QA team has been doing this manually for a long time, and it’s second nature for us to find and delete these files. For people that don’t do this for a living, it’s a pretty obnoxious process. To make this easier for the rare occasion someone needs to do this, there’s an option in the Steam deck called “Delete Proton Files” that should do the same thing as manually deleting the relevant files.
But in this case, it didn’t! We found that the game still didn’t work on any Proton version if we manually deleted all of the Proton files before launching. However, we found that the game would magically start working after using “Delete Proton Files.” Now we had two mysteries to solve: why the game was working after deleting files one way (but not the other), and why the game was launching to black screen otherwise.
The black screen issue ended up being a pretty average rendering bug, and our awesome developers were able to come up with a hack to get PooShooter: Toilet Invaders and a few other games working. (I swear, they make everything look easy).
But why was the game working only after using “Delete Proton Files”? Our Pooshooter testing revealed a sneaky bug where some files weren’t actually getting deleted that should have been. More specifically, file names containing `:` were not getting deleted. After the deletion process went wrong, not all files were being properly created the next time the game was run. In the specific case of Pooshooter: Toilet Invaders, this created a perfect environment where the game defaulted to a different rendering backend which allowed the game to run (albeit with worse performance). The game is web-browser based, using Chromium Embedded Framework. Chromium can render using a bunch of different APIs, so the game had a backup available if the files got into a weird state. Our developers were also able to fix this up, and file deletion is now working the way it should.
We suspect that this bug had been causing mayhem for awhile, causing various subtle and hard to reproduce issues. PooShooter: Toilet Invaders finally gave us the opportunity to identify and fix this tricky issue. We were happy to flush this bug away for good. 🚽
About Meredith Johnson
Meredith has been with CodeWeavers since 2019, and currently wears both the QA Manager and CrossOver Product Manager hats. After getting her PhD in linguistics, she somehow stumbled into the Free and Open Source Software world. When she isn't testing CrossOver, she's probably perfecting yet another bean stew, jogging slowly, buying even more linen tunics, doing a face mask or doting on her perfect chunky baby.
