Josh DuBois
Here at CodeWeavers, customers often ask questions about how CrossOver is made. We get inquiries ranging from fairly basic to extremely technical. Involved and curious users are among the joys of basing our product on an open-source project.
With CrossOver 16.2 coming out, I thought it would be a good time to talk about how we manage development and release branches for CrossOver. On the spectrum of things one might say about Wine, the topic is only a moderately technical one. It is a common question from our users, though, on our web forums and in our live chat support channel.
For starters: CrossOver’s ability to run Windows applications comes from Wine. Wine is an open-source software project with a twenty-plus year history. The interesting part of maintaining our internal source code branches is how to integrate with Wine. To understand our release cycles, you have to understand Wine’s development cycle.
Wine’s Development Cycle
Wine’s master git repository is at
git://source.winehq.org/git/wine.git
People submit new patches to Wine every day, and every day Alexandre Julliard, our fearless leader, commits accepted patches to Wine’s master repository. Here at CodeWeavers, our basic workflow is that our own Wine developers send their work to Wine via the regular public
submission process (and hope it gets accepted!). So by default, the code we write goes into the open-source repository first, before it makes it into any of our own git trees.
Wine has a numbered “development” release every two weeks, on alternate Fridays. These releases get a tag and are built and used widely within the open-source community. There is no magic to the development releases – no special testing is done nor milestones met for these releases. They are just conveniently numbered snapshots of Wine development at regularly spaced intervals in time.
At CodeWeavers we call our master git branch the ‘crossover’ branch. Immediately after every wine development release, Alexandre merges wine’s master into our internal crossover branch. This is how we get new work from upstream integrated into the codebase we use to build CrossOver. This means that for our ‘default’ workflow, the bi-weekly Wine merge is the first time a given piece of Wine development work – even work from our own developers – enters our internal source repository.1
Development releases are exciting because they include all the latest changes, but that “excitement” means they can be unstable. The way to manage that is, not surprisingly, a release branch.
Release Branches
We release a major version of CrossOver once per year, sometime in the fall. About two months prior to our release date, we make a new branch off of our ‘crossover’ master branch. This branch will be dedicated to our upcoming release. Our release branches start their lives basically the same as any development release of Wine would – just a snapshot of Wine development taken at a particular time.
We make our release branches more stable by thorough testing. We test and test and test, and fix bugs and test some more. We test in-house for a couple of weeks and then start a beta, where over two-thousand dedicated beta testers hammer on CrossOver and report problems. At the end of our beta cycles, code in our release branch has become much more predictable and reliable.
Usually we do not do further merges of Wine during the course of a major release of CrossOver. That is to say, typically, a given version of CrossOver will always have the version of Wine which it had when we first created the release branch for that version. Of course we fix bugs during the lifetime of a version, but we keep those fixes targeted so that we can be confident of their safety.
Some releases are exceptional, and we occasionally do a full Wine merge in between major releases. This happens rarely. CrossOver 16.2 is one such occasion! That makes it a bit special. When we do a wine merge, we tend to also do another round of beta testing (albeit a bit shorter than the first).
Hacks
Wine has very high standards for what code is allowed into the master repository. As a commercial product, CrossOver’s needs often differ for pragmatic reasons. Usually that means that a developer sees a problem and can tell that the elegant solution will be months of work or highly destabilizing. Then I beg for a solution that works in practice, even if it is ugly or only solves the problem in limited circumstances. Wine will not generally accept such solutions, but they improve things for our customers. We maintain these in our own source repository, as what we call “hacks.”
Hacks are a problem because they cause our source to differ from what is in Wine’s master repository. That makes Alexandre’s life difficult, because it is hard to merge Wine’s master source code into ours when these differences exist. The differences generated by our “hacks” cause conflicts during the merges, and Alexandre must work around them every time a new merge is done. That is tedious, error-prone, and unpleasant.
For this reason, it used to be that we preferred to commit our hacks to release branches. We would start off CrossOver 16 by branching from our master ‘crossover’ branch, then right away we would pull in all of the old hacks from CrossOver 15. That way, we tried to avoid hacks in our master ‘crossover’ branch into which Alexandre does merges every two weeks. Avoiding hacks on that branch made his life easier (and avoided a potential source of error).
There were problems with this approach, however. Hacks tended to get lost from one release branch to another. So we might fix a bug with a hack in CrossOver 15, but if we forgot to apply the hack to our new CrossOver 16 branch, the bug would return. Also, it meant that we were unable to test certain Windows programs with our master ‘crossover’ branch, because some of them needed hacks to function (and these hacks were only available in release branches). These things could go untested for long periods of time, and we might only realize they were broken during a beta cycle for a new release. For these reasons, we started keeping almost all of our hacks in the crossover master branch. That makes it harder to do the bi-weekly merges, but avoids other problems.
Even still, there are exceptions. Some “hacks,” or diffs from upstream wine, are in fact enormous bodies of work, which run into the hundreds of individual patches and touch many, many files. Currently, the “Command Stream” feature of wined3d is the main example of such a large “hack.” It is not realistic to keep that kind of thing in our master ‘crossover’ branch, because merges then would really be too difficult to do every two weeks. So that kind of thing still lives only in the release branches, and is ported from one release branch to the next at the start of every cycle. It’s a hardship, and with the Command Stream in particular, we’re trying to get it upstream so we don’t have to manage it that way any longer. Fortunately, we tend not to have too many such enormous “hacks” at any given time. When we do, it can make things which ought to be simple become prohibitively complicated or expensive. So we try to keep our “hacks” small in scope, or get them upstream.
Summary
So basically:
- CrossOver has one version of Wine for every major release, with rare but significant exceptions. We test these versions rigorously to make them stable.
- Our general workflow is that developers submit patches to upstream Wine before they commit to our internal repositories, so most of our work is available publicly before we are using it ourselves. (Of course if we are in a hurry, we find a way to get it where it needs to be more quickly.)
- Users with privileges to access our nightly builds will see a wine merge every two weeks, because our nightly builds generally come from our master ‘crossover’ branch. These builds will not include the wined3d Command Stream.
If you are curious about Wine, you can visit www.winehq.org, or checkout the source at
git://source.winehq.org/git/wine.git
… and build it for yourself. Patches are welcome!
1. Of course we sometimes need things faster, and we don’t let our ‘default’ workflow hold us back. When we are in a hurry for a particular piece of work, we can always cherry-pick it or apply it directly. ↑
About Josh DuBois
Josh DuBois is an engineer and product manager for CrossOver.
