Unity Troubles
Hey guys,
today we have some not very great, but nonetheless important news about our project.
Over the last few months, we’ve been hard at work to create first a demo we can showcase to you guys, and then an Early Access release that may still be far from finished, but nonetheless enjoyable for you to play. The problem was as we worked, more and more technical issues kept showing up. Every time we added a new feature, three already existing ones broke. We fixed those bugs, two more popped up. Our QA team kept reporting strange crashes and issues we could not reproduce, that seemed to occur at random and defied all attempts at fixing.
To make things worse, performance was far behind where we wanted it to be. The game chugged along, even with map sizes far below our target. Despite us putting considerable time into optimization. On profiling, CPU cycles just seemed to mysteriously vanish, memory usage was through the roof, even GPU load was higher than expected.
At first, we thought it’s just the nature of software development. Every piece of code written will have some bugs here and there, we’ll just work on fixing them. Except there came more bugs, and more, and the fixes would mysteriously stop working. Then we thought, maybe the worm is in our production pipeline. Maybe if we get more stringent in our project planning and documentation, we can avoid situations where changes to one system break another.
As release deadlines came creeping up, we started racing to solve this growing mountain of issues. Work days started getting longer. Working weekends and all-nighters were pulled to get those critical last-minute fixes in. The dev team started to fatigue as none of our solutions worked. Until last Saturday when we finally found the true source of the issue.
Meet Unity
If you keep up with gaming news, you’ve probably heard about Unity 3D. They’re a ready-made game engine many indie studios use to take off the load of writing your own engine from scratch. It features a wide series of features and is well-established, with a market share of over 70%. Even large companies like Nintendo use it to facilitate development.
They recently made some headlines when they enacted a highly controversial pricing change that threatened to bankrupt many developers. While I don’t want to get into the details here, suffice to say the news left us very concerned. Even though Ascent of Ashes was not among the games threatened with immense usage fees, who’s to say that two years down the line they won’t enact another pricing change that would affect us? The trust was broken, and we started looking into how much use we were actually getting out of Unity, and what our options were in case we were forced to switch.
DOTS
The main reason Ascent of Ashes uses Unity is for their new package, the Data-Oriented Technology Stack (DOTS for short). On paper, DOTS is pretty revolutionary technology. It uses a non-standard programming paradigm to more efficiently arrange data within a computer’s memory, and comes with multithreading support out of the box. For those unfamiliar, most modern processors come with four or more cores, all of which can run one “thread” of calculations for a program. Most games only use one or two threads, limiting the amount of CPU power they can utilize. By using multithreading, Ascent of Ashes can essentially leverage four to eight times more CPU power, enabling far greater performance than other games of the genre.
The downside is that this non-standard paradigm is rather difficult to work with. Most programmers are unfamiliar with it, and there are no established solutions. Every feature has to be developed essentially from scratch, as we figure out how to realize common functionality in an uncommon framework. When we reviewed work logs we discovered that as much as half of total programmer time was lost to dealing with the idiosyncrasies of DOTS.
This was supposed to be a tradeoff for superior performance, with tests showing DOTS could outperform traditional programming techniques by orders of magnitude. If you look up prototypes showing its capabilities, you’ll find improvements as high as 100x or even 1000x faster calculations. And while it is true that certain things in DOTS perform much faster, what we discovered is that the overall package was losing so much performance everywhere else that it actually ended up slower than traditional solutions.
Squash one little bug…
While simple inefficiency explained some of the issues, it did not account for many of the bugs and crashes we were facing. This is where a recent package update released by Unity comes in. As we read through the changelog, we found many familiar error messages. Problems we’ve struggled for weeks to diagnose. Crashes we just couldn’t identify despite trying every approach imaginable.
As it turned out, the 1.0 release of DOTS was incredibly unstable, with many deep-seated issues and major memory leaks. For the past months we’ve been putting all our efforts into finding the problems in our code, only to discover that they were never in our code to begin with. All these arcane errors were the result of poor QA on the part of Unity.
Anyone with programming experience knows that game engines, APIs and other libraries need to conform to a high standard of quality control. This is especially the case for a closed source engine like Unity. If an error occurs, it is virtually impossible for a game developer to identify it as a problem within the engine itself. We scrambled to implement this new DOTS update in time for the Next Fest demo, but the fact remains that they let this many issues slip by them already. Who knows what issues this new version has that we don’t even know about?
A Solution in Sight
The good news is, now that we know the source of the problem, we can start to fix it for good. This means saying goodbye to Unity and DOTS for good. Instead, we will write a new engine based on Godot and the well-established principles of Object-Oriented Programming. This means implementing our own multithreading solution and allocating additional time to optimization, but in the end it will still be faster than continuing work with DOTS.
Porting a game to a new engine is a major undertaking. Some parts of our current code base can be migrated to Godot without much trouble. The problem is the underlying code that was written specifically for DOTS. It does not translate well to other paradigms and will essentially require a complete redesign.
That’s why we decided to do this port after the Early Access release. Our first milestone already requires redesign of several code sections to accommodate the new features. It makes sense to combine it with the porting process. Factor in no longer losing half our dev time to DOTS shenanigans and the first major update post-release should only take a month or two longer than it would have otherwise. Long term we will see the benefits of increased stability and much faster development cycles as we no longer have to waste our time trying to solve the unsolvable.
For Early Access, we remain committed to our release date of November 15th. The time until then will be used to implement features on the basis of traditional programming practices, interfacing with DOTS only where necessary. This will speed up development and allow us to more easily port those new features over when the time comes.
One significant change resulting from this decision is mod support. Because any mods built on the current DOTS structure would break when we make the move to a new engine, we have decided to postpone full mod support until after the move. The dev time we had earmarked for it will be used to add more base game features to the Early Access release instead.
As to what those features will be, and what our detailed roadmap looks like, we will make an announcement on that soon so stay tuned for more news!
Closing Thoughts
Nobody likes bad news, and I’d be lying if I said I wasn’t pretty angry over losing weeks if not months of dev time to a shoddy API. At the same time, the fact we finally found the source of our troubles is great news in itself. It means we can finally stop the bleeding and start moving forward. A solution is in sight, and that gives us much reason to be optimistic. We hope that despite this bump in the road, you guys remain as excited to see Ascent of Ashes become a reality as we are.
Until next time, stay safe and keep surviving.