The cover image for this post is by Eden Constantino
This blog post was written by Jamie.
Introduction
A large amount has been written about the failed launch of Cyberpunk 2077. But for those who don’t follow video game news, Cyberpunk 2077 is a video game from Polish video game studio CD Projekt Red. It was released on December 10th, 2020 in a mostly broken state and the launch was unanimously seen by video game journalists and players alike as disastrous.
The game was first announced in 2012, and sat in development for over eight years. Even with eight years to work on the title
most AAA titles have a development time of three to five years
the game was released in a broken state, with some consumers not even able to run the game on the launch day.
Why Did This Happen?
Because CD Projekt Red have been, understandably, quiet about exactly what happened, we will never really know what actually lead to the game being released in a broken state. But one of my favourite pieces of guess work on this is a video from Dave Farley:
Essentially, by not adopting Agile principles - specifically those relating to continuous delivery and always having a shippable build of the game - CD Projekt Red were doomed to fail, says Farley.
Having the ability to automatically deploy the latest working build of software is an essential requirement for any modern software development team. Having releases be automated means that you don’t have to even schedule downtime in order to release. You can release your application as fast as your build and deployment servers can process the code.
A note on nomenclature:
“release” when specifically referring to video games does not mean “getting the game into the hands of the player”. This is because there is usually a lot of work required in order to publish a game to the many different online stores - and is even more so if you are producing a physical product (i.e. shipping the game to physical stores).
“Release” here means to have a working build in the hands of QA or UAT engineers.
Not An Isolated Incident
Interestingly, Cyberpunk 2077 wasn’t the only game to have ever fallen afoul of a disastrous release, and not all of them are related to the quality of the software. For instance, Tiger Woods ‘99 PGA Tour was recalled because it had - essentially - pirated content on 100,000 copies of the disc. The 2005 entry in the series of golf games was also recalled, but this time for quality control reasons.
But one of the biggest stories relating to not actually shipping a game on time is the one around Duke Nukem Forever.
For those who don’t know: Duke Nukem Forever (DNF) is the (current) final game in the Duke Nukem series. The series started as a side scrolling shooter back in 1991; with the most famous entry in the series being 1996s Duke Nukem 3D, which changed the format of the game series to first-person shooter.
Duke Nukem Forever was released in 2011, after spending 14 years in development hell.
Back in May
May 10th, 2022 to be exact
Scott Miller (founder of Apogee, original developers of the Duke Nukem games) wrote a piece titled “The Truth About DNF” in which he states:
IMO, while I was not part of the DNF project, as a company owner I had some good insight into the issues with the game’s development.
- We were always understaffed by at least 50%.
- We did not have a good development roadmap (at least, I never once saw one), and instead the project was adlibbed too much.
- Because of the game’s slow development, when new 3D technology became available, the project–in effect–rebooted to make use of the newest tech causing massive delays over and over.
I’ve added the emphasis on the second bulleted item, as I believe that this is one of the most important points in this story.
Also, I’ve used an archive.org link for Scott’s post, because the text for the post has changed quite a few times since it was originally written. We also only have Scott’s word on this, as neither 3D Realms or Apogee have ever been officially public with the exact reasons for the failures of DNF.
The point that I added emphasis to reads:
We did not have a good development roadmap (at least, I never once saw one), and instead the project was adlibbed too much.
Anyone who has worked on a project, be it a software development project or anything else, which didn’t have a clear roadmap or clearly defined end goals will know that the project is doomed to failure. Also the second point here (“project was adlibbed too much”) is a clear example of feature creep - something which will almost definitely prevent a project from completing on time.
If they had used an Agile and Continuous Delivery-like approach to developing this game
the fact that development began four years before the Agile Manifesto was written, notwithstanding
it might have been releasable in 1998 (it’s original release dates) or even in 2001 (when they released the first trailer at E3), rather than having to restart development several times between 1998 and 2010. In 2010, Gearbox Software stepped in and completed development of the game, releasing it in 2011. This was after they had purchased the Intellectual Property (IP) from 3D Realms/Apogee
not before 3D Realms sold a "Perpetual licence" to Zoom Platform
How Could Agile & Continuous Delivery Have Helped?
Practitioners of Continuous Delivery and Agile or Scrum develop software in a way which reacts to the almost constant changing of complex requirements in an agile manner. This means that when requirements change, they can pivot, alter the code, and release a working application in very little time. When combined with Continuous Delivery
and it’s extreme counter part: releasing multiple times per day
this can mean that the development team can react to a change in requirements, and build and deploy a new version of the software before their customers have realised that there is a change to the software that they are using
Netflix are one of the companies who have perfected this system
Again, in the realm of video games: releasing to a customer will mean releasing to QA or UAT engineers, rather than releasing directly to the consumers of the game. See my earlier point about there being many barriers to publishing a game to the many online stores.
By having a build of the software which could be released at any moment, you stand a much better chance of actually shipping working software to whomever you are shipping it to
assuming that the build you have works, of course
This would have helped with both the “unclear roadmap” and feature creep points raised by Scott Miller, above.
It’s also why we practise Agile and Continuous Delivery at RJJ Software. Our software projects have a main branch which represents the single source of truth for the project and the version of the code which has been deployed to users of the software. We branch off from the main branch and create feature branches (on per new feature to be developed), and continually integrate against our feature branches, raising a pull request (PR) when we are ready for the feature to be included in the main branch. As soon as the PR is completed, we tag the main branch with a release number and release the new software to our various staging environments. Once a feature branch has been merged back into main, it is deleted.
By tagging the main branch commits with a build number, we can easily revert the deployed software if a bug or issue is discovered until a fix is implemented. Fixes are worked on by branching off of the tagged commit to main with a hotfix branch, which is then PRd back into main when ready to release - using the same process as feature branches.
By continually integrating our changes into source control, we know that our feature branches will not deviate from the main branch too much - which would make them harder to merge back in. And by continually deploying the application (from the main branch), we know that our customers/clients/users will always have the latest version of the application available.
This is a process based on Gitflow. It is far from the only branching system, and definitely not the only branching system which works with Agile
technically, you don’t need a branching system to work in an Agile or Scrum way
but it works with our processes.
In Closing
A combination of both Agile or Scrum and Continuous Delivery - as in “[Agile ∨ Scrum] ∧ Continuous Delivery” in mathematical logic - could have helped in the development of a large number of video games; especially those with either disastrous releases or that were stuck in development hell for years.
By all accounts, the video game industry is starting to come round to the way that the rest of us develop software, and the big studios are starting to adopt Agile and Scrum methodologies. One source (from February 2020) shows that Criterion
a studio owned by Electronic Arts
have adopted a version of Agile for their development teams:
Our approach builds upon Agile (Scrum) and Scaled Agile Framework (SAFe). However, off-the-shelf, these do not facilitate the breadth of creative and scientific collaboration needed to make a game. Equally we believe a full implementation of SAFe would suffocate a game team. So, we have adapted the roles and essential rituals of SAFe to work within the context of a triple-A game
So the tide is turning, but slowly.