Pete Hodgson

Software Delivery Consultant

Hello, production

October 4, 2019

Deploying something useless into production, as soon as you can, is the right way to start a new project. It pulls unknown risk forward, opens up parallel streams of work, and establishes good habits.

I spend a fair amount of time helping engineering teams with the initial build and release of a product. There’s one piece of advice that I find myself giving consistently during the very early days: get the simplest version you can think of into prod, as soon as you can. Focus solely on what’s needed to get a “Hello, world” version of your new system up and running in a real production environment. No more, no less. I call this a Hello, Production release.

If your "Hello, production" release has any sort of useful functionality, you're doing too much. If your system is not responding to live traffic in production, you're not doing enough.

When I say “the simplest version”, I really mean it. The “Hello, production” release for a new web app shouldn’t do more than expose a single endpoint in production, which responds with a 200 status code and some static text ("Hello, world", perhaps). Whatever is the bare minimum to validate that you can get something deployed and responding to traffic, in your production environment. If your “Hello, production” release has any sort of useful functionality, you’re doing too much. This release must be a legit deploy into a production environment, though. If your system is not responding to live traffic in production, you’re not doing enough.

What’s the point

You may wonder what’s valuable about delivering something to production with the explicit goal of providing no useful functionality. It’s valuable in three ways - it acts as a Tracer Bullet, it provides a Walking Skeleton, and it gets the Continuous Delivery pipeline flowing.

The Tracer Bullet

To my knowledge, the term Tracer Bullet was first coined in the seminal book “The Pragmatic Programmer”1. The idea is that if you’re exploring a new approach or a new environment, you should create just enough functionality to confirm that you’re “hitting the target” - that the technical approach is appropriate for your context - and to establish a repeatable feedback loop which will ensure that you continue to hit your target. The tracer code that you create is not a throw-away prototype, however. It serves as the basis for future work.

The “Hello, production” release I described above is a Tracer Bullet. It validates the suitability of a system’s initial infrastructure requirements, and validates the release processes needed to get code running with that infrastructure. It provides the team with feedback on what’s required to get any new infrastructure stood up in production, along with the mechanisms to deploy software onto that infrastructure and expose that software to live production traffic. Keeping the scope of the initial release intentionally small (no useful functionality, remember) ensures that our first tracer bullet gets out as soon as possible, getting us this valuable feedback as soon as possible.

Considering production infrastructure and release processes as soon as possible “brings the pain forward”2, giving early warning of any roadblocks that the team must overcome in order to get their product deployed and running live, in production. These types of production roadblocks are quite common. If you’re in an enterprise environment maybe you need to engage with a separate team that manages production infrastructure, or provide some information to a security or compliance person. If you’re at a startup, maybe you need to persuade the CTO to pay from some services with the company credit card, or get approval from an architecture group for the new web framework you’ll be using. It’s a lot more pleasant to work through these processes calmly at the start of development, rather than rushing to get everything squared away in front of a fast-approaching launch deadline. We’ll discuss this more a little later on.

The Walking Skeleton

I got the idea of a Hello, Production release from the wonderful book Growing Object-Oriented Software, Guided by Tests (or GooS), which introduces this concept as an example of a Walking Skeleton3. While the GooS book focuses on the same Tracer Bullet benefits described above, I think the Walking Skeleton metaphor also helps illustrate an additional set of benefits.

A Hello, Production release constitutes an extremely skinny, but functional, initial version of our system - hence “Walking Skeleton”. Once the skeleton of a system is in place, we have various places where we can start fleshing out functionality - putting meat on the bones. One engineer can start working on the front-end for the system, while another engineer can start working on data storage, and a third engineer can perhaps start on integration with a 3rd-party API. With the walking skeleton in place, these three streams of work can move forward in parallel, but only once it’s in place. By focusing on getting an extremely simple version of the system working end-to-end, we’re able to start working in parallel sooner. The team can spread out across a wider codebase, independently building and releasing their respective features and avoiding stepping on each other’s toes.

If you’re starting on a new product, your team might only constitute one or two engineers, and you may think that enabling parallel streams of work doesn’t provide much benefit. However, having multiple streams of work in flight is beneficial, even for a small team. It’s not unusual for one stream of work to get temporarily blocked, particularly in the early stages of building a product. Perhaps we can’t make progress on the third-party integration because that third party hasn’t quite finished their API yet, or we can’t work more on the front-end because we’re waiting for our designer to create wireframes for some of the screens. Having a walking skeleton in place gives us the option to flip over to an alternate stream of work, while we wait for our initial stream of work to become unblocked.

Get the CD pipes flowing

A third benefit of getting an initial version of the system released into production ASAP is that it begins to establish Continuous Delivery practices early.

In addition to driving an initial pass at the new system’s infrastructure, a Hello, Production release should also establish a basic delivery pipeline for that system - the mechanisms for delivering changes to that system into production with safety. Initially, this delivery pipeline will be quite crude - there won’t be a sophisticated sequence of automated testing stages, and perhaps some of the deployment steps are still manual (although hopefully done by following a checklist, at least).

Getting started with a primitive delivery pipeline is OK - it’s a Walking Skeleton just like your Hello, World codebase is. Having the beginnings of a pipeline in place opens the door for the team to continuously improve that pipeline over time - automating manual processes, adding additional quality gates, improving the visibility of the delivery process, optimizing slow stages in the pipeline, and so on. It’s only once you start using a delivery pipeline regularly that you start to see the opportunities for improvement and the rough edges that need smoothing. With the Walking Skeleton of a pipeline in place, the team can start incrementally making these improvements. Any delay to your first release is a delay in getting started with this continuous improvement work.

Besides acting as a Walking Skeleton, even a basic delivery pipeline will, in and of itself, start providing some of the value of Continuous Delivery. It provides fast feedback on changes to your codebase and helps to establish good habits around incrementally building small chunks of production-ready functionality.

Starting with a CD pipeline that goes all the way to production nudges a team into the habit of frequent production releases. In my experience, the go-live launch for systems built out this way tends to go extremely smoothly, to the point of being anti-climactic. The “launch” of such a system often just involves some small configuration changes which enable public access to the system in production.

How minimal is minimal?

It’s hard to get comfortable with just how minimalist a Hello, Production release should be when you’re new to the concept - cutting the scope of something all the way down to “useless” feels strange. In my experience, engineers often need to take a few passes at slicing the scope of the initial system down before they get to a truly minimal release. Even when you’ve got to what you think is a minimal version of your system, ask yourself, “could I do less and still have something that can respond to traffic in production”.

Conversely, it can be easy to gloss over the part about being live in production. A large part of the value in this practice is in bushwacking that initial path to production. You’re not done until your (minimalist) system is live, responding to traffic in production.

Sounds good, but…

As I mentioned earlier, I find myself advising teams to focus on initial Hello, Production release quite frequently. In doing so, I hear some common counter-arguments. Let’s address them.

“Our Product Manager wants to see useful features.”

Emphasizing the frequent release of useful, working software is great. And an initial release which by design has no useful features might seem counter to that. But creating an initial Hello, Production release doesn’t mean that we’re delaying the release of useful features. We’re merely opting to make an extra release even sooner than that first release of useful software.

Some teams might choose to skip showcasing their first Hello, Production release, with the subsequent release of their first useful features constituting the first “real” release, as far as external stakeholders are concerned.

“It’d be more efficient to hold off and do a bigger initial release.”

The idea that releasing in bigger batches is more “efficient” comes up fairly often. This idea has been refuted quite effectively by advocates of Lean processes, both in the general context of manufacturing and in the specific context of software product development. Additionally, the research outlined in Accelerate4 shows that organizations with a faster production release tempo also tend to exhibit higher organizational performance (be more profitable and productive).

In short, making small, frequent releases to production is demonstrably a better approach. An initial Hello, Production release helps to gets a team into that tempo from the start.

“It’s going to be super hard to get something actually running live in production.”

The effort required to get something stood up in prod is absolutely a legitimate issue, particularly when working within a larger enterprise organization. There can be many hurdles to overcome in getting a new system into production. And that’s all the more reason to get started with down that path as soon as you can! Having a Hello, Production release that’s sitting, ready to go, can be very helpful in moving forward through the various processes and conversations involved in getting into prod. I’ve found that there’s often a marked difference in the response to “What will we need to do once I’m ready for my first production release?” vs. “We have a production release ready to go, now. What’s left to do before we can get it live?”

In a larger organization, it may well take a while to get that first release live in production. Moving that work forward will likely need to be an ongoing effort that’s done in parallel as your team continues to build out product features. However, it’s essential to keep pushing on getting something serving live traffic in production. You won’t fully understand what’s needed to get features into your users’ hands until you’ve completed that “last mile”. The harder it is to get something live, the more benefit you get from forging that path early.

“We can’t go live until it’s finished.”

Many organizations aren’t used to the idea of deploying incomplete software into production environments. People may well question why you’d want to go live before your team is close to finishing the first version of your product. However, I’ve found that - with a little explaining - most folks are open to the concept of going into production early, even if it’s not the norm.

You’ll probably need to come up with a way to prevent public access to your live, unfinished product. At smaller, scrappier organizations, I’ve achieved this by placing all web pages behind a hardcoded username and password (usually implemented with some basic auth middleware). At larger organizations, we’ve done things like not creating production DNS entries until we’re ready to go live, or not adding the production servers to a public subnet. Note that with these approaches, internal users must still be able to access the live production system (even if it’s only exposed on the corporate network), so that we can demo the system, do showcases, perform UAT, and so on.

As discussed earlier, the “go-live” launch of products which begin with a Hello, Production release tend to be non-events, from an operational perspective. Often the only Operations task needed is the creation of a DNS entry, or a simple networking config change to expose the live system to the public internet. This is in stark contrast to the high drama that is often forced upon Operations folks when the first production deployments have been deferred until close to product launch. I’ve seen quite a few Ops people really embrace the Hello, Production concept after seeing that it leads to an enjoyably boring go-live launch!

Start as you mean to go on

In a product’s lifecycle, the pre-launch phase constitutes a short-lived, atypical state of affairs. The overwhelming majority of any successful product’s feature development is post-launch, with new features only being considered complete once they are deployed into production. A team that gets into the groove of regular, post-launch practices early is setting itself up for sustainable success.

Getting a minimalist version of a new system into production as soon as possible yields many benefits. It provides early feedback on your architecture and your delivery pipeline. It also allows your team to “spread out” and work on different aspects of the system in parallel. Most importantly, it front-loads the risk associated with the often unknown work required to get something live in your production environment.

Any team starting on a product or feature which requires significant new pieces of infrastructure should start with a Hello, Production release.

  1. I highly recommend reading the entire Tracer Bullets section in “The Pragmatic Programmer”, which does a wonderful job conveying the concept and the benefits in more detail. In fact, I highly recommend reading the entire book, if you haven’t already. It’s a classic, and a new edition was recently released in celebration of the book’s 20th anniversary. [return]
  2. per the Continuous Delivery heuristic “If It Hurts, Do It More Frequently, and Bring the Pain Forward”. [return]
  3. The GooS authors, in turn, credit the general Walking Skeleton concept to Alistair Cockburn, from his book “Crystal Clear: A Human-Powered Methodology for Small Teams.” [return]
  4. Forsgren, N., Humble, J., Kim, G. 2018. Accelerate: The Science of Lean Software and DevOps: Building and Scaling High-performing Technology Organizations. Portland, Oregon: IT Revolution Press. [return]