Jan Bosch is a research center director, professor, consultant and angel investor in start-ups. You can contact him at jan@janbosch.com.
During the summer, I reflected on the main differences between software engineering in the 1990s, when I became a professor, and today. The best way to describe what I feel is the key difference is perhaps through an analogy. In the 90s and early 2000s, software engineering was akin to designing and building a jet engine first. Once you felt you had a good engine, you’d hook it up to a plane and let the plane fly with it. These days, you design and build the “minimal viable engine,” get the plane flying with it and then start tinkering with it through continuous upgrades (DevOps), experimentation (A/B testing) and automated experimentation by the system itself (online learning, reinforcement learning, etcetera).
By and large, I think this development is really good. I’ve often described the traditional way of system development, where we’d spend a year or more building to a specification, as similar to pointing your car as accurately as possible towards a light at the horizon, closing your eyes and putting the pedal to the metal. Anyone who takes a step back to reflect on software and system development realizes that any system, including an R&D system, needs a feedback loop and constant adjustment. The whole notion that we can build a system based on specs that are a year or more old and expect that the results are exactly what the customer wanted is one of the fundamental fallacies in engineering. The rule of thumb in software engineering is that requirements change at a rate of 1 percent per month.
The risk associated with sprints, continuous deployment (DevOps), DataOps, MLOps and the like is that we never lift our eyes beyond the immediate next sprint. Teams become reticent to take on any task that takes longer. Product management focuses increasingly on the small improvements that can be accomplished in one sprint. The entire company increasingly falls into a reactionary pattern where customer requests that fit in the sprint mindset are all that get prioritized and the organization loses control over its own destiny.
The consequence of this is, of course, that the company accumulates a large amount of debt. This debt includes technical debt, eg architectural and infrastructural, but also strategic debt, as there’s no North Star to guide the organization by, as well as process debt, as process improvements are typically not prioritized in these organizations. The adage of “failing to plan is planning to fail” once again proves true.
Most companies that I work with recognize this challenge in some form and seek to address it. Quite some, unfortunately, reach the wrong conclusion: they let go of Agile practices in R&D, product management and so on, and fall back in the old, waterfall style of operating. This is a fundamental mistake that might easily kill the company: Agile practices in the organization are intended to support business agility. In a fast-changing world, where the future is hard to predict, business agility is one of the most important characteristics a company can have.
The better way forward, in my experience, is to separate planning and execution: we want to plan long-term and execute short-term. This means that we plan system functionality, architecture refactorings, infrastructure changes, process improvements, innovation efforts and so on for a longer period (quarters and years), but we execute those plans in sprints and allow for constant adaptation based on changes in the environment, the company and the market.
In several cases, I’ve noticed that, especially traditional, managers gather quite a bit of comfort from creating plans and then executing them come hell or high water. It provides a sense of being in control of what might otherwise feel like a chaotic situation. What the Stoics teach us, though, is that most of the things in our lives that we think we can control are completely outside of it. Nobody could have predicted the Covid-19 situation, nobody knows when the next economic recession or even depression will hit, nobody knows when the next technological breakthrough is putting our company at risk of disruption. We don’t even know whether the stock market goes up or down tomorrow! Instead, we need to respond to the things we can’t control by focusing our energy on the things we can.
Concluding, the world has moved from planned and offline to real-time and online. The risk is that we lose track of the long term in favor of an overly strong sprint focus. Rather than giving up on Agile practices, we need to complement agile execution with systematic, long(er)-term planning to garner the best of both worlds. It allows us to control the destiny of our company while responding to the societal and industry changes that we don’t control. Agile with purpose!