Planning for business agility

During the summer, I reflected on the main differences between software engineering in the 1990s, when I became a professor, and today. The best way to describe what I feel is the key difference is perhaps through an analogy. In the 90s and early 2000s, software engineering was akin to designing and building a jet engine first. Once you felt you had a good engine, you’d hook it up to a plane and let the plane fly with it. These days, you design and build the “minimal viable engine,” get the plane flying with it and then start tinkering with it through continuous upgrades (DevOps), experimentation (A/B testing) and automated experimentation by the system itself (online learning, reinforcement learning, etcetera).

By and large, I think this development is really good. I’ve often described the traditional way of system development, where we’d spend a year or more building to a specification, as similar to pointing your car as accurately as possible towards a light at the horizon, closing your eyes and putting the pedal to the metal. Anyone who takes a step back to reflect on software and system development realizes that any system, including an R&D system, needs a feedback loop and constant adjustment. The whole notion that we can build a system based on specs that are a year or more old and expect that the results are exactly what the customer wanted is one of the fundamental fallacies in engineering. The rule of thumb in software engineering is that requirements change at a rate of 1 percent per month.

The risk associated with sprints, continuous deployment (DevOps), DataOps, MLOps and the like is that we never lift our eyes beyond the immediate next sprint. Teams become reticent to take on any task that takes longer. Product management focuses increasingly on the small improvements that can be accomplished in one sprint. The entire company increasingly falls into a reactionary pattern where customer requests that fit in the sprint mindset are all that get prioritized and the organization loses control over its own destiny.