Goodbye manual test cases, hello model-based testing
“We have to stop creating test cases by hand. Especially in the high-tech industry, systems have become so complex that it takes thousands and thousands of cases just to get decent coverage. To write, execute and maintain these requires a huge number of test engineers – good luck finding those in an already extremely tight labor market – or the available engineers would need to do a lot more work. Either way doesn’t scale,” says Julien Schmaltz, software engineering expert and principal consultant at CGI. “Our only solution is automation, through model-based testing.”
Likewise, Bryan Bakker observes a lot of attention in industry for elaborate unit testing at the bottom and extensive customer-facing end-to-end testing at the top, creating an hourglass-shaped testing approach. “But those end-to-end test cases are very expensive to make, execute and analyze,” asserts the test architect at Sioux Technologies. “They’re also very unstable in that they easily break down, for example at the slightest change in timing. Model-based testing can really help narrow things down at the intermediate integration level. Transforming the hourglass into a pyramid, MBT can greatly reduce the number of test cases required at the top and thereby the costs of end-to-end testing.”
“Model-based testing supports shift-left, and finding problems earlier in the development process saves money,” acknowledges Johan Poelsma, software architect at ICT Group. “But MBT also lets you get the best out of your domain experts. When problems are found earlier, those experts can focus on where they can add real value, like checking the usability or other non-functionals of a system.”
Gains
Bakker and Poelsma are currently stationed at Thermo Fisher Scientific in Eindhoven, where they’re guiding the introduction of MBT into the software R&D process. Until recently, when still working for ICT, Schmaltz was involved in this project as well. “Initially, there was quite some resistance to adding yet another tool to their toolbox,” Poelsma recalls. “For lack of comparable cases in high tech, we didn’t have much to show for, but we managed to convince them to give MBT a try. Now, slowly but surely, we’re seeing it pay off.”
After careful consideration of multiple solutions, Thermo Fisher Scientific selected the MBT platform of the Amsterdam-based software quality experts Axini. “Model-based testing comes in a variety of flavors,” elaborates Schmaltz. “The principal idea is that you have some kind of abstract representation, a model, of your system environment, from which test cases are automatically derived to check whether your system behaves as expected. This is offered in one way or another by a multitude of platforms, including Tosca, Smartesting, MBTsuite and Axini, to name a few. Some use UML activity diagrams as models, others use user data, Axini uses interface specifications. For systems with many parallel components, lots of interfaces and complex behavior, like the electron microscopes of Thermo Fisher Scientific, Axini is the go-to MBT platform.”
At his current employer, CGI, Schmaltz is using his 15+ years of MBT experience to support Prorail, the organization responsible for the Dutch railway infrastructure. “They’ve already been working with Axini for many years. Their systems also have a lot of interfaces and complex behavior. In one of their projects, developing a key component of ERTMS, the new European Rail Traffic Management System, applying MBT has already saved them 5,000 hours of testing. This clearly shows the added value in terms of software development efficiency.”
Bakker thinks model-based testing has similar gains to offer high-tech companies like Thermo Fisher Scientific. “Primarily in the development of critical software components, which are aplenty in high tech,” he stipulates, while stressing that MBT isn’t a silver bullet that can and should be applied everywhere. “When the impact of a possible bug is low, we can perfectly make do with less elaborate means to limit the risk. And for legacy code, there’s usually already a test set with adequate coverage. Or people don’t see the point of investing in modeling a system that isn’t causing any major problems in the field. It works, so why do anything differently? However, when it’s time for a grand redesign of one or more components, MBT can also be of great value there. You can model the interfaces to the rest of your old system, swap in the revised code and automatically check whether everything is still working properly.”
“At Thermo Fisher Scientific, we see MBT really excel at catching integration issues,” Poelsma adds. “Functionally testing components in isolation isn’t rocket science. It’s a craft mastered by many an engineer and supported by many a tool. It’s when components are integrated that the nasty problems arise. That’s where MBT with a platform like Axini comes to the rescue, by exhaustively testing all the dependency interfaces.”
Awareness
While Prorail, but also insurance provider Achmea and Pension Fund Campina, already see its considerable promise, MBT has yet to gain a foothold with high-tech companies. That’s partly because of all the legacy code keeping them in the status quo, but it’s also a simple lack of awareness. Many companies familiar with model-based development think model-based testing is more of the same. “Why should we use Axini? We’re already working with Verum,” is something Bakker hears a lot. “It’s our job to educate them.” More success stories would certainly help raise awareness. “At Thermo Fisher Scientific, we’re well underway to quantify the benefits,” Poelsma points out, “but we need more pioneers with such audacity and commitment.”
Without hard proof of the benefits, MBT platforms are quickly dismissed as insufficiently mature. All too often, the bar is set too high. “Not all complexity can be handled yet,” acknowledges Bakker. “However, as the latest research advances are incorporated, new possibilities open up, significantly expanding the applicability. We’ve seen the same happen with model-based development tools like Verum’s ASD, now Dezyne. It took them a while, too, to be perceived as good enough for industrial use.”
In fact, Bakker claims, the successes in other markets show that MBT platforms like Axini’s have already reached a sufficient level of maturity. “They just don’t always perfectly align with the expectations of this new user group from high tech. For example, to accommodate its existing customers in other markets, Axini had neatly concealed the integration with the Git version control system. Users always got to see the latest and greatest as all code revisions were automatically incorporated everywhere. In high tech, however, we generally want a more controlled rollout. At the request of Thermo Fisher Scientific, that has since been changed.”
High-tech companies also highly value a seamless embedding into their continuous integration and delivery (CI/CD) pipelines. “CI/CD is a hot topic, with companies looking to test everything all the time,” continues Bakker. “When doing MBT, however, you sometimes just want to run a quick model check without going through the whole pipeline. MBT suppliers are working hard, also in research collaborations with universities, to ensure a smooth integration into the existing development environment.”
Such agility is crucial for the adoption of model-based testing, says Schmaltz. “It’s good to see a company like Axini having evolved from inward facing, building a platform on its own, to outward looking, teaming up with partners such as CGI, ICT and Sioux to pick up on market needs and promptly address them. Market needs like the integration with version control, CI/CD and other customer processes and ways of working. Together, we’re furthering the maturity of the platform and the awareness of MBT.”
Cookbook
Even with all the technology advances, the modeling part of MBT remains largely human work, which poses an additional challenge. “Making models is not for everyone,” contends Bakker. “We already saw that at Sioux with our ASD and Dezyne customers and we’re experiencing it again with MBT at Thermo Fisher Scientific. A test engineer isn’t necessarily proficient in automation, let alone in modeling. You need to be able to think analytically and abstract away from the details. People can learn to model, but only if there’s already a foundation there.”
Schmaltz sees room for improvement in the craft of model-based testing itself. “We need to create better frameworks to model more systematically and thus lower the threshold. MBT consultants can help only so far. At a certain point, engineers are on their own. That’s when things like best practices and design patterns can really make their lives easier. We need to write a cookbook containing pre-defined solutions so that people don’t have to reinvent the wheel. For that, we need to set up more communities to bring all the knowledge together.”
Communities like the MBT working group at Testnet, the Dutch association of test engineers. There, Bakker and Schmaltz are sharing their model-based testing experiences with peers from other companies. As such, it provides an excellent setting to bring the envisioned cookbook a step closer to reality. “A logical next step would be to also reflect on possible practices and patterns as input for a common knowledge base, irrespective of the selected MBT flavor and platform,” argues Bakker.
With the help of a cookbook for model-based testing, Schmaltz believes, anyone with a higher education should be able to master the craft. At the same time, he calls on higher education to take up the gauntlet and increase the MBT proficiency of its students. “The Open University does a bit of model-based testing. And I know Axini is working with the University of Amsterdam. Overall, however, universities could do a lot more, and industry should support them with the practical handles.”
Poelsma concurs. “Such patterns are indispensable for the further adoption of MBT. The good thing is that, within the context of a single customer, you wouldn’t need a whole lot of them. As they tie into the specific software architecture of that customer, a small set of patterns often suffices to get the MBT ball rolling. We’re seeing that at Thermo Fisher Scientific as well. For one of the software components, it took Axini a couple of iterations to arrive at the right solution for the dependency interfaces, but now we’ve got something that easily scales to other components. These patterns have given us a flying start to modeling and realizing the company’s ambition to do more for less.”
Change
In the end, the adoption of MBT is a matter of change management. “It’s all about getting it into the process of the customer,” Schmaltz notes. “That requires experts like us, who know the technology and can apply it in a broad range of environments. But we also need a lot of change agents, who know the business of the customer and who can quickly ascertain how they can make MBT work there.”
“It’s about getting commitment across the organization,” adds Bakker. “Our modus operandi is both bottom-up and top-down. It starts with running small pilots and building it up from there, carefully selecting where we can expand next. At the same time, we’re taking a broad perspective: if we seek to apply MBT across the board, we’re going to need specialists at all the crucial locations in the software development environment. Approaching it from both sides and bringing them together is key.”
“As with any introduction of a new testing technology, it’s very hard to tell beforehand what it’s going to bring. That only becomes clear in hindsight,” indicates Poelsma. “This is where the success stories come in, like the one we’re building at Thermo Fisher Scientific. With proof of the benefits, we can really convince others.”
Those others will come – Schmaltz is sure of it. “Model-based testing has too much potential to fail.”
This article was written in close collaboration with Axini.
