“Without statistics, you’re unlikely to find your best performance and realize consistency”

When Wendy Luiten was taught to program, she used punch cards that were fed into giant computers. Nowadays, she can do the most complex calculations and design simulations at the touch of a button on her laptop. During her career, she witnessed an unprecedented increase in processing power.

Graduating from the University of Twente in 1984, Luiten started a distinguished career as a thermal expert and Six Sigma Master Black Belt at Philips. Presently, she works as a consultant. Her extensive experience has given her a better view of statistics and computing than most engineers.

New software offers great opportunities for creating design simulations and digital twins. However, the apparent ease of these new methods makes it easy to forget that a simulation isn’t reality. A simulation model needs to be validated to ensure that it represents reality to a sufficient degree. Moreover, simulations describe an ideal world, without random variation. In the real world, random variation can make a product unreliable and disappointing to end customers.

“Some people don’t repeat but go on a single measurement,” Luiten says. “Based on that, they decide whether the design is good or not. That’s risky. You don’t know how good the measurement is, you have no idea about the measurement error, you don’t know how representative the prototype is, you don’t know how representative the use case is.”

People tend to be very optimistic about their measurement error. “I’ve seen cases where people thought their temperature measurement error was in the tenths of a degree, but a repeat measurement showed the difference to be 10 degrees. In the thermal world, that’s a huge difference. If your repeat measurement shows such a large deviation, you really can’t be sure how well the product performs and you need to dig deeper for the root cause of this difference.”

This is why Luiten is launching her new “Applied statistics for R&D” training at High Tech Institute, focusing on measurement statistics. In this course, she delves into key statistical skills that have proven their worth in her 30+ years of industry R&D experience. “First, you need to see how good your measurements are,” she explains. “Next, you need to be able to estimate the sample size – the number of measurements you need to demonstrate a particular effect with sufficient probability. Once you can measure output performance accurately and with sufficient precision, you can explore different design configurations and choose the best one. Finally, you investigate the output mean and variation, which is ultimately how you achieve a successful design.”

Repeat measurements

Luiten notes that engineers often are already aware of the systematic measurement error. “Systematic errors are a well-known field,” she says. “You can measure a golden sample and correct the results, which is a de facto calibration of your measurement. That’s a well-known procedure and part of many lab courses in higher education.”

Most people, however, don’t consider the random error or blindly use a standard value of 1, 5 or 10 percent. In reality, the random error depends on the measuring instrument and also on who’s performing the measurements. The statistical method to find out the random error is not usually part of a lab course, so this is less common. But the first time such a test is done, the results are often surprising.”

Luiten mentions some cases she’s dealt with herself. “I’ve seen cases where people were confident that they had almost no random error because they had very expensive automated measuring equipment. However, it turned out that the operators acquired the measurement sample differently, which caused a large random error. In another case, different development labs all claimed a 5 percent measurement error – but when they measured the same devices in a round-robin test, there was a factor 2 difference because of differences in the measurement setup that were thought to be irrelevant. I’ve seen that apparent fluctuations in product quality can be linked to the operator doing the measurements. In all cases, people were convinced that they had a negligible random error in the measurements, but the results were totally unexpected. You only know your random error and its cause if you do the statistical test for it.”

The random measurement error is especially important when it comes to so-called statistical power – the probability of measuring a certain effect if it’s present. If the effect you want to measure is about the same size as your measurement error and you do your measurement again, the probability of proving that effect is below 10 percent. So, if a design change gives you a 5-degree lower temperature, and your random measurement error is 5 degrees Celsius, you’ll see that in 1 out of 10 measurements. On average 9 out of 10 times, the results are inconclusive, even if you do the measurement twice. If you want to improve the power, you either need to lower the measurement error or do more repeats.

Sometimes people see repeat measurements as a waste of effort, but Luiten disagrees. “The true waste is running underpowered experiments, going through all the effort of setting up and executing an experiment and then finding out that the result is inconclusive.”

Navigating the solution space

In addition to measurement errors and sample size estimation, testing different designs is a key element of Luiten’s training course. You can do that in hardware, but that might not be the most effective option. “Nowadays, you can do a lot by virtual testing,” Luiten points out. “Before you even make a prototype, you can experiment with your designs using computer simulations. The inputs can range from materials and dimensions to power and control software. For example, you can model the impact of different materials or dimensions, the use of a different mechanical layout or different settings in a control algorithm. In every product, there are lots of choices to be made, both in the architecture and implementation. Finding out what inputs are the most important and how these determine your performance is key because you don’t want to find out in a later stage that an earlier decision was wrong. Trial-and-error is often too expensive and time-consuming.”

The statistical approach involves setting up a series of experiments in a special way, varying multiple inputs at the same time and not comparing single experiments but groups of experiments to tease out the effect of a single input or interactions between two inputs. “This is a very powerful approach, especially in combination with computer simulations. For a small number of inputs, you can also do this in hardware. If you run the experiments in hardware, the calculated sample size from the earlier stages determines the number of repeats for the different experiments. If the experiments are done virtually, through computer simulations, the sample size is used for the validation experiments for the computer model.”

The next tool in the statistics toolbox is optimization – making the best choice. Once you’ve found the key input parameters and how they relate to the performance, you can make a data-driven decision about what design configuration best suits your purpose. There are often multiple outputs to consider, for example, if you want high strength but at the same time low weight. Multiple response optimization is a well-known tool for this.

“Once you know the impact of an input, it’s also important to look at its variation, and in turn what variation it causes in the performance,” Luiten continues. “This is also something people are less familiar with, but once you know how to do this, it’s not that difficult. And it’s important. For a design to be a success, it’s not just peak performance that matters, but also that you consistently achieve that performance. Using statistical simulations, you can create a statistical model to link the mean and variance of your output to the statistical distribution of your inputs.”

People sometimes say this is pointless because they don’t know the input variation. But if an input is important, not considering its variation is risky in terms of consistent product quality. If the statistical model shows that the input is important, you have a good reason to discuss with the supplier what the distribution is and how much variation it has. This is common practice in the automotive industry. In fact, they have formal procedures in place defining exact requirements not only on the mean but also on the standard deviation of components and sub-assemblies.

Statistical methods, in other words, help navigate all possible configurations that make up the solution space. “You don’t reach your optimal performance by accident. If you have two inputs that can be high or low, that leaves four possibilities. With five inputs, there are 32. And many modern-day designs have more inputs than that. And that’s without even taking all the possible tolerances into account and all different user cases. Without a structural, statistics-based approach, the chances of finding the optimal consistent performance are slim.”

Driving a car

The components of Luiten’s course are closely interrelated; it’s a chain of tools. “For example, if you want to validate simulation results, you also need measurement statistics to tell you what your random error is. This in turn shows how large your sample needs to be so that the experimental setup is correct. Only then can you decide whether you can trust your validation.”

Luiten takes a pragmatic approach. For her, statistics is an applied skill, a means to an end. “Statistics in university is taught in a very theoretical way,” she says. “I saw this in my own studies, and I saw it when my children were in university. It’s taught in a way that had limited practical use in my line of work. I compare it to driving a car: you don’t need to know exactly how the engine works to get from A to B. The aim of the course is not to become a statistical expert but to be able to achieve your goal with statistics.” And mathematical tools like Excel and statistics software make the application much more accessible these days.

Luiten is a Master Black Belt in Design for Six Sigma, and her career has given her a deep understanding and rich experience in applying statistics to innovation processes. “In my experience, many engineers learn by doing, and that makes sense. You can’t learn to swim by watching the Olympics; you have to get in the water yourself, even if it’s only to learn to float. So we have practice exercises, either in Excel or a dedicated statistics tool.” Statistics for Luiten is a general-purpose tool and familiarity with the techniques and tools covered in her course is key for engineers working in a variety of fields, from technical experts, to designers and team leads, to system architects.

“This is a general course for people in innovation, who develop products and do research. If you measure output in continuous numerical parameters, it doesn’t matter what technical field you’re in. I’ve applied these techniques in thermal applications, but any field can use them, from mechanics and electronics to optics and even software. It’s mathematics. It’s for you to decide what to use it for.”

This article was written in close collaboration with High Tech Institute.