Beauty of Testing

Steven Sinofsky wrote a nice post about software testing:

Steven defines testing as “verification that a product does what it is intended to do and does so elegantly, efficiently, and correctly” and as “the conscience of a product“. Does it sound about right? Yes, usually software testers verify that software works as defined in specification/requirements/user stories/whatever. But it is not all that can be done. It is quite easy to build a product which does exactly as defined in the spec, but at the same time is useless for customers, or is not profitable for the company which develops it. Systems engineering defines that lyfecycle of the system contains verification and validation as separate logical stages. Validation here is checking that a product satisfies stakeholder needs. Validation is probably something testers must do as well.

Steven uses movie production as a good analogy to show why testers and developers should be separate roles: there are so many things to worry about that one person can’t remember all the details and deal with all the issues, hence specialization of the roles. And it makes total sense, it is very hard for one person to do everything well. I can imagine further specialization, which probably happens now in some companies: performance testing, user acceptance testing, security testing, telemetry analysis, etc.

Steven notes a trend of moving testing upstream to earlier phases of a product lifecycle, but mentions that it can harm if it is got to extreme. If testing is started when there is no clear understanding of product, then it leads to chaos of enormous amount of meaningless bugs.

On another side of system lifecycle diagram, testing does not stop when product is delivered to customers. Nowadays telemetry helps to test software on production. Steven tells about Microsoft experience of gathering error reports using Watson tool, which helped Microsoft a lot to stabilize their products and even help 3rd party developers to make their software work better on Windows platform. Telemetry is even easier now because software moves to the cloud, and gives developers better option of monitoring.