About a year ago Microsoft laid off much of their internal quality assurance team (article) and with the advent of Windows 10 we can see how that decision has affected the quality of Windows 10. I’ll give you a hint… it’s suffered badly, quite badly.
I bet you’re asking what Microsoft replaced the QA team with. The answer is… The Windows Insider Preview Program. Yeah, that thing.
Outsourcing your quality assurance to outside your company is a recipe for disaster. What do you think the The Windows Insider Preview Program is? That’s right, the outsourcing of quality assurance onto the backs of amateurs. Do you see how this could cause things to blow up and go wrong?
Let’s explain some things here. To do proper software testing you need a team that’s willing to run a beta version of the software full time and not only that but a team made up of intelligent people who are capable of reporting bugs in a concise and detailed manner. They also need to understand that things aren’t going to work right all the time and that things may suddenly blow up in their faces at any moment taking their data with it. Guess what? That’s not The Windows Insider Preview Program. If you look at some of the complaints on The Windows Insider Preview Program they read like some whiny sixteen year old kid wrote them, not the kind of concise and detailed reports that developers need to fix software bugs. “It doesn’t work” or “It crashes” is not helpful but that’s the kind of things that you’ll often read as comments and reports in The Windows Insider Preview Program. Are you starting to see why The Windows Insider Preview Program was destined for failure? Yeah…
As for the bugs that made their way into the Windows 10 Anniversary Update, Microsoft claims that the Anniversary Update was the most highly tested version of Windows to date which is a flat-out lie. Nobody in their right mind runs those preview builds as a full time OS (unless you’re a masochist and you like punching yourself in the gut). So no wonder why so many bugs got through. And let’s not forget that to install an upgrade from one Insider Preview Program build to the next involves a process that’s very similar to installing a whole new operating system. Again, nobody in their right mind would run an Insider Preview Program build as a full time OS, even at home. Only a masochist would agree to running one of these builds as a full time OS.
A couple of months ago people were complaining on the Insider Preview Program that Microsoft was releasing too many builds back-to-back and these were people who were just testing this in lab conditions or people like me who were testing this stuff in virtual machines. Now imagine if someone was actually running these builds as a full-time OS? They would be pulling their hair out.
And as I said above, you need the kind of people who are willing to test something full time and don’t care if something goes horribly wrong. You have to have a dedicated team to do this kind of testing, you can’t expect normal people to be testing this stuff. You need people who are willing (and paid) to run test builds of an OS that don’t care if their system is thrown into complete chaos and thus forced to be reinstalled clean.
The Windows Insider Preview Program (and others like it) is not and will never be the answer to proper software testing. Software companies still need internal testing teams to do the hard work before releasing beta versions. What passes as Insider Preview Program Builds these days smell more like very early beta versions, I’d even go so far as to say they’re alpha versions. Unfortunately Microsoft fired much of their internal testing teams to be replaced by the developers themselves and that’s another recipe for disaster. Developers make the worst testers just like medical professionals make the worst patients. They know how things are supposed to work and thus they can’t test something adequately because they’re testing it from the perspective of someone who knows how it should work and aren’t testing for those edge-cases. You need people who are willing to bang on something and intentionally break it. And sometimes, breaking something is an art. I know people who can break stuff easily while I sit back and scratch my head wondering just what the hell they did.