In gesprek met Bj Rollison

Sprak ik twee weken terug nog met James Whittaker van Google. Ja, dan kan Microsoft niet achter blijven natuurlijk. Deze keer is het de beurt aan Bj Rollison, één van de meest prominente testprofessionals bij Microsoft. Hij schrijft boeken, geeft presentaties, is een grote voorstander van een gedegen testopleiding en geeft graag een kijkje in de testkeuken van Microsoft. Daarnaast is hij een fanatiek zeiler, sleutelt hij graag aan zijn ’72 Corvette, doet hij aan kleiduivenschieten en blogt hij over zijn dagelijkse beslommeringen onder de naam I.M Testy. Hieronder de vragen die ik hem gesteld heb en zijn antwoorden.

1. Can you introduce yourself and explain how you became a tester after you studied cultural anthropology in college
[Bj ] That’s a great question. I don’t think that most of us in the testing discipline thought…”hey, when I grow up I am going to be a software tester!” I suspect like many of us I was presented an opportunity to explore a new job at a transition point in my life and walked through that door. In all honesty cultural anthropology was very interesting, but I never thought I would become a Franz Boas. But, I learned quite a bit and developed a great respect for the values and traditions of different cultures and peoples around the world. Then, after spending approximately 13 years in the U.S. Air Force I had the opportunity to work in a small shop in Okinawa, Japan building computers from the motherboards up to installation in small and medium sized businesses on the island. I taught myself Q-Basic when I bought my first computer in 1979, and later I learned the C language and just enough Assembly to be dangerous. I was also fascinated with memory management because the Japanese DOS required so many device drivers on boot. It was a real challenge to configure the various device driver load sequence in the config.sys and autoexecute.bat files, and then test the system in order to ‘squeeze’ in as much memory usage as we could into the upper memory blocks. But, even then I didn’t think of myself as a tester. I didn’t think about testing as a career choice until 1994 when I was invited to interview at Microsoft for a software testing position. So, in short, my road to a career in software testing has had many twists and turns, but for me it has been a fun and rewarding ride.

2. Can you tell us how many Microsoft is testing, just as an answer to all the criticism?
[Bj ] There are approximately 9000 testers at Microsoft working on approximately 300 products that ship annually in about 39 different languages around the world. The dev to test ratio is about 1.2: 1. This seems like a very high ratio but, I would also say that some of our testers fill unique roles. For example, we design, develop, and maintain many of the test tools and frameworks we use internally. We have whole teams of ‘testers’ whose job is to build and maintain testing infrastructure. Many of our testers are also engaged in low level API testing and code reviews and inspections which is sometimes considered to be the responsibility of the developers by some people. Then of course we have our sustained engineering or maintenance teams. Compatibility testing for service packs or hot fixes takes a considerable amount of resources. By no means am I advocating a 1:1 ratio for developers and testers in an organization. There is no magic formula for determining a ratio for developers to testers. I suspect that similar to many companies we are constantly looking for ways to push quality upstream, improve our processes, and find innovative ways to improve our product quality and provide value to our customers.

3. Is it hard to deal with all the criticism? Do you dare to tell your friends that you are testing at Microsoft?
[Bj ] Criticism? I thought everyone loved Microsoft! In all seriousness Microsoft is really quite a unique company. I suspect that many  of the critics really don’t understand what goes on inside of Microsoft or the magnitude and diversity of our business. Actually, I am quite proud that I work at a company that has had such a tremendous impact on the industry overall. Certainly we are not perfect, but I think Microsoft is a pretty resilient company and especially in recent years is much better at addressing customer issues in a more visible way. Personally, it is also quite an honor to get to work alongside some of the people who are helping to drive creative innovation in the software world. I can’t think of any other software company that builds both home and enterprise level products using a wide diversity of technologies in as many different business segments and in as many global languages as Microsoft. We develop operating system, server systems, productivity applications, mobile device platforms, K-12 educational resources, search engines, client + cloud solutions, developer tools, and games. Our customers around the world rely on Microsoft products in their homes, businesses, and governments. Of course, similar to all software products we can always improve. I think you will find no bigger critics of our software than the testers at Microsoft who, along with teammates constantly think about ways to improve our software quality and deliver value to our customers around the world.

4. How is testing managed at Microsoft. From one place or per country or per application or … ?
[Bj] At Microsoft testing is mostly managed on a per application team basis. There is no single approach or process that would work across the diversity of our businesses, so there is no central command and control authority that drives or dictates processes. Some business divisions with multiple products share similar processes and approaches, and some of our best practices work across 2 or more of our 7 business divisions. Testers use a variety of approaches and are often engaged throughout the product cycle by participating in design reviews as well as code reviews and inspections. They work alongside developers on feature crews, and we also do a lot of self-hosting of the product we are currently working on as well as dog-food other products from different groups. Some senior testers work with developers to improve unit tests, and we also work with project managers to define and model end-2-end scenarios and user stories. In short, Microsoft testers use a wide variety of approaches to testing because we understand no single approach to testing is sufficient for any project (pesticide paradox), and we need to identify and use the most appropriate approaches in each specific situation for the business and our customers.

5. I read in an interview that you were surprised about how many people in the industry lack any sort of formal training in the profession. How do you cope with that at Microsoft?
[Bj ] Anecdotal surveys by myself, Steve McConnell, and Dorothy Graham found a surprising number of testers not only lack formal training, but also generally did not read books on topics specific to their profession. However, I will also say that there have been a great number of good books on software testing published in the past few years which I hope is an indication that testers are expanding their knowledge by reading various views and perspectives of our profession.
At Microsoft we hire mostly university graduates, and our interview process tends to focus on mental acuity such as problem solving, creative thinking, and ability to learn and adapt new concepts quickly. So, we designed a course for our new testers that sets expectations of the role, teaches the theory and application of foundational techniques that are useful in identifying different types of fault models and are used in various approaches to testing, we cover test case design strategies, and we wrap up the course with a practical exercise involving test case design from the requirements, from an exploratory approach, and from an analysis of code and path coverage information. We also provide on-going education on advanced testing techniques and new technologies around the company to help our testers grow and improve their skills and knowledge. On-going training and career development is one of the most valued benefits by our employees. The training model we use at Microsoft is fairly unique in that our engineering training organization is not part of the Human Resources department; our group (Engineering Excellence) is part of the Microsoft Research division. Another difference is that we don’t hire a lot of  professional training vendors. Instead we use senior level subject matter experts to teach new employees and lead special interest communities. Microsoft has been listed in the top 10 corporate training organizations for the past 3 years, and the Engineering Excellence team has become a cornerstone for leading change and improving the career development of our engineers. But, of course, we realize in the end it is the individuals in the product teams who ultimately drive the success of our company. As for me, I am quite honored that that I get to work for the many talented testers at the company and try to help them meet new challenges in their career.

6. Isn’t testing at Microsoft like trap shooting, which is one of your hobbies?  You try to shoot as many defects as you can, but you know upfront that you will always miss some of them.
[Bj ] Interesting. Perhaps the shooting sports are in some ways similar to software testing. Some people think that trap shooting or target shooting is as simple as pointing and pulling a trigger. But, anyone can throw a lot of lead down range and eventually hit something. These are mostly recreational shooters and count success by how many clay targets they break or how many holes they get on a target. Similarly, anyone who approaches testing from a ‘let me try this’ standpoint will eventually find a bug of some sort, and they probably regard success based on how many bugs they find. However, people who take shooting sports seriously spend a lot of time studying ballistics and analyzing shot patterns. They learn to use various types of tools to craft loads in an attempt to tighten up a shotgun pattern or precision shots at a target. A big advantage of reloading your own ammunition is that it helps reduce variables, and drives quality upstream. Likewise, I suspect that many professional testers constantly strive to improve their knowledge and skills of the systems. But unlike precision shooters, I think many testers are stuck in the defect finding mentality and unfortunately spend too little time thinking about defect prevention. At the end of the day we will still sometimes miss. And we often miss not because of the gun, the ammunition, or the environment conditions (as long as we factor them in). We miss because we  get distracted (which causes shooters to flinch). Similarly in testing I suspect that we miss some bugs not necessarily because of some technique or tool, but because we get distracted and use the wrong technique,  tool, or approach in the wrong context.

7. I hear you will giving a keynote at the Swiss Testing Day. Can you give us a sneak preview on what it will be about?
[Bj ] The talk is titled “How We Test At Microsoft” and is a general glimpse into our testing organizations. We produced this talk because our team (Engineering Excellence) often gets requests from enterprise level customers such as the U.S. Air Force and the Internal Revenue Service asking about our testing approach. I also found that when I met folks around the world they would often ask, “What is Microsoft’s approach to testing?” I started to realize that while most people knew about Microsoft (at least the big products), very few really understood our approach to designing, developing, testing, and delivering software or the vast variety of products our engineers work on around the world. This talk provides a brief background on Microsoft and then discusses how we hire and train our testers, and how testers work alongside developers and project managers in our development lifecycles. I will also discuss some challenges that our discipline faces, and a glimpse of what we think the future of testing might look like.

8. Where is Bj stands for in your name. I could not find it anywhere on the Internet. Is it a secret?
[Bj ]  It’s not a secret, it’s just a nickname that was given to me when I was in the U.S. Air Force. While growing up my family called me ‘Bill’ (except my mother called me ‘William’ when she was angry, and ‘William John’ when she was really angry). At my first duty station my commander said, “There are 7 people in this squadron named William (including himself) so, what should we call you?” William, Will, Bill, etc. were already taken so I could either choose numbers or letters, so I choose the first letter of my short name ‘Bill’ and my middle name ‘John’ and Bj stuck. All my military tours of duty were overseas, and I neglected to inform my family that my first name had essentially changed to Bj. When I got to my 3rd overseas assignment my mother called the squadron asking for Bill Rollison. Our squadrons were small and relatively informal, so everyone I worked with only knew me as Bj. The person who answered the phone told my mother there was nobody assigned at that squadron by that name (I guess not thinking to check on the surname). Three days later the person who answered the phone saw my nametag and asked, “Do you sometimes go by Bill?” I said my family called me that, and he replied, “Oh no, someone called here the other day asking for you and I told them there was nobody here by that name.” I called my mother, reassured her that I was not involved in some type of Philadelphia Experiment, and that I was in fact stationed where I told her I would be stationed. These day if someone calls me Bj then I know they know me, if someone calls me Bill it’s my family, and if someone calls me William its either the federal government, and only my mother calls me William John.

9. Last one. A dillema question: If you had to give up your sailboat or your ’72 Corvette, what would you choose and why?
[Bj ] I don’t even have to think about that…the Corvette goes. The Corvette was a summer project and a boyhood dream of owning a classic Vette. But as for sailing; that is a way of life!

Bj Rollison
Test Architect in the Engineering Excellence group at Microsoft