Provides practical insight into the world of software testing and quality assurance.; Many software testers have no formal training and therefore turn to other resources for insight into best practices.; Most large software companies are so committed to quality they have one or more testers for each programmer.; Learn how to find problems in any computer program, how to plan an effective test approach, and how to tell when the software is ready for release. Software testing is one of the invisible jobs in the software industry. Everyone has heard of computer programmers but few people realize there are nearly as many people behind the scenes with job titles such as Software Tester, Software Quality Assurance Engineer, Software Test Engineer, and Software Test Technician. Microsoft alone hires hundreds of people for these positions each year. There are also many companies whose sole purpose is providing software test consulting and software testing services. The first edition of Software Testing was published in November 2000. Although the processes and techniques used in testing computer software are timeless, this title will be brought up-to-date by adding a chapter that specifically deals with testing software for security bugs and revisiting the rest of the book to update examples and references.
Table of Contents
Introduction. About the Second Edition Who Should Use This Book? What This Book Will Do for You Software Necessary to Use This Book How This Book Is Organized Part I: The Big Picture Part II: Testing Fundamentals Part III: Applying Your Testing Skills Part IV: Supplementing Your Testing Part V: Working with Test Documentation Part VI: The Future Appendix Conventions Used in This Book I. THE BIG PICTURE. 1. Software Testing Background. Infamous Software Error Case Studies Disney's Lion King, 1994-1995 Intel Pentium Floating-Point Division Bug, 1994 NASA Mars Polar Lander, 1999 Patriot Missile Defense System, 1991 The Y2K (Year 2000) Bug, circa 1974 Dangerous Viewing Ahead, 2004 What Is a Bug? Terms for Software Failures Software Bug: A Formal Definition Why Do Bugs Occur? The Cost of Bugs What Exactly Does a Software Tester Do? What Makes a Good Software Tester? Summary Quiz 2. The Software Development Process. Product Components What Effort Goes Into a Software Product? What Parts Make Up a Software Product? Software Project Staff Software Development Lifecycle Models Big-Bang Model Code-and-Fix Model Waterfall Model Spiral Model Summary Quiz 3. The Realities of Software Testing. Testing Axioms It's Impossible to Test a Program Completely Software Testing Is a Risk-Based Exercise Testing Can't Show That Bugs Don't Exist The More Bugs You Find, the More Bugs There Are The Pesticide Paradox Not All the Bugs You Find Will Be Fixed When a Bug's a Bug Is Difficult to Say Product Specifications Are Never Final Software Testers Aren't the Most Popular Members of a Project Team Software Testing Is a Disciplined Technical Profession Software Testing Terms and Definitions Precision and Accuracy Verification and Validation Quality and Reliability Testing and Quality Assurance (QA) Summary Quiz II. TESTING FUNDAMENTALS. 4. Examining the Specification. Getting Started Black-Box and White-Box Testing Static and Dynamic Testing Static Black-Box Testing: Testing the Specification Performing a High-Level Review of the Specification Pretend to Be the Customer Research Existing Standards and Guidelines Review and Test Similar Software Low-Level Specification Test Techniques Specification Attributes Checklist Specification Terminology Checklist Summary Quiz 5. Testing the Software with Blinders On. Dynamic Black-Box Testing: Testing the Software While Blindfolded Test-to-Pass and Test-to-Fail Equivalence Partitioning Data Testing Boundary Conditions Sub-Boundary Conditions Default, Empty, Blank, Null, Zero, and None Invalid, Wrong, Incorrect, and Garbage Data State Testing Testing the Software's Logic Flow Testing States to Fail Other Black-Box Test Techniques Behave Like a Dumb User Look for Bugs Where You've Already Found Them Think like a Hacker Follow Experience, Intuition, and Hunches Summary Quiz 6. Examining the Code. Static White-Box Testing: Examining the Design and Code Formal Reviews Peer Reviews Walkthroughs Inspections Coding Standards and Guidelines Examples of Programming Standards and Guidelines Obtaining Standards Generic Code Review Checklist Data Reference Errors Data Declaration Errors Computation Errors Comparison Errors Control Flow Errors Subroutine Parameter Errors Input/Output Errors Other Checks Summary Quiz 7. Testing the Software with X-Ray Glasses. Dynamic White-Box Testing Dynamic White-Box Testing Versus Debugging Testing the Pieces Unit and Integration Testing An Example of Module Testing Data Coverage Data Flow Sub-Boundaries Formulas and Equations Error Forcing Code Coverage Program Statement and Line Coverage Branch Coverage Condition Coverage Summary Quiz III. APPLYING YOUR TESTING SKILLS. 8. Configuration Testing. An Overview of Configuration Testing Isolating Configuration Bugs Sizing Up the Job Approaching the Task Decide the Types of Hardware You'll Need Decide What Hardware Brands, Models, and Device Drivers Are Available Decide Which Hardware Features, Modes, and Options Are Possible Pare Down the Identified Hardware Configurations to a Manageable Set Identify Your Software's Unique Features That Work with the Hardware Configurations Design the Test Cases to Run on Each Configuration 136 Execute the Tests on Each Configuration Rerun the Tests Until the Results Satisfy Your Team Obtaining the Hardware Identifying Hardware Standards Configuration Testing Other Hardware Summary Quiz 9. Compatibility Testing. Compatibility Testing Overview Platform and Application Versions Backward and Forward Compatibility The Impact of Testing Multiple Versions Standards and Guidelines High-Level Standards and Guidelines Low-Level Standards and Guidelines Data Sharing Compatibility Summary Quiz 10. Foreign-Language Testing. Making the Words and Pictures Make Sense Translation Issues Text Expansion ASCII, DBCS, and Unicode Hot Keys and Shortcuts Extended Characters Computations on Characters Reading Left to Right and Right to Left Text in Graphics Keep the Text out of the Code Localization Issues Content Data Formats Configuration and Compatibility Issues Foreign Platform Configurations Data Compatibility How Much Should You Test? Summary Quiz 11. Usability Testing. User Interface Testing What Makes a Good UI? Follows Standards and Guidelines Intuitive Consistent Flexible Comfortable Correct Useful Testing for the Disabled: Accessibility Testing Legal Requirements Accessibility Features in Software Summary Quiz 12. Testing the Documentation. Types of Software Documentation The Importance of Documentation Testing What to Look for When Reviewing Documentation The Realities of Documentation Testing Summary Quiz 13. Testing for Software Security. WarGames-the Movie Understanding the Motivation Threat Modeling Is Software Security a Feature? Is Security Vulnerability a Bug? Understanding the Buffer Overrun Using Safe String Functions Computer Forensics Summary Quiz 14. Website Testing. Web Page Fundamentals Black-Box Testin Text Hyperlinks Graphics Forms Objects and Other Simple Miscellaneous Functionality Gray-Box Testing White-Box Testing Configuration and Compatibility Testing Usability Testing Introducing Automation Summary Quiz IV. SUPPLEMENTING YOUR TESTING. 15. Automated Testing and Test Tools. The Benefits of Automation and Tools Test Tools Viewers and Monitors Drivers Stubs Stress and Load Tools Interference Injectors and Noise Generators Analysis Tools Software Test Automation Macro Recording and Playback Programmed Macros Fully Programmable Automated Testing Tools Random Testing: Monkeys and Gorillas Dumb Monkeys Semi-Smart Monkeys Smart Monkeys Realities of Using Test Tools and Automation Summary Quiz 16. Bug Bashes and Beta Testing. Having Other People Test Your Software Test Sharing Beta Testing Outsourcing Your Testing Summary Quiz V. WORKING WITH TEST DOCUMENTATION. 17. Planning Your Test Effort. The Goal of Test Planning Test Planning Topics High-Level Expectations People, Places, and Things Definitions Inter-Group Responsibilities What Will and Won't Be Tested Test Phases Test Strategy Resource Requirements Tester Assignments Test Schedule Test Cases Bug Reporting Metrics and Statistics Risks and Issues Summary Quiz 18. Writing and Tracking Test Cases. The Goals of Test Case Planning Test Case Planning Overview Test Design Test Cases Test Procedures Test Case Organization and Tracking Summary Quiz 19. Reporting What You Find. Getting Your Bugs Fixed Isolating and Reproducing Bugs Not All Bugs Are Created Equal A Bug's Life Cycle Bug-Tracking Systems The Standard: The Test Incident Report Manual Bug Reporting and Tracking Automated Bug Reporting and Tracking Summary Quiz 20. Measuring Your Success. Using the Information in the Bug Tracking Database Metrics That You'll Use in Your Daily Testing Common Project-Level Metrics Summary Quiz VI. THE FUTURE. 21. Software Quality Assurance. Quality Is Free Testing and Quality Assurance in the Workplace Software Testing Quality Assurance Other Names for Software Testing Groups Test Management and Organizational Structures Capability Maturity Model (CMM) ISO 9000 Summary Quiz 22. Your Career as a Software Tester. Your Job as a Software Tester Finding a Software Testing Position Gaining Hands-On Experience Formal Training Opportunities Websites Professional Organizations Dedicated to Software or Software Quality Further Reading Summary Quiz APPENDIX. A. Answers to Quiz Questions. Chapter 1 Chapter 2 Chapter 3 Chapter 4 Chapter 5 Chapter 6 Chapter 7 Chapter 8 Chapter 9 Chapter 10 Chapter 11 Chapter 12 Chapter 13 Chapter 14 Chapter 15 Chapter 16 Chapter 17 Chapter 18 Chapter 19 Chapter 20 Chapter 21 Chapter 22 Index.
Author Biography
Ron Patton is a software consultant living in Washington State. His software test experience is wide and varied from mission critical systems to painting programs for kids. In 1992 he joined Microsoft as a Software Test Lead in the Systems Group for Multimedia Viewer, the authoring tool and multimedia display engine used by Encarta, Cinemania, and Bookshelf. He moved on to become the Software Test Manager of the Kids Product Unit. Most recently, he was the Software Test Manager of the Microsoft Hardware Group responsible for the software shipped with the mouse, keyboard, gaming, telephony, and ActiMates product lines.
