Software Testing Notes

Chapters 1–5 • Exam Preparation

Chapter 1

Introduction to Software Testing

12 sections • 8 flashcards • 10 quiz questions

1.1 Introduction

LOW

Software has pervaded our society — from modern households to spacecrafts. It has become an essential component of any electronic device or system. As software becomes more complex, the demand for quality in software products has increased dramatically.

Software testing has gained immense popularity in the last decade. Job trends have shifted from development to software testing. Today, organizations have separate testing groups with proper hierarchy. Software development is driven with testing outputs — if the testing team claims the presence of bugs, the development team cannot release the product.

Key Reality:

There is still a gap between academia and industry demand. Software testing is mature in industry but not in academia. Students often only study software engineering without dedicated testing courses, creating a skills gap when they enter the workforce.

Key Points

  • ●Software testing is now a separate discipline from software engineering
  • ●Testing consumes almost half the total development effort
  • ●Testing is NOT just a single phase after coding — it starts from requirements
  • ●CMM measures development process; TMM (Testing Maturity Model) measures testing process

1.2 Evolution of Software Testing

HIGH

Gelperin and Hetzel characterized the growth of software testing into 6 phases:

Phase 1: Debugging-Oriented Phase (Before 1957)

  • Testing basics were unknown
  • Programs were tested until programmers were "sure" all bugs were removed
  • The term used was "checkout" — focused on getting the system to run
  • No clear distinction between development, testing, and debugging

Phase 2: Demonstration-Oriented Phase (1957–78)

  • Charles Baker pointed out that checkout should also demonstrate correctness
  • Scope increased from "program runs" to "program correctness"
  • Purpose was to show the ABSENCE of errors
  • Misconception: software could be tested exhaustively

Phase 3: Destruction-Oriented Phase (1979–82)

  • Revolutionary turning point! Myers changed the view
  • From "testing is to show absence of errors" → "testing is to FIND more and more errors"
  • Debugging was separated from testing
  • Importance of effective testing over exhaustive testing
  • Importance of early testing was realized

Phase 4: Evaluation-Oriented Phase (1983–87)

  • Bugs found early = cheaper to debug
  • Quality of software should be evaluated at every stage
  • Early testing established as Verification & Validation (V&V) activities
  • 1983: National Bureau of Standards released guidelines for V&V

Phase 5: Prevention-Oriented Phase (1988–95)

  • Focus shifted from bug-detection to bug-prevention
  • Prevention model includes test planning, test analysis, and test design
  • Experience from similar projects helps prevent bugs in new projects

Phase 6: Process-Oriented Phase (1996 onwards)

  • Testing established as a complete process running parallel to SDLC
  • TMM (Testing Maturity Model) developed to measure testing process performance
  • Emphasis on quantification of testing parameters

Also: Nguyen & Pirozzi Classification

  • Software Testing 1.0 — Single phase after coding, no test organization, limited tools
  • Software Testing 2.0 — Testing gained importance, early testing started, many tools available
  • Software Testing 3.0 — Strategic process-based effort, driven by quality goals, management actively involved

Key Points

  • ●Evolution: Debugging → Demonstration → Destruction → Evaluation → Prevention → Process
  • ●Myers (1979) was the revolutionary turning point — testing to FIND errors, not show their absence
  • ●Prevention is superior to detection — bug prevention > bug detection
  • ●Testing is now a complete process parallel to SDLC

Exam Tip

Long questions often ask to "Explain the evolution of software testing" — describe all 6 phases with timeline and key characteristics. Also know the difference between Testing 1.0, 2.0, and 3.0.

1.3 Software Testing — Myths and Facts

HIGH
MYTH: Testing is a single phase in SDLC
FACT: Testing starts as soon as requirements are specified and continues throughout SDLC, even post-implementation
MYTH: Testing is easy
FACT: Testers must plan, develop test cases manually, understand project design thoroughly. It can be harder than development.
MYTH: Software development is worth more than testing
FACT: Testing has equal status and importance. Testing is a complete process like development.
MYTH: Complete testing is possible
FACT: Input domain of even a small program is too large. "Complete testing" replaced with "effective testing" — selecting test cases to uncover severe bugs first.
MYTH: Testing starts after program development
FACT: Testing begins when specifications arrive. Verification is done at end of every SDLC phase.
MYTH: Purpose of testing is to check functionality only
FACT: Testing is driven by quality goals — not just functionality but reliability, performance, usability, etc.
MYTH: Anyone can be a tester
FACT: Testing requires training for STLC phases, test case design techniques, tools, etc. Certified tester courses exist.

Key Points

  • ●Complete/exhaustive testing is IMPOSSIBLE — replaced by effective testing
  • ●Testing is NOT easy — requires deep understanding and systematic approach
  • ●Testing has EQUAL importance to development
  • ●Testing starts from requirements, NOT after coding

Exam Tip

Questions often ask "List and explain myths and facts of software testing." Write at least 5-6 myths with their corresponding facts.

1.4 Goals of Software Testing

HIGH

Goals of software testing are classified into 3 major categories:

1. Immediate / Short-term Goals

  • Bug Discovery: Find errors at any stage of software development. More bugs discovered early = better success rate.
  • Bug Prevention: Consequent action of bug discovery. Learn from discovered bugs so they are not repeated. Bug prevention is a superior goal of testing.

2. Long-term Goals

  • Quality: Primary goal from users' perspective. Thorough testing ensures superior quality. Quality depends on correctness, integrity, efficiency, etc.
  • Reliability: Confidence that software will not fail. Increases with rigorous testing. Reliability → Quality (chain: Testing → Reliability → Quality)
  • Customer Satisfaction: Prime concern from users' perspective. Testing must satisfy all specified and unspecified requirements.
  • Risk Management: Identify and reduce risks associated with software. Risk is a problem that could threaten the success of the project.

3. Post-implementation Goals

  • Reduced Maintenance Cost: Thorough testing before release reduces post-release bugs, thus reducing maintenance cost.
  • Improved Testing Process: Learn from testing experience to improve testing process for future projects.

Key Points

  • ●Bug prevention is SUPERIOR to bug discovery (prevention > detection)
  • ●Chain: Testing → Reliability → Quality
  • ●Customer satisfaction requires satisfying both specified AND unspecified requirements
  • ●Risk management identifies problems that could threaten project success

Exam Tip

Questions often ask "Explain the goals of software testing." Always divide into 3 categories with examples. Emphasize that bug prevention is a superior goal.

1.5 Psychology for Software Testing

HIGH

The mindset of a tester is crucial. There are two possible psychological approaches:

Constructive Approach (WRONG)

Testing to show that the program works — trying to demonstrate the absence of errors. This leads to subconsciously avoiding test cases that might break the program.

This is like walking on a bridge and saying "it didn't fall" — you haven't actually tested it properly!

Destructive Approach (CORRECT)

Testing to find errors — trying to break the program. A successful test case is one that finds a bug, not one that shows the program works.

Like stress-testing a bridge — you WANT to find its breaking point!

Myers' Principle:

"Testing is the process of executing a program with the intent of finding errors." A good test case has a high probability of finding an as-yet-undiscovered error. A successful test is one that uncovers an as-yet-undiscovered error.

Testing should be done by independent testers rather than developers, because:

  • Developers have a constructive mindset — they want their code to work
  • Developers may unconsciously avoid test cases that might break their code
  • Independent testers bring a fresh, destructive perspective
  • However, developers can also test effectively with the right mindset

Key Points

  • ●Testing must follow a DESTRUCTIVE mindset — find errors, not show their absence
  • ●A successful test case is one that FINDS a bug
  • ●Independent testers are preferred over developers for testing
  • ●Myers defined testing with intent of finding errors

Exam Tip

Very important for long answers! Explain both approaches with examples. Explain WHY independent testing is preferred.

1.6 Software Testing Definitions

MEDIUM

IEEE Definition (1983):

"Testing is the process of exercising or evaluating a system or system component by manual or automated means to verify that it satisfies specified requirements."

G. J. Myers Definition:

"Software testing is the process of executing a software system to determine whether it matches its specifications and executes in its intended environment."

BCS Definition:

"Testing is the process of executing a program with the specific intention of finding errors."

Key Takeaway:

All definitions emphasize that testing is a process (not a single activity) and its purpose is to find errors/verify requirements. The BCS definition is the most aligned with the destructive approach.

Key Points

  • ●IEEE: Testing verifies specified requirements
  • ●Myers: Testing determines if software matches specifications and runs in intended environment
  • ●BCS: Testing is specifically to FIND ERRORS

1.7 Model for Software Testing

LOW

A model for software testing consists of the following elements:

1
Software Specifications: Requirements that define what the software should do
2
Software Code: The actual implementation
3
Testing Techniques: Methods to design test cases (black-box and white-box)
4
Test Cases: Designed using testing techniques to find errors
5
Test Execution: Running test cases and comparing actual vs. expected results
6
Test Results: Analysis of pass/fail and bug reporting

1.8 Effective vs. Exhaustive Software Testing

HIGH

Why Exhaustive Testing is Impossible

Exhaustive testing means testing the software with ALL possible input combinations. This is practically impossible because:

  • The input domain is too large: Even a simple program accepting two 32-bit integers has 2³² × 2³² = 2⁶⁴ possible input combinations
  • Too many possible paths: The number of paths through a program grows exponentially with the number of decision points
  • Time constraints: Running all test cases may take years
  • Resource constraints: Not enough manpower or computational resources

Effective Testing (The Solution)

Since exhaustive testing is impossible, we practice effective testing:

  • Select a representative subset of test cases that has the highest probability of finding bugs
  • Use testing techniques (BVA, Equivalence Class, etc.) to design smart test cases
  • Prioritize test cases — test critical functions first
  • Focus on areas most likely to have bugs
  • Use risk-based testing approach

Example to Prove Exhaustive Testing is Impossible:

Consider a simple program that takes two integers as input. Each integer can be any value from -2,147,483,648 to 2,147,483,647. The total combinations = (2³²)² ≈ 1.8 × 10¹⁹. If we execute one test per millisecond, it would take approximately 570 million years to test all combinations!

Factors Determining the Limit of Testing:

  • Available time and budget
  • Risk assessment — higher risk areas need more testing
  • Complexity of the software
  • Quality goals and standards
  • Customer requirements and expectations
  • Availability of testing tools and resources

Key Points

  • ●Exhaustive testing is IMPOSSIBLE — proven with mathematical examples
  • ●Effective testing = smart selection of test cases with highest probability of finding bugs
  • ●Two 32-bit integers = 2⁶⁴ combinations ≈ 570 million years to test all!
  • ●Risk-based approach helps prioritize what to test

Exam Tip

Very common long question: "Explain why exhaustive testing is not possible. What is effective testing?" Always give the mathematical example and explain effective testing as the alternative.

1.9 Effective Testing is Hard

LOW

Effective testing is challenging because of several reasons:

  • No clear definition of "enough testing": When do we stop testing? There is no clear-cut answer.
  • Selecting right test cases: Choosing the most effective test cases from an infinite domain requires expertise.
  • Understanding specifications: Incomplete or ambiguous specifications make it hard to design effective tests.
  • Time pressure: Limited time means we must be strategic about what to test.
  • Changing requirements: Requirements keep changing, making test design a moving target.
  • Complexity of software: Complex interactions between modules make it hard to predict where bugs might be.

1.10 Software Testing as a Process

MEDIUM

Software testing is not a single phase — it is a complete process that runs parallel to SDLC. The testing process includes:

  • Requirements Phase: Review requirements for completeness, consistency, and testability
  • Design Phase: Review design documents, plan test strategies
  • Coding Phase: Perform code reviews, unit testing
  • Testing Phase: Execute test cases, integration testing, system testing
  • Maintenance Phase: Regression testing, bug fixes verification

Testing runs parallel to SDLC, not after it. The Software Testing Life Cycle (STLC) has its own phases that correspond to SDLC phases.

1.11 Schools of Software Testing

MEDIUM

Bret Pettichord identified 5 schools of software testing:

Analytical School

Focuses on mathematical and formal methods. Testing is based on formal specifications and logic. Emphasizes test case design techniques and coverage criteria.

Standard School

Focuses on standards and processes. Emphasizes following defined procedures, test plans, and documentation. Values traceability and repeatability.

Quality School

Focuses on quality assurance and process improvement. Views testing as part of a larger quality management system. Emphasizes process maturity models.

Context-Driven School

Believes testing approaches must be adapted to the context of each project. No single best approach exists. Values skill, judgment, and cooperation.

Agile School

Integrates testing into the development process. Testing happens continuously. Values working software over comprehensive documentation.

Key Points

  • ●5 schools: Analytical, Standard, Quality, Context-Driven, Agile
  • ●Each school has different philosophy and approach to testing
  • ●Context-Driven: adapt to project context
  • ●Agile: continuous testing integrated with development

1.12 Software Failure Case Studies

HIGH

These famous failures demonstrate the critical importance of software testing:

Ariane 5 Crash (1996)

Ariane 5 rocket exploded 37 seconds after launch due to a software bug. A 64-bit floating point number was converted to a 16-bit signed integer, causing overflow. Cost: $500 million. Root cause: Reused Ariane 4 software without proper testing for new trajectory.

Mars Climate Orbiter (1999)

Lost due to a unit conversion error — one team used metric units while another used imperial units. Cost: $327 million. Demonstrates importance of requirements verification.

Therac-25 Radiation Machine

A radiation therapy machine that gave lethal radiation doses due to software bugs. Race condition in the software. Killed 6 patients. Critical safety-critical software testing failure.

Pentium FDIV Bug (1994)

Intel Pentium processor had a floating-point division bug. Cost Intel $475 million in replacements. Shows that even hardware testing must be thorough.

Exam Tip

Questions often ask "Explain the importance of software testing with case studies." Mention 2-3 case studies with details about the bug, its impact, and the lesson learned.

Review Questions with Detailed Answers