Module 4
Session 4.5
Regression Testing
Regression Testing Fundamentals
Differentiate progressive vs regressive testing and restate objectives — ISTQB Foundation Level §5.2 | Chapter 8: Regression Testing
Learning Objectives
- Define regression testing and explain why it is a software maintenance necessity.
- Distinguish between progressive testing and regressive testing with examples.
- State the three core objectives of regression testing.
- Identify the contexts that trigger a regression testing cycle.
- Explain why regression testing is both necessary and challenging (the regression testing problem).
What Is Regression Testing?
ISTQB Definition:
"Regression testing is the selective retesting of a system or component to verify that modifications have not caused unintended effects and that the system or component still complies with its specified requirements."
Chowdhury (Software Testing, 2nd ed.) Definition:
"Regression testing is a software maintenance task performed on a modified program to instil confidence that changes are correct and have not adversely affected the unchanged portions of the program."
"Regression testing is the selective retesting of a system or component to verify that modifications have not caused unintended effects and that the system or component still complies with its specified requirements."
Chowdhury (Software Testing, 2nd ed.) Definition:
"Regression testing is a software maintenance task performed on a modified program to instil confidence that changes are correct and have not adversely affected the unchanged portions of the program."
Key word: Selective
Regression testing does not re-run every test that has ever been written. It selects tests relevant to the change made — a fundamental challenge in practice.
Regression testing does not re-run every test that has ever been written. It selects tests relevant to the change made — a fundamental challenge in practice.
Key word: Unintended effects
The primary fear in regression is that a correct fix in one area silently breaks something that was working correctly before.
The primary fear in regression is that a correct fix in one area silently breaks something that was working correctly before.
Key word: Modified program
Regression applies whenever the software changes: bug fix, new feature, configuration change, library upgrade, or database change.
Regression applies whenever the software changes: bug fix, new feature, configuration change, library upgrade, or database change.
Simple analogy: Imagine a bridge engineer repairs a crack in one support column. Before reopening the bridge, they inspect all other columns, connections, and load-bearing components to confirm the repair did not affect them. That inspection is regression testing.
Why Regression Testing Is Needed
Software changes constantly. Every change — no matter how small — carries the risk of introducing new defects or reactivating previously fixed ones. Regression testing is the safety net that catches these regressions before they reach users.
Without regression testing: A fix for Bug A inadvertently breaks Feature B. Since Feature B was not retested, the team ships the broken feature to production, creating a new incident.
With regression testing: After fixing Bug A, a targeted regression cycle re-executes tests covering Feature B and surrounding functionality. The break in Feature B is detected before release.
Why changes cause regressions:
- Shared code paths: A function used by Feature A and Feature B is modified for Feature A, unknowingly altering Feature B's behaviour.
- State dependencies: A change to a database schema or shared variable affects multiple downstream modules.
- Configuration interactions: A configuration change for one environment affects another.
- Integration side effects: A library upgrade fixes one issue but changes behaviour elsewhere.
- Incorrect fix scope: The fix addresses the symptom but not the root cause, leaving related failures dormant.
ISTQB Principle 3 — Early Testing, and Principle 6 — Testing is Context Dependent: Regression testing must happen at every test level (unit, integration, system) after each significant change. Early regression catches regressions at the cheapest fix point.
Progressive vs Regressive Testing
These two concepts are frequently confused. They are complementary, not alternatives, and both are necessary in every development cycle that involves change.
Progressive Testing
Testing performed to verify new functionality that was just added or to verify a previously untested area of the system.
- Purpose: Confirm new code works as specified.
- Trigger: New feature implemented or new requirement added.
- Test basis: New or updated requirements/user stories.
- Example: A payment module was just built. Tests are written to verify payment processing for the first time.
Regressive Testing
Testing performed to verify that previously working functionality still works correctly after a change to the system.
- Purpose: Detect unintended side effects of a change.
- Trigger: Bug fix, feature change, or any software modification.
- Test basis: Existing test cases from previous test cycles.
- Example: A discount calculation was changed. Existing checkout tests are re-executed to confirm other pricing logic still works.
Key difference table:
| Dimension | Progressive Testing | Regressive Testing |
|---|---|---|
| What is tested | New or changed functionality | Previously passing functionality |
| Test cases | New test cases written | Existing test cases re-executed |
| Objective | Confirm new code works | Confirm nothing was broken |
| Triggered by | New requirements / new code | Any modification to existing code |
| Risk if skipped | New features shipped untested | Hidden regressions reach production |
In practice: Both types run together after every sprint or build. Progressive tests verify new stories; regression tests verify existing functionality was not broken by those new stories. Skipping either creates risk.
Objectives of Regression Testing
The objectives of regression testing are directly traceable to the software quality goals it supports. Three core objectives are widely accepted in the ISTQB and academic literature:
Objective 1: Verify the Bug Has Been Addressed
After a defect is reported and fixed, regression tests re-execute the test cases that originally exposed the problem to confirm the fix is effective. This is called bug-fix confirmation.
Without this, a "fixed" defect may still fail in edge cases not initially tested.
After a defect is reported and fixed, regression tests re-execute the test cases that originally exposed the problem to confirm the fix is effective. This is called bug-fix confirmation.
Without this, a "fixed" defect may still fail in edge cases not initially tested.
Objective 2: Find Other Related Bugs
Fixing one bug can introduce new bugs or reveal related defects in the same code area. Regression testing broadens the scope to cover the changed module and its immediate dependencies, looking for newly introduced issues.
Related bugs often share root causes (e.g., a misunderstood business rule) that affect multiple code paths.
Fixing one bug can introduce new bugs or reveal related defects in the same code area. Regression testing broadens the scope to cover the changed module and its immediate dependencies, looking for newly introduced issues.
Related bugs often share root causes (e.g., a misunderstood business rule) that affect multiple code paths.
Objective 3: Check the Effect on Other Parts of the Program
This is the core regression concern: verifying that the change did not adversely affect previously correct functionality in other parts of the system. This is called side-effect regression or stability regression.
Most regression test failures fall into this category: untouched code that broke due to a shared dependency.
This is the core regression concern: verifying that the change did not adversely affect previously correct functionality in other parts of the system. This is called side-effect regression or stability regression.
Most regression test failures fall into this category: untouched code that broke due to a shared dependency.
Chowdhury (Chapter 8) states: Regression Testing Produces Quality Software. The discipline of running regression tests after every change is itself a quality assurance practice — it prevents quality degradation over successive software releases.
Software Maintenance Context
Regression testing is fundamentally a software maintenance activity. Software maintenance includes all activities that modify a system after initial delivery. Four maintenance types each trigger regression testing:
Corrective Maintenance
Fixing defects discovered after delivery. Bug-fix regression tests confirm the fix and check for side effects.
Fixing defects discovered after delivery. Bug-fix regression tests confirm the fix and check for side effects.
Adaptive Maintenance
Modifying software to work in a changed environment (new OS, DB version, regulatory change). Regression tests verify existing behaviour under new conditions.
Modifying software to work in a changed environment (new OS, DB version, regulatory change). Regression tests verify existing behaviour under new conditions.
Perfective Maintenance
Adding new features or improving performance. Progressive tests cover new functionality; regression tests protect existing functionality.
Adding new features or improving performance. Progressive tests cover new functionality; regression tests protect existing functionality.
Preventive Maintenance
Refactoring, code restructuring, documentation improvement. No functional change is intended — regression testing verifies this is true.
Refactoring, code restructuring, documentation improvement. No functional change is intended — regression testing verifies this is true.
ISTQB and maintenance testing: ISTQB Foundation Level defines maintenance testing as testing of changes to an operational system. Regression testing is the primary technique within maintenance testing. The scope of regression testing should be proportional to the risk and size of the change.
When Regression Testing Is Triggered
Regression testing is not a one-time event. It must be performed whenever the software is modified. Key triggering events include:
Bug-fix regression
Performed after a reported bug has been fixed. Re-executes the original failing test case(s) plus related tests to confirm the fix is effective and stable. Corrective maintenance
Performed after a reported bug has been fixed. Re-executes the original failing test case(s) plus related tests to confirm the fix is effective and stable. Corrective maintenance
New feature integration
When new functionality is added to an existing system. Regression ensures the new code does not break existing functionality. Perfective maintenance
When new functionality is added to an existing system. Regression ensures the new code does not break existing functionality. Perfective maintenance
Rapid iterative development (Agile/CI)
In sprint-based and continuous integration environments, regression tests run after every build or commit. Automated regression suites are essential here. Agile / DevOps
In sprint-based and continuous integration environments, regression tests run after every build or commit. Automated regression suites are essential here. Agile / DevOps
First step of integration
When newly completed modules are integrated with existing modules, regression testing verifies existing module behaviour is preserved. Integration phase
When newly completed modules are integrated with existing modules, regression testing verifies existing module behaviour is preserved. Integration phase
Compatibility assessment
When the system is deployed to a new OS, browser version, database, or hardware platform. Regression confirms parity with the known-good baseline. Adaptive maintenance
When the system is deployed to a new OS, browser version, database, or hardware platform. Regression confirms parity with the known-good baseline. Adaptive maintenance
Refactoring / code restructuring
When internal code is reorganised without changing external behaviour, regression tests verify the behaviour guarantee holds. Preventive maintenance
When internal code is reorganised without changing external behaviour, regression tests verify the behaviour guarantee holds. Preventive maintenance
Key principle: The frequency and scope of regression testing should be proportional to (a) the rate of change, and (b) the risk of the changed component. High-frequency change environments (CI/CD pipelines) require automated regression suites to make this sustainable.
Regression Testing & Software Quality
Regression testing is one of the most direct mechanisms for maintaining and increasing software quality across a product's lifecycle. Without it, software quality degrades with each release as undetected side effects accumulate.
How regression improves quality
- Creates a verified quality baseline after each change.
- Prevents accumulated technical debt from causing cascading failures.
- Supports safe refactoring by providing a safety net.
- Builds stakeholder confidence in release quality.
- Reduces production incidents caused by regression defects.
Quality metrics regression supports
- Defect escape rate: Fewer regressions reaching production.
- Defect density: Tracks regression defect density per release.
- Test coverage stability: Ensures passing tests remain passing.
- Mean time to detect (MTTD): Regression testing reduces MTTD for side-effect defects.
ISO/IEC 25010 connection: Regression testing directly supports the reliability quality characteristic (specifically fault tolerance and recoverability sub-characteristics) by ensuring that individual module failures or changes do not propagate into system-wide quality degradation.
The Regression Testing Problem
Regression testing is necessary, but it creates a fundamental challenge: the test suite grows with each release, making full re-execution increasingly expensive and time-consuming.
Formal problem statement (Chowdhury / Regression Testing literature):
"Given a program P, its modified version P′, and a test set T that was used to previously test P, find a way to utilize T to gain sufficient confidence in the correctness of P′."
This means: we cannot always re-run all of T (too expensive). We need strategies to select, prioritize, and reduce T intelligently.
"Given a program P, its modified version P′, and a test set T that was used to previously test P, find a way to utilize T to gain sufficient confidence in the correctness of P′."
This means: we cannot always re-run all of T (too expensive). We need strategies to select, prioritize, and reduce T intelligently.
The scale problem
A mature product may have thousands of test cases. Re-running all of them after every change is impractical, especially in Agile environments with daily builds.
A mature product may have thousands of test cases. Re-running all of them after every change is impractical, especially in Agile environments with daily builds.
The selection problem
Which subset of the existing test suite is sufficient to expose any regressions introduced by the change? Selecting too few risks missing regressions; selecting too many wastes time.
Which subset of the existing test suite is sufficient to expose any regressions introduced by the change? Selecting too few risks missing regressions; selecting too many wastes time.
The prioritization problem
If we cannot run all tests, in what order should we run them? High-priority tests should detect regressions as early as possible in the test run.
If we cannot run all tests, in what order should we run them? High-priority tests should detect regressions as early as possible in the test run.
The obsolescence problem
As the program changes, some old test cases may become invalid or redundant. Running them wastes time and can produce misleading results.
As the program changes, some old test cases may become invalid or redundant. Running them wastes time and can produce misleading results.
Solutions addressed in Sessions 4.6 and 4.7:
- Regression testability — designing the system and tests to support efficient regression (Session 4.6).
- Regression test selection — choosing a minimal subset of tests relevant to the change (Session 4.7).
- Test case prioritization — ordering tests to detect regressions early (Session 4.7).
- Test suite reduction — eliminating redundant tests to reduce test suite size (Session 4.7).
Worked Example: E-Commerce Order Processing
Context: An e-commerce system has the following modules: User Registration, Login, Product Catalogue, Shopping Cart, Checkout, Payment Processing, and Order History. A bug is reported: "10% loyalty discount is not applied correctly for premium members."
Fix applied: The discount calculation function in
checkout.py is corrected. The condition if user.tier == 'premium' was checking the wrong field; it now checks user.membership_tier.
Regression Testing Plan for this Fix:
| Objective | Tests to Execute | Reason |
|---|---|---|
| 1. Verify bug is fixed | TC-CHK-047: Premium member checkout with 10% discount | Direct confirmation of the reported defect fix |
| 2. Find related bugs | TC-CHK-045 to TC-CHK-052: All discount-related checkout tests | Other discount tiers (student, staff) use similar logic; may be similarly broken |
| 3. Check side effects | TC-CHK-001 to TC-CHK-030: Full checkout test suite TC-PAY-001 to TC-PAY-015: Payment processing tests TC-ORD-001 to TC-ORD-010: Order creation tests |
The checkout function is called by payment and order creation; changes may have affected these |
Progressive vs Regressive in this scenario:
- Progressive: If the team simultaneously adds a "gift voucher" feature, new test cases are written for voucher application (TC-CHK-060 onwards).
- Regressive: TC-CHK-001 to TC-CHK-052 and payment/order tests are re-executed to verify the discount fix did not break existing functionality.
ISTQB Connections
ISTQB Principle 2: Exhaustive Testing Is Impossible
This principle directly motivates regression test selection. Since we cannot re-run every test after every change, we must select intelligently based on risk and change impact.
This principle directly motivates regression test selection. Since we cannot re-run every test after every change, we must select intelligently based on risk and change impact.
ISTQB Principle 3: Early Testing
Running regression at unit and integration level (not just system level) finds regressions at the cheapest fix point.
Running regression at unit and integration level (not just system level) finds regressions at the cheapest fix point.
ISTQB Principle 5: Absence-of-Errors Fallacy
A fully passing regression suite does not mean the system is correct — only that it behaves as it previously did. New requirements may not be covered.
A fully passing regression suite does not mean the system is correct — only that it behaves as it previously did. New requirements may not be covered.
ISTQB §5.2 Maintenance Testing
ISTQB classifies regression testing as the primary technique within maintenance testing. Impact analysis should precede regression test selection.
ISTQB classifies regression testing as the primary technique within maintenance testing. Impact analysis should precede regression test selection.
Common Mistakes
Skipping regression after "small" changes
No change is too small to cause a regression. Off-by-one fixes, typo corrections in configuration, and dependency version bumps have all caused production outages.
No change is too small to cause a regression. Off-by-one fixes, typo corrections in configuration, and dependency version bumps have all caused production outages.
Confusing regression with retesting
Retesting re-executes a specific failed test after a fix. Regression testing re-executes a broader set of tests to check for side effects. Both are needed but they serve different purposes.
Retesting re-executes a specific failed test after a fix. Regression testing re-executes a broader set of tests to check for side effects. Both are needed but they serve different purposes.
Only testing the fixed area
Running only the test cases for the changed module satisfies Objective 1 but ignores Objective 3 (side effects). The most dangerous regressions are in untouched code.
Running only the test cases for the changed module satisfies Objective 1 but ignores Objective 3 (side effects). The most dangerous regressions are in untouched code.
Never pruning the regression suite
A regression suite that grows without pruning becomes bloated with obsolete tests. Running invalid or redundant tests wastes time and can create noise.
A regression suite that grows without pruning becomes bloated with obsolete tests. Running invalid or redundant tests wastes time and can create noise.
No baseline established
Without a known-good baseline (test results from the last stable build), it is impossible to determine whether a test failure is a regression or a pre-existing condition.
Without a known-good baseline (test results from the last stable build), it is impossible to determine whether a test failure is a regression or a pre-existing condition.
Manual regression only in CI/CD
In rapid-release environments, manual regression cannot keep pace with the change rate. Lack of automation makes regression testing a bottleneck or causes it to be skipped.
In rapid-release environments, manual regression cannot keep pace with the change rate. Lack of automation makes regression testing a bottleneck or causes it to be skipped.
Class Activity
Scenario: A university student portal has 5 modules: Login, Course Registration, Fee Payment, Grade Viewing, and Transcript Download. The development team has just made the following two changes:
- Bug fix: Course registration now correctly prevents students from registering for more than 6 courses per semester.
- New feature: A "dark mode" toggle has been added to the UI settings page.
Tasks (in pairs):
- Classify each change as the trigger for progressive testing, regressive testing, or both. Justify your answer.
- For the bug fix (change 1), identify tests that satisfy each of the 3 regression objectives.
- List 3 modules (other than Course Registration) where side-effect regression tests should be considered, and explain why each is at risk.
- What would you include in a regression test baseline for this system?
Evaluation Rubric (10 marks)
- 2 marks: Correct classification of progressive vs regressive with justification.
- 3 marks: Regression tests correctly mapped to all 3 objectives for change 1.
- 3 marks: Reasonable risk justification for 3 side-effect modules.
- 2 marks: Appropriate regression baseline definition.
Exit Ticket
- In one sentence, state the formal problem that regression testing addresses (use the P, P′, T formulation).
- What is the difference between retesting and regression testing?
- Name the three objectives of regression testing and give a one-line description of each.
- Give two real-world maintenance scenarios (one corrective, one adaptive) that would trigger a regression test cycle.
Summary & Assignment
Regression testing is the safety net that ensures software quality does not degrade as a system evolves. It is triggered by any change — bug fix, new feature, refactor, or environment update — and serves three core objectives: confirm the fix, find related bugs, and check side effects. The fundamental challenge is that re-running the entire test suite after every change is impractical at scale.
Key takeaways:
- Regression testing is a maintenance activity, not a one-off event.
- Progressive testing verifies new code; regressive testing protects existing code.
- Three objectives: verify fix, find related bugs, check side effects.
- The regression testing problem (P, P′, T) motivates the need for selection, prioritization, and reduction strategies.
Assignment:
- For your mini-project, identify the last code change made (bug fix or new feature) and map it to one of the four maintenance types.
- Design a regression test plan for that change addressing all three regression objectives. List at least 8 test cases (IDs and descriptions).
- Identify which modules in your system are most at risk of side-effect regression for that change, and justify your answer using the concept of shared code paths or shared data.