Session 4.6 - Regression Testability & Timing

What Is Regression Testability?

Testability is the degree to which a system or component supports testing in a given test context. Regression testability specifically refers to how easily and efficiently the system supports repeated testing after changes.

ISTQB Definition of Testability (ISO 25010):
"The degree of effectiveness and efficiency with which test criteria can be established for a system, product, or component, and tests can be performed to determine whether those criteria have been met."

High Regression Testability

Modular, loosely coupled architecture.
Each module has clear, stable interfaces.
Test automation suite covers critical paths.
Test environments are reproducible and stable.
Change impact is isolated and traceable.
Regression runs complete in minutes (automated).

Low Regression Testability

Monolithic, tightly coupled codebase.
Any change can affect any other part.
Only manual tests exist; slow and error-prone.
Test environment unstable or hard to configure.
No traceability between code and tests.
Regression runs take days; often skipped.

Why testability matters for regression: A system with low testability forces teams to run either too many tests (slow) or too few tests (risky). Improving testability is a direct investment in sustainable software quality.

Factors Affecting Regression Testability

Multiple dimensions of the system and the test infrastructure determine how testable the system is for regression purposes.

Controllability
The degree to which the system can be placed into a specific state for testing. High controllability means test preconditions can be set up reliably (e.g., seeding a database with known data). Low controllability means tests depend on unpredictable prior state.

Observability
The degree to which test outcomes can be determined from the system's outputs. If a module modifies a database record but provides no observable output, verifying its behaviour requires querying the database directly — adding test complexity.

Isolatability
The degree to which a component can be tested independently. High isolatability allows testing a single module with mocks/stubs for dependencies, enabling targeted regression without running the whole system.

Stability
The degree to which the test environment and test data remain consistent between runs. Unstable environments cause false positives (tests fail due to environment issues, not code issues), making regression results unreliable.

Understandability
How well the system's architecture and design are documented. Good documentation supports impact analysis: testers can determine which tests are relevant to a given change.

Automation-readiness
The degree to which the system can be driven by automated test tools. Systems without testable APIs, CLIs, or UI automation hooks are hard to automate, making high-frequency regression impractical.

Testability Assessment Checklist:

Factor	Assessment Question	Impact if Poor
Controllability	Can test preconditions be set up reliably?	Flaky, unrepeatable tests
Observability	Can expected outcomes be verified programmatically?	Manual verification bottleneck
Isolatability	Can individual modules be tested without the full system?	Long, slow end-to-end tests only
Stability	Does the environment produce consistent results across runs?	High false-positive rate; low trust in results
Understandability	Is traceability from code to tests documented?	Over-broad or under-targeted regression scope
Automation-readiness	Does the system expose an automatable interface?	Manual-only regression; timing constraints cause skipping

Impact Analysis

Impact analysis is the process of evaluating the consequences of a proposed or implemented change on the system, including which components are affected and which regression tests should be run as a result. It is the precursor to regression test selection.

ISTQB on Impact Analysis: ISTQB §5.2 states that impact analysis should be performed before maintenance testing (regression) to identify the extent of the change and the scope of testing required.

Impact Analysis Process

Identify the change: What was changed? (function, class, module, schema, configuration)
Identify direct dependencies: What other modules call or depend on the changed component?
Identify indirect dependencies: What modules depend on the direct dependents? (transitive impact)
Map to test cases: Which existing test cases exercise the affected components?
Assess risk: What is the business criticality of each affected area?
Define regression scope: Select the minimum test set that covers the risk with acceptable confidence.

Tools for impact analysis:

Dependency graphs: Visual maps of module-to-module dependencies.
Call graphs: Show which functions call which — direct impact visible at function level.
Version control diff: git diff identifies exactly what code changed.
Requirements Traceability Matrix (RTM): Maps code components to requirements to test cases.
Code coverage data: Shows which tests exercise which code paths — helps identify tests relevant to the changed code.

Example impact analysis result:

Changed Component	Direct Dependents	Indirect Dependents	Risk	Regression Scope
`discount_calculator()`	Checkout module, Cart summary	Order creation, Invoice generation	High (financial)	Full checkout + payment + order regression suite
UI colour theme config	Header component, Footer component	None (display only)	Low	Smoke test: verify page loads, spot-check layout

Designing for Regression Testability

Testability should be designed into the system from the start, not retrofitted. The following design principles directly improve regression testability:

Single Responsibility Principle
Each module or function has one clearly defined responsibility. Changes are localised, minimising transitive impact and making impact analysis simpler.

Loose Coupling
Modules interact through well-defined interfaces, not internal state. Changes to one module do not ripple unexpectedly into others. Stubs and mocks can replace dependencies in tests.

Dependency Injection
Dependencies are passed in rather than hard-coded. Allows tests to inject test doubles (mocks, stubs, fakes), enabling isolated module testing without full system setup.

Separation of Concerns
Business logic separated from UI and database access layers. Business logic can be unit-tested independently, enabling fast and reliable regression at unit level.

Idempotent test design
Tests produce the same result regardless of execution order or how many times they run. Each test sets up its own state and cleans up afterwards, preventing inter-test dependencies.

Deterministic test data
Tests use fixed, controlled data rather than random or shared production data. Reproducible data ensures reproducible results, making failure diagnosis straightforward.

Code example: Design for testability

# Poor testability: hard-coded dependencies
def calculate_discount(user_id):
    db = DatabaseConnection()  # hard to mock
    user = db.get_user(user_id)
    return 0.1 if user.tier == 'premium' else 0.0

# Good testability: dependency injection
def calculate_discount(user, discount_rules):
    return discount_rules.get_rate(user.membership_tier)
# Test can inject mock user and mock rules — no DB needed

Role of Test Automation in Regression Testability

Manual regression testing is unsustainable at the pace of modern software delivery. Test automation is the primary enabler of practical regression testability in all but the smallest projects.

Benefits of automated regression

Consistent, repeatable execution without human error.
Can run hundreds of tests in minutes.
Enables regression on every build (CI/CD).
Immediate failure notification to the team.
Frees testers for exploratory and high-value activities.

Limitations of automated regression

Tests must be maintained as code changes (test debt).
Cannot assess usability or visual appearance reliably.
Flaky tests erode trust in the suite over time.
Initial investment is high; not all scenarios are automatable.
Automation cannot cover what is not specified or programmed.

Test Automation Pyramid — and its impact on regression speed:

Layer	Test Type	Count	Speed	Regression Role
Base	Unit Tests	Many (100s–1000s)	Very fast (seconds)	First regression gate; catches logic errors immediately
Middle	Integration Tests	Moderate (10s–100s)	Fast (minutes)	Verifies interface contracts and data flows
Top	End-to-End Tests	Few (10s)	Slow (10s of minutes)	Smoke test of critical user journeys

Key insight: Most regression confidence should come from the base of the pyramid (fast unit tests), not the top (slow E2E tests). Inverting the pyramid creates a slow, fragile regression suite.

When to Run Regression Tests

Deciding when to run regression tests requires balancing thoroughness (more tests = more confidence) against time and cost (more tests = slower delivery). The following triggers and criteria guide this decision.

After a Bug Fix

Always. Run targeted regression covering the fixed area (Obj 1), related modules (Obj 2), and dependent modules (Obj 3). Scope: impact-analysis-driven.

After a New Feature

Always. Progressive tests cover new code; regression tests cover existing functionality that the new feature may have affected. Scope: integration points of new feature.

After Refactoring

Full regression of all refactored modules. Since no functional change is intended, any regression failure indicates the refactoring changed behaviour. Scope: all tests for touched modules.

After Environment Change

When OS, DB, library, or deployment configuration changes. Run a representative regression suite to detect compatibility regressions. Scope: functional smoke + known compatibility-sensitive areas.

Before a Major Release

Full regression suite run against the release candidate. Acts as final quality gate before production deployment. Scope: entire critical-path regression suite.

On Every CI Build (Automated)

In CI/CD pipelines, automated unit + integration regression runs on every commit. E2E regression runs on merge to main branch. Scope: tiered (fast suite every commit; full suite at merge).

Decision framework: Should we run regression now?

Factor	Run More Regression If…	Run Less Regression If…
Change size	Large change touching many modules	Trivial localised fix (e.g., UI label correction)
Business risk	Critical module (payments, authentication)	Low-risk feature (reporting display)
Release proximity	Near production release date	Early in sprint; many changes still ahead
Change frequency	First change in a stable area (unexpected risk)	Frequently changed area (already regression-tested recently)
Coupling	Changed component is a shared utility	Changed component is a standalone leaf module

Regression Frequency by Development Context

The appropriate frequency of regression testing varies dramatically by development model:

Development Context	Regression Frequency	Typical Scope	Automation Required?
Waterfall	At each major milestone or before release	Full regression suite	Optional (manual feasible)
Agile / Scrum	Every sprint (1–2 weeks)	Sprint scope + risk-based full suite	Strongly recommended
Continuous Integration	Every commit / build (multiple per day)	Unit + integration regression (fast)	Mandatory
Continuous Delivery/Deployment	Every commit (automated gate to production)	Full automated suite must pass before deploy	Mandatory
Maintenance / Legacy	After each patch or change request	Impact-analysis-driven; often manual	Beneficial where feasible

Scope & Depth Decisions

Not all regression cycles need the same depth. Choosing the appropriate scope is a risk-based decision:

Smoke Regression (Sanity Check)
A small set of critical tests covering the most important user journeys. Executed first to determine if the build is stable enough for further testing. If smoke tests fail, deeper regression is blocked.
Typical size: 20–50 tests. Duration: 5–15 minutes.

Targeted Regression (Change-Impact Driven)
Tests selected by impact analysis to cover changed modules and their direct dependents. Most common daily regression scope in Agile environments.
Typical size: 10–30% of full suite. Duration: proportional to impact scope.

Full Regression
The entire regression test suite is executed. Used before major releases, after large-scale changes, or at defined milestones. Provides maximum confidence but maximum time cost.
Typical size: 100% of suite. Duration: hours to days without automation.

Choosing regression scope — risk matrix:

Change Size	Low Business Risk	High Business Risk
Small / Localised	Smoke + targeted	Targeted + extended
Medium / Moderate	Targeted + adjacent modules	Full regression of critical paths
Large / Cross-cutting	Full regression	Full regression + exploratory

Regression in CI/CD Pipelines

Continuous Integration and Continuous Delivery represent the highest-frequency regression context. Understanding how regression fits into these pipelines is essential for modern testers.

Typical CI/CD Regression Pipeline

Developer commits code → CI server detects commit.
Build triggered → Code compiled / dependencies resolved.
Unit test regression → All unit tests run (fast, <5 min). Build fails if any unit test fails.
Integration test regression → Integration tests run against the built artifact (15–30 min).
Code merged to main branch → Triggers E2E regression suite (slower, 30–60 min).
All regression gates pass → Artifact promoted to staging environment.
Pre-release full regression → Comprehensive suite on staging before production push.

Key CI/CD regression principles:

Fail fast: Fast tests run first; slow tests gate later pipeline stages.
Zero tolerance for failures: A failed regression gate blocks the pipeline; no manual override without explicit sign-off.
Parallel execution: Tests run in parallel across multiple agents to reduce elapsed time.
Test quarantine: Flaky tests are quarantined (not deleted) while investigated, to prevent false failures from blocking delivery.
Regression baseline: The "green" state of the main branch is the regression baseline. Any commit that makes previously green tests fail is a regression and must be fixed or reverted.

Worked Example: Banking Application Change

System: A retail banking application with modules: Account Management, Transaction Processing, Interest Calculation, Loan Servicing, Reporting, and Customer Notifications.

Change: The interest calculation formula for savings accounts has been updated to use a tiered rate structure (previously flat rate).

Step 1: Assess Testability

Testability Factor	Assessment	Action Required
Controllability	Good — test DB can be seeded with specific balances	None
Observability	Good — interest calculation returns computed value	None
Isolatability	Moderate — calculation module has DB dependency	Use repository mock to isolate
Stability	Poor — interest rates change frequently in config	Pin test config to fixed rate values
Automation-readiness	Good — REST API available for all operations	None; existing API tests can be extended

Step 2: Impact Analysis

Direct impact: Interest Calculation module, Account Management (displays accrued interest), Reporting (interest income reports).
Indirect impact: Customer Notifications (interest credit notification email), Loan Servicing (if savings are used as collateral).
Risk level: High (financial calculation; regulatory compliance).

Step 3: Regression Scope Decision

Layer	Tests to Run	Justification
Unit	All unit tests for `interest_calculator.py`	Directly changed module; all logic paths must pass
Integration	Integration tests: Interest Calc ↔ Account Mgmt; Interest Calc ↔ Reporting	Direct dependents identified in impact analysis
System	End-to-end: interest calculation scenario + notification scenario	High business risk; regulatory compliance requires E2E verification
Skipped	Transaction Processing, Loan Servicing unit tests (no connection to formula)	No dependency on the formula; impact analysis showed no risk

Step 4: Timing Decision
Given high business risk, regression runs immediately after the developer's unit tests pass. Does not wait until end of sprint. A second full regression run is planned 48 hours before the release date as the final quality gate.

Common Mistakes

No impact analysis before selecting tests
Running the same fixed regression suite regardless of change type wastes time (running irrelevant tests) and creates false security (real risks may not be covered).

Treating all changes as equally risky
A UI text change and a payment processing algorithm change do not deserve the same regression depth. Risk-based scoping is essential.

Ignoring testability until regression fails
Testability problems (flaky tests, environment instability) are discovered when regression results are needed most. Testability should be monitored continuously.

Running E2E tests for every commit
E2E tests are slow. Running them on every commit creates hour-long feedback loops. Fast unit/integration regression should be the daily gate; E2E tests gate merges.

Not maintaining the regression suite
Tests that reference deleted features, outdated APIs, or changed data schemas produce false failures. Regression suites require active maintenance alongside the code they test.

Equating "tests passed" with "no regressions"
A passing regression suite only proves the tested scenarios still work. Untested scenarios may contain regressions. Coverage analysis is needed to quantify the confidence level.

Class Activity

Scenario: Your team maintains an online learning platform with these modules: User Registration, Course Catalogue, Video Streaming, Quiz Engine, Certificate Generation, and Payment Processing. The following changes are made in a single sprint:

Bug fix in Quiz Engine: Multiple-choice answer validation was including the wrong answer as correct in certain question configurations.
New feature: Payment Processing now supports UPI payments (in addition to existing credit/debit card).
Refactoring: Certificate Generation module was restructured to use a new PDF library.

Tasks (in groups of 3):

For each change, perform a brief impact analysis: identify direct and indirect affected modules.
Decide the regression scope for each change (smoke / targeted / full). Justify your decision using the risk matrix.
Rank the 3 changes in order of regression urgency and justify the ranking.
Identify 2 testability factors that are at risk in this system and suggest one improvement action for each.

Evaluation Rubric (10 marks)

3 marks: Impact analysis identifies correct direct and indirect dependents for all 3 changes.
3 marks: Regression scope decisions are consistent with the risk matrix and justified.
2 marks: Urgency ranking is logical and well-argued.
2 marks: Testability factors are correctly identified and improvement actions are actionable.

Exit Ticket

Define regression testability in your own words and name two factors that determine it.
What is impact analysis and why must it precede regression test selection?
In a CI/CD pipeline, at which stage should unit regression tests run, and why should they run before E2E tests?
A team is debating whether to run a full regression suite after fixing a single typo in an error message. What decision framework would you apply and what would you recommend?

Summary & Assignment

Regression testability determines whether a team can run meaningful regression tests efficiently enough to keep pace with software change. Impact analysis transforms the "run everything" default into a targeted, risk-proportionate approach. CI/CD environments make automation mandatory and turn regression into a continuous, pipeline-embedded activity rather than a periodic event.

Key takeaways:

Testability factors (controllability, observability, isolatability, stability, automation-readiness) directly determine regression efficiency.
Impact analysis should precede every regression cycle to define scope proportional to risk.
Regression scope decisions follow a risk matrix: change size × business risk.
CI/CD pipelines require tiered regression (fast unit gate on commit; full suite on merge/release).

Assignment:

Assess the regression testability of your mini-project using the 6-factor checklist from this session. Rate each factor (High / Medium / Low) and identify the most critical improvement action.
Choose one recent change in your mini-project and perform a written impact analysis: changed component → direct dependents → indirect dependents → risk rating → regression scope decision.
Design a 3-tier regression pipeline for your project (unit, integration, E2E) specifying which tests run at each tier and the expected duration of each tier.

Regression Testability & Timing

Learning Objectives