Session 5.2 - Why Test Suites Grow

The Test Suite Growth Problem

A test suite starts small and purposeful. Over the life of a product, it grows. Some growth is necessary — new features require new tests. Some growth is wasteful — redundant cases accumulate without adding coverage value. Unmanaged, the two types are indistinguishable inside the suite.

The fundamental tension:

Growth Type	Driver	Coverage Effect	Suite Effect
Necessary growth	New features, new requirements, new risks	Increases coverage	Suite grows — justified
Accidental growth	Redundancy, poor design, process failures	No increase in coverage	Suite grows — wasteful

Industry evidence: Studies of mature software products (Chowdhury, Chapter 9) show that test suites left unmanaged for 3–5 years typically exhibit:

20–40% of cases testing the same partition as another case (data-clone redundancy).
5–15% of cases testing features that no longer exist in the product (orphaned cases).
10–25% of cases with overlapping structural coverage (subsumption redundancy).
Execution time growing 3×–10× faster than the number of features.

Primary Growth Drivers

Five distinct drivers cause test suite growth. The first three are mostly legitimate; the last two are mostly wasteful:

1. Feature Creep

Each new feature requires new test cases. If the product grows, the suite should grow proportionally — but often over-proportionally due to poor scoping.

2. Bug-Fix Accretion

Every confirmed defect prompts a new regression test case. Over years, thousands of defect-specific cases accumulate — many covering the same boundary.

3. Compliance Coverage

Audits, certifications (ISO, SOC2), and regulatory requirements mandate explicit test evidence. Cases are added to satisfy audit checklists regardless of coverage overlap.

4. Process & Org Failures

No review gate before adding cases; multiple teams adding overlapping cases; cases inherited from merged projects; no traceability to prevent duplicate creation.

5. Poor Design

Anti-patterns from Session 5.1: omnibus cases split without pruning redundancy, data-clone cases, happy-path repetition. Each creates immediate redundancy.

Driver 1: Feature Creep

Feature creep is the most natural and mostly legitimate driver of suite growth. Each product increment adds new requirements that must be tested.

Legitimate growth: A product ships with 50 features and 500 tests. Version 2.0 adds 20 new features. Adding 200 new tests (proportional growth) is fully justified.

Where feature growth becomes wasteful:

Over-specification of new features — A new feature is specified with 30 test cases when EP + BVA + one decision table yields 8. The extra 22 are data-clone redundancy created at the point of feature delivery.

No impact analysis on existing cases — A new feature modifies shared behaviour (e.g., a shared calculation function). Existing tests for that function remain unchanged even though some are now obsolete or overlap with the new feature's tests.

Feature scope expansion mid-sprint — A feature grows during development but test scope is not recalculated. Cases written for the original spec plus cases written for the extended spec coexist, many overlapping.

Control discipline: Apply the quality checklist from Session 5.1 at feature delivery. Before merging new cases, verify: does each case cover a partition not already covered by an existing case? If not, the case should not be added.

Driver 2: Bug-Fix Accretion

Every confirmed defect should produce at least one regression test case to prevent recurrence. Over a product's lifetime, thousands of defects are fixed and thousands of cases accumulate.

Year 1: Product ships with 300 cases. 80 defects reported; 80 regression cases added. Suite: 380 cases.

Year 2: 120 more defects. 120 regression cases added. Suite: 500 cases.

Year 3: 90 more defects. 90 regression cases added. However, 40 of these cover boundaries already covered by Year 1 cases. Suite: 590 cases (+40 redundant).

Year 5: Suite is at 800 cases. Analysis reveals: 200 cases cover the same 15 boundary values as the original 300 cases, just with different defect-report IDs attached.

Why this happens:

The engineer fixing the defect writes a new test case but does not search whether an existing case already covers the same boundary.
Defect tracking and test case management systems are not integrated: no query like "does a test case already exist for input X = boundary value Y?"
Different defect descriptions lead to different-looking test cases that exercise identical code paths.

Control discipline: Before writing a bug-fix regression case, run a coverage query: which existing test cases exercise the code path affected by this fix? If an existing case already covers the boundary, update its expected result or preconditions rather than adding a duplicate.

Driver 3: Compliance Coverage

Regulatory standards, certifications, and audit requirements mandate that certain behaviours are explicitly tested and documented. This is a legitimate driver but often creates coverage overlap.

Common compliance mandates and their test impact:

Standard / Requirement	Test Mandate	Overlap Risk
PCI DSS (payment card security)	Explicit test evidence for card data encryption, access control, audit logging	High: overlaps with existing security and functional tests
ISO 9001 (quality management)	Traceability matrix linking each requirement to at least one test case	Medium: may force creation of cases that duplicate existing coverage
GDPR / Data Privacy	Explicit test evidence for data deletion, consent withdrawal, data export	Medium: may duplicate existing user-management tests
DO-178C (avionics software)	MC/DC coverage evidence for all safety-critical conditions	Low: MC/DC is usually more stringent than existing cases
Internal audit checklists	Line-item test cases for each checklist item	Very High: checklists are often written without checking existing coverage

Management approach: Tag compliance-mandated test cases with the relevant standard (e.g., tag:pci-dss). When adding a compliance case, first search for existing cases with the same coverage. If an existing case covers the requirement, add the compliance tag to the existing case rather than creating a new one. This satisfies the audit without adding redundancy.

Driver 4: Process & Organisational Failures

These drivers are entirely wasteful — they add cases that provide no additional coverage value:

No entry gate for test cases
Developers and testers add cases directly to the suite with no review. Redundancy accumulates unchecked.

Multiple teams, no coordination
Team A and Team B both write tests for a shared authentication module. Neither knows the other's tests exist.

Inherited suites from mergers / acquisitions
Two product suites are merged. Cases from both products are combined without deduplication. Overlap is often 30–50%.

Copy-paste proliferation
A tester copies TC-001 to create TC-002 and changes only the test data. If the data is in the same equivalence class, TC-002 is pure redundancy.

Fear of deletion
"We might need this case later." Teams are reluctant to remove cases they did not write. The suite grows monotonically; it never shrinks.

Metric gaming
Teams are rewarded for the number of test cases created. More cases = higher score = more redundancy.

Driver 5: Poor Design

Design anti-patterns (covered in Session 5.1) create immediate redundancy at the moment cases are written. This is the most preventable driver:

Anti-Pattern	How it Inflates the Suite	Typical Inflation Factor
Data-clone cases	Same partition tested with 3–10 different but equivalent input values	3×–10×
Omnibus case splitting without pruning	One 10-objective case is split into 10 cases, but the original is also kept	2×
Happy-path-only suites with later gap-fill	Invalid partition cases are added by different testers without checking for overlap	1.5×–2×
No equivalence class awareness	Testers add cases intuitively; multiple cases hit the same partition by coincidence	2×–4×
Missing expected result specs	Vague results make it impossible to deduplicate; cases that look different may test identical behaviour	Indirect: enables all other types

Consequences of Uncontrolled Growth

Growth is not a cosmetic problem. Each additional case has a recurring cost across every regression cycle:

Execution Time

If a suite of 500 cases runs in 2 hours, a suite of 1,500 cases (3× growth, 60% redundant) runs in approximately 6 hours — with no additional defects found. In CI/CD, this breaks the fast-feedback loop.

Impact: Critical

Maintenance Cost

Every product change that affects a feature or API must be propagated to all test cases for that area. With 3× redundancy, maintenance effort for that area is 3× higher. A single API signature change may break 90 redundant cases instead of 30 unique ones.

Impact: High

Failure Noise

When a defect is introduced, it may cause 20 redundant cases to fail instead of 2. Teams spend time triaging 20 failures that all point to the same root cause. Redundant failures dilute the signal-to-noise ratio.

Impact: High

Cognitive Load

A suite of 2,000 cases is impossible for a tester to understand holistically. It becomes a black box. Teams stop reasoning about coverage and start adding cases by intuition, accelerating the redundancy problem.

Impact: Medium-High

Execution time growth model:

Year	Suite Size	Execution Time	New Defects Found vs Year 1
Year 1	300 cases	1.0 hr	Baseline (100%)
Year 2	500 cases	1.7 hr	+15%
Year 3	800 cases	2.7 hr	+8%
Year 4	1,200 cases	4.0 hr	+4%
Year 5	1,800 cases	6.0 hr	+2%

The suite grew 6× over 5 years; defect-finding yield increased by only 29% total. Each additional case after Year 2 had diminishing returns on defect detection.

A Simple Growth Model

We can model suite growth as a combination of necessary growth and accidental growth:

Growth decomposition:

Suite size at time T = Initial cases + Necessary cases(T) + Accidental cases(T)

Where:

Necessary cases(T) = cases added for new features + cases added for legitimate compliance + non-redundant bug-fix regression cases
Accidental cases(T) = data-clone redundancy + subsumption redundancy + orphaned cases (feature removed) + process-failure duplicates

Practical implication: If you can measure the ratio Accidental / Total, you know the potential gain from minimization (Session 5.3). A suite where 35% of cases are accidental can be reduced to 65% of its current size with no loss of coverage.

Redundancy types to measure:

Redundancy Type	Definition	Detection Method
Data-clone redundancy	Two cases test the same partition with different but equivalent input values	Partition mapping; compare input values against EP classification
Subsumption redundancy	Case A's coverage is a strict subset of Case B's coverage	Coverage matrix analysis; structural subsumption check
Duplicate cases	Two cases are identical in preconditions, steps, data, and expected result	Hash comparison of case attributes
Orphaned cases	Case traces to a requirement that no longer exists in the product	Traceability audit: requirement exists? feature still shipped?

Diagnosing Your Suite

Before minimization (Session 5.3) can be applied, the suite must be diagnosed. A four-step diagnosis process:

Step 1: Traceability audit — For every test case, check: does a live requirement or user story exist that it traces to? Cases without a traceability link or with links to removed requirements are orphan candidates.

Step 2: Coverage overlap analysis — Run all cases through a coverage tool. Identify pairs of cases that cover identical sets of code statements or branches. One of each pair is a subsumption-redundancy candidate.

Step 3: Partition classification — Map each case's test data to the equivalence partitions of its target feature. Cases in the same partition with the same expected result are data-clone candidates.

Step 4: Last-failure audit — Query the test execution history: when did each case last fail due to a product defect (not a maintenance issue)? Cases that have not found a defect in 24+ months and cover the same partition as cases that have are low-value candidates for removal.

Case classification output:

Class	Definition	Action
Value-Adding	Unique partition coverage; active traceability link; found defect within 24 months	Keep; include in every regression cycle
Low-Value	Unique partition coverage but no defect found in 24+ months; or compliance-only tag	Keep but consider deprioritizing; include in release gates, not CI gates
Redundant	Duplicate, data-clone, or subsumed by another case; orphaned traceability	Minimization candidate: archive and verify before permanent removal

Worked Example: E-Learning Platform Suite Audit

System: An e-learning platform with modules: Authentication, Course Catalogue, Video Player, Quiz Engine, Results, Certificates, and Payments.

Current suite: 650 cases built over 4 years. The team suspects significant redundancy.

Traceability audit results:

Finding	Cases Affected	Root Cause
No traceability link	72 cases (11%)	Added by contractors; link not required at that time
Traces to removed feature ("Flash Video Player")	45 cases (7%)	Feature removed 2 years ago; cases never deleted
Traces to requirement changed but case not updated	31 cases (5%)	Requirement changed; no case-update workflow triggered

Coverage overlap analysis (Quiz Engine module, 140 cases):

Finding	Cases	Driver
Data-clone cases (same timer boundary, different quiz names)	28 cases covering 4 unique boundaries	Bug-fix accretion: each timer defect produced a new case
Subsumed cases (shorter path already covered by longer path case)	17 cases	Feature creep: new scenarios added without impact analysis
Orphaned (Flash quiz player tests)	22 cases	Feature removal without case cleanup

Summary:

Classification	Cases	% of Suite
Value-Adding	387	59.5%
Low-Value (compliance-only, no recent failure)	76	11.7%
Redundant / Orphaned	187	28.8%

Implication: The suite could be reduced from 650 to approximately 463 cases (removing the 187 redundant/orphaned cases) with no loss of coverage — a 29% reduction in execution time and maintenance effort.

Common Mistakes

Treating all growth as necessary
Teams often assume that a larger suite means better coverage. Growth is only justified if new cases cover new partitions, new boundaries, or new risks. Redundant growth adds cost, not quality.

No case retirement process
Cases are added but never removed. A formal retirement process (traceability audit + coverage verification + archive) is as important as the case creation process.

Diagnosing without historical execution data
Identifying low-value cases requires knowing when each case last failed. Without execution history (stored in CI), diagnosis is guesswork. Invest in execution telemetry from day one.

Mixing redundancy removal with feature deletion
Removing a redundant case is safe. Removing a case because "the feature seems obvious" without coverage verification is dangerous. The two must be separate processes.

Class Activity

Scenario: The following 10 test cases exist in the Authentication module of a food-delivery app. Analyse them for growth drivers and classify each:

ID	Description	Last Defect Found	Traceability
TC-A01	Login with valid email and password	18 months ago	REQ-AUTH-01
TC-A02	Login with valid email and password (different user: bob@food.com)	Never	REQ-AUTH-01
TC-A03	Login with empty password field	6 months ago	REQ-AUTH-02
TC-A04	Login with wrong password (3 attempts → account lock)	12 months ago	REQ-AUTH-03
TC-A05	Login via Facebook OAuth	Never	REQ-AUTH-08 (removed)
TC-A06	Login with email containing uppercase (ALICE@FOOD.COM)	3 months ago	REQ-AUTH-01
TC-A07	Login with email exactly 60 chars (boundary)	9 months ago	REQ-AUTH-04
TC-A08	Login with email 61 chars (above boundary)	9 months ago	REQ-AUTH-04
TC-A09	Login with valid email and password (different user: carol@food.com)	Never	REQ-AUTH-01
TC-A10	Session token expires after 30 minutes of inactivity	2 months ago	REQ-AUTH-09 (PCI DSS)

Tasks:

Classify each growth driver (feature creep, bug-fix, compliance, process failure, poor design) for each case where applicable.
Classify each case as Value-Adding, Low-Value, or Redundant. Justify each classification.
For cases you classify as redundant, identify which specific redundancy type applies (data-clone, subsumption, orphaned, duplicate).
What is the minimum set of cases that maintains full coverage? List the IDs.
Which organisational process failure is most evident in this suite? Propose one process change that would have prevented it.

Exit Ticket

Name all five drivers of test suite growth. Which two are entirely wasteful? Which two are mostly legitimate?
A suite grows from 200 to 1,000 cases over 3 years. The product added 15 new features. Approximately how many of the 800 new cases are likely necessary vs. accidental? What additional information would you need to answer precisely?
What is the difference between data-clone redundancy and subsumption redundancy? Give a one-sentence definition of each.
Why is execution history (when each case last found a defect) essential for suite diagnosis?

Summary & Assignment

Test suites grow for five reasons: feature creep, bug-fix accretion, compliance coverage, process failures, and poor design. The first three are mostly legitimate; the last two are entirely wasteful. Uncontrolled growth inflates execution time, maintenance cost, and failure noise without proportionally improving defect detection. A diagnosis framework (traceability audit + coverage overlap + partition mapping + failure history) classifies cases as value-adding, low-value, or redundant, and quantifies the potential gain from minimization.

Key takeaways:

Not all growth is necessary: distinguish legitimate from accidental growth before deciding to minimise.
Bug-fix accretion is the most insidious driver: each case is individually justified but collectively redundant.
Compliance cases can be tags on existing cases rather than new cases — satisfy the audit without adding redundancy.
Redundancy types: data-clone, subsumption, exact duplicate, orphaned. Each requires a different detection method.
Suite diagnosis requires execution history: a case that has never found a defect and covers the same partition as one that has is a candidate for removal.

Assignment (Session 5.2):

For your mini-project's test suite, perform a traceability audit. Report: how many cases lack a traceability link? How many trace to requirements that have changed or been removed?
Identify the three most significant growth drivers in your suite. For each, provide 2–3 specific examples of cases that were added because of that driver.
Classify every case in your suite as Value-Adding, Low-Value, or Redundant using the four-step diagnosis framework. Report the count and percentage in each category.
Estimate the execution-time reduction you would achieve by removing all redundant cases. Express this as a percentage and as an absolute time saving per CI run.

Why Test Suites Grow

Learning Objectives