Session 5.7 - Measuring Prioritized Suite Effectiveness

Why Measure Prioritized Suite Effectiveness?

Prioritization is an investment: it requires calculation time, tooling, and ongoing recalculation. Without measurement, there is no evidence that the chosen ordering is better than random. Metrics are needed to:

Validate the ordering
Confirm that the chosen technique (general, version-specific, combined) actually detects faults earlier than an unprioritized or randomly ordered suite.

Compare techniques
Objectively decide whether additional-coverage greedy outperforms FER-only ordering for this specific product and history.

Track improvement over time
Monitor whether APFD is improving or degrading as the suite evolves. A falling APFD indicates the prioritization model needs updating.

Communicate to stakeholders
Provide a single number that answers: "How good is our test ordering?" Easier to communicate than raw pass/fail counts.

APFD: Average Percentage of Faults Detected

APFD (Elbaum et al., 2002)

What it measures: The speed at which a prioritized test suite detects faults, expressed as the area under the fault-detection curve (FDC). An APFD of 1.0 means all faults are detected immediately (ideal). An APFD of 0.5 means faults are detected at the same rate as a random ordering (the baseline). An APFD below 0.5 means the ordering is worse than random.

Intuition: Imagine running the test suite in order. Plot the percentage of total faults detected (y-axis) vs. the percentage of tests run (x-axis). APFD is the area under this curve divided by the total possible area (1.0).

APFD Formula:
APFD = 1 − (TF₁ + TF₂ + … + TF_m) / (n × m) + 1 / (2n)

Formula notation:

n = total number of test cases in the suite
m = total number of faults seeded/present in the version under test
TF_i = the position (1-indexed) in the ordered suite of the first test case that detects fault f_i
∑TF_i = sum of the first-detection positions for all m faults

Special cases:

Ideal ordering: Every fault is detected by the first test case (TF_i = 1 for all i). APFD = 1 − m/(n×m) + 1/(2n) = 1 − 1/n + 1/(2n) ≈ 1.0 for large n.
Worst ordering: Every fault is detected by the last test case (TF_i = n for all i). APFD = 1 − n×m/(n×m) + 1/(2n) = 1/(2n) ≈ 0.0 for large n.
Random ordering: Expected TF_i ≈ n/2. APFD ≈ 0.5. This is the no-skill baseline.

APFD Worked Example

Suite: 5 test cases (n=5). Faults: 3 faults (m=3): F1, F2, F3.

Each test case detects the following faults:

Fault detection matrix:

Position	Test Case	Detects F1?	Detects F2?	Detects F3?
1	TC-A	Yes	No	No
2	TC-B	No	No	Yes
3	TC-C	No	No	No
4	TC-D	No	Yes	No
5	TC-E	No	No	No

First-detection positions: TF_F1 = 1 (detected at position 1); TF_F2 = 4; TF_F3 = 2.

APFD = 1 − (1 + 4 + 2) / (5 × 3) + 1 / (2 × 5)

= 1 − 7/15 + 1/10 = 1 − 0.4667 + 0.1000 = 0.633

APFD = 0.633 means this ordering detects faults 26.6% faster than a random ordering (0.5 baseline). All three faults are detected; the ordering accelerates early detection of F1 and F3.

Comparing with an alternative ordering:

Position	Test Case	F1	F2	F3
1	TC-D	No	Yes	No
2	TC-C	No	No	No
3	TC-E	No	No	No
4	TC-A	Yes	No	No
5	TC-B	No	No	Yes

TF_F1 = 4; TF_F2 = 1; TF_F3 = 5.

APFD = 1 − (4+1+5)/(5×3) + 1/(2×5) = 1 − 10/15 + 0.1 = 1 − 0.667 + 0.1 = 0.433

Conclusion: Ordering 1 (APFD=0.633) is significantly better than Ordering 2 (APFD=0.433). Ordering 2 is actually slightly worse than random (0.5). The metric provides an objective basis for choosing between orderings.

Interpreting APFD Values

APFD Range	Interpretation	Action
< 0.50	Ordering is worse than random. High-priority faults are detected late.	Investigate immediately: priority scores may be inverted or stale
0.50 – 0.60	Slight improvement over random. Prioritization provides minimal benefit.	Review priority scoring model; consider if weights need adjustment
0.60 – 0.70	Moderate improvement. Prioritization is working but not optimally.	Acceptable; monitor trend over time
0.70 – 0.85	Good prioritization. Early fault detection is substantially better than random.	Good state; maintain and recalibrate periodically
> 0.85	Excellent prioritization. Near-ideal fault detection speed.	Validate that measurements are correct; exceptional results may indicate overfitting to past fault patterns

APFD limitations:

Treats all faults equally: A critical payment fault and a minor UI spacing fault contribute equally to APFD. Cost-weighted APFD (APFD_c) addresses this.
Requires known fault locations: Computing APFD requires knowing which test cases detect which faults. In practice, this requires fault seeding, mutation testing, or retrospective analysis of historical failures.
Does not account for test execution cost: A suite where all faults are detected early but each test takes 20 minutes may have high APFD but poor practical performance. APFD_c incorporates execution cost.
Sensitive to suite size: For very small suites (n < 10), the 1/(2n) correction term dominates and APFD values are less meaningful.

APFD_c: Cost-Aware Extension

APFD_c (Elbaum et al., 2002)

Motivation: Basic APFD assumes each test case takes the same amount of time to execute. In reality, execution times vary dramatically. A case that detects a fault at position 3 but takes 30 minutes should be valued differently from one at position 3 that takes 1 minute.

APFD_c = (∑_i=1^m (cost_detected_i − cost(TF_i)/2)) / (total_cost × m)

Notation:

cost(t_j) = execution time of test case t_j
cost_detected_i = cumulative execution time up to and including the test that first detects fault f_i
total_cost = total execution time of the entire suite
m = total number of faults

APFD_c worked micro-example:

5 cases, 2 faults. Execution times: TC-A=2min, TC-B=10min, TC-C=1min, TC-D=5min, TC-E=3min. Total = 21 min.

Ordering: TC-A, TC-B, TC-C, TC-D, TC-E. F1 first detected by TC-A; F2 first detected by TC-D.

cost_detected_F1 = 2 (TC-A). cost(TF_F1) = 2. Contribution = 2 − 2/2 = 1.
cost_detected_F2 = 2+10+1+5 = 18 (TC-D). cost(TF_F2) = 5. Contribution = 18 − 5/2 = 15.5.

APFD_c = (1 + 15.5) / (21 × 2) = 16.5 / 42 = 0.393. The high cost of TC-B (10 min, detects no faults) penalizes this ordering. If TC-B were moved to the end (after TC-D), APFD_c would improve significantly.

Time-to-First-Failure (TTFF)

TTFF

Definition: The wall-clock time elapsed from the start of the test run until the first test failure is observed (assuming at least one fault is present in the version under test).

Formula: TTFF = ∑_j=1^{pos(first_failure)} exec_time(t_j)

Where pos(first_failure) is the position in the ordered suite of the first test that fails.

Why it matters: TTFF directly measures how quickly a developer can begin investigating a defect. In CI pipelines, TTFF determines how quickly the pipeline can be aborted and the developer notified. A lower TTFF means faster developer feedback.

TTFF vs. APFD:

Metric	Measures	Best For
APFD	Overall fault detection speed across all faults	Comparing two orderings holistically; research benchmarking
TTFF	How quickly the first fault is detected	CI pipeline abort-early logic; developer notification latency

A suite can have high APFD (good average detection speed) but poor TTFF (first fault detected late) if all faults happen to be concentrated in later tests. Both metrics together give a more complete picture.

Rate of Fault Detection (RFD)

RFD Curve

Definition: The Rate of Fault Detection (RFD) is the slope of the fault-detection curve at each point in the test run. It describes how the rate of fault detection changes as tests execute — not just the cumulative total.

RFD_k = (Faults detected by tests 1–k) − (Faults detected by tests 1–(k-1))

A good prioritization strategy produces an RFD curve that is front-loaded: high detection rate early in the run, tapering off as the high-priority tests exhaust the available faults.

RFD curve shapes and their interpretation:

Curve Shape	Interpretation
Steeply front-loaded (high early RFD, rapid decline)	Excellent prioritization: high-value tests run first; all faults detected early
Uniform / flat	Ordering is essentially random: no prioritization benefit
Back-loaded (low early RFD, spike at end)	Inverted priority: low-value tests run first; equivalent to reverse ordering
Bimodal (spike at start, flat middle, spike at end)	Two clusters of fault-detecting tests; version-specific P1 cases and general P1 cases are separated by low-value tests in the middle

Metric Comparison Summary

Metric	Formula Complexity	Data Required	Best Insight	Key Limitation
APFD	Low	Fault-detection positions	Overall ordering quality	Ignores execution cost; treats all faults equally
APFD_c	Medium	Fault-detection positions + exec times + fault severity	Cost-adjusted ordering quality	Requires accurate execution time data
TTFF	Very Low	First failure time	CI fast-feedback; developer wait time	Only measures first fault; ignores remaining faults
RFD	Low	Per-test fault detection count	Shape of detection curve; identifies ordering anomalies	Qualitative; not a single summary number

Recommended metric set by context:

CI pipeline monitoring: TTFF (primary) + APFD (trend tracking).
Comparing prioritization algorithms: APFD (primary) + APFD_c if execution times vary significantly.
Diagnosing a poor-performing suite: RFD curve (to identify where in the run faults cluster).
Safety-critical release gates: APFD_c weighted by fault severity (high-severity faults detected early are worth more).

Benchmarking Two Prioritization Orderings

The standard benchmarking process for comparing two orderings A and B:

Step 1: Define the comparison set — Choose the suite, version, and fault set (seeded faults or historical failure data from N recent runs). Both orderings use the same cases and faults.

Step 2: Calculate APFD for each ordering — Using the formula. If execution times differ significantly across cases, also calculate APFD_c.

Step 3: Calculate TTFF for each ordering — Report wall-clock time to first failure under each ordering.

Step 4: Statistical significance — For production decisions, repeat across multiple fault sets (different versions) and test for statistical significance (Wilcoxon signed-rank test is standard for non-parametric comparisons). A single APFD measurement may be influenced by the specific fault set used.

Step 5: Interpret and decide — Prefer the ordering with higher APFD (and APFD_c if applicable). APFD difference > 0.05 is generally considered practically significant. Smaller differences may not justify the overhead of the more complex algorithm.

Module 5 Capstone: The Complete Suite Management Lifecycle

Module 5 has built a complete test suite management lifecycle. This capstone ties all sessions together:

Session	Activity	Output
5.1 Design Foundations	Apply EP, BVA, decision tables, white-box heuristics	Well-formed, non-redundant initial test suite
5.2 Why Suites Grow	Diagnose growth drivers; classify cases	Redundancy classification: value-adding / low-value / redundant
5.3 Minimization	Apply greedy/ILP; safe archive process	Minimized suite T′ with full coverage
5.4 Benefits	Quantify ROI; present to stakeholders	ROI model; execution-time and maintenance savings
5.5 Prioritization Goals	Define goal mix; build priority score model	Weighted priority scores for each case
5.6 Prioritization Types	Apply general + version-specific algorithms	Ordered suite: version-specific P1 + general P2–P4
5.7 Measurement (this session)	Calculate APFD, APFD_c, TTFF; compare orderings	Validated effectiveness metrics; evidence for ordering choice

The lifecycle in one diagram:

Design right (5.1) → Prevent redundancy at creation time

Diagnose (5.2) → Identify what has accumulated since last governance

Minimise (5.3–5.4) → Permanently remove redundancy with safe process; quantify ROI

Prioritise (5.5–5.6) → Order minimized suite for maximum early fault detection

Measure (5.7) → Validate ordering with APFD/TTFF; feed measurements back to improve scoring model

The lifecycle repeats. Measurement outcomes inform the next design and prioritization cycle. A well-governed suite improves continuously rather than degrading over time.

Common Mistakes

Calculating APFD once and never again
APFD is only meaningful when calculated regularly (per sprint or release). A single historical APFD tells you how good the ordering was; regular measurement tells you whether it is improving or degrading.

Treating APFD as a coverage metric
APFD measures fault detection speed, not coverage. A suite with high APFD and 50% branch coverage is good at detecting known faults early but may miss faults in uncovered branches entirely. APFD and coverage are orthogonal.

Using APFD to compare suites of different sizes
APFD is sensitive to suite size (the 1/(2n) term). Comparing APFD across suites of significantly different sizes (e.g., before and after minimization) requires careful interpretation. APFD_c is more robust for cross-suite comparison.

Not accounting for fault severity in safety-critical contexts
Basic APFD weights all faults equally. For safety-critical systems, a weighted APFD where high-severity faults contribute more to the score is essential. Using unweighted APFD may optimise for early detection of minor faults while delaying critical ones.

Class Activity

Scenario: A healthcare appointment booking system has 6 test cases (n=6) and 4 faults (m=4): F1 (booking conflict), F2 (payment failure), F3 (notification not sent), F4 (audit log missing). Two orderings are being compared:

Position	Ordering A	F1	F2	F3	F4	Exec (min)
1	TC-H03	Yes	No	No	No	3
2	TC-H01	No	Yes	No	No	8
3	TC-H05	No	No	No	No	2
4	TC-H02	No	No	Yes	No	4
5	TC-H04	No	No	No	Yes	6
6	TC-H06	No	No	No	No	5

Position	Ordering B	F1	F2	F3	F4	Exec (min)
1	TC-H01	No	Yes	No	No	8
2	TC-H06	No	No	No	No	5
3	TC-H03	Yes	No	No	No	3
4	TC-H04	No	No	No	Yes	6
5	TC-H05	No	No	No	No	2
6	TC-H02	No	No	Yes	No	4

Tasks:

Calculate APFD for both Ordering A and Ordering B. Show all steps: identify TF_i for each fault, compute ∑TF_i, apply the formula.
Calculate TTFF for both orderings (time in minutes to first failure).
Calculate APFD_c for Ordering A. (Total suite execution time = 3+8+2+4+6+5 = 28 min.)
Which ordering is better? Justify your answer using all three metrics.
F2 (payment failure) is classified as a critical severity fault. If you were to modify APFD to give F2 twice the weight of other faults (weighted APFD), would this change which ordering is preferred? Recalculate.

Exit Ticket

A test suite has n=8 cases and m=2 faults. Ordering X has TF_F1=1, TF_F2=2. Ordering Y has TF_F1=7, TF_F2=8. Calculate APFD for both. Which is better and by how much?
A suite achieves APFD = 0.48. What does this tell you about the ordering? What corrective action would you take?
Why does APFD_c penalize orderings that place long-running tests that detect no faults early in the queue?
What is the relationship between APFD measurement and the prioritization recalibration cycle? How often should APFD be measured, and what threshold would trigger a recalibration?

Summary & Module 5 Close

APFD is the primary metric for measuring the effectiveness of a prioritized test suite. It quantifies early fault detection as a value between 0 and 1, with 0.5 as the random baseline. APFD_c extends this to account for varying execution costs. TTFF measures how quickly the CI pipeline can give actionable feedback. The RFD curve reveals the shape of fault detection over time. Together, these metrics provide the evidence needed to validate prioritization investment and continuously improve the ordering.

Key takeaways:

APFD = 1 − ∑TF_i/(n×m) + 1/(2n). Range: ~0 (worst) to ~1 (best). Baseline: 0.5 (random).
APFD > 0.70 is a good target for most commercial software prioritization.
APFD_c accounts for execution cost; prefer it when test execution times vary significantly.
TTFF is the operationally critical metric for CI/CD abort-early decisions.
RFD curves diagnose ordering shape: front-loaded is good; flat or back-loaded needs correction.
Measure APFD regularly; a declining trend signals that priority scores need recalibration.

Module 5 Capstone Assignment:

Bring together all Module 5 sessions for your mini-project suite:

(5.1) Verify your suite passes the 8-point quality checklist. Report any cases revised.
(5.2) Complete the growth diagnosis: report redundancy percentage by type.
(5.3–5.4) Apply greedy minimization; report T′ size, coverage verification result, and 2-year ROI model.
(5.5–5.6) Produce the combined prioritization ordering (version-specific + general) for T′. Include the priority score table and weight rationale.
(5.7) Calculate APFD for your prioritized ordering using fault data from your project's defect tracker (historical failures as proxy faults). Compare with a random ordering APFD to quantify the improvement.
Report: Submit a 1–2 page "Test Suite Management Report" summarising: initial suite size, minimized suite size, APFD before and after prioritization, execution time saving, and one governance recommendation to prevent re-bloat.

Measuring Prioritized Suite Effectiveness

Learning Objectives