Session 5.5 - Test Suite Prioritization

What is Test Case Prioritization?

Test case prioritization (TCP) is the process of ordering test cases within a test suite so that those with higher value — by some criterion — are executed before those with lower value. The suite size is unchanged; only the execution order changes.

Prioritization vs. related techniques:

Technique	Suite Size	What Changes	Goal
Prioritization (this session)	Unchanged	Execution order	Detect defects as early as possible in the execution window
Selection (Session 4.7)	Reduced per cycle	Which cases run	Reduce cost of a specific regression cycle
Minimization (Session 5.3)	Permanently reduced	Which cases exist	Permanently reduce suite size without coverage loss

Why does order matter?

If a regression run is time-boxed (e.g., "run as many tests as possible in 2 hours before the release gate"), the cases that run in the first 2 hours determine what defects are found before the release decision. Prioritization ensures the highest-value cases execute within the available window — regardless of whether the full suite completes.

Even when the full suite always completes, prioritization improves time-to-first-failure: the earlier a defect is detected in the run, the sooner a developer can begin diagnosing it.

The Five Prioritization Goals

Chowdhury (Chapter 10) identifies five goals that a prioritization strategy can optimise for. Each represents a different definition of "value":

Goal	Prioritisation Question	Best Applied When
1. Early Defect Detection	Which cases are most likely to find a defect, soonest?	Time-boxed regression; CI fast-feedback gate
2. Risk-Based Priority	Which areas carry the highest risk if a defect escapes?	Release decisions; safety-critical releases
3. Business Value	Which features are most important to the customer?	Customer-facing releases; SLA-critical systems
4. Coverage Rate	Which cases maximise new coverage gain per unit time?	When coverage metric must be met within a time window
5. Fault Exposure Rate	Which cases historically expose the most faults per run?	Mature suites with rich execution history; CI pipelines

These goals often conflict: A case testing a low-usage but safety-critical feature may score high on Risk but low on Business Value. A case with high historical fault-exposure rate may cover low-risk stable code. A good prioritization strategy selects a goal mix appropriate for the release context.

Goal 1: Early Defect Detection

Early Defect Detection

Principle: Run cases that are most likely to detect a defect first. This maximises the probability that a defect is found early in the execution window, giving developers maximum time to fix it before a deadline.

Proxy metrics for "likely to detect a defect":

Historical failure rate: Cases that failed more frequently in recent execution history are more likely to fail again.
Changed code proximity: Cases that exercise code modified in the current change set are more likely to reveal regression defects.
Defect density of target area: Cases covering modules with historically high defect density.

The APFD metric (preview — detailed in Session 5.7):

APFD (Average Percentage of Faults Detected) measures how quickly a prioritized suite detects faults as a percentage of the total execution. An APFD of 1.0 means all faults are detected immediately; APFD of 0.5 is the expected value for a random ordering.

Early-defect-detection prioritization directly maximises APFD. This is the most widely studied prioritization goal in the research literature (Elbaum et al., Rothermel et al.).

Goal 2: Risk-Based Priority

Risk-Based Prioritization

Principle: Run cases for high-risk features first. Risk is the product of likelihood of failure and impact if failure occurs.

Risk = Likelihood × Impact

Cases covering high-risk areas are prioritized regardless of how often they have historically failed. A case that has never failed but covers a safety-critical payment flow is high priority because the impact of it failing in production is catastrophic.

Risk classification matrix for prioritization:

Likelihood	Low Impact	Medium Impact	High Impact
High	P3	P2	P1
Medium	P4	P3	P2
Low	P4	P4	P3

Risk factors that elevate a test case's priority:

Covers a financial transaction or data integrity function.
Covers a security control (authentication, authorisation, encryption).
Covers a feature used by the highest proportion of users.
Covers code changed in the current release (regression risk is elevated for changed code).
Covers a module with the highest historical defect density.

Goal 3: Business Value

Business Value Priority

Principle: Run cases for features that are most valuable to the customer or business first. If the full suite cannot complete before the release decision, at least the most business-critical functionality has been tested.

Business value proxies:

Revenue impact: Features directly generating revenue (checkout, subscription, payments) > supporting features.
User frequency: Features used by 80% of users daily > features used by 5% monthly.
SLA coverage: Features governed by SLAs or contractual obligations have highest business priority.
Customer-reported priority: Features listed in the customer's acceptance criteria or UAT plan.

Business value vs. risk: Business value prioritizes what matters to the customer; risk prioritizes what matters for safety/reliability. They often align but sometimes diverge: a low-usage but safety-critical feature may have low business value but very high risk. Both must be considered together — which is why a weighted scoring model is used in practice.

Goal 4: Coverage Rate

Coverage Rate Prioritization

Principle: Order cases so that coverage accumulates as quickly as possible. The case that covers the most new (not yet covered) branches or statements is run first.

Algorithm: This is the greedy set-cover ordering from Session 5.3, but applied to ordering rather than removal. In each step, select the case that maximises the number of new requirements covered, given what has already been covered by cases run so far.

When to use: When a coverage threshold (e.g., "we need 80% branch coverage before the release gate") must be reached as quickly as possible within a time window.

Goal 5: Fault Exposure Rate

Fault Exposure Rate

Principle: Order cases by their historical rate of detecting faults. Cases with the highest fault-detection frequency per execution are run first.

Metric: Fault Exposure Rate (FER)

FER(t_i) = (Number of distinct faults detected by t_i over last N executions) / N

Cases with highest FER run first.

Limitation: FER reflects past behaviour. A case that has never failed (FER = 0) may still detect a future defect. FER-only prioritization can starve low-FER cases — which is why FER is best used as one factor in a multi-factor model, not as the sole criterion.

Priority Scoring: A Multi-Factor Model

In practice, prioritization uses a weighted multi-factor score that combines several of the goals above. Each factor is scored and weighted according to the release context:

Priority Score Formula:

Score(t_i) = w₁×Risk(t_i) + w₂×BizValue(t_i) + w₃×FER(t_i) + w₄×ChangedCode(t_i) + w₅×(1/ExecTime(t_i))

Where all factors are normalised to [0,1] and weights w₁–w₅ sum to 1.0.

Factor descriptions:

Risk(t_i) Risk score of the feature covered by t_i from the risk matrix. Higher = run first.

BizValue(t_i) Business value of the feature covered (revenue impact, user frequency, SLA). Higher = run first.

FER(t_i) Historical fault exposure rate over last N runs. Higher = run first.

ChangedCode(t_i) 1 if t_i covers code changed in the current version; 0 otherwise. Changed = run first.

1/ExecTime(t_i) Inverse of execution time. Faster cases run earlier to get quick feedback; all else being equal, prefer short tests over long ones.

Context-dependent weights:

Context	w₁ Risk	w₂ BizValue	w₃ FER	w₄ Changed	w₅ Speed
CI commit gate (fast feedback)	0.15	0.10	0.25	0.40	0.10
Nightly regression (full suite)	0.25	0.20	0.30	0.15	0.10
Pre-release gate	0.35	0.30	0.20	0.10	0.05
Hotfix regression (time-critical)	0.20	0.15	0.15	0.45	0.05

Prioritization in Time-Boxed Regression

A time-boxed regression has a fixed deadline: run as many tests as possible within a time budget T_budget. Prioritization determines which tests run within the budget.

Time-box execution model:

Sort cases by descending priority score.
Execute in priority order, tracking cumulative execution time.
When cumulative time reaches T_budget, stop. The remaining cases are deferred.
Report: "X of Y cases run; P1 and P2 cases fully covered; P3 coverage at Z%."

Example: 2-hour release gate with 580-case suite (after minimization from Session 5.3)

Priority Band	Cases	Cumulative Time	Risk Coverage
P1 (score ≥ 0.75)	65 cases	35 min	100% P1 risk areas
P2 (score 0.50–0.74)	110 cases	1h 20min	100% P2 risk areas
P3 (score 0.25–0.49)	180 cases (partial)	2h 00min (budget hit)	73% P3 areas within budget
P4 (score < 0.25)	225 cases	Deferred	Deferred to next run

All P1 and P2 cases (the highest-risk 30% of the suite) are completed in the first 80 minutes. The release decision is well-informed even if P3/P4 cases do not complete.

Prioritization in CI/CD Pipelines

CI/CD pipelines use layered regression gates where each layer applies prioritization with a different goal mix:

Gate 1: Pre-commit (developer machine) — Run the 15–30 unit tests that cover the changed function. Goal: speed. Time budget: <30 seconds. Criterion: ChangedCode weight = 0.8.

Gate 2: CI fast gate (pull request) — Run P1 cases + cases covering changed modules. Goal: early defect detection. Time budget: <10 minutes. Criterion: FER + ChangedCode dominant.

Gate 3: Nightly regression — Run full suite in priority order. Goal: risk + business value + coverage rate. Time budget: 2–4 hours. All five goals contribute to scoring.

Gate 4: Release gate — Run full suite. Goal: risk + business value dominant. All P1 and P2 cases must pass before release. P3/P4 failures are risk-assessed individually.

Worked Example: Prioritizing a Food Delivery App Suite

System: Food delivery app. Six test cases to prioritize. Score using the CI commit gate weights (w₁=0.15, w₂=0.10, w₃=0.25, w₄=0.40, w₅=0.10).

Factor scores (normalised 0–1):

Test Case	Feature	Risk	BizValue	FER	Changed	1/Time	Score
TC-01	Place order	0.9	1.0	0.7	1	0.8	0.15(0.9)+0.10(1.0)+0.25(0.7)+0.40(1)+0.10(0.8) = 0.830
TC-02	Search restaurants	0.5	0.8	0.4	0	0.6	0.15(0.5)+0.10(0.8)+0.25(0.4)+0.40(0)+0.10(0.6) = 0.315
TC-03	Payment processing	1.0	1.0	0.8	1	0.4	0.15(1.0)+0.10(1.0)+0.25(0.8)+0.40(1)+0.10(0.4) = 0.890
TC-04	Order history	0.3	0.4	0.2	0	0.9	0.15(0.3)+0.10(0.4)+0.25(0.2)+0.40(0)+0.10(0.9) = 0.224
TC-05	Delivery tracking	0.7	0.9	0.6	1	0.5	0.15(0.7)+0.10(0.9)+0.25(0.6)+0.40(1)+0.10(0.5) = 0.700
TC-06	User profile update	0.2	0.3	0.1	0	1.0	0.15(0.2)+0.10(0.3)+0.25(0.1)+0.40(0)+0.10(1.0) = 0.188

Prioritized execution order:

P1 TC-03 (Payment, score 0.890)
P1 TC-01 (Place order, score 0.830)
P2 TC-05 (Delivery tracking, score 0.700)
P3 TC-02 (Search, score 0.315)
P4 TC-04 (Order history, score 0.224)
P4 TC-06 (Profile update, score 0.188)

If the CI gate has a 3-minute budget: TC-03 and TC-01 complete (critical payment and ordering paths verified). TC-05 starts but may not complete. TC-02, TC-04, TC-06 are deferred to the nightly run.

Common Mistakes

Single-factor prioritization
Using only execution speed (shortest tests first) or only FER runs fast cases and high-history cases first, but may defer critical low-FER tests that cover safety features. Always use a multi-factor model.

Static priority never updated
Priority scores must be recalculated for each release cycle. A test that was P4 last sprint may become P1 this sprint if its module was changed. Stale priority ordering is worse than random ordering in some scenarios.

Ignoring execution time in the model
A high-priority but 45-minute test can block all other tests. Execution time as a factor (1/time) ensures fast tests of similar priority run before slow ones, improving time-to-first-failure.

Confusing prioritization with selection
Prioritization means all cases eventually run; selection means only a subset runs. Treating prioritization as "we only run P1 and P2" is actually selection. The distinction matters for coverage and defect-detection guarantees.

Class Activity

Scenario: You have 8 test cases for an online banking application. A hotfix release is scheduled in 90 minutes. Use the hotfix weights (w₁=0.20 Risk, w₂=0.15 BizValue, w₃=0.15 FER, w₄=0.45 Changed, w₅=0.05 Speed):

ID	Feature	Risk	BizVal	FER	Changed?	Exec Time
TC-B01	Fund transfer	1.0	1.0	0.8	Yes	8 min
TC-B02	Balance enquiry	0.6	0.7	0.5	Yes	2 min
TC-B03	Statement download	0.4	0.5	0.3	No	5 min
TC-B04	Login (2FA)	0.9	0.9	0.7	No	3 min
TC-B05	Bill payment	0.8	0.8	0.6	Yes	6 min
TC-B06	Loan application	0.5	0.6	0.4	No	15 min
TC-B07	Profile update	0.2	0.2	0.1	No	2 min
TC-B08	Notification settings	0.1	0.1	0.05	No	1 min

1/ExecTime: normalize by max time (15 min). E.g., TC-B01: 1/8 normalised to (1/8)/(1/1) = 1/8; use rank normalization or (min_time/exec_time) as the score.

Tasks:

Calculate the priority score for each test case using the hotfix weights. Show your working for at least two cases.
Rank the 8 cases by descending score. Assign P1 (≥0.70), P2 (0.50–0.69), P3 (0.30–0.49), P4 (<0.30) bands.
Given a 90-minute time box, identify which cases complete within the budget (based on execution times). Does the prioritization ensure the most critical cases are covered?
If the weights were changed to the nightly regression weights (w₁=0.25, w₂=0.20, w₃=0.30, w₄=0.15, w₅=0.10), how does the ranking change? Which cases swap positions?
A stakeholder says: "Just run P1 and P2 cases — skip the rest." Is this prioritization or selection? What is the implication for the remaining cases?

Exit Ticket

Prioritization does not reduce the suite size. What exactly does it change, and why does that matter in a time-boxed regression?
Name the five prioritization goals. For each, give a one-sentence description of the scenario where it would be the dominant goal.
A test case covers a low-usage admin feature but has the highest historical fault exposure rate in the suite. How should this case be handled in a pre-release gate prioritization vs. a CI fast gate?
Why is a single-factor (e.g., FER-only) prioritization potentially dangerous for a safety-critical release?

Summary & Assignment

Test case prioritization reorders a suite to maximise value delivered within an execution window. Five goals — early defect detection, risk, business value, coverage rate, and fault exposure rate — define different dimensions of value. In practice, a weighted multi-factor score combines these goals with context-dependent weights (CI gate vs. nightly vs. release gate). Prioritization does not remove cases; all cases eventually run. It ensures the highest-value cases run first so that if time runs out, the most important coverage is complete.

Key takeaways:

Prioritization changes order, not size. It complements minimization (which changes size).
The five goals are: early detection, risk, business value, coverage rate, fault exposure rate.
Use a weighted multi-factor score; weights change by context (CI vs. nightly vs. release).
In CI/CD, layer prioritization across gates: commit gate (fast/changed), nightly (full risk+FER), release gate (risk+business value dominant).
Priority scores must be recalculated every cycle; stale priorities are as bad as no priorities.

Assignment (Session 5.5):

For your mini-project suite (after minimization from Session 5.3), assign factor scores (Risk, BizValue, FER, ChangedCode, 1/ExecTime) to each test case. Document your scoring rationale for each factor.
Calculate priority scores using two weight sets: CI commit gate weights and pre-release gate weights. Produce two ranked orderings of your suite.
Simulate a 30-minute time-boxed regression using the pre-release gate ordering. Report which cases complete, which are deferred, and what risk areas are covered vs. not covered within the time box.
Design a 3-layer CI/CD prioritization pipeline for your project. For each layer, specify: trigger condition, time budget, weight set used, and the coverage guarantee each layer provides.

Test Suite Prioritization

Learning Objectives