Prioritization Types & Techniques

Type	Information Used	Stability	Best For
General Prioritization	Properties of test cases independent of any specific version: coverage, historical FER, execution cost	Stable across versions — recalculate infrequently	Nightly regression; stable products; when change data is unavailable
Version-Specific Prioritization	Change information for the current version P′: which code changed, impact analysis results	Changes every version — recalculate per release	CI/CD gates; targeted regression after a specific change; hotfix releases
Model-Based Prioritization	Trained ML/statistical model using historical suite + defect + coverage data	Improves over time with more data	Large mature suites (>1,000 cases); products with rich execution history

Algorithm 1: Total Coverage Greedy

Principle: Rank each case by its total statement/branch coverage count. Cases covering more code run first.

Advantage: Simple; no iteration needed. Disadvantage: Two cases covering the same 50 branches are equally ranked even though together they cover only 50 branches, not 100. The additional algorithm corrects this.

Algorithm 2: Additional Coverage Greedy (Recommended)

Principle: At each step, select the case that covers the most additional (not yet covered by previously selected cases) branches/statements. This is the greedy set-cover ordering.

Advantage: Maximises coverage rate — the fastest accumulation of coverage as tests execute. Research (Rothermel et al.) shows additional coverage greedy produces APFD values 10–20% higher than total coverage greedy.

Algorithm 3: FER-Based Greedy

Principle: Sort cases by descending historical fault exposure rate. The case most likely to find a defect (based on history) runs first.

Best for: Mature stable products where the code doesn't change much but defect patterns are well-established. Less effective for products under active development where change information should dominate.

Key insight: A test case that exercises changed code is more likely to detect a regression than a test case that exercises unchanged, stable code. Version-specific prioritization exploits this by elevating changed-code cases regardless of their general coverage or historical FER.

Algorithm 4: Change-Impact Prioritization

Principle: Assign each test case a change-impact score based on how much of the changed code it exercises.

Cases with CI = 0 (exercise no changed code) are deprioritized to the end of the queue — but still run. This differs from selection (where CI=0 cases would be excluded).

Algorithm 5: Modification-Traversing (Firewall) Prioritization

Principle: Identify the "firewall" — all modules modified or directly dependent on a modified module. Test cases that traverse the firewall (exercise at least one firewall entity) are elevated to P1.

Advantage: Simpler than CI(t_i) calculation; can be performed with module-level impact analysis tools without needing statement-level coverage data for the changed code.

Version-specific algorithm comparison:

Algorithm	Granularity	Tool Requirement	Accuracy
Change-Impact (CI score)	Statement/branch level	Statement coverage tool + diff	Highest: precise fraction of changed code exercised
Modification-Traversing (Firewall)	Module level	Module dependency graph + diff	Medium: binary (firewall or not) — misses intra-module change proximity

Model-based prioritization workflow:

When model-based prioritization is justified:

• Suite size > 1,000 cases (heuristics degrade; model learns patterns heuristics miss).
• Rich execution history available (≥ 200 historical runs with defect labels).
• Team has ML engineering capacity to build, deploy, and maintain the model.
• APFD gain over greedy > 10% (validate this before committing to model maintenance overhead).

Dimension	General	Version-Specific	Model-Based
Recalculation frequency	Per sprint or release	Per commit / change set	Per run (inference) + periodic retraining
Data required	Coverage + FER history	Diff + impact analysis	Rich historical execution log
Implementation complexity	Low (sort by score)	Medium (requires diff integration)	High (ML pipeline)
APFD vs. random	Moderate improvement	High improvement for targeted changes	Highest improvement at scale
Robustness to cold start	Good (works with minimal history)	Good (works with any change diff)	Poor (needs significant history)
Best suited for	Stable products; nightly regression	Active development; CI gates; hotfixes	Large mature CI systems

Coverage and FER data:

TC	Feature	Branches	Changed Branches (Δ)	CI Score	FER
TC-I01	Reorder point (standard)	B1, B2, B3, B7	B2, B3	2/4 = 0.50	0.6
TC-I02	Reorder point (seasonal)	B1, B3, B4, B7	B3, B4	2/4 = 0.50	0.4
TC-I03	Reorder point (zero lead time)	B2, B3, B5	B2, B3	2/3 = 0.67	0.7
TC-I04	Stock receipt	B6, B8, B9	(none)	0.00	0.5
TC-I05	Stock issue	B8, B10, B11	(none)	0.00	0.3
TC-I06	Low-stock alert	B7, B12, B13	B7 (indirect)	1/3 = 0.33	0.8
TC-I07	Supplier order	B9, B14, B15	(none)	0.00	0.2
TC-I08	Audit trail	B16, B17	(none)	0.00	0.1

Changed branches Δ = {B2, B3, B4, B7} (4 total). Note: B7 is touched only indirectly by the fix.

Step 1: Version-specific tier (T_changed ordered by CI descending, FER tiebreak):

1. P1 TC-I03 (CI=0.67, FER=0.7)
2. P1 TC-I01 (CI=0.50, FER=0.6) — tie on CI, higher FER wins
3. P1 TC-I02 (CI=0.50, FER=0.4)
4. P2 TC-I06 (CI=0.33, FER=0.8)

Step 2: General tier — additional coverage greedy on T_unchanged {TC-I04, TC-I05, TC-I07, TC-I08}:

Branches already covered by T_changed: {B1, B2, B3, B4, B5, B7, B12, B13}.

Iteration	Selected	New Branches	Reasoning
1	TC-I04	B6, B8, B9 (3 new)	Most new branches; B8,B9 not yet covered
2	TC-I05	B10, B11 (2 new)	B8 already covered; 2 new branches
3	TC-I07	B14, B15 (2 new)	B9 already covered; 2 new branches
4	TC-I08	B16, B17 (2 new)	Last remaining

Final combined execution order:

1. P1 TC-I03 — Reorder point (zero lead time)
2. P1 TC-I01 — Reorder point (standard)
3. P1 TC-I02 — Reorder point (seasonal)
4. P2 TC-I06 — Low-stock alert
5. P3 TC-I04 — Stock receipt
6. P3 TC-I05 — Stock issue
7. P4 TC-I07 — Supplier order
8. P4 TC-I08 — Audit trail

All cases that exercise the changed code run in the first 4 positions. If a 30-minute time box only completes TC-I03 to TC-I05, all changed-code coverage is verified and a significant portion of the unchanged code is also checked.

Scenario: An e-commerce platform has 10 test cases. The current change modifies the apply_coupon() function. Changed statements: S3, S5, S7, S9 (4 statements).

TC	Feature	Statements Covered	FER
TC-E01	Apply 10% coupon	S1, S3, S5, S7	0.7
TC-E02	Apply expired coupon	S2, S3, S6	0.5
TC-E03	Apply coupon at threshold	S3, S5, S8, S9	0.6
TC-E04	Apply invalid coupon code	S2, S4, S6	0.4
TC-E05	Apply 50% coupon	S1, S5, S7, S9	0.8
TC-E06	Cart subtotal calculation	S10, S11, S12	0.3
TC-E07	Payment processing	S13, S14, S15	0.9
TC-E08	Order confirmation email	S16, S17	0.2
TC-E09	Apply coupon + loyalty points	S3, S7, S9, S18	0.5
TC-E10	Coupon + free shipping combo	S5, S9, S19, S20	0.4

Tasks:

1. Compute the Change-Impact score CI(t_i) for each test case. (|Δ| = 4 changed statements: S3, S5, S7, S9.)
2. Partition into T_changed (CI > 0) and T_unchanged (CI = 0).
3. Order T_changed by descending CI score. Break ties with FER.
4. Apply the additional coverage greedy algorithm to T_unchanged. Show all iterations.
5. Produce the final combined execution order. If a 20-minute time box fits 6 cases (assume 3-4 min each), which cases are completed? What risk areas are covered?

1. What is the key difference between general and version-specific prioritization? Give one example of when each should be the dominant approach.
2. A test case has CI(t) = 0 after a change. Should it be excluded from the regression run? Justify your answer in terms of the difference between prioritization and selection.
3. Describe the "additional coverage greedy" algorithm in 3 steps. Why does it produce higher APFD values than the "total coverage greedy" algorithm?
4. Under what conditions would you recommend model-based prioritization over the combined general+version-specific heuristic approach?

Key takeaways:

• General prioritization: stable, periodic; best for nightly regression on stable products.
• Version-specific: dynamic, per-commit; essential for CI/CD on active development.
• Additional coverage greedy outperforms total coverage greedy for APFD.
• Change-Impact score CI(t_i) = |cov(t_i) ∩ Δ| / |Δ|; cases with CI > 0 form the version-specific P1 tier.
• Combine both: version-specific T_changed first; general additional-coverage for T_unchanged.
• Model-based: only justified for suites >1,000 cases with ≥100 historical defect-labelled runs.

Assignment (Session 5.6):

1. For your mini-project's most recent feature change, compute CI(t_i) for all test cases. Report the T_changed and T_unchanged partition sizes.
2. Apply the combined prioritization algorithm (version-specific tier + additional coverage greedy tier) to your suite. Produce the complete ordered list.
3. Compare your combined ordering with a FER-only ordering for the same suite. Calculate and compare the estimated APFD for each ordering (assume each test case that exercises a changed branch has a 70% probability of detecting the change-induced defect).
4. If your team were to adopt model-based prioritization in the future, what features would you include in the model? What historical data would you need to collect starting now?