Session 4.4 - System & Acceptance Testing

Overview & Relationship Between Test Levels

System testing and acceptance testing are the final two test levels before a software product reaches production. Although they are closely related and sometimes overlap, they serve distinct purposes and are owned by different stakeholders.

System Testing

Who: Independent QA/test team
What: Complete integrated system against specifications
Basis: System requirements (SRS), design docs
Environment: Test environment mimicking production
Goal: Find defects before acceptance

Acceptance Testing

Who: Customer / end users / business stakeholders
What: System against business needs & user expectations
Basis: Business requirements, user stories, contracts
Environment: Production-like or actual production
Goal: Formal approval / sign-off for release

ISTQB Key Point: Both system testing and acceptance testing are validation activities. System testing asks "Did we build the system right?" (against the specification), while acceptance testing asks "Did we build the right system?" (against business needs). Both questions must be answered before release.

Position in the V-Model:
System testing corresponds to the System Design phase; acceptance testing corresponds to the Business Requirements phase. Together they form the right arm of the V-Model — the validation side.

System Testing In Depth

System testing exercises the entire system as a whole. At this level, the goal is to validate all functional behaviours and non-functional properties documented in the system requirements.

Scope of System Testing

Aspect	Detail
Test Object	The complete, integrated system (all modules assembled)
Test Basis	System Requirement Specification (SRS), system design docs, use cases, regulations
Test Types	Functional, performance, security, usability, reliability, compatibility
Defect Targets	Incorrect system behaviour, missing requirements, integration failures, non-functional violations
Typical Team	Independent test team; testers with domain knowledge
Entry Condition	Integration testing complete & passed; system deployed to test environment
Exit Condition	All planned tests executed; no open critical defects; coverage targets met

System testing verifies end-to-end workflows that span multiple modules, databases, external services, and user interfaces. This is the first time the full system is exercised as a single entity.

Functional system tests
Verify that the system implements all functional requirements correctly across complete user workflows.

Interface verification
Validate that all internal and external interfaces (APIs, databases, file exchanges) behave as specified.

Data integrity checks
Confirm data is correctly created, read, updated, and deleted across system boundaries and storage layers.

Error handling & recovery
System handles unexpected inputs, resource failures, and network interruptions without data corruption.

Non-Functional Testing at System Level

System testing is also the primary level for non-functional testing — verifying how well the system performs its functions rather than just what it does.

Performance Testing — Evaluates response time, throughput, and resource utilisation under expected and peak load conditions. Subtypes: load testing, stress testing, spike testing, soak testing.

Security Testing — Identifies vulnerabilities: authentication weaknesses, authorisation flaws, injection risks (SQL, XSS), sensitive data exposure, and insecure configurations.

Usability Testing — Evaluates how easily real users can learn, operate, and complete tasks with the system. Produces usability defect reports and UX improvement recommendations.

Reliability / Availability Testing — Verifies the system meets its uptime and mean-time-between-failure (MTBF) requirements under continuous operation.

Compatibility Testing — Confirms the system operates correctly across required browsers, operating systems, devices, and database versions.

Maintainability & Portability Testing — Assesses ease of modification, deployment across environments, and code standards compliance.

ISTQB classification: Non-functional characteristics are documented in the ISO/IEC 25010 product quality model. ISTQB Foundation Level requires awareness of the major non-functional test types and their purposes, not their detailed execution techniques.

Acceptance Testing In Depth

Acceptance testing is the final test level performed before release. Its purpose is not primarily to find defects (defects should have been found earlier) but to confirm the system is fit for purpose and to provide the formal basis for the customer's decision to accept or reject the delivery.

Key characteristics of acceptance testing:

Performed by or on behalf of the customer or end user, not the development/QA team.
Based on business requirements and user needs, not technical specifications.
Uses real-world scenarios and representative data.
Results in a formal acceptance decision (accept, conditionally accept, or reject).
Defects found at this stage are costly — they indicate failures in earlier testing phases.

ISTQB Principle 3 — Early Testing: Finding defects first in acceptance testing is a sign of process failure. The goal of all earlier test levels is to ensure that acceptance testing is a confirmation activity, not a primary defect-discovery activity.

Types of Acceptance Testing

ISTQB identifies several distinct forms of acceptance testing, each with a different focus and context:

1. User Acceptance Testing (UAT)

The most common form. Real end users verify the system in representative business scenarios to confirm it meets their operational needs before formal deployment.

Business users End-to-end scenarios Formal sign-off

2. Operational Acceptance Testing (OAT)

Also known as production acceptance testing. Performed by operations/system administration teams to verify the system is ready to be managed and maintained in the production environment.

Backup & restore Disaster recovery Installation & configuration Monitoring & logging

3. Alpha Testing

Performed by internal users (employees of the developing organisation) at the development site, simulating end-user behaviour. Often the first exposure of the product to real usage patterns.

Internal users Controlled environment Before beta

4. Beta Testing

Performed by a selected group of external customers in their real environments. The development team typically has no control over the test conditions. Used to gather real-world feedback before general release.

External users Real environment Before general release

5. Contract Acceptance Testing

Performed against the acceptance criteria defined in a contract between the customer and supplier. Tests are designed to verify that contractual obligations have been met. Failure means the supplier must remediate before payment is released.

Contract terms Formal criteria Legal obligation

6. Regulatory Acceptance Testing

Performed to confirm compliance with government regulations, safety standards, or industry regulations (e.g., FDA, GDPR, ISO, aviation standards). Often requires an independent audit body.

Regulatory compliance Safety standards Certification body

User Acceptance Testing (UAT) — Deep Dive

UAT is typically the most visible and business-critical form of acceptance testing. It is the customer's formal opportunity to validate the delivered system before committing to deployment.

UAT Process Steps

Define acceptance criteria early — agreed between customer and supplier before development begins.
Develop UAT test cases based on business processes and user stories, typically by business analysts with user input.
Prepare the UAT environment — should closely mirror production (data, configuration, interfaces).
Execute UAT — end users perform real business tasks; testers/BAs support and observe.
Log defects and issues — in the defect management system; business users provide feedback notes.
Evaluate results against acceptance criteria — determine pass/fail/conditional.
Formal sign-off decision — stakeholders formally accept or reject the system.

UAT challenges

Business users have limited testing time and expertise.
Test cases are often informal or missing.
Late defect discovery creates pressure to accept with known issues.
Environment or data issues delay execution.
"It works for me" mentality limits coverage.

UAT best practices

Define acceptance criteria before sprint/project starts.
Involve users in test case design, not just execution.
Provide test scripts to guide non-technical users.
Use production-representative data.
Plan UAT as a formal project phase with schedule and resources.

Acceptance Criteria & Formal Sign-off

Acceptance criteria are the conditions that a software system must satisfy to be accepted by a customer, user, or other authorised entity. They must be defined, agreed, and documented before testing begins.

Characteristics of Good Acceptance Criteria (SMART)

Attribute	Meaning	Example
Specific	Clearly states what is required	"Login response time < 2 seconds"
Measurable	Can be objectively verified	"99.5% uptime over 30 days"
Agreed	Accepted by both customer and supplier	Signed in project charter or contract
Realistic	Achievable given the constraints	Based on current system architecture
Time-bound	Verified within a defined test period	"During 5-day UAT window"

Typical acceptance criteria categories:

Functional completeness
All critical use cases execute without error

Defect thresholds
Zero critical/high defects open at sign-off

Performance
Page load < N seconds under M concurrent users

Data accuracy
Migrated data matches source with <0.01% variance

Security
No high-severity security vulnerabilities open

Regulatory
Audit trail captures all required fields per regulation X

Formal Sign-off Process

Evaluate all UAT test results against each acceptance criterion.
Review open defect list — classify by severity and agree handling (fix, defer, waive).
Prepare a UAT Summary Report (tests executed, passed, failed, open issues).
Present to the sign-off authority (product owner, customer representative, steering committee).
Record a formal decision: Accept, Conditionally Accept (with agreed remediation), or Reject.
Obtain signatures on the sign-off document. This authorises deployment to production.

Conditional Acceptance: Common in practice. The system is accepted subject to identified low-priority defects being fixed by an agreed date. Critical and high defects must always be resolved before unconditional sign-off.

Entry & Exit Criteria

Formal test phases should only begin when specified preconditions are met (entry criteria) and only end when defined completion conditions are achieved (exit criteria). This discipline prevents premature testing and premature closure.

System Testing Entry Criteria

Integration testing completed and integration test report available.
All planned modules integrated and deployed to system test environment.
System test environment configured and verified.
Test data prepared and loaded.
System test plan and test cases reviewed and approved.
No open critical or high defects from integration testing.

System Testing Exit Criteria

All planned test cases executed (at minimum, all critical paths).
Requirements coverage target achieved (e.g., 100% of critical FRs).
Zero open critical defects; agreed threshold for high/medium.
System test summary report produced and reviewed.
All deferred defects logged with agreed resolution plan.

Acceptance Testing Entry Criteria

System testing exit criteria met; system test report available.
UAT environment deployed with production-representative data.
Acceptance criteria formally agreed by all stakeholders.
UAT test scenarios/scripts prepared and approved by business users.
End users briefed, trained, and scheduled for UAT.

Acceptance Testing Exit Criteria

All UAT test scenarios executed by authorised users.
All acceptance criteria evaluated and results documented.
Zero open critical defects; open minor/cosmetic issues agreed.
UAT summary report produced and signed off.
Formal acceptance sign-off document signed by authorised stakeholder.

Test Environments

The test environment must be appropriate for the test level. Using the wrong environment invalidates test results.

Environment requirements by test level

Test Level	Environment Characteristics	Data
Unit Testing	Developer workstation; mocked dependencies	Synthetic / hard-coded test data
Integration Testing	Shared dev/test server; real module interfaces; stubs for external systems	Controlled test data sets
System Testing	Dedicated test environment; all modules; near-production configuration; external system simulators	Realistic anonymised data; production-scale volumes
Acceptance Testing	Production-identical environment (or actual production in some cases); all external integrations live	Actual or high-fidelity representative production data

Environment risks to manage:

Environment differences from production invalidate test results ("it passed in UAT but failed in production").
Shared environments cause test interference between teams.
Inadequate data volume masks performance issues.
Use of real personal data in test environments creates GDPR/privacy compliance risks.

Worked Example: Hospital Patient Appointment System

Context: A hospital has commissioned a new patient appointment booking system. The project is entering system testing, followed by UAT with clinical staff.

System Test Scenarios (selected)

ID	Scenario	Type	Requirement
ST-01	Book appointment: valid patient, available slot	Functional (+ve)	FR-010
ST-02	Book appointment: no available slots for requested date	Functional (-ve)	FR-010
ST-03	Cancel appointment within 24 hours — penalty applied	Functional (business rule)	FR-015
ST-04	System responds to 500 concurrent bookings within 3 seconds	Performance	NFR-02
ST-05	Patient data encrypted in transit and at rest	Security	NFR-07
ST-06	Appointment reminder SMS sent 24h before appointment	Functional (integration)	FR-022

UAT Acceptance Criteria (agreed pre-project)

#	Acceptance Criterion	Threshold
AC-01	All critical booking workflows complete without error	100% pass rate
AC-02	Appointment booking response time	< 2 seconds (95th percentile)
AC-03	Open critical defects at UAT completion	Zero
AC-04	Open high defects at UAT sign-off	Zero (or agreed waiver with remediation plan)
AC-05	SMS reminders dispatched correctly in UAT data set	≥ 99% delivery rate
AC-06	System available during UAT 5-day window	≥ 99.5% uptime

UAT Sign-off Decision Scenario

After 5 days of UAT by clinical staff:

AC-01 to AC-03: All met. No critical defects open.
AC-04: 2 high defects open (cosmetic display issues on mobile browser).
AC-05: SMS delivery rate = 99.7%. Met.
AC-06: Uptime = 100%. Met.

Decision: Conditional Acceptance — System accepted subject to the 2 high defects being resolved within 10 working days post-deployment. Remediation plan signed by both parties.

Defects & Risks at System and Acceptance Level

Functional requirement not implemented
A feature in the SRS was never built. Discovered by coverage analysis or direct scenario execution.

End-to-end workflow failure
Individual modules work but data is lost, corrupted, or misinterpreted when passed between them.

Performance degradation under load
System works correctly for one user but fails timeouts, errors, or crashes at expected concurrent user numbers.

Security vulnerability
Unauthorised access, data leakage, or injection attacks found during security testing at system level.

Data migration errors
Historical data migrated to the new system is incomplete, transformed incorrectly, or causes processing failures.

Usability failure
System is technically correct but users cannot complete tasks without unacceptable effort — discovered in UAT.

Cost of late defect discovery: ISTQB and industry data consistently show that defects found at acceptance testing cost 10–100× more to fix than equivalent defects found at unit or integration testing. The later a defect is found, the more rework, retesting, and stakeholder communication is required.

ISTQB Test Level Comparison

Property	Unit Testing	Integration Testing	System Testing	Acceptance Testing
Test Object	Single module / class	Module interfaces / subsystems	Complete system	Complete system + business process
Test Basis	Code, detailed design	Interface specs, architectural design	System requirements, SRS	Business requirements, contracts, user needs
Performed by	Developer	Developer / QA	Independent QA team	Customer / end users
Primary goal	Code correctness	Interface correctness	System behaviour vs specification	Fitness for purpose / sign-off
Key artefacts	Unit test report	Integration test report	System test report, defect log	UAT summary, sign-off document

Common Mistakes

Starting UAT without entry criteria
Accepting a system into UAT before system testing is complete results in users finding defects that QA should have caught, destroying trust and wasting business time.

Undefined acceptance criteria
Without agreed, written criteria, sign-off decisions become subjective and disputes arise about whether the system "passes".

Skipping system testing
Moving directly from integration to UAT leaves functional, performance, and security gaps that will surface in production.

UAT with unrealistic data
Tests that pass on synthetic data may fail with real production-scale or production-quality data. UAT environments must use representative data.

No formal sign-off document
Verbal acceptance is not acceptance. Without a signed document, there is no contractual record of the delivery being approved.

Accepting with too many open defects
Pressure to deploy causes teams to accept systems with high-severity open defects, creating known technical debt and production risk.

Class Activity

Scenario: Your team has been asked to plan system testing and UAT for a new university student portal. The portal allows students to view grades, register for courses, pay fees, and download transcripts. The system has passed integration testing.

Tasks (in groups of 3):

System Test Plan Outline: Identify 6 system test scenarios covering at least 2 functional areas and 2 non-functional areas. Label each as functional or non-functional.
Acceptance Criteria: Write 4 measurable acceptance criteria for the UAT phase. Ensure at least one performance criterion and one defect threshold criterion.
Entry Criteria Check: List 4 items that must be true before UAT can begin.
Sign-off Decision: Given: UAT completed; all functional criteria met; 1 high defect (slow PDF generation, 8 seconds vs 3-second target) open. What is your recommended sign-off decision and why?

Evaluation Rubric (10 marks)

2 marks: System test scenarios are realistic and correctly classified as functional/non-functional.
3 marks: Acceptance criteria are specific, measurable, and cover required categories.
2 marks: Entry criteria are appropriate and cover environment, data, and prior test completion.
3 marks: Sign-off decision is justified using the acceptance criteria framework; trade-offs are acknowledged.

Exit Ticket

State two differences between system testing and acceptance testing (purpose, ownership, or test basis).
Name and briefly describe three types of acceptance testing other than UAT.
A customer signs off on a system with 3 medium defects still open. What document should record this, and what conditions should accompany the sign-off?
Why is finding a critical defect in UAT considered a process failure, not just a testing event?

Summary & Assignment

System testing and acceptance testing are the final gatekeepers before production deployment. System testing validates the complete system against its specification using an independent QA team; acceptance testing formally confirms fitness for business purpose and results in a signed acceptance decision.

Key takeaways:

System testing covers both functional and non-functional properties of the complete system.
Acceptance testing is owned by the customer and results in a formal accept/reject decision.
Multiple acceptance test types exist (UAT, OAT, alpha, beta, contract, regulatory) — all serve distinct purposes.
Acceptance criteria must be defined before testing begins and must be measurable and agreed by all parties.
Formal entry and exit criteria prevent premature test start and premature closure.
Defects found at acceptance level are the most expensive to fix — strong earlier testing minimises this risk.

Assignment: Using your mini-project system:

Write a one-page System Test Plan outline (scope, test types, environment, entry/exit criteria).
Define 5 measurable acceptance criteria covering functional, performance, and defect threshold categories.
Design 3 UAT test scenarios as would be executed by a business user (not a developer). Write them in plain language without technical jargon.
Prepare a template UAT Sign-off document with fields for: project name, test summary, acceptance criteria results, open issues with severity, decision, and stakeholder signatures.

System & Acceptance Testing

Learning Objectives