Module 6
Session 6.8
Testing Mobile & Web Ecosystems
Testing Mobile & Web Ecosystems
Plan tests for mobile systems integrated with web platforms and services — Pressman Ch. 19 | Black (2011)
Learning Objectives
- Distinguish between native, hybrid, and mobile-web applications and explain the testing implications of each type.
- Enumerate mobile-specific testing challenges: device fragmentation, OS versioning, network variability, battery/performance, sensor inputs, and platform store requirements.
- Design a device testing strategy using a risk-based device matrix rather than exhaustive coverage.
- Apply functional, performance, and network condition testing strategies to mobile applications.
- Plan integration testing for a mobile-web ecosystem where a mobile app consumes a shared REST API backend.
- Explain ecosystem-level testing: end-to-end tests that span mobile, web, and backend in a single scenario.
The Mobile Testing Landscape
Mobile represents more than half of global web traffic and is the primary computing platform for billions of users. Yet mobile testing is significantly more complex than web testing due to the variety of device hardware, operating system versions, and usage contexts.
The testing complexity: Even if a team could test 1 device per minute, testing 3,000 devices would take 50 hours. Exhaustive device testing is impossible. Risk-based device selection and automated cloud testing are the only viable approaches.
App Types & Testing Implications
Native Applications
Written in the platform's native language (Swift/Objective-C for iOS; Kotlin/Java for Android). Compiled to native code; full access to device APIs (camera, GPS, biometrics, notifications, file system).
Testing implications:
- Separate codebases for iOS and Android require separate (though conceptually parallel) test suites
- Unit tests in Swift/Kotlin (XCTest, JUnit) for business logic and data layer
- UI automation via XCUITest (iOS) or Espresso (Android) for UI layer
- Cross-platform: Appium can drive both platforms via a unified API
- Device sensor and hardware API testing (camera, GPS, accelerometer) often requires physical devices
Hybrid Applications
Web content (HTML/CSS/JavaScript) packaged in a native container (Cordova, Capacitor, Ionic). Single codebase for both platforms with native API access via plugins.
Testing implications:
- Business logic can be unit-tested in JavaScript (Jest, Mocha)
- Web content layer: test in browser before device (faster feedback)
- Native plugin interactions require device or emulator testing
- Performance is typically lower than native; battery usage must be tested
- Appium supports hybrid apps; can switch context between native and WebView within a session
Mobile-Web Applications
Responsive web applications accessed via the mobile browser. No app store distribution; updated instantly server-side.
Testing implications:
- Extends web testing (Session 6.7) with mobile-specific considerations: touch events, viewport sizes, mobile browser quirks
- Chrome DevTools Device Toolbar provides emulation for rapid responsive testing
- Real device testing is still required for accurate touch interaction, font rendering, and mobile browser behavior
- Progressive Web App (PWA) features (service workers, offline mode, push notifications) require additional test scenarios
Mobile-Specific Testing Challenges
Device Fragmentation
Thousands of device models with varying screen sizes, resolutions, RAM, CPU speeds, and hardware capabilities. A layout that works on a Pixel 8 may overflow on a budget Android device with a smaller screen.
Thousands of device models with varying screen sizes, resolutions, RAM, CPU speeds, and hardware capabilities. A layout that works on a Pixel 8 may overflow on a budget Android device with a smaller screen.
OS Version Diversity
Android users especially span many OS versions. New APIs may not be available on older OS versions; deprecated APIs may behave differently. Test on both current and n-2 OS versions at minimum.
Android users especially span many OS versions. New APIs may not be available on older OS versions; deprecated APIs may behave differently. Test on both current and n-2 OS versions at minimum.
Network Variability
Mobile users experience transitions between WiFi, 5G, 4G/LTE, 3G, and no connectivity. Applications must handle all states gracefully, including in-flight request cancellation when connectivity drops.
Mobile users experience transitions between WiFi, 5G, 4G/LTE, 3G, and no connectivity. Applications must handle all states gracefully, including in-flight request cancellation when connectivity drops.
Battery & Performance
Background processing, wakelocks, and excessive CPU/network usage drain batteries and trigger OS-level throttling. Performance testing on mobile must include battery impact measurement.
Background processing, wakelocks, and excessive CPU/network usage drain batteries and trigger OS-level throttling. Performance testing on mobile must include battery impact measurement.
Sensor & Hardware Inputs
Camera, GPS, accelerometer, gyroscope, biometric sensors, NFC. Functional flows that depend on sensor inputs require physical devices or sophisticated emulator configuration.
Camera, GPS, accelerometer, gyroscope, biometric sensors, NFC. Functional flows that depend on sensor inputs require physical devices or sophisticated emulator configuration.
OS Interrupts
Incoming calls, notifications, low battery warnings, and permission dialogs can interrupt the application mid-flow. Testing must cover how the app handles and recovers from these interruptions.
Incoming calls, notifications, low battery warnings, and permission dialogs can interrupt the application mid-flow. Testing must cover how the app handles and recovers from these interruptions.
Device Testing Strategy
Because exhaustive device coverage is impossible, a risk-based device matrix selects representative devices that maximize defect detection probability.
Building the Device Matrix
- Analyze your user base: Use analytics to identify the actual devices and OS versions your users run. Prioritize the devices that represent 80% of your user base.
- Select representative tiers: Choose at least one low-end device (limited RAM/CPU), one mid-range, and one high-end. Defects often appear only on low-end hardware.
- Cover the latest OS versions: Always include the current and previous major OS releases for both iOS and Android.
- Include at least one tablet: Layouts designed for phone screens often break on tablet screen sizes.
- Review the matrix per release: Device market share shifts; update the matrix at least annually.
| Tier | Example Device | OS | Priority | Test Scope |
|---|---|---|---|---|
| High-end flagship | iPhone 15 Pro | iOS 17 | P1 | Full regression |
| Mid-range popular | Samsung Galaxy A54 | Android 14 | P1 | Full regression |
| Previous OS version | iPhone 13 | iOS 16 | P1 | Critical flows |
| Low-end budget | Motorola Moto G (2023) | Android 13 | P2 | Critical flows + performance |
| Tablet | iPad Air | iPadOS 17 | P2 | Layout and navigation |
| Older Android | Samsung Galaxy A32 | Android 12 | P3 | Compatibility spot-check |
Emulators / Simulators
Android Emulator (AVD) and iOS Simulator (Xcode). Fast and free; support most API surface area. Limitations: no real hardware sensors, different GPU rendering, not representative of real-world performance. Use for development and initial testing.
Physical Devices & Cloud Farms
Real devices provide accurate rendering, performance, and hardware behavior. Cloud farms (Firebase Test Lab, BrowserStack App Automate, AWS Device Farm) provide access to hundreds of physical devices without hardware investment. Essential for pre-release testing.
Functional Mobile Testing
Mobile functional testing covers areas analogous to web functional testing plus mobile-specific interaction patterns.
Touch Interactions
Tap, long press, swipe, pinch-to-zoom, two-finger tap. Verify that all touch targets meet the minimum 44×44 point size and that gestures work correctly under fast or imprecise input.
Tap, long press, swipe, pinch-to-zoom, two-finger tap. Verify that all touch targets meet the minimum 44×44 point size and that gestures work correctly under fast or imprecise input.
Screen Orientation
Test all flows in portrait and landscape orientation. Verify that app state is preserved on rotation and that no UI elements overflow or clip in landscape mode.
Test all flows in portrait and landscape orientation. Verify that app state is preserved on rotation and that no UI elements overflow or clip in landscape mode.
Deep Links & App Links
Verify that deep links (from notifications, marketing emails, other apps) navigate to the correct in-app screen and that invalid deep links are handled gracefully.
Verify that deep links (from notifications, marketing emails, other apps) navigate to the correct in-app screen and that invalid deep links are handled gracefully.
Push Notifications
Verify notification receipt, content, badge count, and navigation when tapping the notification from both foreground and background (app killed) states.
Verify notification receipt, content, badge count, and navigation when tapping the notification from both foreground and background (app killed) states.
Permissions & Privacy
Test the full permission lifecycle: first request, grant, deny, "ask every time," and revocation in system settings. Verify the app handles denied permissions without crashing.
Test the full permission lifecycle: first request, grant, deny, "ask every time," and revocation in system settings. Verify the app handles denied permissions without crashing.
Background / Foreground Transitions
Verify that the app saves state when backgrounded and restores correctly. Test long-running operations (download, upload) across foreground/background transitions.
Verify that the app saves state when backgrounded and restores correctly. Test long-running operations (download, upload) across foreground/background transitions.
Mobile Performance Testing
Mobile performance testing must account for constraints that don't exist on desktop: limited RAM, CPU throttling under thermal load, battery depletion, and slower storage I/O.
Key Mobile Performance Metrics
- App launch time: Cold start (from killed) and warm start (from background). Industry targets: cold <2s, warm <1s
- Frame rate: UI should render at 60fps (or 90/120fps on high-refresh devices). Frames dropped below 60fps produce visible jank
- Memory usage: Monitor peak and sustained memory; excessive usage triggers OS-level app termination
- Battery drain: Run soak tests (app active for 30 min); measure battery percentage drop vs. baseline
- Network payload: Monitor request count and payload size; mobile users on metered connections are sensitive to data usage
Performance Testing Tools for Mobile
- Android Profiler (Android Studio): Real-time CPU, memory, network, and energy profiling
- Xcode Instruments (iOS): Time Profiler, Leaks, Network, Energy Log instruments
- Firebase Performance Monitoring: Real-user monitoring (RUM) data from production devices; tracks network request latency and screen rendering times
- Perfetto: System-level tracing for Android; identifies scheduling, I/O, and rendering bottlenecks
Network Condition Testing
Mobile applications must be tested under varied and degraded network conditions that are representative of real-world usage.
Network States to Test
- No connectivity: Airplane mode. Verify graceful error messages; verify cached content is displayed where appropriate; verify queued operations resume when connectivity returns.
- Slow connection (3G/low bandwidth): Use network throttling (Charles Proxy, Android Emulator network profiles). Verify timeout handling and loading state UI.
- Connectivity transition: Switch from WiFi to 4G mid-operation. Verify in-flight requests are either completed or retried without data corruption.
- High latency: Simulate 500ms+ round-trip latency. Verify that the UI shows loading indicators and that users cannot submit the same form twice during a slow response.
- Packet loss: Simulate 5–10% packet loss. Verify retry logic and data integrity.
Tools for network condition simulation:
- Charles Proxy / mitmproxy: Intercept and throttle HTTP/S traffic; simulate latency, bandwidth limits, and connection drops
- Android Emulator: Built-in network profiles (GPRS, EDGE, 3G, 4G, 5G) in AVD settings
- Network Link Conditioner (iOS Simulator): Simulate various network conditions including full packet loss
- tc (Linux traffic control): Kernel-level network condition simulation for CI environments
API & Backend Integration Testing
Most mobile applications are thin clients that consume a shared REST or GraphQL API. The API layer is the most critical integration point in a mobile-web ecosystem.
Why Mobile Clients Place Special Demands on the API
- Mobile networks have higher latency and lower reliability than server-to-server connections
- Multiple app versions may be in production simultaneously; the API must support backward compatibility
- Mobile clients send API version headers; the backend must handle requests from both current and older app versions
- Background sync operations may retry requests; the API must handle duplicate requests idempotently
- Push notification delivery depends on the backend registering device tokens and calling the appropriate push service (FCM for Android, APNs for iOS)
API Tests Specific to Mobile Integration
- Test API responses with a slow connection mock: verify the mobile client handles partial responses and timeouts correctly
- Test API version negotiation: send requests with older client version headers and verify the API returns compatible responses
- Test push notification registration endpoint: verify token registration, token refresh, and deregistration (logout)
- Test file upload endpoints with degraded network: verify partial upload recovery and resumable uploads
- Test that the API returns paginated results compatible with the mobile client's infinite-scroll implementation
Ecosystem-Level Testing
An ecosystem test validates a complete user journey that spans multiple platforms: a user starts on the mobile app, triggers a backend event, and the result is visible on the web dashboard. These tests verify cross-platform consistency and integration of all system components.
Example Ecosystem Test: Mobile Order → Web Dashboard
- Mobile (Appium): User places a food order via the iOS app
- API assertion: Verify the POST /orders API received the order and returned HTTP 201
- Backend verification: Directly query the database to confirm order record was created with correct fields
- Web (Playwright): Log into the restaurant's web dashboard; verify the order appears in the "New Orders" list within 5 seconds
- Push notification: Verify the mobile app received a push notification confirming the order
Orchestrating ecosystem tests: These tests require multiple automation tools to run in coordination. Use a test orchestration framework (Robot Framework with multiple libraries, Cucumber with parallel step runners, or a custom pytest fixture that initializes all clients) to drive all layers from a single test script.
Cross-Platform Data Consistency
Data created on mobile must appear identically on web (and vice versa). Verify that: timestamps are timezone-correct on both platforms; character encoding (emoji, non-ASCII) is handled correctly through the full stack; currency and number formatting is consistent.
Shared Authentication Testing
If mobile and web share an OAuth2 / JWT authentication system, test: login on mobile, access web without re-login (SSO); logout on web invalidates the mobile session; token refresh works correctly when the mobile app has been inactive overnight.
Mobile Testing Tools
| Tool | Purpose | Platform | Deployment |
|---|---|---|---|
| Appium | UI automation for native and hybrid apps | iOS + Android | OSS |
| XCUITest | Native iOS UI automation (Apple-recommended) | iOS only | OSS (Xcode) |
| Espresso | Native Android UI automation (Google-recommended) | Android only | OSS |
| Detox | E2E testing for React Native apps | iOS + Android | OSS |
| Firebase Test Lab | Cloud device farm; run Espresso/XCUITest on real devices | iOS + Android | Cloud (Google) |
| BrowserStack App Automate | Cloud device farm for Appium tests; 3,000+ real devices | iOS + Android | Cloud (commercial) |
| AWS Device Farm | Cloud device farm; supports Appium, XCUITest, Espresso | iOS + Android | Cloud (AWS) |
| Charles Proxy | HTTP/S proxy for network condition simulation and mocking | iOS + Android | Commercial |
| Android Profiler | CPU, memory, network, energy profiling | Android only | OSS (Android Studio) |
| Xcode Instruments | Performance profiling (Leaks, Time Profiler, Energy) | iOS only | OSS (Xcode) |
Module 6 Capstone
Module 6 covered the full testing automation and special domains landscape:
- 6.1 Need for Automation: Manual testing ceilings, industry pressures, value dimensions, ROI, automation pyramid
- 6.2 Tool Categorization: Unit, API, functional/UI, performance, security, static analysis, and management tool categories
- 6.3 Tool Selection & Cost: Weighted scoring model, TCO, ROI calculation, vendor risk, tool pilot process
- 6.4 Automation Guidelines: Six core guidelines, Page Object Model, code demos (JUnit, Playwright, k6), CI integration
- 6.5 OO Testing Basics: Encapsulation, inheritance, polymorphism, dynamic binding, testability design, OO coverage criteria
- 6.6 OO Testing Techniques: State-based testing, RTS, behavioural testing, inheritance hierarchy strategies, OO integration
- 6.7 Web Testing: Web challenges, quality attributes, content/navigation, security (OWASP), accessibility (WCAG), API layer
- 6.8 Mobile & Ecosystem: App types, device fragmentation, device matrix, mobile functional/performance/network testing, ecosystem-level testing
Common Mistakes
Testing only on the latest flagship device: Budget and mid-range devices represent the majority of the Android market. Memory constraints, slower CPUs, and older GPU drivers on these devices produce defects that are invisible on flagships.
Relying solely on emulators: Emulators are excellent for functional and API-level testing but do not accurately simulate real-world performance, battery drain, GPU rendering, or hardware sensor behavior. Physical device testing is required before release.
Ignoring network condition testing: Testing only on a stable WiFi connection misses defects that appear for the majority of mobile users who use cellular networks with variable reliability.
Testing mobile in isolation from the web platform: In a mobile-web ecosystem, a change to the shared API can break both the mobile app and the web app. Ecosystem-level regression tests are essential to catch cross-platform integration breaks.
Not testing OS interrupt scenarios: The most common source of crash reports in production mobile apps is unhandled interruptions (incoming calls, low memory warnings, permission dialogs appearing mid-transaction). Always include interrupt scenario tests in the regression suite.
Class Activity
Mobile-Web Ecosystem Test Planning (30 minutes)
You are the QA lead for a ride-sharing application. The system consists of: a Rider mobile app (iOS + Android), a Driver mobile app (iOS + Android), a web admin dashboard, a shared REST API, and a real-time WebSocket service for location updates.
Design a test plan that addresses:
- Device matrix: Define a device matrix for both the Rider and Driver apps. Which devices would you prioritize for each user type and why?
- Critical user journeys: List three end-to-end journeys that span Rider app → API → Driver app. What would each journey test verify?
- Network condition tests: Describe two specific test scenarios related to network conditions that are particularly critical for a ride-sharing app (e.g., what happens when the driver loses connectivity during a trip?)
- Performance requirements: Define SLAs for: app launch time, map rendering time, driver location update latency, and ride request API response time. How would you test each?
- Tool selection: Choose one UI automation tool for each: iOS testing, Android testing, API testing, and performance testing. Justify each choice using the Session 6.3 selection criteria.
Exit Ticket
- Explain the difference between a native app, a hybrid app, and a mobile-web app. Which testing approach (XCUITest, Appium, Playwright) is most appropriate for each?
- Why is exhaustive device coverage impossible for mobile testing? Describe the risk-based approach used to select a practical device matrix.
- List three network states that must be tested for a mobile application. For each, describe one specific defect that could manifest only under that condition.
- What is an ecosystem-level test? Give a concrete example from the ride-sharing domain that spans all three tiers (mobile app, API, web dashboard).
- What are the two most important metrics for monitoring mobile app performance under load, and which tools would you use to measure them on each platform?
Summary & Module Close
Key takeaways from Session 6.8:
- Three app types (native, hybrid, mobile-web) have different testing approaches; Appium provides a cross-platform bridge for native and hybrid automation.
- Mobile testing challenges: device fragmentation, OS versioning, network variability, battery/performance, sensor inputs, OS interrupts.
- Risk-based device matrix: select devices by user base analytics, hardware tier, and OS version; use cloud farms for breadth.
- Functional mobile testing: touch interactions, orientation, deep links, push notifications, permissions, background/foreground transitions.
- Network condition testing is essential: test no connectivity, slow connections, and mid-operation transitions.
- Ecosystem-level tests span mobile app + API + web dashboard in a single test scenario; require multi-tool orchestration.
Module 6 complete. You have now covered the full span of testing automation and special domains: the business case for automation, the tool landscape, selection and cost analysis, automation guidelines, OO testing techniques, web testing, and mobile-web ecosystem testing. The course continues in Module 7 with software quality metrics, measurement-based testing strategies, and defect analysis.