ISA Compliance & Testing Strategy
💬 Discuss Testing: Join #powercommons:matrix.org to discuss ISA compliance testing strategies, share test results, and collaborate on verification efforts.
When Compliance Testing Happens
Location: WS9 (Integration & Verification)
Timing: Weeks 36-56 of project
Duration: 20 weeks
Week 36-40: Initial test runs, identify failures
Week 41-48: Fix bugs, iterative re-testing
Week 49-52: Final verification, achieve 100% pass
Week 53-56: Linux validation, certification
Test Suites
1. OpenPOWER ISA Compliance Tests
- Source: https://github.com/OpenPOWER-Foundation
- Coverage: Book I (user), Book II (virtual), Book III (privileged)
- Purpose: Official ISA certification
2. Linux Test Project (LTP)
- Purpose: OS-level ISA validation
- Tests: System calls, threading, signals
3. glibc Test Suite
- Purpose: Validate VMX/VSX usage in standard library
- Critical for: LCS compliance
4. KVM Selftests
- Purpose: Hypervisor functionality
- Tests: Guest isolation, nested paging
Testing Platforms
| Platform | When Available | Purpose |
|---|---|---|
| RTL Simulation | Week 1+ | Early testing, detailed debug |
| FPGA Emulation | Week 20+ | Faster testing, pre-silicon validation |
| Silicon | Post tape-out | Final validation, performance |
Test Execution Process
Phase 1: Discovery (Weeks 36-40)
- Run full compliance suite on all platforms
- Categorize failures by workstream
- Assign to responsible teams
Phase 2: Resolution (Weeks 41-48)
- Fix bugs in parallel
- Re-test incrementally
- Track pass rate improvement
- Target: 95%+ pass rate
Phase 3: Certification (Weeks 49-52)
- Achieve 100% pass rate
- Document any exceptions
- Performance validation
- Generate compliance report
Phase 4: Validation (Weeks 53-56)
- Boot Linux distributions (Fedora, Ubuntu)
- Run application workloads
- KVM multi-guest testing
- Final sign-off
Success Criteria
- Pass 100% of ISA compliance tests (Book I/II/III)
- Boot unmodified Linux distributions
- KVM functional with multi-threaded guests
- Performance within 90% of baseline
Test Coverage by Workstream
| Workstream | Tests |
|---|---|
| WS1 | Instruction execution, encoding |
| WS2 | Hypervisor, LPAR, privilege levels |
| WS3 | Interrupts, exceptions, vectors |
| WS4 | MMU, page tables, TLB |
| WS5 | Debug facilities, breakpoints |
| WS6 | Performance counters, events |
| WS7 | VMX/VSX instructions, registers |
| WS8 | Prefixed instructions |
| WS9 | System integration, Linux boot |
Detailed Test Issues
See WS9 section in WS5-WS9_Additional_Workstreams.md for:
- Issue #371: ISA compliance test suite (30 days)
- Issue #372: Regression suite (20 days)
- Issue #373: Linux kernel bring-up (15 days)
- Issue #374: Distribution testing (20 days)
- Issue #375: KVM testing (15 days)