generated from pulumi/pulumi-provider-boilerplate
-
Notifications
You must be signed in to change notification settings - Fork 6
Open
Labels
kind/documentationImprovements or additions to documentationImprovements or additions to documentationkind/taskWork that's part of an ongoing epicWork that's part of an ongoing epic
Description
Context
As testing practices mature, we need comprehensive documentation to guide contributors and maintain consistency.
Part of Epic: #572
Documentation Needed
1. CONTRIBUTING.md (Update)
Comprehensive guide covering:
- Overview of testing strategy
- Types of tests (unit, integration, regression)
- How to write unit tests
- How to write integration tests
- Test patterns and best practices
- How to run tests locally
- How to debug failing tests
- CI/CD test pipeline explanation
2. Update CLAUDE.md
- Add testing requirements for new resources
- Reference TESTING.md
- Add test review checklist
- Update example patterns with tests
3. PR Template Updates
- Add testing checklist to PR template
- Require test evidence for bug fixes
- Require test additions for new features
CONTRIBUTING.md Outline
# Testing Guide for pulumi-pulumiservice
## Overview
[Purpose, strategy, coverage goals]
## Test Types
### Unit Tests
- Location: `provider/pkg/resources/*_test.go`
- Purpose: Test resource implementations in isolation
- Speed: Fast (<10 seconds per resource)
- Dependencies: None (uses mocks)
- Run: `cd provider/pkg && go test ./...`
### Integration Tests
- Location: `examples/examples_yaml_pulumitest_test.go`
- Purpose: Test end-to-end functionality against real API
- Speed: Slow (3 hours total, sharded)
- Dependencies: PULUMI_ACCESS_TOKEN, test organization
- Run: `cd examples && go test -tags=yaml -v -timeout 10m`
### Regression Tests
- Purpose: Prevent known bugs from recurring
- Location: Embedded in unit/integration tests
- Marked with: Issue reference in comment
## Writing Unit Tests
### Pattern 1: Mock-Based Testing
[Example with code]
### Pattern 2: Table-Driven Tests
[Example with code]
### Required Test Cases
- Read (found, not found, error)
- Create (success, error)
- Update (success, noop, error)
- Delete (success, not found, error)
- Diff (no changes, changes detected)
- Check (valid, invalid)
## Writing Integration Tests
### Using pulumitest Framework
[Example with code]
### Test Lifecycle
1. Setup: Create resources
2. Verify: Check outputs
3. Update: Modify resources
4. Verify: Check idempotency
5. Cleanup: Destroy resources
### Local Testing
[Environment setup, .env file, etc.]
## Best Practices
### DO:
- ✅ Use descriptive test names
- ✅ Test error cases
- ✅ Use subtests for organization
- ✅ Mock external dependencies
- ✅ Clean up resources
- ✅ Add comments for complex tests
### DON'T:
- ❌ Test other languages (only YAML)
- ❌ Make real API calls in unit tests
- ❌ Leave hardcoded tokens
- ❌ Commit .env files
- ❌ Skip error handling tests
## Debugging Tests
### Common Issues
[List of common problems and solutions]
### Getting Help
[Where to ask questions]
## CI/CD Pipeline
### Test Stages
1. Unit tests (2 minutes)
2. Integration tests (3 hours, 6 shards)
### Test Sharding
[Explanation of how tests are sharded]
### Flaky Tests
[How to identify and fix]
## Reference
- Example unit test: `provider/pkg/resources/team_test.go`
- Example integration test: `examples/examples_yaml_test.go`
- Testing epic: #572Acceptance Criteria
- CLAUDE.md updated with testing requirements
- Test templates created
- PR template includes testing checklist
- Documentation reviewed by team
Estimated Effort
1 week
Metadata
Metadata
Assignees
Labels
kind/documentationImprovements or additions to documentationImprovements or additions to documentationkind/taskWork that's part of an ongoing epicWork that's part of an ongoing epic