Skip to content

Create comprehensive testing documentation #580

@rshade

Description

@rshade

Context

As testing practices mature, we need comprehensive documentation to guide contributors and maintain consistency.

Part of Epic: #572

Documentation Needed

1. CONTRIBUTING.md (Update)

Comprehensive guide covering:

  • Overview of testing strategy
  • Types of tests (unit, integration, regression)
  • How to write unit tests
  • How to write integration tests
  • Test patterns and best practices
  • How to run tests locally
  • How to debug failing tests
  • CI/CD test pipeline explanation

2. Update CLAUDE.md

  • Add testing requirements for new resources
  • Reference TESTING.md
  • Add test review checklist
  • Update example patterns with tests

3. PR Template Updates

  • Add testing checklist to PR template
  • Require test evidence for bug fixes
  • Require test additions for new features

CONTRIBUTING.md Outline

# Testing Guide for pulumi-pulumiservice

## Overview
[Purpose, strategy, coverage goals]

## Test Types

### Unit Tests
- Location: `provider/pkg/resources/*_test.go`
- Purpose: Test resource implementations in isolation
- Speed: Fast (<10 seconds per resource)
- Dependencies: None (uses mocks)
- Run: `cd provider/pkg && go test ./...`

### Integration Tests
- Location: `examples/examples_yaml_pulumitest_test.go`
- Purpose: Test end-to-end functionality against real API
- Speed: Slow (3 hours total, sharded)
- Dependencies: PULUMI_ACCESS_TOKEN, test organization
- Run: `cd examples && go test -tags=yaml -v -timeout 10m`

### Regression Tests
- Purpose: Prevent known bugs from recurring
- Location: Embedded in unit/integration tests
- Marked with: Issue reference in comment

## Writing Unit Tests

### Pattern 1: Mock-Based Testing
[Example with code]

### Pattern 2: Table-Driven Tests
[Example with code]

### Required Test Cases
- Read (found, not found, error)
- Create (success, error)
- Update (success, noop, error)
- Delete (success, not found, error)
- Diff (no changes, changes detected)
- Check (valid, invalid)

## Writing Integration Tests

### Using pulumitest Framework
[Example with code]

### Test Lifecycle
1. Setup: Create resources
2. Verify: Check outputs
3. Update: Modify resources
4. Verify: Check idempotency
5. Cleanup: Destroy resources

### Local Testing
[Environment setup, .env file, etc.]

## Best Practices

### DO:
- ✅ Use descriptive test names
- ✅ Test error cases
- ✅ Use subtests for organization
- ✅ Mock external dependencies
- ✅ Clean up resources
- ✅ Add comments for complex tests

### DON'T:
- ❌ Test other languages (only YAML)
- ❌ Make real API calls in unit tests
- ❌ Leave hardcoded tokens
- ❌ Commit .env files
- ❌ Skip error handling tests

## Debugging Tests

### Common Issues
[List of common problems and solutions]

### Getting Help
[Where to ask questions]

## CI/CD Pipeline

### Test Stages
1. Unit tests (2 minutes)
2. Integration tests (3 hours, 6 shards)

### Test Sharding
[Explanation of how tests are sharded]

### Flaky Tests
[How to identify and fix]

## Reference
- Example unit test: `provider/pkg/resources/team_test.go`
- Example integration test: `examples/examples_yaml_test.go`
- Testing epic: #572

Acceptance Criteria

  • CLAUDE.md updated with testing requirements
  • Test templates created
  • PR template includes testing checklist
  • Documentation reviewed by team

Estimated Effort

1 week

Metadata

Metadata

Assignees

Labels

kind/documentationImprovements or additions to documentationkind/taskWork that's part of an ongoing epic

Type

No type

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions