Add Nextflow Development Constitution

.specify/memory/constitution.md

@@ -1,50 +1,198 @@
-# [PROJECT_NAME] Constitution
-<!-- Example: Spec Constitution, TaskFlow Constitution, etc. -->
+# Nextflow Development Constitution
+
+<!--
+SYNC IMPACT REPORT
+==================
+Version Change: INITIAL → 1.0.0 (Initial constitution)
+Modified Principles: N/A (new constitution)
+Added Sections:
+  - All principles (I-VII)
+  - Development Workflow
+  - Quality Standards
+  - Governance
+
+Removed Sections: N/A (initial version)
+
+Templates Status:
+  ✅ plan-template.md - Reviewed, aligned with modular architecture and testing principles
+  ✅ spec-template.md - Reviewed, aligned with user scenario focus and requirements structure
+  ✅ tasks-template.md - Reviewed, aligned with test-driven and parallel development principles
+  ✅ agent-file-template.md - Reviewed, no agent-specific conflicts
+  ✅ checklist-template.md - Reviewed, compatible with quality standards
+
+Follow-up TODOs:
+  - None at this time
+==================
+-->
 
 ## Core Principles
 
-### [PRINCIPLE_1_NAME]
-<!-- Example: I. Library-First -->
-[PRINCIPLE_1_DESCRIPTION]
-<!-- Example: Every feature starts as a standalone library; Libraries must be self-contained, independently testable, documented; Clear purpose required - no organizational-only libraries -->
+### I. Modular Architecture
+
+Nextflow MUST maintain a clear separation between core functionality and extensions through its modular architecture:
+
+- **Core modules** (`modules/`) contain essential functionality: core workflow runtime & plugin system (nextflow), shared utilities (nf-commons), language parsing (nf-lang), HTTP filesystem support (nf-httpfs), and lineage tracking (nf-lineage)
+- **Core plugins** (`plugins/`) provides cloud provider integrations (AWS, Azure, GCP), execution platforms (Kubernetes), and specialized services (Seqera Platform, Wave container management)
+- New features MUST be evaluated for placement: core features belong in `modules/`, specialized/cloud-specific features belong in `plugins/`
+- Each module and plugin MUST be independently buildable and testable
+- Plugin dependencies MUST be explicitly declared in `build.gradle` with semantic versioning
+
+**Rationale**: This architecture enables independent development of cloud provider features without core engine changes, supports third-party plugin development, and maintains a clean separation of concerns across a large multi-module codebase.
+
+### II. Test-Driven Quality Assurance (NON-NEGOTIABLE)
+
+Testing MUST be comprehensive and multi-layered before any code is merged:
+
+- **Unit tests** MUST use Spock Framework for all Groovy code, be independently executable, and achieve meaningful coverage (measured via JaCoCo)
+- **Integration tests** (`tests/` directory) MUST validate end-to-end workflows using actual `.nf` scripts with expected outputs
+- **Smoke tests** (`make smoke` or `NXF_SMOKE=1`) MUST be available to skip long-running and cloud-dependent tests during rapid development
+- **Cloud validation tests** (`validation/` directory) MUST verify cloud provider integrations end-to-end before release
+- **Documentation tests** (`docs/snippets/`) MUST ensure all documentation examples remain functional
+- All tests MUST pass before commits, and `make test` MUST be run locally before pushing
+
+**Rationale**: Scientific workflows demand reliability and reproducibility. Multi-layered testing catches issues at appropriate levels: unit tests for logic, integration tests for workflow correctness, and validation tests for cloud provider compatibility.
+
+### III. Dataflow Programming Model
+
+Nextflow's core abstraction MUST adhere to following principles and guidelines:
+
+- The dataflow programming model is the fundamental abstraction in the Nextflow programming model.
+- Workflows are defined as dataflow graphs where data flows between processes.
+- Processes MUST be stateless, side-effect-free transformations that communicate via channels.
+- Workflows should be defined in a platform agnostic manner to enable portability across clusters and clouds.
+- The DSL MUST prioritize expressiveness for concurrent and parallel pipeline definition.
+- Changes to the language parser (ANTLR grammars in `nf-lang`) MUST preserve backward compatibility with existing pipelines unless explicitly versioned (DSL1 vs DSL2).
+- Concurrency primitives (GPars actors/dataflow) MUST be used correctly to maintain the dataflow semantics.
+
+**Rationale**: The dataflow model is Nextflow's fundamental abstraction, enabling automatic parallelization and distribution. Preserving this model ensures existing scientific pipelines continue to work and users can reason about workflow behavior.
+
+### IV. Apache 2.0 License Compliance
+
+All source code MUST include Apache 2.0 license headers:
+
+- Every source file MUST begin with the Apache 2.0 license header
+- All contributions MUST comply with Apache 2.0 terms
+- Third-party dependencies MUST use compatible licenses
+- License compliance MUST be verified during code review
+
+**Rationale**: Legal clarity protects both contributors and users. Consistent licensing enables academic and commercial use, which is critical for scientific software adoption.
+
+### V. Developer Certificate of Origin (DCO) Sign-off
+
+All commits MUST be signed with DCO certification:
+
+- Contributors MUST certify they have the right to submit the code by using `git commit -s` or `git commit --signoff`
+- Every commit message MUST include a `Signed-off-by` line
+- The DCO bot MUST verify sign-off before any PR can be merged
+- Contributors MUST NOT bypass the DCO requirement
 
-### [PRINCIPLE_2_NAME]
-<!-- Example: II. CLI Interface -->
-[PRINCIPLE_2_DESCRIPTION]
-<!-- Example: Every library exposes functionality via CLI; Text in/out protocol: stdin/args → stdout, errors → stderr; Support JSON + human-readable formats -->
+**Rationale**: DCO provides legal protection and clear chain of custody for contributions, which is essential for open-source projects with diverse contributors.
 
-### [PRINCIPLE_3_NAME]
-<!-- Example: III. Test-First (NON-NEGOTIABLE) -->
-[PRINCIPLE_3_DESCRIPTION]
-<!-- Example: TDD mandatory: Tests written → User approved → Tests fail → Then implement; Red-Green-Refactor cycle strictly enforced -->
+### VI. Semantic Versioning and Release Discipline
 
-### [PRINCIPLE_4_NAME]
-<!-- Example: IV. Integration Testing -->
-[PRINCIPLE_4_DESCRIPTION]
-<!-- Example: Focus areas requiring integration tests: New library contract tests, Contract changes, Inter-service communication, Shared schemas -->
+Version management MUST follow strict semantic versioning with calendar-based releases:
 
-### [PRINCIPLE_5_NAME]
-<!-- Example: V. Observability, VI. Versioning & Breaking Changes, VII. Simplicity -->
-[PRINCIPLE_5_DESCRIPTION]
-<!-- Example: Text I/O ensures debuggability; Structured logging required; Or: MAJOR.MINOR.BUILD format; Or: Start simple, YAGNI principles -->
+- **Project versions** use calendar-based scheme: `YY.MM.PATCH` where April (`.04.`) and October (`.10.`) are stable releases, all other months use `-edge` suffix (e.g., `25.09.0-edge`)
+- **Plugin versions** MUST use semantic versioning (`MAJOR.MINOR.PATCH`)
+- Version changes MUST be documented in `changelog.txt` files (both project root and per-plugin)
+- Breaking changes MUST increment MAJOR version for plugins and be clearly documented
+- Release process MUST follow the documented procedure in `CLAUDE.md` including: updating changelogs, version files, running `make releaseInfo`, using `[release]` tag in commit message
 
-## [SECTION_2_NAME]
-<!-- Example: Additional Constraints, Security Requirements, Performance Standards, etc. -->
+**Rationale**: Predictable versioning enables users to understand compatibility and stability expectations. Calendar-based versioning for the main project makes release timing transparent, while semantic versioning for plugins enables clear communication of breaking changes.
 
-[SECTION_2_CONTENT]
-<!-- Example: Technology stack requirements, compliance standards, deployment policies, etc. -->
+### VII. Groovy Idioms and Code Standards
 
-## [SECTION_3_NAME]
-<!-- Example: Development Workflow, Review Process, Quality Gates, etc. -->
+Code MUST follow Groovy best practices and Nextflow conventions:
 
-[SECTION_3_CONTENT]
-<!-- Example: Code review requirements, testing gates, deployment approval process, etc. -->
+- Use Groovy idioms (closures, operator overloading, DSL builders) appropriately
+- Follow existing code patterns and conventions from similar modules
+- Leverage Groovy's dynamic capabilities judiciously without sacrificing type safety where beneficial
+- Use Groovy's `@CompileStatic` where performance is critical or type safety is desired
+- AST transformations (in `modules/nextflow`) MUST be well-documented due to their compile-time magic
+- Code MUST be formatted consistently (consider CodeNarc configuration in `gradle/codenarc.groovy`)
+
+**Rationale**: Groovy enables powerful DSL capabilities that make Nextflow's language expressive, but requires discipline to maintain readability and debuggability. Consistency across the large codebase improves maintainability.
+
+## Development Workflow
+
+### Build and Development Process
+
+- **Build tool**: Gradle with wrapper (`./gradlew`) is the authoritative build system
+- **Quick commands**: Makefile provides convenience targets (`make compile`, `make test`, `make assemble`, `make check`, `make clean`)
+- **Development testing**: Use `./launch.sh run script.nf` for testing changes against real workflows without full installation
+- **Local installation**: `make install` publishes to Maven local for integration testing
+- **Dependency management**: All dependencies MUST be declared in `build.gradle` with explicit versions; use `make deps` to analyze dependency trees
+
+### Git Workflow
+
+- **Branch management**: Work on feature branches, never commit directly to `master`
+- **Commit sign-off**: Always use `git commit -s` to add DCO sign-off
+- **CI control tags**: Use special commit message tags to control CI behavior:
+  - `[ci skip]` - Skip CI tests entirely
+  - `[ci fast]` - Run only unit tests, skip integration tests
+  - `[e2e stage]` - Run end-to-end tests against Seqera platform staging environment
+  - `[e2e prod]` - Run end-to-end tests against production platform
+  - `[release]` - Trigger release automation
+- **Pull requests**: Must pass all CI checks, require code review, and have DCO verification
+
+### Architecture Decision Records (ADRs)
+
+- Significant structural and technical decisions MUST be documented as ADRs in the `adr/` directory
+- ADRs MUST follow the template format: date prefix + descriptive name (e.g., `20251114-module-system.md`)
+- ADRs provide historical context for why architectural decisions were made
+- When changing fundamental architecture, review existing ADRs and create new ones documenting the rationale
+
+## Quality Standards
+
+### Code Review Requirements
+
+- All changes MUST go through pull request review
+- Reviewers MUST verify:
+  - Tests are included and passing
+  - Code follows Groovy idioms and project conventions
+  - License headers are present
+  - DCO sign-off is present
+  - Changes align with modular architecture principles
+  - Breaking changes are appropriately versioned and documented
+
+### Testing Gates
+
+- `make test` MUST pass before committing locally
+- All CI tests MUST pass before merging
+- Integration tests MUST be run for changes affecting workflow execution
+- Cloud validation tests MUST be run before releases touching cloud provider plugins
+- Smoke tests enable rapid iteration but MUST NOT replace full test execution
+
+### Performance and Compatibility
+
+- Target platform: Java 17 runtime compatibility (development uses Java 21 toolchain)
+- Performance-critical paths SHOULD be profiled and optimized
+- Memory usage SHOULD be monitored for large-scale workflows
+- Backward compatibility MUST be maintained for existing DSL features unless a new DSL version is introduced
 
 ## Governance
-<!-- Example: Constitution supersedes all other practices; Amendments require documentation, approval, migration plan -->
 
-[GOVERNANCE_RULES]
-<!-- Example: All PRs/reviews must verify compliance; Complexity must be justified; Use [GUIDANCE_FILE] for runtime development guidance -->
+### Amendment Process
+
+This constitution supersedes all other development practices. Amendments require:
+
+1. **Proposal**: Submit amendment proposal via GitHub issue or pull request
+2. **Discussion**: Community discussion period (minimum 1 week for major changes)
+3. **Approval**: Approval from core maintainers
+4. **Documentation**: Update this constitution with version bump following semantic versioning:
+   - **MAJOR**: Backward incompatible governance changes, principle removal/redefinition
+   - **MINOR**: New principle added or materially expanded guidance
+   - **PATCH**: Clarifications, wording improvements, typo fixes
+
+### Compliance and Review
+
+- All pull requests and code reviews MUST verify compliance with these principles
+- Deviations from principles MUST be explicitly justified in PR description
+- Complexity additions MUST be justified against the "simplicity first" principle
+- Constitution compliance is enforced through code review and CI automation where possible
+
+### Ratification and Version History
+
+**Version**: 1.0.0 | **Ratified**: 2025-11-17 | **Last Amended**: 2025-11-17
 
-**Version**: [CONSTITUTION_VERSION] | **Ratified**: [RATIFICATION_DATE] | **Last Amended**: [LAST_AMENDED_DATE]
-<!-- Example: Version: 2.1.1 | Ratified: 2025-06-13 | Last Amended: 2025-07-16 -->
+This constitution was derived from the Nextflow project's documented practices in `CLAUDE.md`, `CONTRIBUTING.md`, and the project's existing architectural patterns. It codifies the development principles that have made Nextflow a successful scientific workflow management system.