Pipeline
@ap/pipeline is a domain-agnostic multi-stage pipeline orchestrator with Monte Carlo simulation support. Extracted from Aura’s financial forecasting pipeline, it is designed to serve any AP app that needs staged data processing.
Overview
The package implements a four-stage processing pattern — Import → Analyze → Generate → Report — where each stage is a pure function with typed I/O. The orchestrator manages stage execution order, error handling, and run lifecycle. Storage keys are parameterized via createStorageKeys(namespace) to prevent domain-specific prefix collisions.
The financial domain (transactions, goals, assets, transfers) is the reference implementation under src/domains/financial/. Downstream apps such as Athenaeum and Sentinel implement their own domain types against the generic stage interfaces without touching pipeline core.
Architecture
| Layer | Path | Responsibility |
|---|---|---|
| Orchestrator | orchestrator.ts | Stage execution order, error handling, run lifecycle |
| Stages | stages/ | Import, Analyze, Generate, Report — pure functions with typed I/O |
| Adapters | adapters/ | CLI, CI, UI — execution context variants |
| Reporters | reporters/ | Console, JSON, Markdown — output formatting |
| Config | config.ts | Zod schemas for pipeline configuration validation |
| Financial domain | src/domains/financial/ | Reference implementation (Transaction, Goal, Asset, Transfer) |
| Generic interfaces | src/stages.ts | ImportStage, AnalysisStage — base contracts for custom domains |
| Storage | src/services/storage.ts | createStorageKeys(namespace) — parameterized key generation |
Tech Stack
| Dependency | Version | Purpose |
|---|---|---|
| TypeScript | ^5.6.0 | Language / type system (strict mode, no any) |
| zod | ^3.22.4 | Runtime schema validation for pipeline configs |
| nanoid | ^5.1.6 | Unique ID generation for pipeline runs |
| papaparse | ^5.5.3 | CSV parsing in the import stage |
| vitest | ^4.0.18 | Test runner |
| OpenTelemetry SDK | ^0.212.0 | Observability / telemetry |
| simple-git | ^3.31.1 | Git integration for repo analysis scripts |
Key Decisions
Domain-Agnostic Extraction (2026-02-06)
The pipeline was extracted from apps/Aura/src/pipeline/ as a generic orchestrator rather than a financial-forecasting-specific tool. The four-stage pattern (import → analyze → generate → report) accepts typed configs and produces typed outputs, making it reusable across Athenaeum (document processing), Sentinel (monitoring), and future apps needing staged processing with Monte Carlo simulation.
Trade-off: extraction required replacing Aura-specific type imports with generic interfaces — additional upfront work, but prevents duplicate pipeline implementations across the ecosystem.
Source App Remains Intact (2026-02-06)
The original Aura pipeline files were not modified or removed during extraction. Aura continues to operate with its own copy. Migration to the shared package is a separate, deliberate step.
Domain Separation Complete (2026-02-12)
Financial types moved to src/domains/financial/types.ts. Generic stage interfaces (ImportStage, AnalysisStage) added in src/stages.ts. Storage keys parameterized via createStorageKeys(namespace) — no more hardcoded aura_ prefix. All existing code paths preserved with backward-compatible re-exports.
Test Suite Fix (2026-02-12)
__tests__/pipeline.test.ts had incorrect relative imports (referenced ./orchestrator instead of ../src/pipeline/orchestrator) and storage tests relied on localStorage without a browser environment. Fixed by correcting import paths, adding a localStorage mock via vi.stubGlobal, and adding vitest.config.ts. All 42 tests pass.
Current Status
Phase: Active Development (Staging) — Last updated 2026-02-12
Domain-agnostic refactor complete. Typecheck clean. 42/42 tests passing. Comprehensive audit completed 2026-02-15. Athenaeum and Sentinel are unblocked for integration.
Next Steps
- Integration test with Athenaeum as first consumer
- Integration test with Sentinel as second consumer
- Evaluate whether zod and papaparse should be peer dependencies
- Compliance review → graduate to
apps/
See Also
- Aura — origin app; financial forecasting pipeline extracted from here
- Athenaeum — first planned consumer of the shared pipeline package
- Sentinel — second planned consumer of the shared pipeline package
- Monte Carlo Goal Forecasting — simulation pattern used in the generate stage