Stock Art Collector

A CLI and web tool for collecting and managing stock art assets from multiple provider APIs. Features async downloads, rate limiting, duplicate detection, license verification, and attribution management. Graduated from staging to apps/ zone on 2026-01-29.

Overview

Stock Art Collector (SAC) aggregates stock media assets — photos, videos, vectors, audio — from Unsplash, Pexels, Pixabay, Freepik, and Coverr through a uniform adapter interface. Assets are stored in a local SQLite catalog with full metadata, quality thresholds, and license tracking. A FastAPI + HTMX web interface (port 3070) provides search and collection management alongside the Click CLI.

The project dogfoods the Source Adapter Interface pattern with a dynamic registry that auto-discovers adapters at runtime — no hardcoded provider lists.

Current phase: Phase 6 — UX Polish + Performance Tuning Maturity: L3 (Verified) — Multi-source E2E confirmed 2026-03-11 Test coverage: >85% — 214/215 tests passing (1 known flaky serve test)

Tech Stack

Technology	Version	Purpose
Python	3.11+	Runtime — async support
SQLite / SQLAlchemy	3.x / 2.0+	Local catalog, async ORM
aiosqlite	0.x	Async SQLite driver
Click	8.x	CLI framework
FastAPI	0.109+	Web interface (port 3070)
HTMX	1.9+	Server-driven frontend
httpx	0.27+	Async HTTP client for API calls
Pydantic	2.x	Type-safe config with SecretStr
imagehash	4.x	Perceptual hashing for duplicate detection
Pillow	10.x	Image metadata / EXIF extraction
FFprobe	—	Video metadata (FFmpeg suite)
Mutagen	1.47+	Audio metadata — ID3, duration, bitrate
Alembic	—	Database migrations

Source adapters: Unsplash, Pexels, Pixabay, Freepik, Coverr

Key Decisions

Decision	Rationale	Outcome
SQLite for local storage	Simplicity, portability, no external dependencies	Implemented via aiosqlite
HTMX for web interface	Minimal JS, server-driven interactions	Phase 7 — implemented
Click for CLI framework	Mature, well-documented, composable	Implemented
Pydantic for configuration	Type-safe settings, env var support	Implemented (CB-SAC-001)
imagehash for duplicate detection	Multiple hash algorithms, pure Python	Implemented (CB-SAC-006)
Token bucket for rate limiting	Simple, effective, per-source time windows	Implemented (CB-SAC-003)
Adapter pattern for sources	Uniform interface, easy to add providers	Implemented — 5 adapters
Dynamic source registry	Centralised auto-discovery, no hardcoded lists	Implemented (SRC-01)
Streaming downloads	httpx stream + chunked aiofiles writes, prevents OOM on video	Implemented (DL-01)
SecretStr for API keys	Prevents accidental logging of credentials	Implemented
Path traversal protection	Path.resolve() + is_relative_to() on all user-supplied paths	Implemented (Phase 8)

Circuit Breakers

6 circuit breakers guard production integrity:

ID	Trigger	Priority	Response
CB-SAC-001	Hardcoded API key detected	P1 CRITICAL	Block commit, rotate key
CB-SAC-002	Asset without verified license	P1 CRITICAL	Quarantine, require manual review
CB-SAC-003	3 consecutive 429 responses	P2 HIGH	15-min cooldown, exponential backoff
CB-SAC-004	CC-BY asset missing attribution	P2 HIGH	Block export until attribution added
CB-SAC-005	Asset below quality threshold	P3 MEDIUM	Warning on import, optional override
CB-SAC-006	>10% duplicates in batch	P3 MEDIUM	Review before import completes

Patterns Established

Pattern	Description	Reusable
Source adapter interface	Abstract base with search, get_asset, download, check_health	Yes
Dynamic source registry	Auto-discovery singleton, eliminates hardcoded source lists	Yes
Circuit breaker	Configurable thresholds, HTTP fault tolerance	Yes
Token bucket rate limiting	Per-source windows (per-hour, per-minute)	Yes
Async session context manager	Commit/rollback, prevents lazy loading in async ORM	Yes
Perceptual hash similarity	Multi-algorithm (2/3 required for match)	Yes
SecretStr for credentials	Pydantic pattern to prevent secret logging	Yes
Path traversal protection	Validate user paths against allowed roots	Yes

Lessons Learned

• Empty SecretStr config bug: KEY= in .env becomes SecretStr(''), not None. Sources with empty keys falsely reported as configured and failed at runtime. Fix: _has_api_key() checks bool(value.get_secret_value()).
• Pydantic-settings + tests: Config tests fail when .env exists because pydantic-settings loads it automatically. Use Settings(_env_file=None) for tests asserting default/unconfigured state.
• Download-to-catalog gap: sac download saves file to disk but never inserts the Asset record. sac list --downloaded-only returns empty. Pipeline resolver (sac://) depends on catalog — persistence step needed.
• async SQLAlchemy relationship loading: N+1 queries if not explicit. Always use selectinload/joinedload in async context.
• FastAPI lifespan pattern: on_event("startup") is deprecated — use asynccontextmanager lifespan.
• Click async wrapper: Click is sync-only. Async CLI commands require run_async() wrapper pattern.

Technical Debt

Item	Priority	Notes
Download-to-catalog gap	High	sac download skips catalog insert — pipeline resolver broken
Flaky serve test	Medium	TestServeCommand blocks pytest — needs pytest-timeout or mock
Video metadata requires FFmpeg	Medium	External dep, should degrade gracefully
No persistent rate limit state	Low	BG-007 — state resets on restart
Export hardcodes limit=1000	Low	Configurable via --limit flag added in Phase 1 tech debt

Status

Phase 6 — UX Polish + Performance Tuning (active as of 2026-03-11)

• All 5 source adapters verified: Unsplash, Pexels, Pixabay, Freepik, Coverr
• 214/215 tests passing; 1 known flaky serve test (BG-008)
• Rate limit persistence pending (BG-007)
• Web UI (port 3070): search results template optimization + asset detail view in progress

Phase	Status	Key Deliverables
0: Compliance	Complete	CI, Dependabot, pre-commit, concerns.md
1: Core Infrastructure	Complete	Config, models, DB session, migrations
2: Source Adapters	Complete	Unsplash, Pexels, Pixabay adapters
3: Metadata & Organization	Complete	Photo/video/audio metadata, hashing, duplicates
4: CLI Implementation	Complete	All commands functional
5: New Source Adapters	Complete (2026-03-11)	Freepik, Coverr, parallel aggregation
6: UX Polish	In Progress	Port 3070 UI, asset detail, export UX

Related

• Circuit Breaker — Fault tolerance pattern used for external APIs
• Token Bucket Rate Limiting — Rate limiting implementation
• Source Adapter Interface — Uniform API across stock providers
• Async Session Context Manager — SQLAlchemy async pattern
• Perceptual Hash Similarity — Duplicate detection
• Source Adapter — Companion project page