Data Integrity Patterns

Patterns for maintaining data consistency and preventing drift in knowledge systems.

Cross-Reference Validation Pattern

Verify all internal links before adding or modifying content:

## Before Adding a Link
 
1. Check target exists: `ls obsidian/path/to/target.md`
2. Verify frontmatter matches expected type
3. Add bidirectional link in target if appropriate

When to Apply: Every time you add a [[wikilink]] or markdown link to another page.

Anti-pattern: Creating links to pages that don’t exist, expecting them to be created later.

Single Source of Truth Pattern

Maintain one authoritative source for each piece of data:

Data Type	Single Source	Consumers
Project status	.management/knowledge.md in each repo	project-status.md
Technical debt	Project .management/knowledge.md	technical-debt.md registry
Version info	Frontmatter in project page	Index pages, dashboards
Research outcomes	research/completed/*.md	Project pages

Sync Rule: Aggregated views pull from sources; never manually edit aggregated data.

# Good: Sync from source
python scripts/sync-knowledge.py
 
# Bad: Manual edit in aggregated file
# This will be overwritten on next sync

Stub Page Lifecycle Pattern

When a 404 is detected, follow this lifecycle instead of creating empty stubs:

┌─────────────────────────────────────────────────────┐
│ 1. 404 DETECTED                                     │
│    └─→ Check if page should exist                   │
│        ├─→ No: Remove the broken link               │
│        └─→ Yes: Continue to step 2                  │
├─────────────────────────────────────────────────────┤
│ 2. LOCATE SOURCE CONTENT                            │
│    └─→ Find authoritative source for this content   │
│        ├─→ Found: Create page with real content     │
│        └─→ Not found: Create as "draft" with TODO   │
├─────────────────────────────────────────────────────┤
│ 3. DRAFT PAGE (if source not found)                 │
│    status: draft                                    │
│    tags: [needs-content]                            │
│    Body: "TODO: Source content from X"              │
├─────────────────────────────────────────────────────┤
│ 4. REVIEW DRAFTS                                    │
│    Quarterly audit of all status: draft pages       │
│    Either populate or delete                        │
└─────────────────────────────────────────────────────┘

Anti-pattern: Creating stub pages with placeholder content like “Coming soon” or “TBD”.

Frontmatter Consistency Pattern

Required frontmatter fields by content type:

Content Type	Required Fields
project	title, created, updated, type, status, version
research	title, created, updated, type, status, outcome
reference	title, created, updated, type, tags
registry	title, created, updated, type, source

Validation Checks:

1. status must be one of: draft, active, completed, archived
2. version must follow semver pattern: v1.0, v0.3
3. updated must be ≥ created
4. source for synced content must reference a valid path

Quarterly Audit Checklist

Run this checklist every quarter:

• Run link validation: npm run validate:links
• Check for stale sync timestamps (>30 days)
• Verify project-status.md against source repos
• Review all status: draft pages
• Check for orphaned pages (no incoming links)
• Validate frontmatter consistency
• Update index pages if projects added/removed

Audit Findings (January 2026)

Lessons learned from the comprehensive Knowledge Vault audit:

Sync Process Duplicate Prevention

The ops-core sync process can create duplicates when research moves from active to completed:

Issue	Cause	Prevention
Duplicate research docs	Active and completed versions synced separately	Sync script should check for existing file in alternate directory before creating
Status mismatch	Active version not deleted when completed synced	Include cleanup step in sync process

Pattern: Before syncing a research document, check if it exists in the opposite status directory (active vs completed). If completed version exists, skip or delete the active version.

Missing Project Page Links

Links to project pages can break when:

1. New Library submodules added without corresponding project pages
2. Project page paths referenced before creation
3. Renamed projects with outdated references

Prevention: Run link validation after adding new submodules or project references.

Blog Publication Workflow

Blog posts require explicit status updates:

Status	Meaning
draft	Work in progress, not ready for publication
published	Complete content, ready for public consumption

Pattern: Complete content + status: draft = needs review. Update to status: published after final review.

Best Practices

Practice	Why
Validate before commit	Catch broken links early
Use source syncing	Prevents manual drift
Never create empty stubs	Better to fix the link
Quarterly audits	Catch accumulated drift
Bidirectional linking	Improves discoverability
Check for duplicates	Sync can create pairs
Explicit publication	Blog posts need workflow

Related

• Knowledge Vault Data Integrity Audit
• Data Integrity Initiatives
• Ops-Core Patterns