Knowledge Vault

Infrastructure sync discipline

2 min read

Infrastructure sync discipline

Update ALL related scripts, configs, and documentation atomically when infrastructure changes.

Context

Infrastructure often has multiple touchpoints: provisioning scripts, cleanup scripts, monitoring configs, documentation, and capacity planning. When any component changes, the others can easily fall out of sync.

Problem

# cleanup-runners.sh - outdated list
RUNNERS="runner-1 runner-2 runner-3"
 
# Reality: 7 runners now exist after adding Library repos
# runner-1 through runner-7 are actually deployed
 
# Result: 4 runners never get cleaned up, accumulating stale data

When Library repos were added and required additional runners, the cleanup script wasn’t updated. This created a drift between deployed infrastructure and maintenance automation.

Solution

Treat infrastructure changes as atomic operations that must update all related files:

  1. Create a checklist for infrastructure changes:
## Runner Addition Checklist
 
- [ ] Add runner to docker-compose.yml
- [ ] Update cleanup script with new runner name
- [ ] Update monitoring/health check script
- [ ] Update capacity documentation (CPU/RAM)
- [ ] Update runner count in README
  1. Use variables or config files:
# config/runners.conf
RUNNER_COUNT=7
RUNNER_PREFIX="runner"
 
# cleanup-runners.sh - reads from config
source config/runners.conf
for i in $(seq 1 $RUNNER_COUNT); do
  cleanup "${RUNNER_PREFIX}-${i}"
done
  1. Add validation scripts:
# validate-infrastructure-sync.sh
DEPLOYED=$(docker ps --format '{{.Names}}' | grep runner | wc -l)
CONFIGURED=$(grep -c "runner-" cleanup-runners.sh)
if [ "$DEPLOYED" != "$CONFIGURED" ]; then
  echo "DRIFT DETECTED: $DEPLOYED deployed vs $CONFIGURED configured"
  exit 1
fi

Reusability

  • Score: 5/5
  • Used in: aloysiusproductions-ops-core runner management

Trade-offs

  • Pro: Prevents infrastructure drift
  • Pro: Makes dependencies explicit
  • Pro: Catches issues before they become outages
  • Con: Adds overhead to infrastructure changes
  • Con: Requires discipline to maintain checklists

Source

Discovered during workflow audit on 2026-01-23. Commit 5132a1a corrected runner count from 3 to 7 in cleanup scripts.

Graph View

Backlinks