Organization as Code Infrastructure

Summary

Organization as Code (OaC) elevates infrastructure management from manual operations to continuous reconciliation, where GitHub acts as the operational brain and single source of truth for physical infrastructure. The architecture shifts from “optimistic push” to deterministic pull, enabling self-healing systems where machines continuously align themselves with declared state, runners automatically resurrect after failures, and backup systems operate without disrupting production workloads.

Core Operating Model: Pull-Based Reconciliation

The fundamental architectural pattern is that GitHub records desired state, but does not push state to machines. Instead, physical infrastructure must continuously reconcile itself to declared state.

The Sync Gap

Traditional CI/CD workflows push changes once and disappear. If a machine is offline during deployment, it misses the update permanently. OaC closes this gap through scheduled agents running on every node.

Ansible Pull Mode

Each machine runs a periodic task (every 15-30 minutes via cron/Task Scheduler):

git pull origin main
ansible-playbook site.yml

This creates automatic convergence: 15 minutes after a merge to main, the entire fleet aligns itself with the new standard without human intervention. The agent aggressively reverts unauthorized changes, deleted files, or stopped services, ensuring reality matches declaration.

GitOps Philosophy

Configuration management must be pull-based rather than push-based:

Machines “pull” their instructions rather than receiving “pushed” commands
Disconnected or broken nodes automatically recover upon reconnection
Drift is overwritten every 15 minutes rather than accumulating until failure
Humans declare state in Git; machines enforce it continuously

This pattern is linked to ssh-fleet-management and organization-as-platform-architecture for broader infrastructure coordination.

Self-Hosted Runner Stabilization

Docker-based GitHub Actions runners on consumer hardware exhibit three primary failure modes, all solvable through architectural discipline.

Host Power Management

Consumer hardware prioritizes energy savings over uptime. The “Stay Awake” protocol forces high-power states:

macOS (M2/M4 Mini):

sudo pmset -a sleep 0 displaysleep 0

Or use System Settings > Energy Saver > “Prevent automatic sleeping when display is off”

Windows:

Control Panel > Power Options > High Performance
Set “Put the computer to sleep” to Never

Linux (ThinkPad T14s):

systemctl mask sleep.target suspend.target hibernate.target hybrid-sleep.target

This single change eliminates ~40% of runner instability.

Container Immortality: Docker Restart Policies

Standard Docker containers are ephemeral by design. Runners die when processes crash, updates occur, or the Docker daemon restarts. The OaC standard mandates Docker Compose with resurrection policies.

Required docker-compose.yml pattern:

version: '3'
services:
  github-runner:
    image: my-runner-image:latest
    restart: unless-stopped  # Auto-restart on crash/reboot
    volumes:
      - /var/run/docker.sock:/var/run/docker.sock  # Sibling container support
      - ./work:/_work  # Persistent workspace
    environment:
      - ACTIONS_RUNNER_REQUIRE_JOB_CONTAINER=true

Additional requirements:

Runner auto re-registration scripts
Health check watchdogs
Graceful shutdown handling for token refresh

Without these, Docker may restart the container while GitHub still marks the runner as dead, creating “zombie” queues.

Resource Contention from Backups

Heavy I/O spikes from backup cycles saturate network/disk, causing runners to timeout or Docker containers to exit under load. Symptoms include “random” runner stops, network stalls, and disk I/O spikes during backup windows.

Solution: Traffic Isolation

Schedule backups during off-peak hours
Throttle I/O bandwidth for backup processes
Use dedicated VLANs for backup traffic
Isolate backup cycles from CI/CD hours completely

Hardware Architecture: All Hands on Deck

Specialized hardware roles prevent cascading failures and maximize efficiency across mixed architectures (ARM64/x86_64) and operating systems.

The Commander: Mac Mini M4 Pro (Headless Server)

Role: Always-On Headless Orchestrator / Infrastructure Anchor

Primary responsibilities:

Runner coordination and monitoring
Backup aggregation hub
Local Docker registry mirror / brew cache
Ansible control plane (if not using Pull mode)
Central monitoring dashboard (Homepage/Glances)
Emergency failover brain

Critical constraints:

Never sleeps
Rarely reboots
Never used for experimental work
Strictly protected from heavy background databases

This machine manages “cattle,” but is not itself cattle.

The Clean Room: Mac Mini M2

Role: Dedicated iOS/macOS Build Agent

Optimization:

Single-purpose: Apple ecosystem builds only (Xcode, Swift)
No heavy background databases
No backup origination
No experimental workloads

Rationale: If this box flakes, iOS builds halt entirely. Isolation protects build determinism.

The Heavy Lifter: WinPC

Role: Heavy Compute & Windows Truth Source

Best setup:

WSL2 Ubuntu for Linux-based GitHub runners
Docker builds and compute-intensive tasks in WSL2
Native Windows runner only for PowerShell/Windows-specific builds

Benefits:

Linux parity without separate hardware
Windows coverage when required
Better stability than Docker-on-Windows alone

The Roaming Node: ThinkPad T14s

Role: Ephemeral Contributor Node

Configuration:

Registers as runner only when:
- Plugged into AC power
- On trusted network
- Docked physically
Deregisters gracefully on suspend/disconnect

Rationale: Prevents stuck queues waiting for a laptop in a backpack. Dynamic registration/deregistration scripts prevent CI queue failures.

Machine role specialization links to sovereign-meta-engineering infrastructure principles.

Secure Backups: 3-2-1 Strategy as Code

Backups must be code-native: versioned, deduplicated, encrypted client-side, and scriptable. Manual folder dragging is prohibited.

Tool Selection: Restic or Borg

Both tools provide:

Deduplication (Git-like snapshots)
Client-side encryption (stolen machines yield unreadable data)
Incremental snapshots
Mountable filesystems for browsing
Infrastructure-native scripting

Restic advantages:

Go-based, static binary (easy deployment)
Multi-threaded performance
Audited cryptography
Works over SFTP without server-side installation

Borg advantages:

Mature, proven in production
Built-in compression
Slightly better deduplication ratios

Backup Hierarchy

Tier 1: Local → M4 Pro

All machines push encrypted snapshots to M4 Pro
Scheduled during off-peak hours
I/O throttled to prevent runner interference

Tier 2: M4 Pro → Offsite

Nightly sync to cloud storage (AWS S3, Backblaze B2, Wasabi, Cloudflare R2)
Bandwidth capped
Completely isolated from CI hours

Security:

Encryption keys stored in password manager or GitHub Secrets
Never stored on machines themselves
Client-side encryption ensures stolen hardware yields no data
Append-only repositories protect against ransomware

Failover Architecture

Primary (M4 Pro): Central hub pulling data from WinPC and T14s

Secondary (M2 Mini): “Warm” failover—GitHub Action runner labels automatically shift traffic to M2 if M4 Pro goes offline

Offsite: S3-compatible bucket prevents single point of failure via power surge, fire, or network failure

Self-Healing Infrastructure Loop

The OaC model creates a continuous reconciliation cycle that eliminates “maintenance” as a manual task.

The Four-Stage Loop

Detect: Monitoring identifies deviation (configuration drift, stopped service, runner offline)
Diagnose: Automated scripts determine root cause (missing file, crashed process, resource exhaustion)
Remediate: Ansible agent or restart policy corrects the issue (restores file, restarts service, clears queue)
Verify: Health checks confirm successful remediation (runner re-registers, service responds, drift eliminated)

Self-Healing Mechanisms

Container resurrection:

Docker Compose restart: unless-stopped auto-restarts crashed runners
No manual intervention required after power flicker or memory pressure spike

Configuration enforcement:

Ansible pull every 15 minutes overwrites manual changes
Deleted configuration files are restored automatically
Unauthorized edits are reverted

Connectivity resilience:

Overlay networks (Tailscale/WireGuard) provide stable private IPs
Mesh VPN auto-routes connections via NAT traversal
Machines maintain connectivity across network changes (home WiFi → coffee shop → office)

Deterministic Blueprints

Infrastructure state is declared, not described. The repository contains:

infra/
  ansible/          # Machine state declarations
  docker/           # Container definitions
  runners/          # Runner configurations
  backups/          # Backup job definitions
  monitoring/       # Dashboard and alert configs
  secrets/          # (Encrypted) Secret management

Nothing is hand-configured. Nothing is undocumented. Nothing relies on memory.

If it exists, it is declared.

Networking: Zero-Trust Mesh Overlay

Physical network fragility is eliminated through identity-based networking rather than location-based.

Tailscale Mesh VPN

Creates secure, private mesh between devices regardless of physical location:

Stable internal IPs (e.g., 100.x.y.z) persist across network changes
NAT traversal handles restrictive networks automatically
M4 Pro maintains static internal IP reachable from T14s at coffee shop
No port forwarding, no brittle static IPs, no broken-on-router-reboot configs

Identity-Based Access

Devices connect via cryptographic identity, not network topology:

Machines maintain “Organization” membership across physical movement
VPN auto-routes connections when direct paths unavailable
Connectivity self-heals without manual reconfiguration

Workflow Synchronization

GitHub Actions can update the remote repository but should almost never do so directly to avoid infinite loops, unclear authorship, broken audit trails, and race conditions.

Correct Pattern: Two-Way Asymmetric

GitHub Actions responsibilities:

Validate code
Test builds
Tag releases
Publish artifacts
Open pull requests (not push directly to main)

Machine responsibilities:

Pull changes
Converge to declared state
Enforce configuration

Explicit Repo Updates

When workflows must update the repo (e.g., automated version bumps), use explicit steps:

- name: Push changes
  uses: ad-m/github-push-action@master
  with:
    github_token: ${{ secrets.GITHUB_TOKEN }}
    branch: main

By default, workflows run in temporary virtual bubbles and disappear after execution—changes are not persisted without explicit push steps.

Cross-Platform Package Management

macOS/Linux:

Brewfile (Homebrew) for package definitions
Committed to repository
Applied via brew bundle in Ansible playbooks

Windows:

winget or Chocolatey scripts
Declarative package lists in repository

Ultimate approach: Nix

Define entire machine state (apps, settings, tools) in single configuration file
Works across macOS and Linux (T14s)
Ensures reproducible environments regardless of OS

Monitoring and Health Checks

Lightweight dashboard options:

Homepage (modern, containerized)
Glances (terminal-based, minimal resources)
Prometheus + Grafana (for production-grade monitoring)

Critical metrics:

Runner online/offline status
Resource utilization (CPU, memory, disk)
Backup job success/failure
Configuration drift detection
Network connectivity status

Automated alerts:

Runner offline >15 minutes
Backup failure
Disk space <10%
Configuration drift detected

Comparison: Traditional vs. Organization as Code

Feature	Traditional Infrastructure	Organization as Code
State Source	Manual sysadmin / scripts	GitHub repository (single truth)
Update Method	Push (SSH loop)	Pull (cron + Ansible)
Drift Handling	Accumulates until failure	Overwritten every 15 min
Crash Handling	Manual restart	`restart: unless-stopped` (auto)
Connectivity	Static IP / port forwarding	Tailscale mesh (identity-based)
Verification	Trust-based	Measured and verified

Implementation Checklist

P0: Critical

Host sleep disabled on all machines
Docker Compose restart policies set
Ansible playbooks created for all machines
Cron jobs configured for 15-minute pull
Backup scripts written and tested
One always-on anchor node (M4 Pro) operational

P1: Important

Runner auto re-registration scripts
Health check watchdogs
Backups throttled or isolated from CI hours
Tailscale mesh configured
Monitoring dashboard deployed

P2: Optimization

Local Docker registry/brew cache on M4 Pro
Ephemeral runner registration for T14s
Disaster recovery drill written as code
Runner lifecycle management documented

Key Principles

Declare everything in Git → Machines pull & converge → GitHub Actions validates
Never force machines to stay in sync; make them want to sync
Backups compete with production; isolate them
Runners are cattle, not pets; automate resurrection
Drift is inevitable; automatic correction is mandatory
The system must self-heal faster than humans can break it

Source Files

/Users/gordie/dev/ap/library/knowledge-vault/research-lab-materials/logs/notebook outputs/Organization as Code - Runners, Backups, Sync.txt
/Users/gordie/dev/ap/library/knowledge-vault/research-lab-materials/logs/notebook outputs/Organization as Code - Self-Hosted Runners & Sync.txt
/Users/gordie/dev/ap/library/knowledge-vault/research-lab-materials/logs/notebook outputs/Organization as Code - The Deterministic Infrastructure Blueprint.txt
/Users/gordie/dev/ap/library/knowledge-vault/research-lab-materials/logs/notebook outputs/Organization as Code - The Self-Healing Infrastructure Framework.txt
/Users/gordie/dev/ap/library/knowledge-vault/research-lab-materials/logs/notebook outputs/Organization as Code - The Self-Healing Infrastructure Model.txt
/Users/gordie/dev/ap/library/knowledge-vault/research-lab-materials/logs/notebook outputs/Organization as Code Setup.txt

Knowledge Vault

Explorer