Commit Graph

3 Commits

Author SHA1 Message Date
pi-agent 33d85a56d2 fix: restore 3-node Docker Swarm + swarm health monitoring in DNS healer
- Rejoined Synology to Contabo-led swarm as manager (reachable)
- Swarm now has 2 managers: Synology (DS3617xs) + Contabo (vmi3361598)
- Saved join token to /var/state/falah/swarm-token for auto-rejoin
- DNS healer v3: checks swarm membership every 2 cycles (~2 min)
  - If swarm inactive, auto-rejoins using saved token
  - Reports node status (Contabo present, Mac Mini offline)
- Mac Mini (colima) was permanently removed from swarm while offline
  (needs manual re-join when back online with same token)

Current swarm:
  DS3617xs (Synology)  → Manager, Reachable, Engine 20.10.3
  vmi3361598 (Contabo)  → Manager, Leader, Engine 29.5.3
  colima (Mac Mini)     → GONE (offline >20d, need re-join)
2026-06-28 04:26:30 +08:00
pi-agent 59f892e73b fix: who watches the watchmen — cross-heal + cron watchdog
- App healer (v7): checks DNS healer every 30s, restarts if down
- DNS healer (v2): checks app healer every 60s, restarts if down
- scripts/watchdog.sh: ultimate fallback cron job (every 5 min)
  revives both healers if both are down simultaneously
- Installed to /usr/local/bin/falah-watchdog + /etc/cron.d/falah-watchdog

Three-layer defense:
  Layer 1: Docker --restart unless-stopped (instant)
  Layer 2: Cross-heal between the two healers (30-60s)
  Layer 3: Cron watchdog (5 min, last resort)
2026-06-28 03:54:45 +08:00
pi-agent ad48309995 feat: DNS & domain routing auto-healer — Cloudflare tunnels, DNS, gateway, routing integrity, SSL, Traefik
- dns-router-healer/Dockerfile: Alpine + curl, docker-cli, bash, jq, drill
- dns-router-healer/dns-router-heal.sh: 8 health checks in 60s cycle
  1. Cloudflare tunnels (primary + fallback) — restart on failure
  2. DNS resolution — verify all falah domains → Cloudflare IPs
  3. Direct service health — local health endpoint
  4. Gateway routing — nginx on port 7878
  5. Routing integrity — gateway response matches direct
  6. Public endpoint via Cloudflare (every 5 cycles)
  7. SSL certificate expiry (every 10 cycles)
  8. Traefik detection (every 5 cycles) — standalone or Swarm
- docker-compose.dns-healer.yml: sidecar deployment config
- .pi/agents/dns-healer.md: pi agent definition
2026-06-28 03:45:15 +08:00