fix: who watches the watchmen — cross-heal + cron watchdog

- App healer (v7): checks DNS healer every 30s, restarts if down
- DNS healer (v2): checks app healer every 60s, restarts if down
- scripts/watchdog.sh: ultimate fallback cron job (every 5 min)
  revives both healers if both are down simultaneously
- Installed to /usr/local/bin/falah-watchdog + /etc/cron.d/falah-watchdog

Three-layer defense:
  Layer 1: Docker --restart unless-stopped (instant)
  Layer 2: Cross-heal between the two healers (30-60s)
  Layer 3: Cron watchdog (5 min, last resort)
This commit is contained in:
2026-06-28 03:54:45 +08:00
parent ad48309995
commit 59f892e73b
3 changed files with 146 additions and 0 deletions
+16
View File
@@ -523,6 +523,22 @@ while true; do
# ── Periodic maintenance (every 10 cycles = 5 min) ─────
CYCLE=$((CYCLE + 1))
[ "$CYCLE" -ge 10 ] && { CYCLE=0; periodic_maintenance; }
# ── Cross-check: DNS healer alive? ────────────────────
if ! docker ps --filter 'name=falah-dns-healer' --format '{{.Status}}' 2>/dev/null | grep -q '^Up'; then
log "⚠️ Sibling DNS healer DOWN — restarting..."
docker restart falah-dns-healer 2>/dev/null || \
docker run -d --name falah-dns-healer --restart unless-stopped \
--network host \
-v /var/run/docker.sock:/var/run/docker.sock:ro \
-v /var/log/falah:/var/log/falah \
-e CHECK_INTERVAL=60 \
falah-dns-healer:latest >> "$AGENT_LOG" 2>&1
sleep 3
if docker ps --filter 'name=falah-dns-healer' --format '{{.Status}}' | grep -q '^Up'; then
alert "INFO" "Cross-heal: restarted DNS healer"
fi
fi
sleep "$CHECK_INTERVAL"
done