fix: who watches the watchmen — cross-heal + cron watchdog
- App healer (v7): checks DNS healer every 30s, restarts if down - DNS healer (v2): checks app healer every 60s, restarts if down - scripts/watchdog.sh: ultimate fallback cron job (every 5 min) revives both healers if both are down simultaneously - Installed to /usr/local/bin/falah-watchdog + /etc/cron.d/falah-watchdog Three-layer defense: Layer 1: Docker --restart unless-stopped (instant) Layer 2: Cross-heal between the two healers (30-60s) Layer 3: Cron watchdog (5 min, last resort)
This commit is contained in:
@@ -350,5 +350,24 @@ while true; do
|
||||
fi
|
||||
|
||||
log "── Cycle $CYCLE complete ─────────────────────────────"
|
||||
|
||||
# ── Cross-check: App healer alive? ─────────────────────
|
||||
if ! docker ps --filter 'name=falah-auto-healer' --format '{{.Status}}' 2>/dev/null | grep -q '^Up'; then
|
||||
log "⚠️ Sibling app healer DOWN — restarting..."
|
||||
docker restart falah-auto-healer 2>/dev/null || \
|
||||
docker run -d --name falah-auto-healer --restart unless-stopped \
|
||||
--network host \
|
||||
-v /var/run/docker.sock:/var/run/docker.sock:ro \
|
||||
-v /var/log/falah:/var/log/falah \
|
||||
-v /var/state/falah:/var/state/falah \
|
||||
-e CONTAINER_NAME=falah-mobile-staging \
|
||||
-e CHECK_INTERVAL=30 \
|
||||
falah-auto-healer:v6 >> "$AGENT_LOG" 2>&1
|
||||
sleep 3
|
||||
if docker ps --filter 'name=falah-auto-healer' --format '{{.Status}}' | grep -q '^Up'; then
|
||||
alert "INFO" "Cross-heal: restarted app healer"
|
||||
fi
|
||||
fi
|
||||
|
||||
sleep "$CHECK_INTERVAL"
|
||||
done
|
||||
|
||||
Reference in New Issue
Block a user