- App healer (v7): checks DNS healer every 30s, restarts if down
- DNS healer (v2): checks app healer every 60s, restarts if down
- scripts/watchdog.sh: ultimate fallback cron job (every 5 min)
revives both healers if both are down simultaneously
- Installed to /usr/local/bin/falah-watchdog + /etc/cron.d/falah-watchdog
Three-layer defense:
Layer 1: Docker --restart unless-stopped (instant)
Layer 2: Cross-heal between the two healers (30-60s)
Layer 3: Cron watchdog (5 min, last resort)
- rollback was using --network bridge explicitly, which fails on this Synology's
Docker daemon when the bridge network is associated with a Swarm overlay
- Fixed: omit --network flag entirely, let Docker use default bridge behavior
- Also found and fixed: gateway container on Swarm overlay network couldn't
route to 192.168.0.11:4014 — attached to bridge network to resolve
- Fixed blocking command in incident reporting (added timeout 5s + --until now)
Was causing the entire agent script to hang forever after saving each incident
- Full SRE flow now works end-to-end:
1. Detect fault → 2. Fetch logs → 3. Analyze against knowledge base
4. Save incident with full context (logs, inspect, events, resources)
5. Execute restore protocol (rollback to :rollback image)
6. Verify service restored
- 3 incidents successfully captured and logged with diagnostics
- All containers healthy, auto-healer monitoring every 30s
- Removed set -e — agent no longer crashes when a remediation step fails
- SIGKILL/OOM → restart first, only rollback if exceeding max restarts/hour
- Added HEALTHY_CYCLES counter: after 10 consecutive healthy cycles (5 min),
reset restart tracking state (system considered stable)
- Rollback now handles the case where Docker restart policy already revived
the container (rm -f before re-creating)
pi-agent