- FIXED core bug: log() used tee -a, writing to both log file AND stdout,
corrupting DIAGNOSIS=$(detect_fault) and causing infinite fault loops
- ADDED 120s cooldown after restore to prevent detection loops during startup
- ADDED swarm service detection to auto-healer (handles both standalone
containers and Docker Swarm services)
- FIXED swarm service detection: awk now uses $2 (service name) not $1 (ID)
- FIXED exact match for service names (grep -x)
- FIXED restore protocol to use docker service update --force for swarm services
- DEPLOYED auto-healers to both Synology (v15) and Contabo (v15)
- DNS healer also deployed to Contabo (v3)
- DNS healer has swarm health monitoring (checks every 2 cycles)
- All 3 nodes in Docker Swarm: Synology (manager), colima (worker), Contabo (leader)
- App healer (v7): checks DNS healer every 30s, restarts if down
- DNS healer (v2): checks app healer every 60s, restarts if down
- scripts/watchdog.sh: ultimate fallback cron job (every 5 min)
revives both healers if both are down simultaneously
- Installed to /usr/local/bin/falah-watchdog + /etc/cron.d/falah-watchdog
Three-layer defense:
Layer 1: Docker --restart unless-stopped (instant)
Layer 2: Cross-heal between the two healers (30-60s)
Layer 3: Cron watchdog (5 min, last resort)
- rollback was using --network bridge explicitly, which fails on this Synology's
Docker daemon when the bridge network is associated with a Swarm overlay
- Fixed: omit --network flag entirely, let Docker use default bridge behavior
- Also found and fixed: gateway container on Swarm overlay network couldn't
route to 192.168.0.11:4014 — attached to bridge network to resolve
- Fixed blocking command in incident reporting (added timeout 5s + --until now)
Was causing the entire agent script to hang forever after saving each incident
- Full SRE flow now works end-to-end:
1. Detect fault → 2. Fetch logs → 3. Analyze against knowledge base
4. Save incident with full context (logs, inspect, events, resources)
5. Execute restore protocol (rollback to :rollback image)
6. Verify service restored
- 3 incidents successfully captured and logged with diagnostics
- All containers healthy, auto-healer monitoring every 30s
- Removed set -e — agent no longer crashes when a remediation step fails
- SIGKILL/OOM → restart first, only rollback if exceeding max restarts/hour
- Added HEALTHY_CYCLES counter: after 10 consecutive healthy cycles (5 min),
reset restart tracking state (system considered stable)
- Rollback now handles the case where Docker restart policy already revived
the container (rm -f before re-creating)
pi-agent