Commit Graph

3 Commits

Author SHA1 Message Date
pi-agent 6d946571b0 fix: auto-healer v5 — fix blocking bug, complete SRE flow verified
- Fixed blocking  command in incident reporting (added timeout 5s + --until now)
  Was causing the entire agent script to hang forever after saving each incident
- Full SRE flow now works end-to-end:
  1. Detect fault → 2. Fetch logs → 3. Analyze against knowledge base
  4. Save incident with full context (logs, inspect, events, resources)
  5. Execute restore protocol (rollback to :rollback image)
  6. Verify service restored
- 3 incidents successfully captured and logged with diagnostics
- All containers healthy, auto-healer monitoring every 30s
2026-06-28 03:17:56 +08:00
pi-agent 4fcfa4375d fix: auto-healer v2 — fix set -e crash, proper restart vs rollback escalation, restart count reset on stability
- Removed set -e — agent no longer crashes when a remediation step fails
- SIGKILL/OOM → restart first, only rollback if exceeding max restarts/hour
- Added HEALTHY_CYCLES counter: after 10 consecutive healthy cycles (5 min),
  reset restart tracking state (system considered stable)
- Rollback now handles the case where Docker restart policy already revived
  the container (rm -f before re-creating)
2026-06-28 03:03:30 +08:00
pi-agent 456295d73e feat: dedicated auto-healing agent — sidecar container + pi agent definition
- auto-healer/Dockerfile: lightweight Alpine container with Docker CLI
- auto-healer/auto-heal-agent.sh: real-time fault detection + remediation engine
  - Detects: crash, OOM, DB disconnect, gateway down, disk pressure
  - Fixes: restart, rollback, nginx reload, prune, escalate
  - Sliding window restart tracking (3/hr max)
- auto-healer/docker-compose.healer.yml: sidecar deployment config
  - Mounts Docker socket for event monitoring
  - Host network mode for health checks
  - Persistent state in /var/state/falah
- .pi/agents/auto-healer.md: pi agent definition for manual invocation
2026-06-28 02:53:52 +08:00