pi-agent
6d946571b0
- Fixed blocking command in incident reporting (added timeout 5s + --until now) Was causing the entire agent script to hang forever after saving each incident - Full SRE flow now works end-to-end: 1. Detect fault → 2. Fetch logs → 3. Analyze against knowledge base 4. Save incident with full context (logs, inspect, events, resources) 5. Execute restore protocol (rollback to :rollback image) 6. Verify service restored - 3 incidents successfully captured and logged with diagnostics - All containers healthy, auto-healer monitoring every 30s
3.6 KiB
3.6 KiB
name, description, tools
| name | description | tools |
|---|---|---|
| auto-healer | SRE-grade auto-healing agent for Falah Mobile — log-aware diagnosis, restore protocol, incident logging | read, bash, grep, find, ls |
Auto-Healer Agent — SRE Protocol
On Fault Detection
FAULT DETECTED
│
▼
┌──────────────────┐
│ 1. FETCH LOGS │ ← docker logs --tail 100
└──────┬───────────┘
│
▼
┌──────────────────┐
│ 2. ANALYZE │ ← Knowledge Base: 20+ error patterns
│ Root Cause │ Map → action: rollback, restart, hotfix
└──────┬───────────┘
│
▼
┌──────────────────────────────────────┐
│ 3. DECIDE │
│ │
│ Can I hotfix? ──→ apply_hotfix() │
│ Need restore? ──→ restore_protocol()│
└──────────┬───────────────────────────┘
│
▼
┌──────────────────────────────────────┐
│ 4. RESTORE PROTOCOL │
│ │
│ a. Save incident report to disk │
│ (logs + inspect + events) │
│ b. Tag current image as :failed │
│ c. Rollback to :rollback image │
│ d. Verify service restored │
│ e. Report incident ID │
└──────────────────────────────────────┘
Error Knowledge Base
| Log Pattern | Root Cause | Action |
|---|---|---|
| "Cannot find module" | Missing dependency | Rollback |
| "unexpected token" / "SyntaxError" | JS parse error (bad deploy) | Rollback |
| "heap out of memory" / "FATAL ERROR" | OOM / memory leak | Rollback |
| "ReferenceError" / "TypeError" | Code bug | Rollback |
| "PrismaClientInitializationError" | DB connection issue | Restart container |
| "ECONNREFUSED" / "ENOTFOUND" | Downstream unreachable | Restart container |
| "rate limit" / "429" | Rate limiter | Hotfix (auto-cleanup) |
| "jwt expired" / "invalid token" | Config issue | Check config → Rollback |
| "500" / "unhandled rejection" | Unhandled exception | Rollback |
| Container exit 137 / OOMKilled | SIGKILL / OOM | Rollback (if repeated) |
| Gateway 502 | nginx down | Reload gateway |
Restore Protocol Steps
# 1. Save incident
cat > /var/log/falah/incidents/incident_$(date +%Y%m%d_%H%M%S).log << 'EOF'
... full diagnostics ...
EOF
# 2. Rollback
docker stop falah-mobile-staging
docker rm -f falah-mobile-staging
docker run -d --name falah-mobile-staging \
--restart unless-stopped \
-p 4014:3000 \
-v /var/services/homes/admin/falah-mobile/data:/app/data \
--network falah-umbrel_falah-system \
falah-mobile:rollback
# 3. Verify
curl -f http://localhost:4014/mobile/api/health
# 4. Report
echo "Incident: $id — rolled back to :rollback"
Hotfix Actions (no rollback needed)
- Rate limit: Restart container (code auto-cleans every 5 min)
- DB connection: Restart container (reconnect)
- Gateway:
nginx -s reload
Escalation
If 3+ restarts/hour or rollback fails:
- Log full incident with all diagnostics
- Do NOT restart again (leave for manual debug)
- Alert via available channels