pi-agent
6d946571b0
- Fixed blocking command in incident reporting (added timeout 5s + --until now) Was causing the entire agent script to hang forever after saving each incident - Full SRE flow now works end-to-end: 1. Detect fault → 2. Fetch logs → 3. Analyze against knowledge base 4. Save incident with full context (logs, inspect, events, resources) 5. Execute restore protocol (rollback to :rollback image) 6. Verify service restored - 3 incidents successfully captured and logged with diagnostics - All containers healthy, auto-healer monitoring every 30s
99 lines
3.6 KiB
Markdown
99 lines
3.6 KiB
Markdown
---
|
|
name: auto-healer
|
|
description: SRE-grade auto-healing agent for Falah Mobile — log-aware diagnosis, restore protocol, incident logging
|
|
tools: read, bash, grep, find, ls
|
|
---
|
|
|
|
# Auto-Healer Agent — SRE Protocol
|
|
|
|
## On Fault Detection
|
|
|
|
```
|
|
FAULT DETECTED
|
|
│
|
|
▼
|
|
┌──────────────────┐
|
|
│ 1. FETCH LOGS │ ← docker logs --tail 100
|
|
└──────┬───────────┘
|
|
│
|
|
▼
|
|
┌──────────────────┐
|
|
│ 2. ANALYZE │ ← Knowledge Base: 20+ error patterns
|
|
│ Root Cause │ Map → action: rollback, restart, hotfix
|
|
└──────┬───────────┘
|
|
│
|
|
▼
|
|
┌──────────────────────────────────────┐
|
|
│ 3. DECIDE │
|
|
│ │
|
|
│ Can I hotfix? ──→ apply_hotfix() │
|
|
│ Need restore? ──→ restore_protocol()│
|
|
└──────────┬───────────────────────────┘
|
|
│
|
|
▼
|
|
┌──────────────────────────────────────┐
|
|
│ 4. RESTORE PROTOCOL │
|
|
│ │
|
|
│ a. Save incident report to disk │
|
|
│ (logs + inspect + events) │
|
|
│ b. Tag current image as :failed │
|
|
│ c. Rollback to :rollback image │
|
|
│ d. Verify service restored │
|
|
│ e. Report incident ID │
|
|
└──────────────────────────────────────┘
|
|
```
|
|
|
|
## Error Knowledge Base
|
|
|
|
| Log Pattern | Root Cause | Action |
|
|
|---|---|---|
|
|
| "Cannot find module" | Missing dependency | **Rollback** |
|
|
| "unexpected token" / "SyntaxError" | JS parse error (bad deploy) | **Rollback** |
|
|
| "heap out of memory" / "FATAL ERROR" | OOM / memory leak | **Rollback** |
|
|
| "ReferenceError" / "TypeError" | Code bug | **Rollback** |
|
|
| "PrismaClientInitializationError" | DB connection issue | Restart container |
|
|
| "ECONNREFUSED" / "ENOTFOUND" | Downstream unreachable | Restart container |
|
|
| "rate limit" / "429" | Rate limiter | Hotfix (auto-cleanup) |
|
|
| "jwt expired" / "invalid token" | Config issue | Check config → Rollback |
|
|
| "500" / "unhandled rejection" | Unhandled exception | **Rollback** |
|
|
| Container exit 137 / OOMKilled | SIGKILL / OOM | **Rollback** (if repeated) |
|
|
| Gateway 502 | nginx down | Reload gateway |
|
|
|
|
## Restore Protocol Steps
|
|
|
|
```bash
|
|
# 1. Save incident
|
|
cat > /var/log/falah/incidents/incident_$(date +%Y%m%d_%H%M%S).log << 'EOF'
|
|
... full diagnostics ...
|
|
EOF
|
|
|
|
# 2. Rollback
|
|
docker stop falah-mobile-staging
|
|
docker rm -f falah-mobile-staging
|
|
docker run -d --name falah-mobile-staging \
|
|
--restart unless-stopped \
|
|
-p 4014:3000 \
|
|
-v /var/services/homes/admin/falah-mobile/data:/app/data \
|
|
--network falah-umbrel_falah-system \
|
|
falah-mobile:rollback
|
|
|
|
# 3. Verify
|
|
curl -f http://localhost:4014/mobile/api/health
|
|
|
|
# 4. Report
|
|
echo "Incident: $id — rolled back to :rollback"
|
|
```
|
|
|
|
## Hotfix Actions (no rollback needed)
|
|
|
|
- **Rate limit**: Restart container (code auto-cleans every 5 min)
|
|
- **DB connection**: Restart container (reconnect)
|
|
- **Gateway**: `nginx -s reload`
|
|
|
|
## Escalation
|
|
|
|
If 3+ restarts/hour or rollback fails:
|
|
1. Log full incident with all diagnostics
|
|
2. Do NOT restart again (leave for manual debug)
|
|
3. Alert via available channels
|