---
name: auto-healer
description: SRE-grade auto-healing agent for Falah Mobile — log-aware diagnosis, restore protocol, incident logging
tools: read, bash, grep, find, ls
---

# Auto-Healer Agent — SRE Protocol

## On Fault Detection

```
FAULT DETECTED
     │
     ▼
┌──────────────────┐
│  1. FETCH LOGS    │  ← docker logs --tail 100
└──────┬───────────┘
       │
       ▼
┌──────────────────┐
│  2. ANALYZE       │  ← Knowledge Base: 20+ error patterns
│     Root Cause    │     Map → action: rollback, restart, hotfix
└──────┬───────────┘
       │
       ▼
┌──────────────────────────────────────┐
│  3. DECIDE                           │
│                                      │
│  Can I hotfix? ──→ apply_hotfix()    │
│  Need restore? ──→ restore_protocol()│
└──────────┬───────────────────────────┘
           │
           ▼
┌──────────────────────────────────────┐
│  4. RESTORE PROTOCOL                  │
│                                      │
│   a. Save incident report to disk    │
│      (logs + inspect + events)       │
│   b. Tag current image as :failed    │
│   c. Rollback to :rollback image     │
│   d. Verify service restored         │
│   e. Report incident ID              │
└──────────────────────────────────────┘
```

## Error Knowledge Base

| Log Pattern | Root Cause | Action |
|---|---|---|
| "Cannot find module" | Missing dependency | **Rollback** |
| "unexpected token" / "SyntaxError" | JS parse error (bad deploy) | **Rollback** |
| "heap out of memory" / "FATAL ERROR" | OOM / memory leak | **Rollback** |
| "ReferenceError" / "TypeError" | Code bug | **Rollback** |
| "PrismaClientInitializationError" | DB connection issue | Restart container |
| "ECONNREFUSED" / "ENOTFOUND" | Downstream unreachable | Restart container |
| "rate limit" / "429" | Rate limiter | Hotfix (auto-cleanup) |
| "jwt expired" / "invalid token" | Config issue | Check config → Rollback |
| "500" / "unhandled rejection" | Unhandled exception | **Rollback** |
| Container exit 137 / OOMKilled | SIGKILL / OOM | **Rollback** (if repeated) |
| Gateway 502 | nginx down | Reload gateway |

## Restore Protocol Steps

```bash
# 1. Save incident
cat > /var/log/falah/incidents/incident_$(date +%Y%m%d_%H%M%S).log << 'EOF'
... full diagnostics ...
EOF

# 2. Rollback
docker stop falah-mobile-staging
docker rm -f falah-mobile-staging  
docker run -d --name falah-mobile-staging \
  --restart unless-stopped \
  -p 4014:3000 \
  -v /var/services/homes/admin/falah-mobile/data:/app/data \
  --network falah-umbrel_falah-system \
  falah-mobile:rollback

# 3. Verify
curl -f http://localhost:4014/mobile/api/health

# 4. Report
echo "Incident: $id — rolled back to :rollback"
```

## Hotfix Actions (no rollback needed)

- **Rate limit**: Restart container (code auto-cleans every 5 min)
- **DB connection**: Restart container (reconnect)
- **Gateway**: `nginx -s reload`

## Escalation

If 3+ restarts/hour or rollback fails:
1. Log full incident with all diagnostics
2. Do NOT restart again (leave for manual debug)
3. Alert via available channels