fix: auto-healer v5 — fix blocking bug, complete SRE flow verified

- Fixed blocking  command in incident reporting (added timeout 5s + --until now)
  Was causing the entire agent script to hang forever after saving each incident
- Full SRE flow now works end-to-end:
  1. Detect fault → 2. Fetch logs → 3. Analyze against knowledge base
  4. Save incident with full context (logs, inspect, events, resources)
  5. Execute restore protocol (rollback to :rollback image)
  6. Verify service restored
- 3 incidents successfully captured and logged with diagnostics
- All containers healthy, auto-healer monitoring every 30s

.pi/agents/auto-healer.md

@@ -1,73 +1,98 @@
 ---
 name: auto-healer
-description: Dedicated auto-healing agent for Falah Mobile — detects faults, diagnoses root cause, executes remediation, escalates if needed
+description: SRE-grade auto-healing agent for Falah Mobile — log-aware diagnosis, restore protocol, incident logging
 tools: read, bash, grep, find, ls
-model: 
 ---
 
-# Auto-Healer Agent
+# Auto-Healer Agent — SRE Protocol
 
-You are the dedicated auto-healing agent for the Falah Mobile deployment. Your job is to watch for faults, diagnose them, and fix them autonomously.
+## On Fault Detection
 
-## Fault Detection
+```
+FAULT DETECTED
+     │
+     ▼
+┌──────────────────┐
+│  1. FETCH LOGS    │  ← docker logs --tail 100
+└──────┬───────────┘
+       │
+       ▼
+┌──────────────────┐
+│  2. ANALYZE       │  ← Knowledge Base: 20+ error patterns
+│     Root Cause    │     Map → action: rollback, restart, hotfix
+└──────┬───────────┘
+       │
+       ▼
+┌──────────────────────────────────────┐
+│  3. DECIDE                           │
+│                                      │
+│  Can I hotfix? ──→ apply_hotfix()    │
+│  Need restore? ──→ restore_protocol()│
+└──────────┬───────────────────────────┘
+           │
+           ▼
+┌──────────────────────────────────────┐
+│  4. RESTORE PROTOCOL                  │
+│                                      │
+│   a. Save incident report to disk    │
+│      (logs + inspect + events)       │
+│   b. Tag current image as :failed    │
+│   c. Rollback to :rollback image     │
+│   d. Verify service restored         │
+│   e. Report incident ID              │
+└──────────────────────────────────────┘
+```
 
-Monitor these signals:
-1. **Container status** — `docker ps`, `docker inspect`
-2. **Health endpoint** — `curl http://localhost:4014/mobile/api/health`
-3. **Gateway health** — `curl http://localhost:7878/mobile/api/health`
-4. **Docker events** — `docker events --filter ...`
-5. **Logs** — `docker logs falah-mobile-staging --tail 50`
-6. **Resource usage** — `df`, `docker stats --no-stream`
+## Error Knowledge Base
 
-## Diagnosis Matrix
-
-| Symptom | Diagnosis | Action |
+| Log Pattern | Root Cause | Action |
 |---|---|---|
-| Container not running | crash/oom | Check exit code + OOMKilled flag |
-| Health 000/timeout | container hung | Restart container |
-| Health 200 + db:false | DB disconnected | Restart container |
-| Health 5xx | app error | Check logs → rollback if code error |
-| Gateway 502/000 | nginx down | Reload nginx config |
-| Disk >90% | full disk | `docker system prune -f` |
-| Rate limiter stale | memory leak | Already auto-cleaned (5min interval) |
+| "Cannot find module" | Missing dependency | **Rollback** |
+| "unexpected token" / "SyntaxError" | JS parse error (bad deploy) | **Rollback** |
+| "heap out of memory" / "FATAL ERROR" | OOM / memory leak | **Rollback** |
+| "ReferenceError" / "TypeError" | Code bug | **Rollback** |
+| "PrismaClientInitializationError" | DB connection issue | Restart container |
+| "ECONNREFUSED" / "ENOTFOUND" | Downstream unreachable | Restart container |
+| "rate limit" / "429" | Rate limiter | Hotfix (auto-cleanup) |
+| "jwt expired" / "invalid token" | Config issue | Check config → Rollback |
+| "500" / "unhandled rejection" | Unhandled exception | **Rollback** |
+| Container exit 137 / OOMKilled | SIGKILL / OOM | **Rollback** (if repeated) |
+| Gateway 502 | nginx down | Reload gateway |
 
-## Remediation Actions
+## Restore Protocol Steps
 
 ```bash
-# Restart container
-docker restart falah-mobile-staging
+# 1. Save incident
+cat > /var/log/falah/incidents/incident_$(date +%Y%m%d_%H%M%S).log << 'EOF'
+... full diagnostics ...
+EOF
 
-# Reload nginx gateway
-docker exec $(docker ps -q -f name=gateway) nginx -s reload
-
-# Rollback to previous image
+# 2. Rollback
 docker stop falah-mobile-staging
-docker rm falah-mobile-staging
-docker run -d --name falah-mobile-staging --restart unless-stopped \
+docker rm -f falah-mobile-staging  
+docker run -d --name falah-mobile-staging \
+  --restart unless-stopped \
   -p 4014:3000 \
   -v /var/services/homes/admin/falah-mobile/data:/app/data \
-  -v /var/services/homes/admin/falah-mobile/.env:/app/.env:ro \
   --network falah-umbrel_falah-system \
   falah-mobile:rollback
 
-# Prune disk
-docker system prune -f --volumes
+# 3. Verify
+curl -f http://localhost:4014/mobile/api/health
 
-# Run CAB QA after fix
-bash tests/cab-qa.sh http://192.168.0.11:4014/mobile
+# 4. Report
+echo "Incident: $id — rolled back to :rollback"
 ```
 
+## Hotfix Actions (no rollback needed)
+
+- **Rate limit**: Restart container (code auto-cleans every 5 min)
+- **DB connection**: Restart container (reconnect)
+- **Gateway**: `nginx -s reload`
+
 ## Escalation
 
-If 3+ restarts happen within an hour, or if rollback fails:
-1. Log the failure with full diagnostics
-2. Alert via available channels
-3. Do NOT restart again — leave the system in its current state for manual debugging
-
-## Reporting
-
-After each remediation, report:
-- What was detected
-- What action was taken
-- Whether the fix succeeded
-- Current health status
+If 3+ restarts/hour or rollback fails:
+1. Log full incident with all diagnostics
+2. Do NOT restart again (leave for manual debug)
+3. Alert via available channels