fix: auto-healer v5 — fix blocking bug, complete SRE flow verified

- Fixed blocking  command in incident reporting (added timeout 5s + --until now)
  Was causing the entire agent script to hang forever after saving each incident
- Full SRE flow now works end-to-end:
  1. Detect fault → 2. Fetch logs → 3. Analyze against knowledge base
  4. Save incident with full context (logs, inspect, events, resources)
  5. Execute restore protocol (rollback to :rollback image)
  6. Verify service restored
- 3 incidents successfully captured and logged with diagnostics
- All containers healthy, auto-healer monitoring every 30s

.pi/agents/auto-healer.md

@@ -1,73 +1,98 @@
 ---
 name: auto-healer
-description: Dedicated auto-healing agent for Falah Mobile — detects faults, diagnoses root cause, executes remediation, escalates if needed
+description: SRE-grade auto-healing agent for Falah Mobile — log-aware diagnosis, restore protocol, incident logging
 tools: read, bash, grep, find, ls
-model: 
 ---
 
-# Auto-Healer Agent
+# Auto-Healer Agent — SRE Protocol
 
-You are the dedicated auto-healing agent for the Falah Mobile deployment. Your job is to watch for faults, diagnose them, and fix them autonomously.
+## On Fault Detection
 
-## Fault Detection
+```
+FAULT DETECTED
+     │
+     ▼
+┌──────────────────┐
+│  1. FETCH LOGS    │  ← docker logs --tail 100
+└──────┬───────────┘
+       │
+       ▼
+┌──────────────────┐
+│  2. ANALYZE       │  ← Knowledge Base: 20+ error patterns
+│     Root Cause    │     Map → action: rollback, restart, hotfix
+└──────┬───────────┘
+       │
+       ▼
+┌──────────────────────────────────────┐
+│  3. DECIDE                           │
+│                                      │
+│  Can I hotfix? ──→ apply_hotfix()    │
+│  Need restore? ──→ restore_protocol()│
+└──────────┬───────────────────────────┘
+           │
+           ▼
+┌──────────────────────────────────────┐
+│  4. RESTORE PROTOCOL                  │
+│                                      │
+│   a. Save incident report to disk    │
+│      (logs + inspect + events)       │
+│   b. Tag current image as :failed    │
+│   c. Rollback to :rollback image     │
+│   d. Verify service restored         │
+│   e. Report incident ID              │
+└──────────────────────────────────────┘
+```
 
-Monitor these signals:
+## Error Knowledge Base
-1. **Container status** — `docker ps`, `docker inspect`
-2. **Health endpoint** — `curl http://localhost:4014/mobile/api/health`
-3. **Gateway health** — `curl http://localhost:7878/mobile/api/health`
-4. **Docker events** — `docker events --filter ...`
-5. **Logs** — `docker logs falah-mobile-staging --tail 50`
-6. **Resource usage** — `df`, `docker stats --no-stream`
 
-## Diagnosis Matrix
+| Log Pattern | Root Cause | Action |
-
-| Symptom | Diagnosis | Action |
 |---|---|---|
-| Container not running | crash/oom | Check exit code + OOMKilled flag |
+| "Cannot find module" | Missing dependency | **Rollback** |
-| Health 000/timeout | container hung | Restart container |
+| "unexpected token" / "SyntaxError" | JS parse error (bad deploy) | **Rollback** |
-| Health 200 + db:false | DB disconnected | Restart container |
+| "heap out of memory" / "FATAL ERROR" | OOM / memory leak | **Rollback** |
-| Health 5xx | app error | Check logs → rollback if code error |
+| "ReferenceError" / "TypeError" | Code bug | **Rollback** |
-| Gateway 502/000 | nginx down | Reload nginx config |
+| "PrismaClientInitializationError" | DB connection issue | Restart container |
-| Disk >90% | full disk | `docker system prune -f` |
+| "ECONNREFUSED" / "ENOTFOUND" | Downstream unreachable | Restart container |
-| Rate limiter stale | memory leak | Already auto-cleaned (5min interval) |
+| "rate limit" / "429" | Rate limiter | Hotfix (auto-cleanup) |
+| "jwt expired" / "invalid token" | Config issue | Check config → Rollback |
+| "500" / "unhandled rejection" | Unhandled exception | **Rollback** |
+| Container exit 137 / OOMKilled | SIGKILL / OOM | **Rollback** (if repeated) |
+| Gateway 502 | nginx down | Reload gateway |
 
-## Remediation Actions
+## Restore Protocol Steps
 
 ```bash
-# Restart container
+# 1. Save incident
-docker restart falah-mobile-staging
+cat > /var/log/falah/incidents/incident_$(date +%Y%m%d_%H%M%S).log << 'EOF'
+... full diagnostics ...
+EOF
 
-# Reload nginx gateway
+# 2. Rollback
-docker exec $(docker ps -q -f name=gateway) nginx -s reload
-
-# Rollback to previous image
 docker stop falah-mobile-staging
-docker rm falah-mobile-staging
+docker rm -f falah-mobile-staging  
-docker run -d --name falah-mobile-staging --restart unless-stopped \
+docker run -d --name falah-mobile-staging \
+  --restart unless-stopped \
   -p 4014:3000 \
   -v /var/services/homes/admin/falah-mobile/data:/app/data \
-  -v /var/services/homes/admin/falah-mobile/.env:/app/.env:ro \
   --network falah-umbrel_falah-system \
   falah-mobile:rollback
 
-# Prune disk
+# 3. Verify
-docker system prune -f --volumes
+curl -f http://localhost:4014/mobile/api/health
 
-# Run CAB QA after fix
+# 4. Report
-bash tests/cab-qa.sh http://192.168.0.11:4014/mobile
+echo "Incident: $id — rolled back to :rollback"
 ```
 
+## Hotfix Actions (no rollback needed)
+
+- **Rate limit**: Restart container (code auto-cleans every 5 min)
+- **DB connection**: Restart container (reconnect)
+- **Gateway**: `nginx -s reload`
+
 ## Escalation
 
-If 3+ restarts happen within an hour, or if rollback fails:
+If 3+ restarts/hour or rollback fails:
-1. Log the failure with full diagnostics
+1. Log full incident with all diagnostics
-2. Alert via available channels
+2. Do NOT restart again (leave for manual debug)
-3. Do NOT restart again — leave the system in its current state for manual debugging
+3. Alert via available channels
-
-## Reporting
-
-After each remediation, report:
-- What was detected
-- What action was taken
-- Whether the fix succeeded
-- Current health status

auto-healer/auto-heal-agent.sh

@@ -1,364 +1,531 @@
 #!/bin/bash
 # ============================================================
-# Falah Mobile — Auto-Healing Agent (Dedicated Container)
+# Falah Mobile — SRE Auto-Healing Agent
 # ============================================================
-# Real-time fault detection + autonomous remediation.
+# Log-aware fault diagnosis + restore protocol.
-# Runs as a sidecar container alongside falah-mobile-staging.
 #
-# Detects:
+# On fault:
-#   Container crash, health 5xx, DB disconnect, OOM,
+#   1. FETCH logs from failing container
-#   Gateway 502, 5xx spikes, disk pressure
+#   2. ANALYZE logs against error knowledge base
+#   3. DECIDE: quick hotfix or restore protocol?
+#   4. RESTORE PROTOCOL: rollback to known-good image
+#   5. While rolling back, capture full diagnostics for later fix
 #
-# Fixes:
+# This is NOT a blind restarter — it's a diagnostic-first SRE agent.
-#   Restart, rollback, nginx reload, prune, escalate
 # ============================================================
 
-# NO set -e — we handle errors explicitly to avoid the entire agent crashing
+# ── Configuration ────────────────────────────────────────────
-# on a single failed remediation step. Each function manages its own errors.
 
 CONTAINER_NAME="${CONTAINER_NAME:-falah-mobile-staging}"
 HEALTH_URL="${HEALTH_URL:-http://localhost:4014/mobile/api/health}"
 GATEWAY_URL="${GATEWAY_URL:-http://localhost:7878/mobile/api/health}"
 CHECK_INTERVAL="${CHECK_INTERVAL:-30}"
 MAX_RESTARTS="${MAX_RESTARTS_PER_HOUR:-3}"
-LOG="/var/log/falah/agent.log"
-STATE_DIR="/var/state/falah"
-STATE_FILE="$STATE_DIR/restart-count"
 ROLLBACK_IMAGE="${ROLLBACK_IMAGE:-falah-mobile:rollback}"
 
-mkdir -p "$STATE_DIR" "$(dirname "$LOG")"
+AGENT_LOG="/var/log/falah/agent.log"
+STATE_DIR="/var/state/falah"
+STATE_FILE="$STATE_DIR/restart-count"
+INCIDENTS_DIR="/var/log/falah/incidents"
+
+mkdir -p "$STATE_DIR" "$(dirname "$AGENT_LOG")" "$INCIDENTS_DIR"
 
 # ── Logging ──────────────────────────────────────────────────
 
 log() {
-  echo "[$(date '+%Y-%m-%d %H:%M:%S')] [agent] $*" | tee -a "$LOG"
+  echo "[$(date '+%Y-%m-%d %H:%M:%S')] [agent] $*" | tee -a "$AGENT_LOG"
 }
 
 alert() {
   local severity="$1" msg="$2"
   log "[$severity] $msg"
-  # Future: emit to webhook, ntfy, Slack, etc.
+  # Future: webhook, ntfy, Slack, PagerDuty
-  # curl -sf -X POST -H "Content-Type: application/json" \
-  #   -d "{\"severity\":\"$severity\",\"message\":\"$msg\",\"service\":\"$CONTAINER_NAME\"}" \
-  #   "$ALERT_WEBHOOK" &
 }
 
-# ── Track restarts (sliding window per hour) ─────────────────
+# ── Restart Tracking (sliding window) ────────────────────────
 
 count_restarts() {
   [ -f "$STATE_FILE" ] || echo 0 > "$STATE_FILE"
-  local now window_start count
+  local now window_start
   now=$(date +%s)
   window_start=$((now - 3600))
-  
-  # Filter entries within the last hour
   awk -v ws="$window_start" '$1 > ws' "$STATE_FILE" 2>/dev/null | wc -l
 }
 
 record_restart() {
   date +%s >> "$STATE_FILE"
-  # Trim old entries
   local now window_start
   now=$(date +%s)
   window_start=$((now - 3600))
   awk -v ws="$window_start" '$1 > ws' "$STATE_FILE" > "${STATE_FILE}.tmp" && mv "${STATE_FILE}.tmp" "$STATE_FILE"
 }
 
-# ── Fault Detection ──────────────────────────────────────────
+# ── Basic Health Checks ──────────────────────────────────────
 
 check_http() {
-  local url="$1" label="$2"
+  local url="$1"
-  local code body
+  local body code
-  body=$(curl -sf --max-time 10 "$url" 2>/dev/null) || true
   code=$(curl -s -o /dev/null -w "%{http_code}" --max-time 10 "$url" 2>/dev/null || echo "000")
-  
+  body=$(curl -sf --max-time 10 "$url" 2>/dev/null) || true
-  if [ "$code" = "200" ]; then
+  [ "$code" = "200" ] && { echo "$body"; return 0; } || return 1
-    echo "$body"
-    return 0
-  fi
-  return 1
-}
-
-check_db() {
-  local body
-  body=$(curl -sf --max-time 10 "$HEALTH_URL" 2>/dev/null) || return 1
-  echo "$body" | grep -q '"db":true' && return 0 || return 1
-}
-
-check_docker_events() {
-  # Check if the container recently crashed
-  local since_seconds="${1:-120}"
-  local since_epoch
-  since_epoch=$(date -d "-${since_seconds} seconds" +%s 2>/dev/null || echo $(( $(date +%s) - since_seconds )))
-  
-  # Use docker events --since to check for recent die/oom events
-  docker events --since "${since_epoch}s" --filter "container=$CONTAINER_NAME" \
-    --filter "event=die" --format '{{.Status}}' 2>/dev/null | head -1
 }
 
 check_disk() {
   df / | awk 'NR==2 {gsub(/%/,"",$5); print $5}'
 }
 
-# ── Fault Diagnosis ──────────────────────────────────────────
+# ── Phase 1: Detect Fault ────────────────────────────────────
 
-diagnose() {
+detect_fault() {
-  local fault_reason=""
+  # Returns: diagnosis string (or "healthy")
-  local health_body
   
-  # 1. Check if container exists
+  # 1. Container exists?
-  if ! docker ps -a --format '{{.Names}}' | grep -q "^${CONTAINER_NAME}$"; then
+  if ! docker ps -a --format '{{.Names}}' 2>/dev/null | grep -q "^${CONTAINER_NAME}$"; then
     echo "container_missing"
     return
   fi
+
+  # 2. Container running?
+  local status exit_code oom
+  status=$(docker inspect "$CONTAINER_NAME" --format '{{.State.Status}}' 2>/dev/null || echo "unknown")
   
-  # 2. Check if container is running
-  local status
-  status=$(docker inspect "$CONTAINER_NAME" --format '{{.State.Status}}' 2>/dev/null || echo "missing")
   if [ "$status" != "running" ]; then
-    local exit_code
     exit_code=$(docker inspect "$CONTAINER_NAME" --format '{{.State.ExitCode}}' 2>/dev/null || echo "0")
-    local oom
     oom=$(docker inspect "$CONTAINER_NAME" --format '{{.State.OOMKilled}}' 2>/dev/null || echo "false")
-    
+    [ "$oom" = "true" ] && { echo "oom_killed"; return; }
-    if [ "$oom" = "true" ]; then
+    [ "$exit_code" = "137" ] && { echo "sigkill"; return; }
-      echo "oom_killed"
+    echo "crashed_exit_$exit_code"
-    elif [ "$exit_code" = "137" ]; then
-      echo "sigkill_oom"
-    elif [ "$exit_code" != "0" ]; then
-      echo "crashed_exit_${exit_code}"
-    else
-      echo "not_running"
-    fi
     return
   fi
-  
+
-  # 3. Check health endpoint
+  # 3. Health endpoint responds?
-  health_body=$(check_http "$HEALTH_URL" "direct" 2>/dev/null) || true
+  local health_body
-  if [ -z "$health_body" ]; then
+  health_body=$(check_http "$HEALTH_URL") || { echo "health_down"; return; }
-    echo "health_unreachable"
+
-    return
+  # 4. DB connected?
-  fi
+  echo "$health_body" | grep -q '"db":false' && { echo "db_down"; return; }
-  
+
-  # 4. Check DB connectivity
+  # 5. Status ok?
-  local db_status
+  echo "$health_body" | grep -q '"status":"ok"' || { echo "health_invalid"; return; }
-  db_status=$(echo "$health_body" | grep -o '"db":false\|"db":true' || echo "unknown")
+
-  if [ "$db_status" = '"db":false' ]; then
+  # 6. Gateway?
-    echo "db_disconnected"
+  check_http "$GATEWAY_URL" > /dev/null || { echo "gateway_down"; return; }
-    return
+
-  fi
-  
-  # 5. Check response content
-  if ! echo "$health_body" | grep -q '"status":"ok"'; then
-    echo "health_invalid"
-    return
-  fi
-  
-  # 6. Check gateway
-  if ! check_http "$GATEWAY_URL" "gateway" > /dev/null 2>&1; then
-    echo "gateway_down"
-    return
-  fi
-  
   echo "healthy"
 }
 
-# ── Remediations ─────────────────────────────────────────────
+# ── Phase 2: Fetch Logs ──────────────────────────────────────
 
-remediate_container_restart() {
+fetch_logs() {
+  local lines="${1:-100}"
+  docker logs "$CONTAINER_NAME" --tail "$lines" 2>&1 || echo "LOG_FETCH_FAILED"
+}
+
+# ── Phase 3: Log Analysis (Knowledge Base) ───────────────────
+
+analyze_logs() {
   local diagnosis="$1"
-  local restarts
+  local logs="$2"
-  restarts=$(count_restarts)
+
+  # Build a diagnostic report
+  local root_cause="unknown"
+  local certainty="low"
+  local suggested_action="rollback"
+  local details=""
+
+  # ── Error Pattern Knowledge Base ───────────────────────────
+  # Each pattern maps to: root_cause | certainty | suggested_action
+
+  # Node.js crashes
+  if echo "$logs" | grep -qi "Cannot find module"; then
+    root_cause="missing_dependency"
+    certainty="high"
+    details=$(echo "$logs" | grep -i "Cannot find module" | head -3 | tr '\n' '; ')
   
-  log "🔄 Remediation: restart container (diagnosis=$diagnosis, restarts_last_hour=$restarts)"
+  elif echo "$logs" | grep -qi "Module not found"; then
+    root_cause="missing_dependency"
+    certainty="high"
+    details=$(echo "$logs" | grep -i "Module not found" | head -3 | tr '\n' '; ')
   
-  if [ "$restarts" -ge "$MAX_RESTARTS" ]; then
+  elif echo "$logs" | grep -qi "unexpected token"; then
-    alert "CRITICAL" "Excessive restarts ($restarts/h) — escalating to rollback"
+    root_cause="js_parse_error"
-    remediate_rollback
+    certainty="high"
-    return
+    suggested_action="rollback"
-  fi
+    details=$(echo "$logs" | grep -i "unexpected token" | head -3 | tr '\n' '; ')
-  
+  
-  docker restart "$CONTAINER_NAME" >> "$LOG" 2>&1
+  elif echo "$logs" | grep -qi "SyntaxError"; then
-  record_restart
+    root_cause="js_syntax_error"
-  sleep 10
+    certainty="high"
-  
+    details=$(echo "$logs" | grep -i "SyntaxError" | head -3 | tr '\n' '; ')
-  # Verify
+  
-  if check_http "$HEALTH_URL" "direct" > /dev/null 2>&1; then
+  elif echo "$logs" | grep -qi "ReferenceError"; then
-    log "✅ Restart successful — container healthy"
+    root_cause="js_reference_error"
-    alert "INFO" "Auto-healed: restart $CONTAINER_NAME (diagnosis=$diagnosis)"
+    certainty="high"
-    return 0
+    details=$(echo "$logs" | grep -i "ReferenceError" | head -3 | tr '\n' '; ')
-  else
+  
-    log "⚠️  Restart didn't fix — escalating"
+  elif echo "$logs" | grep -qi "TypeError"; then
-    remediate_escalate
+    root_cause="js_type_error"
+    certainty="high"
+    details=$(echo "$logs" | grep -i "TypeError" | head -3 | tr '\n' '; ')
+  
+  elif echo "$logs" | grep -qi "heap out of memory\|FATAL ERROR\|Allocation failed"; then
+    root_cause="out_of_memory"
+    certainty="high"
+    details=$(echo "$logs" | grep -i "heap out of memory\|FATAL ERROR\|Allocation failed" | head -3 | tr '\n' '; ')
+  
+  # Prisma / DB errors
+  elif echo "$logs" | grep -qi "PrismaClientInitializationError\|prisma.*connect.*ECONNREFUSED"; then
+    root_cause="db_connection_refused"
+    certainty="high"
+    suggested_action="restart_container"
+    details=$(echo "$logs" | grep -i "PrismaClientInitializationError\|ECONNREFUSED" | head -3 | tr '\n' '; ')
+  
+  elif echo "$logs" | grep -qi "prisma.*ENOENT\|prisma.*does not exist"; then
+    root_cause="prisma_schema_mismatch"
+    certainty="high"
+    details=$(echo "$logs" | grep -i "prisma.*ENOENT\|does not exist" | head -3 | tr '\n' '; ')
+  
+  elif echo "$logs" | grep -qi "Can't reach database server\|getaddrinfo ENOTFOUND.*db\|postgres.*connect"; then
+    root_cause="db_unreachable"
+    certainty="high"
+    suggested_action="restart_container"
+    details=$(echo "$logs" | grep -i "ENOTFOUND\|Can't reach database" | head -3 | tr '\n' '; ')
+  
+  # Network / connectivity
+  elif echo "$logs" | grep -qi "ECONNREFUSED\|ECONNRESET\|ETIMEDOUT\|ENOTFOUND"; then
+    root_cause="downstream_unreachable"
+    certainty="medium"
+    suggested_action="restart_container"
+    details=$(echo "$logs" | grep -i "ECONNREFUSED\|ECONNRESET\|ETIMEDOUT\|ENOTFOUND" | head -3 | tr '\n' '; ')
+  
+  # Rate limiting
+  elif echo "$logs" | grep -qi "rate limit\|too many requests\|429"; then
+    root_cause="rate_limit_exceeded"
+    certainty="medium"
+    suggested_action="hotfix_rate_limit"
+    details=$(echo "$logs" | grep -i "rate limit\|too many" | head -3 | tr '\n' '; ')
+  
+  # Auth / config
+  elif echo "$logs" | grep -qi "jwt expired\|invalid token\|Unauthorized"; then
+    root_cause="auth_config_error"
+    certainty="medium"
+    suggested_action="check_config"
+    details=$(echo "$logs" | grep -i "jwt expired\|invalid token\|Unauthorized" | head -3 | tr '\n' '; ')
+  
+  # 5xx patterns from health endpoint
+  elif echo "$logs" | grep -qi "500\|Internal Server Error\|unhandled rejection\|uncaught exception"; then
+    root_cause="unhandled_exception"
+    certainty="medium"
+    details=$(echo "$logs" | grep -i "unhandled\|uncaught\|500\|Internal Server Error" | head -3 | tr '\n' '; ')
+  
+  # Next.js specific
+  elif echo "$logs" | grep -qi "next.*error\|Error:.*page\|render.*error\|application.*error"; then
+    root_cause="application_error"
+    certainty="medium"
+    details=$(echo "$logs" | grep -i "Error:\|render.*error" | head -3 | tr '\n' '; ')
   fi
+
+  # If diagnosis gives strong signal, incorporate it
+  case "$diagnosis" in
+    oom_killed|sigkill)
+      [ "$root_cause" = "unknown" ] && root_cause="container_killed_sigkill"
+      suggested_action="rollback"
+      ;;
+    db_down)
+      [ "$root_cause" = "unknown" ] && root_cause="database_disconnected"
+      suggested_action="restart_container"
+      ;;
+    health_down)
+      [ "$root_cause" = "unknown" ] && root_cause="container_unresponsive"
+      ;;
+    gateway_down)
+      root_cause="nginx_gateway_down"
+      suggested_action="reload_gateway"
+      certainty="high"
+      ;;
+  esac
+
+  # Return structured report as JSON-like lines
+  echo "ROOT_CAUSE=$root_cause"
+  echo "CERTAINTY=$certainty"
+  echo "ACTION=$suggested_action"
+  echo "DETAILS=$details"
 }
 
-remediate_rollback() {
+# ── Phase 4: Restore Protocol ────────────────────────────────
-  log "🔄 Remediation: rollback to previous image"
+# Fast rollback to last known-good state, while capturing
-  
+# full diagnostics for permanent fix later.
-  if docker images "$ROLLBACK_IMAGE" --format '{{.ID}}' | head -1 | grep -q .; then
+
-    log "🔄 Rolling back to $ROLLBACK_IMAGE..."
+restore_protocol() {
-    
+  local diagnosis="$1"
-    # Stop and remove old container if it exists (may have been auto-restarted by Docker)
+  local root_cause="$2"
-    if docker ps -a --format '{{.Names}}' | grep -q "^${CONTAINER_NAME}$"; then
+  local action="$3"
-      docker stop "$CONTAINER_NAME" >> "$LOG" 2>&1 || true
+  local details="$4"
-      sleep 2
+  local logs="$5"
-      docker rm -f "$CONTAINER_NAME" >> "$LOG" 2>&1 || true
+  local incident_id
-      sleep 2
+  incident_id="incident_$(date +%Y%m%d_%H%M%S)"
-    fi
+
-    
+  log "🚨 RESTORE PROTOCOL triggered — incident: $incident_id"
-    # Extract original run config from inspect (if available) or use defaults
+  log "   Diagnosis:   $diagnosis"
-    local ports volumes network env_vars
+  log "   Root cause:  $root_cause"
-    ports=$(docker inspect "$CONTAINER_NAME" --format '{{range $p, $conf := .HostConfig.PortBindings}}{{$p}} {{end}}' 2>/dev/null || echo "4014:3000")
+  log "   Action:      $action"
-    volumes=$(docker inspect "$CONTAINER_NAME" --format '{{range $v := .Mounts}}{{$v.Source}}:{{$v.Destination}} {{end}}' 2>/dev/null)
+  log "   Details:     $details"
-    network=$(docker inspect "$CONTAINER_NAME" --format '{{.HostConfig.NetworkMode}}' 2>/dev/null || echo "bridge")
+
-    
+  # ── Step 0: Save full diagnostic context for later fix ─────
-    # Run rollback image with same config
+  {
-    docker run -d \
+    echo "=== INCIDENT REPORT: $incident_id ==="
-      --name "$CONTAINER_NAME" \
+    echo "Timestamp:   $(date -u '+%Y-%m-%dT%H:%M:%SZ')"
-      --restart unless-stopped \
+    echo "Container:   $CONTAINER_NAME"
-      -p 4014:3000 \
+    echo "Diagnosis:   $diagnosis"
-      $volumes \
+    echo "Root cause:  $root_cause"
-      --network "$network" \
+    echo "Certainty:   $certainty"
-      "$ROLLBACK_IMAGE" >> "$LOG" 2>&1
+    echo "Action:      $action"
-    
+    echo "Details:     $details"
-    sleep 15
+    echo ""
-    
+    echo "=== Container Logs ==="
-    if check_http "$HEALTH_URL" "direct" > /dev/null 2>&1; then
+    echo "$logs"
-      log "✅ Rollback successful — previous image restored"
+    echo ""
-      alert "INFO" "Auto-healed: rolled back to $ROLLBACK_IMAGE"
+    echo "=== Container Inspect ==="
-      return 0
+    docker inspect "$CONTAINER_NAME" 2>/dev/null || echo "INSPECT_FAILED"
-    else
+    echo ""
-      alert "CRITICAL" "Rollback also failed — manual intervention required"
+    echo "=== Docker Events (last 5 min) ==="
+    timeout 5 docker events --since "${CHECK_INTERVAL}s" --until "$(date +%s)" \
+      --filter "container=$CONTAINER_NAME" \
+      --format '{{.Time}} {{.Type}} {{.Action}} {{.Status}}' 2>/dev/null || echo "EVENTS_FAILED"
+    echo ""
+    echo "=== System Resources ==="
+    df -h / | tail -1
+    free -m 2>/dev/null || echo "free_unavailable"
+    echo "=== END INCIDENT REPORT ==="
+  } > "$INCIDENTS_DIR/$incident_id.log"
+
+  log "📝 Incident saved to $INCIDENTS_DIR/$incident_id.log"
+  alert "INFO" "Incident $incident_id logged — root cause: $root_cause"
+
+  # ── Step 1: Apply action ───────────────────────────────────
+  case "$action" in
+    rollback)
+      log "🔄 RESTORE: rolling back to $ROLLBACK_IMAGE..."
+      if docker images "$ROLLBACK_IMAGE" --format '{{.ID}}' | head -1 | grep -q .; then
+        # Remove old container
+        docker stop "$CONTAINER_NAME" 2>/dev/null || true
+        sleep 2
+        docker rm -f "$CONTAINER_NAME" 2>/dev/null || true
+        sleep 2
+
+        # Recreate from rollback image
+        local network
+        network=$(docker inspect "$CONTAINER_NAME" --format '{{.HostConfig.NetworkMode}}' 2>/dev/null || echo "bridge")
+        docker run -d \
+          --name "$CONTAINER_NAME" \
+          --restart unless-stopped \
+          -p 4014:3000 \
+          -v /var/services/homes/admin/falah-mobile/data:/app/data \
+          --network "$network" \
+          "$ROLLBACK_IMAGE" >> "$AGENT_LOG" 2>&1
+
+        sleep 15
+        if check_http "$HEALTH_URL" > /dev/null 2>&1; then
+          log "✅ RESTORE: rollback successful — service restored"
+          alert "INFO" "Restore protocol: rolled back to $ROLLBACK_IMAGE (incident: $incident_id)"
+          record_restart
+          return 0
+        else
+          log "❌ RESTORE: rollback also failed — escalation needed"
+          alert "CRITICAL" "Rollback failed for incident $incident_id — manual intervention"
+          return 1
+        fi
+      else
+        log "⚠️  RESTORE: no rollback image found — attempting restart instead"
+        docker restart "$CONTAINER_NAME" >> "$AGENT_LOG" 2>&1
+        sleep 10
+        if check_http "$HEALTH_URL" > /dev/null 2>&1; then
+          log "✅ RESTORE: restart successful (fallback)"
+          record_restart
+          return 0
+        fi
+        return 1
+      fi
+      ;;
+
+    restart_container)
+      log "🔄 RESTORE: restarting container..."
+      docker restart "$CONTAINER_NAME" >> "$AGENT_LOG" 2>&1
+      record_restart
+      sleep 10
+      if check_http "$HEALTH_URL" > /dev/null 2>&1; then
+        log "✅ RESTORE: restart successful"
+        return 0
+      else
+        log "⚠️  RESTORE: restart didn't fix — escalating to rollback"
+        restore_protocol "$diagnosis" "restart_failed" "rollback" "$details" "$logs"
+      fi
+      ;;
+
+    reload_gateway)
+      log "🔄 RESTORE: reloading nginx gateway..."
+      local gw
+      gw=$(docker ps --format '{{.Names}}' | grep -i "gateway\|nginx" | head -1)
+      [ -n "$gw" ] && docker exec "$gw" nginx -s reload >> "$AGENT_LOG" 2>&1
+      sleep 3
+      check_http "$GATEWAY_URL" > /dev/null && log "✅ RESTORE: gateway restored" || log "⚠️  Gateway still down"
+      ;;
+
+    hotfix_rate_limit)
+      log "🔄 RESTORE: rate limit issue — applying hotfix"
+      # Rate limiter auto-clears stale entries every 5 min (code-level fix)
+      # For immediate relief, restart container
+      docker restart "$CONTAINER_NAME" >> "$AGENT_LOG" 2>&1
+      record_restart
+      sleep 10
+      ;;
+
+    check_config)
+      log "🔄 RESTORE: config issue detected — rolling back to safe state"
+      restore_protocol "$diagnosis" "config_error" "rollback" "$details" "$logs"
+      ;;
+
+    *)
+      log "🔄 RESTORE: unknown action '$action' — attempting restart as safe default"
+      docker restart "$CONTAINER_NAME" >> "$AGENT_LOG" 2>&1
+      record_restart
+      sleep 10
+      ;;
+  esac
+}
+
+# ── Phase 5: Apply Quick Hotfix (when possible) ──────────────
+
+apply_hotfix() {
+  local root_cause="$1"
+  local logs="$2"
+
+  case "$root_cause" in
+    rate_limit_exceeded)
+      log "🔧 HOTFIX: rate limit exceeded — restarting to clear state"
+      docker restart "$CONTAINER_NAME" >> "$AGENT_LOG" 2>&1
+      sleep 5
+      log "🔧 HOTFIX: rate limiter now auto-cleans every 5 min (code fix in place)"
+      ;;
+
+    db_connection_refused|db_unreachable)
+      log "🔧 HOTFIX: DB connection issue — restarting container"
+      docker restart "$CONTAINER_NAME" >> "$AGENT_LOG" 2>&1
+      sleep 10
+      ;;
+
+    *)
+      log "🔧 HOTFIX: no quick fix for '$root_cause' — relying on restore protocol"
       return 1
-    fi
+      ;;
-  else
+  esac
-    log "⚠️  No rollback image found — cannot rollback"
+  return 0
-    alert "WARN" "No rollback image available"
+}
-    return 1
+
+# ── Periodic Maintenance ────────────────────────────────────
+
+periodic_maintenance() {
+  local disk_pct
+  disk_pct=$(check_disk)
+  if [ "$disk_pct" -gt 90 ]; then
+    log "⚠️  Disk at ${disk_pct}% — running prune"
+    docker system prune -f >> "$AGENT_LOG" 2>&1
+    log "✅ Prune completed"
   fi
 }
 
-remediate_gateway() {
-  log "🔄 Remediation: reload nginx gateway"
-  
-  local gw_container
-  gw_container=$(docker ps --format '{{.Names}}' | grep -i "gateway\|nginx" | head -1)
-  
-  if [ -n "$gw_container" ]; then
-    docker exec "$gw_container" nginx -s reload >> "$LOG" 2>&1
-    sleep 3
-    if check_http "$GATEWAY_URL" "gateway" > /dev/null 2>&1; then
-      log "✅ Gateway reload successful"
-      alert "INFO" "Auto-healed: reloaded nginx in $gw_container"
-      return 0
-    fi
-  fi
-  log "⚠️  Could not reload gateway"
-  return 1
-}
-
-remediate_prune() {
-  log "🔄 Remediation: docker system prune (disk pressure)"
-  docker system prune -f >> "$LOG" 2>&1
-  log "✅ Prune completed"
-  alert "INFO" "Auto-healed: pruned Docker resources"
-}
-
-remediate_escalate() {
-  alert "CRITICAL" "Auto-heal unable to fix — escalation needed"
-  # Future: send to PagerDuty, OpsGenie, Slack, etc.
-  log "🚨 ESCALATION: Manual intervention required for $CONTAINER_NAME"
-}
-
 # ── Main Loop ────────────────────────────────────────────────
 
 log "══════════════════════════════════════════════════════"
-log "🚀 Auto-Healing Agent starting"
+log "🚀 SRE Auto-Healing Agent starting"
 log "   Container:     $CONTAINER_NAME"
 log "   Health URL:    $HEALTH_URL"
 log "   Gateway URL:   $GATEWAY_URL"
 log "   Interval:      ${CHECK_INTERVAL}s"
 log "   Max restarts:  $MAX_RESTARTS/hour"
+log "   Incidents:     $INCIDENTS_DIR"
 log "══════════════════════════════════════════════════════"
 
+CYCLE=0
+HEALTHY_CYCLES=0
+
 while true; do
-  # ── Phase 1: Detect ──
+  # ── Detect ──────────────────────────────────────────────
-  DIAGNOSIS=$(diagnose)
+  DIAGNOSIS=$(detect_fault)
-  
+
-  # ── Phase 2: Act ──
+  if [ "$DIAGNOSIS" = "healthy" ]; then
-  case "$DIAGNOSIS" in
+    # Stability counter — reset restart tracking after 5 min healthy
-    healthy)
+    HEALTHY_CYCLES=$((HEALTHY_CYCLES + 1))
-      # All good — reset restart count if we've had consecutive healthy cycles
+    if [ "$HEALTHY_CYCLES" -ge 10 ]; then
-      if [ -z "$HEALTHY_CYCLES" ]; then HEALTHY_CYCLES=0; fi
+      HEALTHY_CYCLES=0
-      HEALTHY_CYCLES=$((HEALTHY_CYCLES + 1))
+      echo 0 > "$STATE_FILE" 2>/dev/null || true
-      # After 10 healthy cycles (5 min), reset restart counter
+    fi
-      if [ "$HEALTHY_CYCLES" -ge 10 ]; then
+  else
-        HEALTHY_CYCLES=0
+    # ── FAULT DETECTED ────────────────────────────────────
-        # Reset state file — system is stable now
+    HEALTHY_CYCLES=0
-        echo 0 > "$STATE_FILE" 2>/dev/null || true
+    log ""
+    log "⚠️  FAULT DETECTED: $DIAGNOSIS"
+
+    # Step 1: Fetch logs
+    log "📋 Fetching container logs..."
+    CONTAINER_LOGS=$(fetch_logs 100)
+
+    # Step 2: Analyze logs against knowledge base
+    log "🔍 Analyzing logs for root cause..."
+    ANALYSIS=$(analyze_logs "$DIAGNOSIS" "$CONTAINER_LOGS")
+    
+    # Parse analysis output
+    ROOT_CAUSE=$(echo "$ANALYSIS" | grep "^ROOT_CAUSE=" | cut -d= -f2-)
+    CERTAINTY=$(echo "$ANALYSIS" | grep "^CERTAINTY=" | cut -d= -f2-)
+    ACTION=$(echo "$ANALYSIS" | grep "^ACTION=" | cut -d= -f2-)
+    DETAILS=$(echo "$ANALYSIS" | grep "^DETAILS=" | cut -d= -f2-)
+
+    log "   Root cause:  $ROOT_CAUSE  (certainty: $CERTAINTY)"
+    log "   Action:      $ACTION"
+    [ -n "$DETAILS" ] && log "   Details:     $DETAILS"
+
+    # Step 3: Decision — can we hotfix or need restore protocol?
+    restarts=$(count_restarts)
+
+    if [ "$restarts" -ge "$MAX_RESTARTS" ]; then
+      # Too many restarts — go straight to restore protocol
+      log "⚠️  $restarts restarts in last hour — exceeding limit of $MAX_RESTARTS"
+      log "🚨 SKIPPING hotfix attempt — going directly to restore protocol"
+      restore_protocol "$DIAGNOSIS" "$ROOT_CAUSE" "rollback" "$DETAILS" "$CONTAINER_LOGS"
+
+    elif [ "$ACTION" = "rollback" ]; then
+      # Knowledge base says rollback — run restore protocol
+      restore_protocol "$DIAGNOSIS" "$ROOT_CAUSE" "$ACTION" "$DETAILS" "$CONTAINER_LOGS"
+
+    elif [ "$ACTION" = "restart_container" ]; then
+      # Try restart first, escalate if fails
+      log "🔄 Attempting restart (restart #$restarts)..."
+      apply_hotfix "$ROOT_CAUSE" "$CONTAINER_LOGS"
+      sleep 10
+      if [ "$(detect_fault)" != "healthy" ]; then
+        log "⚠️  Restart didn't resolve — escalating to restore protocol"
+        restore_protocol "$DIAGNOSIS" "$ROOT_CAUSE" "rollback" "$DETAILS" "$CONTAINER_LOGS"
       fi
-      ;;
+
-      
+    elif [ "$ACTION" = "reload_gateway" ]; then
-    container_missing)
+      restore_protocol "$DIAGNOSIS" "$ROOT_CAUSE" "$ACTION" "$DETAILS" "$CONTAINER_LOGS"
-      alert "CRITICAL" "Container $CONTAINER_NAME does not exist"
+
-      ;;
+    elif [ "$ACTION" = "hotfix_rate_limit" ]; then
-      
+      apply_hotfix "$ROOT_CAUSE" "$CONTAINER_LOGS"
-    oom_killed|sigkill_oom)
+
-      alert "WARN" "Container killed ($DIAGNOSIS) — restarting"
+    else
-      # If we've had too many restarts, escalate to rollback
+      # No specific action — safe default: restart, then rollback
-      local restarts
+      log "🔄 No specific fix for '$ROOT_CAUSE' — attempting restart"
-      restarts=$(count_restarts)
+      docker restart "$CONTAINER_NAME" >> "$AGENT_LOG" 2>&1
-      if [ "$restarts" -ge "$MAX_RESTARTS" ]; then
+      record_restart
-        alert "CRITICAL" "OOM repeated ($restarts restarts/hour) — rolling back"
+      sleep 10
-        remediate_rollback
+      [ "$(detect_fault)" != "healthy" ] && restore_protocol "$DIAGNOSIS" "$ROOT_CAUSE" "rollback" "$DETAILS" "$CONTAINER_LOGS"
-      else
-        remediate_container_restart "$DIAGNOSIS"
-      fi
-      ;;
-      
-    crashed_*|not_running)
-      alert "WARN" "Container crashed ($DIAGNOSIS)"
-      remediate_container_restart "$DIAGNOSIS"
-      ;;
-      
-    health_unreachable)
-      alert "WARN" "Health endpoint unreachable"
-      remediate_container_restart "$DIAGNOSIS"
-      ;;
-      
-    db_disconnected)
-      alert "WARN" "Database disconnected"
-      remediate_container_restart "$DIAGNOSIS"
-      ;;
-      
-    health_invalid)
-      alert "WARN" "Health response invalid"
-      remediate_container_restart "$DIAGNOSIS"
-      ;;
-      
-    gateway_down)
-      alert "WARN" "Gateway down"
-      remediate_gateway
-      ;;
-      
-    *)
-      alert "WARN" "Unknown diagnosis: $DIAGNOSIS"
-      remediate_container_restart "$DIAGNOSIS"
-      ;;
-  esac
-  
-  # ── Periodic maintenance ──
-  # Check disk every 10 cycles (5 min at 30s interval)
-  CYCLE_COUNTER=$((CYCLE_COUNTER + 1))
-  if [ "$CYCLE_COUNTER" -ge 10 ]; then
-    CYCLE_COUNTER=0
-    DISK_PCT=$(check_disk)
-    if [ "$DISK_PCT" -gt 90 ]; then
-      alert "WARN" "Disk at ${DISK_PCT}% — pruning"
-      remediate_prune
     fi
   fi
-  
+
+  # ── Periodic maintenance (every 10 cycles = 5 min) ─────
+  CYCLE=$((CYCLE + 1))
+  [ "$CYCLE" -ge 10 ] && { CYCLE=0; periodic_maintenance; }
+
   sleep "$CHECK_INTERVAL"
 done