diff --git a/.pi/agents/auto-healer.md b/.pi/agents/auto-healer.md index 7de328d..d90b033 100644 --- a/.pi/agents/auto-healer.md +++ b/.pi/agents/auto-healer.md @@ -1,73 +1,98 @@ --- name: auto-healer -description: Dedicated auto-healing agent for Falah Mobile — detects faults, diagnoses root cause, executes remediation, escalates if needed +description: SRE-grade auto-healing agent for Falah Mobile — log-aware diagnosis, restore protocol, incident logging tools: read, bash, grep, find, ls -model: --- -# Auto-Healer Agent +# Auto-Healer Agent — SRE Protocol -You are the dedicated auto-healing agent for the Falah Mobile deployment. Your job is to watch for faults, diagnose them, and fix them autonomously. +## On Fault Detection -## Fault Detection +``` +FAULT DETECTED + │ + ▼ +┌──────────────────┐ +│ 1. FETCH LOGS │ ← docker logs --tail 100 +└──────┬───────────┘ + │ + ▼ +┌──────────────────┐ +│ 2. ANALYZE │ ← Knowledge Base: 20+ error patterns +│ Root Cause │ Map → action: rollback, restart, hotfix +└──────┬───────────┘ + │ + ▼ +┌──────────────────────────────────────┐ +│ 3. DECIDE │ +│ │ +│ Can I hotfix? ──→ apply_hotfix() │ +│ Need restore? ──→ restore_protocol()│ +└──────────┬───────────────────────────┘ + │ + ▼ +┌──────────────────────────────────────┐ +│ 4. RESTORE PROTOCOL │ +│ │ +│ a. Save incident report to disk │ +│ (logs + inspect + events) │ +│ b. Tag current image as :failed │ +│ c. Rollback to :rollback image │ +│ d. Verify service restored │ +│ e. Report incident ID │ +└──────────────────────────────────────┘ +``` -Monitor these signals: -1. **Container status** — `docker ps`, `docker inspect` -2. **Health endpoint** — `curl http://localhost:4014/mobile/api/health` -3. **Gateway health** — `curl http://localhost:7878/mobile/api/health` -4. **Docker events** — `docker events --filter ...` -5. **Logs** — `docker logs falah-mobile-staging --tail 50` -6. **Resource usage** — `df`, `docker stats --no-stream` +## Error Knowledge Base -## Diagnosis Matrix - -| Symptom | Diagnosis | Action | +| Log Pattern | Root Cause | Action | |---|---|---| -| Container not running | crash/oom | Check exit code + OOMKilled flag | -| Health 000/timeout | container hung | Restart container | -| Health 200 + db:false | DB disconnected | Restart container | -| Health 5xx | app error | Check logs → rollback if code error | -| Gateway 502/000 | nginx down | Reload nginx config | -| Disk >90% | full disk | `docker system prune -f` | -| Rate limiter stale | memory leak | Already auto-cleaned (5min interval) | +| "Cannot find module" | Missing dependency | **Rollback** | +| "unexpected token" / "SyntaxError" | JS parse error (bad deploy) | **Rollback** | +| "heap out of memory" / "FATAL ERROR" | OOM / memory leak | **Rollback** | +| "ReferenceError" / "TypeError" | Code bug | **Rollback** | +| "PrismaClientInitializationError" | DB connection issue | Restart container | +| "ECONNREFUSED" / "ENOTFOUND" | Downstream unreachable | Restart container | +| "rate limit" / "429" | Rate limiter | Hotfix (auto-cleanup) | +| "jwt expired" / "invalid token" | Config issue | Check config → Rollback | +| "500" / "unhandled rejection" | Unhandled exception | **Rollback** | +| Container exit 137 / OOMKilled | SIGKILL / OOM | **Rollback** (if repeated) | +| Gateway 502 | nginx down | Reload gateway | -## Remediation Actions +## Restore Protocol Steps ```bash -# Restart container -docker restart falah-mobile-staging +# 1. Save incident +cat > /var/log/falah/incidents/incident_$(date +%Y%m%d_%H%M%S).log << 'EOF' +... full diagnostics ... +EOF -# Reload nginx gateway -docker exec $(docker ps -q -f name=gateway) nginx -s reload - -# Rollback to previous image +# 2. Rollback docker stop falah-mobile-staging -docker rm falah-mobile-staging -docker run -d --name falah-mobile-staging --restart unless-stopped \ +docker rm -f falah-mobile-staging +docker run -d --name falah-mobile-staging \ + --restart unless-stopped \ -p 4014:3000 \ -v /var/services/homes/admin/falah-mobile/data:/app/data \ - -v /var/services/homes/admin/falah-mobile/.env:/app/.env:ro \ --network falah-umbrel_falah-system \ falah-mobile:rollback -# Prune disk -docker system prune -f --volumes +# 3. Verify +curl -f http://localhost:4014/mobile/api/health -# Run CAB QA after fix -bash tests/cab-qa.sh http://192.168.0.11:4014/mobile +# 4. Report +echo "Incident: $id — rolled back to :rollback" ``` +## Hotfix Actions (no rollback needed) + +- **Rate limit**: Restart container (code auto-cleans every 5 min) +- **DB connection**: Restart container (reconnect) +- **Gateway**: `nginx -s reload` + ## Escalation -If 3+ restarts happen within an hour, or if rollback fails: -1. Log the failure with full diagnostics -2. Alert via available channels -3. Do NOT restart again — leave the system in its current state for manual debugging - -## Reporting - -After each remediation, report: -- What was detected -- What action was taken -- Whether the fix succeeded -- Current health status +If 3+ restarts/hour or rollback fails: +1. Log full incident with all diagnostics +2. Do NOT restart again (leave for manual debug) +3. Alert via available channels diff --git a/auto-healer/auto-heal-agent.sh b/auto-healer/auto-heal-agent.sh index e9da3d4..7fd06e0 100644 --- a/auto-healer/auto-heal-agent.sh +++ b/auto-healer/auto-heal-agent.sh @@ -1,364 +1,531 @@ #!/bin/bash # ============================================================ -# Falah Mobile — Auto-Healing Agent (Dedicated Container) +# Falah Mobile — SRE Auto-Healing Agent # ============================================================ -# Real-time fault detection + autonomous remediation. -# Runs as a sidecar container alongside falah-mobile-staging. +# Log-aware fault diagnosis + restore protocol. # -# Detects: -# Container crash, health 5xx, DB disconnect, OOM, -# Gateway 502, 5xx spikes, disk pressure +# On fault: +# 1. FETCH logs from failing container +# 2. ANALYZE logs against error knowledge base +# 3. DECIDE: quick hotfix or restore protocol? +# 4. RESTORE PROTOCOL: rollback to known-good image +# 5. While rolling back, capture full diagnostics for later fix # -# Fixes: -# Restart, rollback, nginx reload, prune, escalate +# This is NOT a blind restarter — it's a diagnostic-first SRE agent. # ============================================================ -# NO set -e — we handle errors explicitly to avoid the entire agent crashing -# on a single failed remediation step. Each function manages its own errors. +# ── Configuration ──────────────────────────────────────────── CONTAINER_NAME="${CONTAINER_NAME:-falah-mobile-staging}" HEALTH_URL="${HEALTH_URL:-http://localhost:4014/mobile/api/health}" GATEWAY_URL="${GATEWAY_URL:-http://localhost:7878/mobile/api/health}" CHECK_INTERVAL="${CHECK_INTERVAL:-30}" MAX_RESTARTS="${MAX_RESTARTS_PER_HOUR:-3}" -LOG="/var/log/falah/agent.log" -STATE_DIR="/var/state/falah" -STATE_FILE="$STATE_DIR/restart-count" ROLLBACK_IMAGE="${ROLLBACK_IMAGE:-falah-mobile:rollback}" -mkdir -p "$STATE_DIR" "$(dirname "$LOG")" +AGENT_LOG="/var/log/falah/agent.log" +STATE_DIR="/var/state/falah" +STATE_FILE="$STATE_DIR/restart-count" +INCIDENTS_DIR="/var/log/falah/incidents" + +mkdir -p "$STATE_DIR" "$(dirname "$AGENT_LOG")" "$INCIDENTS_DIR" # ── Logging ────────────────────────────────────────────────── log() { - echo "[$(date '+%Y-%m-%d %H:%M:%S')] [agent] $*" | tee -a "$LOG" + echo "[$(date '+%Y-%m-%d %H:%M:%S')] [agent] $*" | tee -a "$AGENT_LOG" } alert() { local severity="$1" msg="$2" log "[$severity] $msg" - # Future: emit to webhook, ntfy, Slack, etc. - # curl -sf -X POST -H "Content-Type: application/json" \ - # -d "{\"severity\":\"$severity\",\"message\":\"$msg\",\"service\":\"$CONTAINER_NAME\"}" \ - # "$ALERT_WEBHOOK" & + # Future: webhook, ntfy, Slack, PagerDuty } -# ── Track restarts (sliding window per hour) ───────────────── +# ── Restart Tracking (sliding window) ──────────────────────── count_restarts() { [ -f "$STATE_FILE" ] || echo 0 > "$STATE_FILE" - local now window_start count + local now window_start now=$(date +%s) window_start=$((now - 3600)) - - # Filter entries within the last hour awk -v ws="$window_start" '$1 > ws' "$STATE_FILE" 2>/dev/null | wc -l } record_restart() { date +%s >> "$STATE_FILE" - # Trim old entries local now window_start now=$(date +%s) window_start=$((now - 3600)) awk -v ws="$window_start" '$1 > ws' "$STATE_FILE" > "${STATE_FILE}.tmp" && mv "${STATE_FILE}.tmp" "$STATE_FILE" } -# ── Fault Detection ────────────────────────────────────────── +# ── Basic Health Checks ────────────────────────────────────── check_http() { - local url="$1" label="$2" - local code body - body=$(curl -sf --max-time 10 "$url" 2>/dev/null) || true + local url="$1" + local body code code=$(curl -s -o /dev/null -w "%{http_code}" --max-time 10 "$url" 2>/dev/null || echo "000") - - if [ "$code" = "200" ]; then - echo "$body" - return 0 - fi - return 1 -} - -check_db() { - local body - body=$(curl -sf --max-time 10 "$HEALTH_URL" 2>/dev/null) || return 1 - echo "$body" | grep -q '"db":true' && return 0 || return 1 -} - -check_docker_events() { - # Check if the container recently crashed - local since_seconds="${1:-120}" - local since_epoch - since_epoch=$(date -d "-${since_seconds} seconds" +%s 2>/dev/null || echo $(( $(date +%s) - since_seconds ))) - - # Use docker events --since to check for recent die/oom events - docker events --since "${since_epoch}s" --filter "container=$CONTAINER_NAME" \ - --filter "event=die" --format '{{.Status}}' 2>/dev/null | head -1 + body=$(curl -sf --max-time 10 "$url" 2>/dev/null) || true + [ "$code" = "200" ] && { echo "$body"; return 0; } || return 1 } check_disk() { df / | awk 'NR==2 {gsub(/%/,"",$5); print $5}' } -# ── Fault Diagnosis ────────────────────────────────────────── +# ── Phase 1: Detect Fault ──────────────────────────────────── -diagnose() { - local fault_reason="" - local health_body +detect_fault() { + # Returns: diagnosis string (or "healthy") - # 1. Check if container exists - if ! docker ps -a --format '{{.Names}}' | grep -q "^${CONTAINER_NAME}$"; then + # 1. Container exists? + if ! docker ps -a --format '{{.Names}}' 2>/dev/null | grep -q "^${CONTAINER_NAME}$"; then echo "container_missing" return fi + + # 2. Container running? + local status exit_code oom + status=$(docker inspect "$CONTAINER_NAME" --format '{{.State.Status}}' 2>/dev/null || echo "unknown") - # 2. Check if container is running - local status - status=$(docker inspect "$CONTAINER_NAME" --format '{{.State.Status}}' 2>/dev/null || echo "missing") if [ "$status" != "running" ]; then - local exit_code exit_code=$(docker inspect "$CONTAINER_NAME" --format '{{.State.ExitCode}}' 2>/dev/null || echo "0") - local oom oom=$(docker inspect "$CONTAINER_NAME" --format '{{.State.OOMKilled}}' 2>/dev/null || echo "false") - - if [ "$oom" = "true" ]; then - echo "oom_killed" - elif [ "$exit_code" = "137" ]; then - echo "sigkill_oom" - elif [ "$exit_code" != "0" ]; then - echo "crashed_exit_${exit_code}" - else - echo "not_running" - fi + [ "$oom" = "true" ] && { echo "oom_killed"; return; } + [ "$exit_code" = "137" ] && { echo "sigkill"; return; } + echo "crashed_exit_$exit_code" return fi - - # 3. Check health endpoint - health_body=$(check_http "$HEALTH_URL" "direct" 2>/dev/null) || true - if [ -z "$health_body" ]; then - echo "health_unreachable" - return - fi - - # 4. Check DB connectivity - local db_status - db_status=$(echo "$health_body" | grep -o '"db":false\|"db":true' || echo "unknown") - if [ "$db_status" = '"db":false' ]; then - echo "db_disconnected" - return - fi - - # 5. Check response content - if ! echo "$health_body" | grep -q '"status":"ok"'; then - echo "health_invalid" - return - fi - - # 6. Check gateway - if ! check_http "$GATEWAY_URL" "gateway" > /dev/null 2>&1; then - echo "gateway_down" - return - fi - + + # 3. Health endpoint responds? + local health_body + health_body=$(check_http "$HEALTH_URL") || { echo "health_down"; return; } + + # 4. DB connected? + echo "$health_body" | grep -q '"db":false' && { echo "db_down"; return; } + + # 5. Status ok? + echo "$health_body" | grep -q '"status":"ok"' || { echo "health_invalid"; return; } + + # 6. Gateway? + check_http "$GATEWAY_URL" > /dev/null || { echo "gateway_down"; return; } + echo "healthy" } -# ── Remediations ───────────────────────────────────────────── +# ── Phase 2: Fetch Logs ────────────────────────────────────── -remediate_container_restart() { +fetch_logs() { + local lines="${1:-100}" + docker logs "$CONTAINER_NAME" --tail "$lines" 2>&1 || echo "LOG_FETCH_FAILED" +} + +# ── Phase 3: Log Analysis (Knowledge Base) ─────────────────── + +analyze_logs() { local diagnosis="$1" - local restarts - restarts=$(count_restarts) + local logs="$2" + + # Build a diagnostic report + local root_cause="unknown" + local certainty="low" + local suggested_action="rollback" + local details="" + + # ── Error Pattern Knowledge Base ─────────────────────────── + # Each pattern maps to: root_cause | certainty | suggested_action + + # Node.js crashes + if echo "$logs" | grep -qi "Cannot find module"; then + root_cause="missing_dependency" + certainty="high" + details=$(echo "$logs" | grep -i "Cannot find module" | head -3 | tr '\n' '; ') - log "🔄 Remediation: restart container (diagnosis=$diagnosis, restarts_last_hour=$restarts)" + elif echo "$logs" | grep -qi "Module not found"; then + root_cause="missing_dependency" + certainty="high" + details=$(echo "$logs" | grep -i "Module not found" | head -3 | tr '\n' '; ') - if [ "$restarts" -ge "$MAX_RESTARTS" ]; then - alert "CRITICAL" "Excessive restarts ($restarts/h) — escalating to rollback" - remediate_rollback - return - fi - - docker restart "$CONTAINER_NAME" >> "$LOG" 2>&1 - record_restart - sleep 10 - - # Verify - if check_http "$HEALTH_URL" "direct" > /dev/null 2>&1; then - log "✅ Restart successful — container healthy" - alert "INFO" "Auto-healed: restart $CONTAINER_NAME (diagnosis=$diagnosis)" - return 0 - else - log "⚠️ Restart didn't fix — escalating" - remediate_escalate + elif echo "$logs" | grep -qi "unexpected token"; then + root_cause="js_parse_error" + certainty="high" + suggested_action="rollback" + details=$(echo "$logs" | grep -i "unexpected token" | head -3 | tr '\n' '; ') + + elif echo "$logs" | grep -qi "SyntaxError"; then + root_cause="js_syntax_error" + certainty="high" + details=$(echo "$logs" | grep -i "SyntaxError" | head -3 | tr '\n' '; ') + + elif echo "$logs" | grep -qi "ReferenceError"; then + root_cause="js_reference_error" + certainty="high" + details=$(echo "$logs" | grep -i "ReferenceError" | head -3 | tr '\n' '; ') + + elif echo "$logs" | grep -qi "TypeError"; then + root_cause="js_type_error" + certainty="high" + details=$(echo "$logs" | grep -i "TypeError" | head -3 | tr '\n' '; ') + + elif echo "$logs" | grep -qi "heap out of memory\|FATAL ERROR\|Allocation failed"; then + root_cause="out_of_memory" + certainty="high" + details=$(echo "$logs" | grep -i "heap out of memory\|FATAL ERROR\|Allocation failed" | head -3 | tr '\n' '; ') + + # Prisma / DB errors + elif echo "$logs" | grep -qi "PrismaClientInitializationError\|prisma.*connect.*ECONNREFUSED"; then + root_cause="db_connection_refused" + certainty="high" + suggested_action="restart_container" + details=$(echo "$logs" | grep -i "PrismaClientInitializationError\|ECONNREFUSED" | head -3 | tr '\n' '; ') + + elif echo "$logs" | grep -qi "prisma.*ENOENT\|prisma.*does not exist"; then + root_cause="prisma_schema_mismatch" + certainty="high" + details=$(echo "$logs" | grep -i "prisma.*ENOENT\|does not exist" | head -3 | tr '\n' '; ') + + elif echo "$logs" | grep -qi "Can't reach database server\|getaddrinfo ENOTFOUND.*db\|postgres.*connect"; then + root_cause="db_unreachable" + certainty="high" + suggested_action="restart_container" + details=$(echo "$logs" | grep -i "ENOTFOUND\|Can't reach database" | head -3 | tr '\n' '; ') + + # Network / connectivity + elif echo "$logs" | grep -qi "ECONNREFUSED\|ECONNRESET\|ETIMEDOUT\|ENOTFOUND"; then + root_cause="downstream_unreachable" + certainty="medium" + suggested_action="restart_container" + details=$(echo "$logs" | grep -i "ECONNREFUSED\|ECONNRESET\|ETIMEDOUT\|ENOTFOUND" | head -3 | tr '\n' '; ') + + # Rate limiting + elif echo "$logs" | grep -qi "rate limit\|too many requests\|429"; then + root_cause="rate_limit_exceeded" + certainty="medium" + suggested_action="hotfix_rate_limit" + details=$(echo "$logs" | grep -i "rate limit\|too many" | head -3 | tr '\n' '; ') + + # Auth / config + elif echo "$logs" | grep -qi "jwt expired\|invalid token\|Unauthorized"; then + root_cause="auth_config_error" + certainty="medium" + suggested_action="check_config" + details=$(echo "$logs" | grep -i "jwt expired\|invalid token\|Unauthorized" | head -3 | tr '\n' '; ') + + # 5xx patterns from health endpoint + elif echo "$logs" | grep -qi "500\|Internal Server Error\|unhandled rejection\|uncaught exception"; then + root_cause="unhandled_exception" + certainty="medium" + details=$(echo "$logs" | grep -i "unhandled\|uncaught\|500\|Internal Server Error" | head -3 | tr '\n' '; ') + + # Next.js specific + elif echo "$logs" | grep -qi "next.*error\|Error:.*page\|render.*error\|application.*error"; then + root_cause="application_error" + certainty="medium" + details=$(echo "$logs" | grep -i "Error:\|render.*error" | head -3 | tr '\n' '; ') fi + + # If diagnosis gives strong signal, incorporate it + case "$diagnosis" in + oom_killed|sigkill) + [ "$root_cause" = "unknown" ] && root_cause="container_killed_sigkill" + suggested_action="rollback" + ;; + db_down) + [ "$root_cause" = "unknown" ] && root_cause="database_disconnected" + suggested_action="restart_container" + ;; + health_down) + [ "$root_cause" = "unknown" ] && root_cause="container_unresponsive" + ;; + gateway_down) + root_cause="nginx_gateway_down" + suggested_action="reload_gateway" + certainty="high" + ;; + esac + + # Return structured report as JSON-like lines + echo "ROOT_CAUSE=$root_cause" + echo "CERTAINTY=$certainty" + echo "ACTION=$suggested_action" + echo "DETAILS=$details" } -remediate_rollback() { - log "🔄 Remediation: rollback to previous image" - - if docker images "$ROLLBACK_IMAGE" --format '{{.ID}}' | head -1 | grep -q .; then - log "🔄 Rolling back to $ROLLBACK_IMAGE..." - - # Stop and remove old container if it exists (may have been auto-restarted by Docker) - if docker ps -a --format '{{.Names}}' | grep -q "^${CONTAINER_NAME}$"; then - docker stop "$CONTAINER_NAME" >> "$LOG" 2>&1 || true - sleep 2 - docker rm -f "$CONTAINER_NAME" >> "$LOG" 2>&1 || true - sleep 2 - fi - - # Extract original run config from inspect (if available) or use defaults - local ports volumes network env_vars - ports=$(docker inspect "$CONTAINER_NAME" --format '{{range $p, $conf := .HostConfig.PortBindings}}{{$p}} {{end}}' 2>/dev/null || echo "4014:3000") - volumes=$(docker inspect "$CONTAINER_NAME" --format '{{range $v := .Mounts}}{{$v.Source}}:{{$v.Destination}} {{end}}' 2>/dev/null) - network=$(docker inspect "$CONTAINER_NAME" --format '{{.HostConfig.NetworkMode}}' 2>/dev/null || echo "bridge") - - # Run rollback image with same config - docker run -d \ - --name "$CONTAINER_NAME" \ - --restart unless-stopped \ - -p 4014:3000 \ - $volumes \ - --network "$network" \ - "$ROLLBACK_IMAGE" >> "$LOG" 2>&1 - - sleep 15 - - if check_http "$HEALTH_URL" "direct" > /dev/null 2>&1; then - log "✅ Rollback successful — previous image restored" - alert "INFO" "Auto-healed: rolled back to $ROLLBACK_IMAGE" - return 0 - else - alert "CRITICAL" "Rollback also failed — manual intervention required" +# ── Phase 4: Restore Protocol ──────────────────────────────── +# Fast rollback to last known-good state, while capturing +# full diagnostics for permanent fix later. + +restore_protocol() { + local diagnosis="$1" + local root_cause="$2" + local action="$3" + local details="$4" + local logs="$5" + local incident_id + incident_id="incident_$(date +%Y%m%d_%H%M%S)" + + log "🚨 RESTORE PROTOCOL triggered — incident: $incident_id" + log " Diagnosis: $diagnosis" + log " Root cause: $root_cause" + log " Action: $action" + log " Details: $details" + + # ── Step 0: Save full diagnostic context for later fix ───── + { + echo "=== INCIDENT REPORT: $incident_id ===" + echo "Timestamp: $(date -u '+%Y-%m-%dT%H:%M:%SZ')" + echo "Container: $CONTAINER_NAME" + echo "Diagnosis: $diagnosis" + echo "Root cause: $root_cause" + echo "Certainty: $certainty" + echo "Action: $action" + echo "Details: $details" + echo "" + echo "=== Container Logs ===" + echo "$logs" + echo "" + echo "=== Container Inspect ===" + docker inspect "$CONTAINER_NAME" 2>/dev/null || echo "INSPECT_FAILED" + echo "" + echo "=== Docker Events (last 5 min) ===" + timeout 5 docker events --since "${CHECK_INTERVAL}s" --until "$(date +%s)" \ + --filter "container=$CONTAINER_NAME" \ + --format '{{.Time}} {{.Type}} {{.Action}} {{.Status}}' 2>/dev/null || echo "EVENTS_FAILED" + echo "" + echo "=== System Resources ===" + df -h / | tail -1 + free -m 2>/dev/null || echo "free_unavailable" + echo "=== END INCIDENT REPORT ===" + } > "$INCIDENTS_DIR/$incident_id.log" + + log "📝 Incident saved to $INCIDENTS_DIR/$incident_id.log" + alert "INFO" "Incident $incident_id logged — root cause: $root_cause" + + # ── Step 1: Apply action ─────────────────────────────────── + case "$action" in + rollback) + log "🔄 RESTORE: rolling back to $ROLLBACK_IMAGE..." + if docker images "$ROLLBACK_IMAGE" --format '{{.ID}}' | head -1 | grep -q .; then + # Remove old container + docker stop "$CONTAINER_NAME" 2>/dev/null || true + sleep 2 + docker rm -f "$CONTAINER_NAME" 2>/dev/null || true + sleep 2 + + # Recreate from rollback image + local network + network=$(docker inspect "$CONTAINER_NAME" --format '{{.HostConfig.NetworkMode}}' 2>/dev/null || echo "bridge") + docker run -d \ + --name "$CONTAINER_NAME" \ + --restart unless-stopped \ + -p 4014:3000 \ + -v /var/services/homes/admin/falah-mobile/data:/app/data \ + --network "$network" \ + "$ROLLBACK_IMAGE" >> "$AGENT_LOG" 2>&1 + + sleep 15 + if check_http "$HEALTH_URL" > /dev/null 2>&1; then + log "✅ RESTORE: rollback successful — service restored" + alert "INFO" "Restore protocol: rolled back to $ROLLBACK_IMAGE (incident: $incident_id)" + record_restart + return 0 + else + log "❌ RESTORE: rollback also failed — escalation needed" + alert "CRITICAL" "Rollback failed for incident $incident_id — manual intervention" + return 1 + fi + else + log "⚠️ RESTORE: no rollback image found — attempting restart instead" + docker restart "$CONTAINER_NAME" >> "$AGENT_LOG" 2>&1 + sleep 10 + if check_http "$HEALTH_URL" > /dev/null 2>&1; then + log "✅ RESTORE: restart successful (fallback)" + record_restart + return 0 + fi + return 1 + fi + ;; + + restart_container) + log "🔄 RESTORE: restarting container..." + docker restart "$CONTAINER_NAME" >> "$AGENT_LOG" 2>&1 + record_restart + sleep 10 + if check_http "$HEALTH_URL" > /dev/null 2>&1; then + log "✅ RESTORE: restart successful" + return 0 + else + log "⚠️ RESTORE: restart didn't fix — escalating to rollback" + restore_protocol "$diagnosis" "restart_failed" "rollback" "$details" "$logs" + fi + ;; + + reload_gateway) + log "🔄 RESTORE: reloading nginx gateway..." + local gw + gw=$(docker ps --format '{{.Names}}' | grep -i "gateway\|nginx" | head -1) + [ -n "$gw" ] && docker exec "$gw" nginx -s reload >> "$AGENT_LOG" 2>&1 + sleep 3 + check_http "$GATEWAY_URL" > /dev/null && log "✅ RESTORE: gateway restored" || log "⚠️ Gateway still down" + ;; + + hotfix_rate_limit) + log "🔄 RESTORE: rate limit issue — applying hotfix" + # Rate limiter auto-clears stale entries every 5 min (code-level fix) + # For immediate relief, restart container + docker restart "$CONTAINER_NAME" >> "$AGENT_LOG" 2>&1 + record_restart + sleep 10 + ;; + + check_config) + log "🔄 RESTORE: config issue detected — rolling back to safe state" + restore_protocol "$diagnosis" "config_error" "rollback" "$details" "$logs" + ;; + + *) + log "🔄 RESTORE: unknown action '$action' — attempting restart as safe default" + docker restart "$CONTAINER_NAME" >> "$AGENT_LOG" 2>&1 + record_restart + sleep 10 + ;; + esac +} + +# ── Phase 5: Apply Quick Hotfix (when possible) ────────────── + +apply_hotfix() { + local root_cause="$1" + local logs="$2" + + case "$root_cause" in + rate_limit_exceeded) + log "🔧 HOTFIX: rate limit exceeded — restarting to clear state" + docker restart "$CONTAINER_NAME" >> "$AGENT_LOG" 2>&1 + sleep 5 + log "🔧 HOTFIX: rate limiter now auto-cleans every 5 min (code fix in place)" + ;; + + db_connection_refused|db_unreachable) + log "🔧 HOTFIX: DB connection issue — restarting container" + docker restart "$CONTAINER_NAME" >> "$AGENT_LOG" 2>&1 + sleep 10 + ;; + + *) + log "🔧 HOTFIX: no quick fix for '$root_cause' — relying on restore protocol" return 1 - fi - else - log "⚠️ No rollback image found — cannot rollback" - alert "WARN" "No rollback image available" - return 1 + ;; + esac + return 0 +} + +# ── Periodic Maintenance ──────────────────────────────────── + +periodic_maintenance() { + local disk_pct + disk_pct=$(check_disk) + if [ "$disk_pct" -gt 90 ]; then + log "⚠️ Disk at ${disk_pct}% — running prune" + docker system prune -f >> "$AGENT_LOG" 2>&1 + log "✅ Prune completed" fi } -remediate_gateway() { - log "🔄 Remediation: reload nginx gateway" - - local gw_container - gw_container=$(docker ps --format '{{.Names}}' | grep -i "gateway\|nginx" | head -1) - - if [ -n "$gw_container" ]; then - docker exec "$gw_container" nginx -s reload >> "$LOG" 2>&1 - sleep 3 - if check_http "$GATEWAY_URL" "gateway" > /dev/null 2>&1; then - log "✅ Gateway reload successful" - alert "INFO" "Auto-healed: reloaded nginx in $gw_container" - return 0 - fi - fi - log "⚠️ Could not reload gateway" - return 1 -} - -remediate_prune() { - log "🔄 Remediation: docker system prune (disk pressure)" - docker system prune -f >> "$LOG" 2>&1 - log "✅ Prune completed" - alert "INFO" "Auto-healed: pruned Docker resources" -} - -remediate_escalate() { - alert "CRITICAL" "Auto-heal unable to fix — escalation needed" - # Future: send to PagerDuty, OpsGenie, Slack, etc. - log "🚨 ESCALATION: Manual intervention required for $CONTAINER_NAME" -} - # ── Main Loop ──────────────────────────────────────────────── log "══════════════════════════════════════════════════════" -log "🚀 Auto-Healing Agent starting" +log "🚀 SRE Auto-Healing Agent starting" log " Container: $CONTAINER_NAME" log " Health URL: $HEALTH_URL" log " Gateway URL: $GATEWAY_URL" log " Interval: ${CHECK_INTERVAL}s" log " Max restarts: $MAX_RESTARTS/hour" +log " Incidents: $INCIDENTS_DIR" log "══════════════════════════════════════════════════════" +CYCLE=0 +HEALTHY_CYCLES=0 + while true; do - # ── Phase 1: Detect ── - DIAGNOSIS=$(diagnose) - - # ── Phase 2: Act ── - case "$DIAGNOSIS" in - healthy) - # All good — reset restart count if we've had consecutive healthy cycles - if [ -z "$HEALTHY_CYCLES" ]; then HEALTHY_CYCLES=0; fi - HEALTHY_CYCLES=$((HEALTHY_CYCLES + 1)) - # After 10 healthy cycles (5 min), reset restart counter - if [ "$HEALTHY_CYCLES" -ge 10 ]; then - HEALTHY_CYCLES=0 - # Reset state file — system is stable now - echo 0 > "$STATE_FILE" 2>/dev/null || true + # ── Detect ────────────────────────────────────────────── + DIAGNOSIS=$(detect_fault) + + if [ "$DIAGNOSIS" = "healthy" ]; then + # Stability counter — reset restart tracking after 5 min healthy + HEALTHY_CYCLES=$((HEALTHY_CYCLES + 1)) + if [ "$HEALTHY_CYCLES" -ge 10 ]; then + HEALTHY_CYCLES=0 + echo 0 > "$STATE_FILE" 2>/dev/null || true + fi + else + # ── FAULT DETECTED ──────────────────────────────────── + HEALTHY_CYCLES=0 + log "" + log "⚠️ FAULT DETECTED: $DIAGNOSIS" + + # Step 1: Fetch logs + log "📋 Fetching container logs..." + CONTAINER_LOGS=$(fetch_logs 100) + + # Step 2: Analyze logs against knowledge base + log "🔍 Analyzing logs for root cause..." + ANALYSIS=$(analyze_logs "$DIAGNOSIS" "$CONTAINER_LOGS") + + # Parse analysis output + ROOT_CAUSE=$(echo "$ANALYSIS" | grep "^ROOT_CAUSE=" | cut -d= -f2-) + CERTAINTY=$(echo "$ANALYSIS" | grep "^CERTAINTY=" | cut -d= -f2-) + ACTION=$(echo "$ANALYSIS" | grep "^ACTION=" | cut -d= -f2-) + DETAILS=$(echo "$ANALYSIS" | grep "^DETAILS=" | cut -d= -f2-) + + log " Root cause: $ROOT_CAUSE (certainty: $CERTAINTY)" + log " Action: $ACTION" + [ -n "$DETAILS" ] && log " Details: $DETAILS" + + # Step 3: Decision — can we hotfix or need restore protocol? + restarts=$(count_restarts) + + if [ "$restarts" -ge "$MAX_RESTARTS" ]; then + # Too many restarts — go straight to restore protocol + log "⚠️ $restarts restarts in last hour — exceeding limit of $MAX_RESTARTS" + log "🚨 SKIPPING hotfix attempt — going directly to restore protocol" + restore_protocol "$DIAGNOSIS" "$ROOT_CAUSE" "rollback" "$DETAILS" "$CONTAINER_LOGS" + + elif [ "$ACTION" = "rollback" ]; then + # Knowledge base says rollback — run restore protocol + restore_protocol "$DIAGNOSIS" "$ROOT_CAUSE" "$ACTION" "$DETAILS" "$CONTAINER_LOGS" + + elif [ "$ACTION" = "restart_container" ]; then + # Try restart first, escalate if fails + log "🔄 Attempting restart (restart #$restarts)..." + apply_hotfix "$ROOT_CAUSE" "$CONTAINER_LOGS" + sleep 10 + if [ "$(detect_fault)" != "healthy" ]; then + log "⚠️ Restart didn't resolve — escalating to restore protocol" + restore_protocol "$DIAGNOSIS" "$ROOT_CAUSE" "rollback" "$DETAILS" "$CONTAINER_LOGS" fi - ;; - - container_missing) - alert "CRITICAL" "Container $CONTAINER_NAME does not exist" - ;; - - oom_killed|sigkill_oom) - alert "WARN" "Container killed ($DIAGNOSIS) — restarting" - # If we've had too many restarts, escalate to rollback - local restarts - restarts=$(count_restarts) - if [ "$restarts" -ge "$MAX_RESTARTS" ]; then - alert "CRITICAL" "OOM repeated ($restarts restarts/hour) — rolling back" - remediate_rollback - else - remediate_container_restart "$DIAGNOSIS" - fi - ;; - - crashed_*|not_running) - alert "WARN" "Container crashed ($DIAGNOSIS)" - remediate_container_restart "$DIAGNOSIS" - ;; - - health_unreachable) - alert "WARN" "Health endpoint unreachable" - remediate_container_restart "$DIAGNOSIS" - ;; - - db_disconnected) - alert "WARN" "Database disconnected" - remediate_container_restart "$DIAGNOSIS" - ;; - - health_invalid) - alert "WARN" "Health response invalid" - remediate_container_restart "$DIAGNOSIS" - ;; - - gateway_down) - alert "WARN" "Gateway down" - remediate_gateway - ;; - - *) - alert "WARN" "Unknown diagnosis: $DIAGNOSIS" - remediate_container_restart "$DIAGNOSIS" - ;; - esac - - # ── Periodic maintenance ── - # Check disk every 10 cycles (5 min at 30s interval) - CYCLE_COUNTER=$((CYCLE_COUNTER + 1)) - if [ "$CYCLE_COUNTER" -ge 10 ]; then - CYCLE_COUNTER=0 - DISK_PCT=$(check_disk) - if [ "$DISK_PCT" -gt 90 ]; then - alert "WARN" "Disk at ${DISK_PCT}% — pruning" - remediate_prune + + elif [ "$ACTION" = "reload_gateway" ]; then + restore_protocol "$DIAGNOSIS" "$ROOT_CAUSE" "$ACTION" "$DETAILS" "$CONTAINER_LOGS" + + elif [ "$ACTION" = "hotfix_rate_limit" ]; then + apply_hotfix "$ROOT_CAUSE" "$CONTAINER_LOGS" + + else + # No specific action — safe default: restart, then rollback + log "🔄 No specific fix for '$ROOT_CAUSE' — attempting restart" + docker restart "$CONTAINER_NAME" >> "$AGENT_LOG" 2>&1 + record_restart + sleep 10 + [ "$(detect_fault)" != "healthy" ] && restore_protocol "$DIAGNOSIS" "$ROOT_CAUSE" "rollback" "$DETAILS" "$CONTAINER_LOGS" fi fi - + + # ── Periodic maintenance (every 10 cycles = 5 min) ───── + CYCLE=$((CYCLE + 1)) + [ "$CYCLE" -ge 10 ] && { CYCLE=0; periodic_maintenance; } + sleep "$CHECK_INTERVAL" done