1

Falah OS Backend QA Gap Analysis

wmj2024 edited this page 2026-07-07 15:08:52 +08:00

Table of Contents

• Falah OS Backend — QA, Testing & Post-Deployment Gap Analysis
• 🔴 TIER 1 — CRITICAL GAPS
• 1. FALAH_OS_TEST_FIX_LOG.md Contains False Claims
• 2. Netlify Functions Have Zero Test Coverage
• 3. Critical Security Issues
• 4. Docker Compose Race Condition
• 🟡 TIER 2 — QA INFRASTRUCTURE GAPS
• 5. No Cross-Service Integration Tests
• 6. No Tests for 5 of 11 Docker Services
• 7. Fragile Test Infrastructure
• 8. No Performance / Load / Stress Tests
• 9. No Security Tests
• 🟠 TIER 3 — PRE-DEPLOYMENT GAPS
• 10. No Deployment Pipeline
• 11. QA Directory Is Incomplete
• 12. Komodo Config May Be Outdated
• 🔵 TIER 4 — POST-DEPLOYMENT GAPS
• 13. No Monitoring Infrastructure Deployed
• 14. No Backup Automation
• 15. No Rollback Plan
• 16. No Post-Deploy Smoke Tests
• 17. No Incident Response
• ⚪ TIER 5 — ARCHITECTURAL CONCERNS
• 18. Dual Code Paths = Double Maintenance
• 19. The Mobile App Has Better QA Than the Backend
• 📊 Summary Scorecard
• 🎯 Action Plan
• Immediate (before any production deploy)
• Short-term
• Medium-term (v1.4)

Falah OS Backend — QA, Testing & Post-Deployment Gap Analysis

Source: github.com/Falah-Consultancy-Limited/Falah-OS-Master
Analyzed by: pi-agent
Date: 2026-07-04
Tools used: e2e-qa-test-agent, pre-deploy-cab, systematic-debugging, review-and-debug

---

🔴 TIER 1 — CRITICAL GAPS

1. FALAH_OS_TEST_FIX_LOG.md Contains False Claims

The fix log claims 110/110 tests across 8 suites plus 23/23 QA tests — but the test files it references do not exist on disk:

Claimed test file	Actual existence
tests/certificates.test.js	❌ DOES NOT EXIST
tests/products.test.js	❌ DOES NOT EXIST
tests/screening.test.js	❌ DOES NOT EXIST
tests/finance.test.js	❌ DOES NOT EXIST

Actual test files: 8 files (ummahid, wallet, ramz, mocknet + 4 admin variants). The log reports tests that cannot possibly pass because they don't exist.

The fix log also claims "No service code was modified" while describing fixes that touch tests/setup.js and node_modules/.

2. Netlify Functions Have Zero Test Coverage

• 6 function files (auth, certificates, finance, products, revocations, screening) + 2 lib files
• These use a different runtime (Deno/Web Standard: crypto.subtle, Response, Request)
• Docker service tests (Jest + supertest) cannot test them
• No test runner, no test file, no CI pipeline for the Netlify path
• The netlify.toml routes ALL API calls to /.netlify/functions/api but there is no api.mjs file

3. Critical Security Issues

Finding	Location	Severity
iBaaS hardcoded key	docker-compose.yml: IBAAS_API_KEY=${IBAAS_API_KEY:-falah-os-ibaas-key-2026}	🔴 CRITICAL — called out as blocker in fix log, still unfixed
HA compose hardcoded creds	docker-compose.ha.yml: POSTGRES_PASSWORD=falahsecure	🔴 CRITICAL — contradicts v1.3 security changelog
CORS wide open	All services: app.use(cors()) with no origin restriction	🔴 Known gap
Shared admin secret	All 4 services share the same x-admin-secret static token	🟡 HIGH

4. Docker Compose Race Condition

api-gateway depends_on references ibaas with condition: service_healthy, but ibaas is the last service defined in the file. The nginx upstream for ibaas returns 502 until ibaas is healthy, and if ibaas fails, the entire gateway stays down.

---

🟡 TIER 2 — QA INFRASTRUCTURE GAPS

5. No Cross-Service Integration Tests

All 8 test files test services in isolation (each require()-ed directly). Zero tests:

• Hit the API gateway and verify routing
• Test complete user flows across services (identity → wallet → contract)
• Verify nginx rate limiting
• Verify security headers
• Guard POST /api/__reset in non-test environments

6. No Tests for 5 of 11 Docker Services

Service	Has Tests?
ummahid	✅
wallet	✅
ramz	✅
mocknet	✅
falahd (system daemon)	❌ None
ibaas (Islamic Banking)	❌ None
app (frontend server)	❌ None
community	❌ None
prayer	❌ None
chat	❌ None
api-gateway (nginx)	❌ None (routes only)

7. Fragile Test Infrastructure

• Admin test files use bare require('./setup') — relies on Node.js module caching
• process.env is mutated globally, affecting all tests if load order changes
• --runInBand required because in-memory state and ports conflict in parallel
• In-memory Maps grow unbounded — no test validates memory behavior

8. No Performance / Load / Stress Tests

• k6 load testing roadmapped for Phase 5 but not implemented
• No benchmark baselines
• No memory leak detection for in-memory Maps
• No chaos testing (Mock-Net has chaos infra but no tests exercise it)

9. No Security Tests

• No JWT tampering, replay, or expiration boundary tests
• No SQL injection tests (critical when Postgres is wired in v1.4)
• x-admin-secret has no rate limiting — brute-forceable via admin endpoints
• No CSRF protection on any endpoint
• No input size validation beyond nginx's client_max_body_size 10M

---

🟠 TIER 3 — PRE-DEPLOYMENT GAPS

10. No Deployment Pipeline

Capability	Status
make up (Docker)	✅
npm run deploy (Netlify)	✅
GitHub Actions / CI	❌ None
Automated deploy gate	❌ None
Env promotion (dev → staging → prod)	❌ None
Pre-deploy checklist automation	❌ None
Rollback automation	❌ None

11. QA Directory Is Incomplete

qa/qa.test.js is a good persona-based test (REGULATOR → MERCHANT → CONSUMER) but:

• Tests a single Docker Compose stack — doesn't validate the Netlify path
• Requires services to be require()-able, not running via Docker
• Not integrated into any deploy workflow

12. Komodo Config May Be Outdated

komodo.toml exists but its URLs should be verified against current infrastructure.

---

🔵 TIER 4 — POST-DEPLOYMENT GAPS

13. No Monitoring Infrastructure Deployed

Component	Status
Prometheus config	⚠️ Stub exists, not deployed
Grafana dashboards	❌ Not created
Loki log aggregation	❌ Not deployed
Alertmanager	⚠️ Config stub, not deployed
Health dashboard	❌ None — only CLI health-check.sh
Uptime monitoring	❌ None
Error rate tracking	❌ None

14. No Backup Automation

• scripts/backup.sh backs up nothing meaningful (Postgres unused in v1.3)
• No cron / scheduled backup
• No off-site backup (S3 vars in .env.example are optional/empty)
• No restore drill documented

15. No Rollback Plan

Called out as a "Production Blocker" in the fix log. Still unfilled.

16. No Post-Deploy Smoke Tests

No automated check runs after deploy to verify:

• All services healthy
• Critical user flows work
• Error rates within baseline
• API contracts unchanged

17. No Incident Response

• No alerting channels configured (Slack webhook URL is Optional in .env.example)
• No runbooks for common failures
• No on-call rotation

---

⚪ TIER 5 — ARCHITECTURAL CONCERNS

18. Dual Code Paths = Double Maintenance

Aspect	Netlify Functions	Docker Services
Runtime	Deno (Edge)	Node.js (Express)
JWT auth	Custom crypto.subtle HMAC	jsonwebtoken npm package
Responses	Response.json()	res.json()
State	Mutable module-level arrays	Map objects
Auth flow	verify(token) throws on bad	jwt.verify() returns payload
Users	Hardcoded DEMO_USERS	Express auth endpoint

Netlify functions mutate module-level arrays (CERTS.push(...)) but Netlify function instances are ephemeral — mutations are unreliable.

19. The Mobile App Has Better QA Than the Backend

Ironically, the Falah Mobile app (pi-agent/falah-mobile at git.falahos.my) has proper QA maturity:

QA Artifact	Falah Mobile	Falah OS Backend
Layered test strategy	✅ qa-strategy.md	❌ None
HTTP contract tests	✅ qa-test-suite.py (8 layers)	❌ None
Pre-deploy CAB checklist	✅ cab-qa.sh	❌ None
Visual regression	✅ qa-visual.py	❌ None
E2E persona scenarios	✅ qa-test-suite.py	⚠️ Basic qa/qa.test.js

The backend should adopt the mobile app's QA framework.

---

📊 Summary Scorecard

Category	Status	Score
Unit tests (Docker)	8/11 services, missing test files claimed	50%
Unit tests (Netlify)	No test infrastructure exists	0%
Integration tests	No cross-service or gateway-level tests	0%
Security tests	None	0%
Performance tests	None	0%
CI/CD pipeline	None	0%
Monitoring	Stubs exist, nothing deployed	10%
Backup/DR	No backup, no rollback	0%
Post-deploy verification	No smoke tests	0%
Documentation accuracy	Test fix log makes false claims	30%

Overall QA Readiness: ~10%

---

🎯 Action Plan

Immediate (before any production deploy)

1. Delete the fabricated test log — audit what tests actually exist
2. Fix iBaaS hardcoded key in docker-compose.yml
3. Remove hardcoded creds from docker-compose.ha.yml
4. Create missing test files (certificates, products, screening, finance)
5. Add Netlify function tests (Deno-compatible runner)

Short-term

6. Run e2e-qa-test-agent against the repo for real baselines
7. Set up GitHub Actions with npm test gate
8. Add tests for untested services (falahd, ibaas, community, prayer, chat)
9. Wire CAB protocol into the deploy process
10. Deploy Prometheus + Grafana from existing config stubs

Medium-term (v1.4)

11. Apply TDD discipline for all new code
12. Unify the two API code paths or document canonical
13. Add rollback procedures and automated backup
14. Adopt mobile app's QA framework for backend

---

Generated by pi-agent using e2e-qa-test-agent, pre-deploy-cab, systematic-debugging, and review-and-debug skills.