Agentic AI Chatbot Platform
Production multi-agent assistant for cybersecurity & case management — a router agent dispatching to five specialist agents across hybrid RAG, text-to-SQL and action execution, on self-hosted vLLM.
role: Architect & lead developer
Problem
Analysts and enterprise clients needed one assistant that could answer questions from a large knowledge base, query live case data, and take actions — across 4000+ monitored brands and 400+ clients, each with strict data boundaries.
Architecture
A router agent classifies every query and dispatches it to one of five specialist agents. Everything runs on self-hosted inference so no client data leaves the platform.
Key engineering
- Hybrid retrieval — BM25 + dense search with cross-encoder reranking over a 200+ document multi-page knowledge base; chunking and metadata filtering tuned per document type. Measurably better relevance than dense-only, fewer domain hallucinations.
- Self-learning few-shot engine — successful queries are auto-harvested into Redis as exemplars and injected for semantically similar inputs → text-to-SQL accuracy 60% → 95% with zero fine-tuning cost.
- Context engineering — conversation memory, token-window budgeting, summarisation-based compaction and dynamic prompt assembly keep 10+ turn sessions grounded without overflow.
- Semantic caching across vLLM and SGLang holds p95 at 5–6s under 500–600 concurrent requests.
- RBAC at retrieval & tool layer so answers respect per-tenant, per-role boundaries.
- MCP servers standardise tool and data-source exposure — onboarding a new internal tool no longer needs bespoke glue.
- Eval & guardrail harness — golden-set regression, groundedness/relevance scoring, prompt-injection and PII filters gate every prompt or model change.
Deployment
Docker + gRPC microservices behind an async worker architecture; Grafana dashboards for inference throughput, agent health, cache hit-rate, token spend and latency.
Tool-Selection LLM — LoRA Fine-tune on xLAM
Teaching a small LLM to pick the right tool/API for an agent — and say how confident it is. LoRA-tuned Qwen3.5-2B with Unsloth on xLAM function-calling data: 50.6% → 100% accuracy, ECE 0.220 → 0.002.
DocuMind AI — Grounded Multimodal Document Q&A at Scale
System design for a 1M-user document Q&A product: async multimodal ingestion, hybrid retrieval, a deterministic confidence gate that refuses instead of hallucinating, and hot/warm/cold session tiering.
RedForge — AI Red-teaming & GenAI Eval Platform
Target-agnostic platform that attacks any LLM chatbot with 200+ OWASP-LLM-Top-10 test cases and an adversarial mutation engine. Built on personal time, adopted as a company product.