3B Tokens/Day Vision-LLM Inference on One GPU
Scaled self-hosted vision-LLM screenshot analysis to 3 billion input and 50 million output tokens per day on a single 96GB GPU — no third-party API spend.
role: Developer
Levers that mattered
- Batching tuned for screenshot-heavy prompts
- KV-cache reuse for shared prompt prefixes
- Quantization to fit throughput targets in 96GB
- Concurrency limits to keep latency stable under load
The result: 3B input / 50M output tokens per day on a single GPU, avoiding third-party API spend entirely.
Tool-Selection LLM — LoRA Fine-tune on xLAM
Teaching a small LLM to pick the right tool/API for an agent — and say how confident it is. LoRA-tuned Qwen3.5-2B with Unsloth on xLAM function-calling data: 50.6% → 100% accuracy, ECE 0.220 → 0.002.
DocuMind AI — Grounded Multimodal Document Q&A at Scale
System design for a 1M-user document Q&A product: async multimodal ingestion, hybrid retrieval, a deterministic confidence gate that refuses instead of hallucinating, and hot/warm/cold session tiering.
Agentic AI Chatbot Platform
Production multi-agent assistant for cybersecurity & case management — a router agent dispatching to five specialist agents across hybrid RAG, text-to-SQL and action execution, on self-hosted vLLM.