System design lesson: size hot storage by concurrency, not users

While designing a 1M-user document Q&A system I sized the vector store as "keep 5 sessions warm per user" → ~10.5 TB.

The fix

Hot storage tracks concurrent access; cold storage tracks total history.

• 5–10k concurrent sessions × ~2.1 MB (embeddings + extracted text) ≈ 10.5–21 GB hot
• Everything else lives in S3 → Glacier and is rehydrated on access (cache-aside at session level)

That's a 500–1000× difference from conflating the two.

Second lesson: refusal belongs in code

"Only answer from the context" in a system prompt is not a guarantee. A deterministic confidence gate on retrieval scores — skipping the LLM entirely when low — is.

Full case study: documind-ai-rag-system