From Prototype to Production: Hardening a University RAG Assistant

Context

NYU's Berkley Center for Entrepreneurship serves 1,100+ startup teams per year, and staff were drowning in repetitive program questions. I scoped and built an AI knowledge assistant grounded exclusively in verified institutional content, with streaming, citation-backed answers and multi-turn follow-ups, then led the production-hardening effort that took it from working prototype to reliable internal tool.

Problem

A RAG prototype that answers correctly in a demo is not a production system. The real challenges: hallucination risk against institutional trust, abuse and cost exposure on the expensive LLM path, latency, and reliability under concurrent load, all on a university budget.

Approach

Grounded-first architecture: every answer must trace to approved sources with citations. Then production hardening in layers: a semantic cache in front of the LLM, a guardrail register covering rate limits and abuse controls executed before any expensive work, structured telemetry for product analytics, and a maker/checker review loop that drove open findings to zero.

Frameworks

Retrieval-Augmented Generation (FAISS, query expansion, semantic chunking)OWASP LLM Top 10 threat modelingSemantic Caching (Postgres + pgvector, HNSW)Test-Driven Production Hardening

Implementation

  • •End-to-end pipeline: web scraping, semantic chunking, embeddings, FAISS similarity search, query expansion, streamed cited responses
  • •Postgres + pgvector semantic cache with similarity-threshold gating, TTL, and content-versioned invalidation
  • •Guardrail register (15 cases): session/IP rate limits, burst limits, concurrent-stream caps, prompt-size and loop detection, all executed before cache, retrieval, or generation
  • •Natively async re-architecture (AsyncOpenAI, async connection pools, async SSE) proven under a 25-concurrent-stream test
  • •Redis-backed sliding-window rate limiting with bounded in-memory fallback and fail-open degradation
  • •Structured per-query telemetry: answer rate, cache hits, latency, cost, and failure reasons

Outcomes

  • ✓55% semantic cache hit rate; cached answers 41x faster (357 ms vs ~14.7 s) at 98% lower cost per hit
  • ✓Resolved ~25 review findings, including 5 high-severity issues (one a cross-conversation cache leak), to zero open issues
  • ✓Test suite grown from 33 to 207 backend tests plus 21 frontend tests, hermetic and deterministic
  • ✓One slow request can no longer freeze others, verified with sabotage-tested concurrency suites
  • ✓Serving 1,100+ startup teams per year with grounded, citation-backed answers

Learnings

  • →The distance between "RAG demo" and "production RAG" is mostly guardrails, caching, telemetry, and tests
  • →Caching semantic answers requires content-versioned invalidation; stale answers are trust killers
  • →Follow-up questions must bypass shared caches: history-dependent answers leak across conversations
  • →A maker/checker agent review loop is a genuinely effective way to drive findings to zero