ServicesWorkAboutBlogContact
← Back to work
CASE STUDY · AI / RAG

A compliance chatbot that answers from 20,000+ documents - and proves every word.

For a compliance-EdTech client, we built a production retrieval-augmented (RAG) chatbot that answers regulatory and compliance questions with a verified citation on every response - reaching 95%+ correct answers across a knowledge base of more than 20,000 documents.

ClientCompliance-EdTech (confidential)
IndustryEdTech · Compliance
EngagementCustom RAG system - ingestion to production API
95%+Correct answers
20K+Documents ingested
100%Answers cited
DailyFreshness checks
The chatbot returning a structured, cited answer to a compliance question - conclusion, status, and required actions
Every answer comes back structured and cited - a clear conclusion, a status, and the exact required actions, grounded in the source regulation.
THE CHALLENGE

In compliance, a confident wrong answer is worse than no answer.

The client needed reliable answers to regulatory and compliance questions sourced from hundreds of official URLs and over 20,000 documents. Off-the-shelf chatbots fall short here in three ways: they hallucinate, they can't cite where an answer came from, and they don't understand that a statute outranks a sub-regulatory note. On top of that, the source material changes over time - and a stale answer in this domain is a liability, not just a bug.

THE APPROACH

What happens on every single question

A ten-stage pipeline turns a raw question into a cited, confidence-scored, audit-logged answer:

  1. 01

    Validate & preprocess

    Authenticate the request by API key, then normalize legal references and expand acronyms so the question matches how the source material is written.

  2. 02

    Embed the question

    Encode the query with OpenAI text-embedding-3-large - the exact model used at ingestion, so similarity is consistent.

  3. 03

    Hybrid retrieval

    Run semantic search in Pinecone and keyword (BM25) search in PostgreSQL in parallel, so both meaning and exact terminology are covered.

  4. 04

    Reciprocal Rank Fusion

    Merge the two result sets into a single ranked list that rewards passages strong in both searches.

  5. 05

    Authority reranking

    Weight sources by hierarchy - statute over regulation over sub-regulatory guidance - so the most authoritative passage wins.

  6. 06

    Assemble context

    Pull the full text of the top 8–10 chunks from PostgreSQL with clear source labels for the model to ground on.

  7. 07

    Generate (temperature 0)

    Call the OpenAI ChatGPT model with a system prompt that forbids answering outside the provided context and requires citations on every claim.

  8. 08

    Verify citations

    Strip any citation the model did not actually ground in a retrieved source - no fabricated references survive.

  9. 09

    Confidence scoring

    Tag every answer HIGH / MEDIUM / LOW so low-confidence responses are visibly flagged rather than presented as fact.

  10. 10

    Log everything

    Record the query, answer, sources cited, chunks used, confidence and latency for a complete audit trail.

THE ARCHITECTURE

A lean, pay-per-use stack built on what the client already owned

Because the client was already on AWS, ingestion runs as parallel batch jobs in their own account, while the live query path stays simple and managed. Vectors live in Pinecone; everything authoritative - text, citations, logs - lives in PostgreSQL.

Ingestion - AWS Batch + Fargate

Parallel scrapers pull hundreds of URLs and PDFs, store raw files in S3, then parse → chunk → embed → upsert to Pinecone. Progress is tracked in PostgreSQL so a run is fully resumable.

Freshness - EventBridge

A scheduled daily task re-checks every source with hash comparison; changed documents are flagged for review and re-ingested on approval, with old chunks retired.

Vector search - Pinecone

3072-dim embeddings with filterable metadata (source type, authority level, section), keyed by stable UUIDs shared with PostgreSQL for idempotent updates.

Relational store - Supabase / PostgreSQL

Single source of truth for chunk text, citations, query logs, chat sessions and ingestion status - plus optional BM25 keyword search via tsvector.

Backend - FastAPI on Railway

Runs the full RAG pipeline and admin endpoints (content freshness, logs, manual re-ingestion, health) behind API-key auth. The client keeps end-user auth on their side.

LLM & embeddings - OpenAI

text-embedding-3-large for retrieval and a ChatGPT-class model at temperature 0 for deterministic, regulation-faithful answers.

RELIABILITY

Designed to fail safely

  • OpenAI outageCircuit breaker returns a clear “temporarily unavailable” instead of hanging.
  • Pinecone outageDegrade gracefully to keyword (BM25) search and queue retries.
  • Wrong or fabricated citationsAutomatic citation verification plus subject-matter review.
  • Stale knowledge baseDaily source diff checks with alerting on repeated empty diffs.
  • Runaway costPer-API-key rate limits and billing alerts.
THE OUTCOME

Answers you can trust - and verify

The system reached 95%+ correct answers across the pilot, with every answer carrying verifiable citations and a confidence label. Responses are deterministic (temperature 0), the knowledge base stays current through automated daily checks, and every interaction is fully audit-logged - all running on a lean stack that costs only what it uses.

AWS BatchAWS FargateAmazon S3EventBridgePineconeSupabasePostgreSQLFastAPIRailwayOpenAINext.js

Want a chatbot that answers from your data - accurately?

We build custom RAG systems grounded in your own documents, with citations and confidence built in.