Building an AI search tech stack from scratch - what components do you actually need?

I’ve built this stack multiple times. Here’s the framework I use:

Core Architecture (RAG Pattern):

User Query
    ↓
Query Embedding (embedding model)
    ↓
Vector Search (vector DB)
    ↓
Candidate Retrieval
    ↓
Reranking (cross-encoder)
    ↓
Context Assembly
    ↓
LLM Generation
    ↓
Response

Component Recommendations for Your Scale (500K docs):

Budget reality check:

At 500K docs, you’re looking at:

• Vector DB: $100-500/month managed
• Embedding costs: One-time ~$50-100 to embed corpus
• LLM costs: Usage dependent, plan for $500-2000/month

For 2 engineers, managed services are 100% worth it.

Been running AI search in production for 18 months. Here’s what I’d do differently:

Mistakes we made:

1. Started with self-hosted vector DB - Wasted 3 months on infrastructure. Should have used managed from day 1.

2. Cheap embedding model - Saved $20/month, lost significant retrieval quality. Quality embeddings are worth it.

3. No hybrid search initially - Pure vector search missed exact-match queries. Hybrid (vector + BM25) solved this.

4. Underestimated monitoring needs - Hard to debug when you can’t see retrieval quality metrics.

What we run now:

• Pinecone (vector) + Elasticsearch (BM25) hybrid
• OpenAI embeddings (ada-002, upgrading to 3)
• Cohere reranker
• Claude for generation
• Custom monitoring dashboard tracking retrieval metrics

Latency breakdown:

• Embedding: 30ms
• Hybrid search: 40ms
• Rerank: 60ms
• LLM: 800ms (streaming helps UX)

Total perceived latency is fine because we stream LLM output.

Adding the data pipeline perspective that often gets overlooked:

Document processing matters A LOT:

Before anything touches your vector DB, you need:

1. Chunking strategy - How do you split documents?
2. Metadata extraction - What attributes do you capture?
3. Cleaning pipeline - Remove boilerplate, normalize text
4. Update mechanism - How do new/changed docs flow through?

Chunking advice:

The trap:

People spend weeks on vector DB selection and days on chunking. It should be the opposite. Bad chunking = bad retrieval no matter how good your vector DB is.

Vector database comparison based on your requirements:

For 500K docs + 2 engineers + sub-200ms:

Pinecone:

• Pros: Fully managed, excellent docs, predictable pricing
• Cons: Vendor lock-in, limited customization
• Fit: Perfect for your constraints

Qdrant:

• Pros: Great performance, good hybrid support, cloud or self-host
• Cons: Newer managed offering
• Fit: Good option, especially if you might need hybrid search

Weaviate:

• Pros: Great hybrid search, built-in vectorization
• Cons: More complex setup
• Fit: Better for larger teams

Milvus:

• Pros: Most scalable, fully open source
• Cons: Requires infrastructure expertise
• Fit: Overkill for your scale, pass

My recommendation:

Start with Pinecone. It’s boring (in a good way). You’ll have time to evaluate alternatives once you understand your actual needs better.

Don’t forget MLOps and observability:

What you need to track:

1. Retrieval metrics

   • Precision@K (are top K results relevant?)
   • Recall (are we finding all relevant docs?)
   • Latency distribution

2. Generation metrics

   • Response relevance (does answer match query?)
   • Groundedness (is answer supported by context?)
   • Hallucination rate

3. System metrics

   • Query latency p50/p95/p99
   • Error rates
   • Cost per query

Tools:

• Weights & Biases for experiment tracking
• Datadog/Grafana for system monitoring
• LangSmith for LLM observability
• Custom dashboard for business metrics

The thing nobody tells you:

You’ll spend more time on monitoring and debugging than building the initial system. Plan for it from day 1.

Startup reality check:

If you’re building this for a business (not research), consider:

Build vs Buy:

• Building RAG from scratch: 2-3 months dev time
• Using existing RAG platform: Days to production

Platforms that package this:

• LlamaIndex + managed vector DB
• Vectara (full RAG-as-a-service)
• Cohere RAG endpoints

When to build custom:

• Need extreme customization
• Data sensitivity requirements
• Scale economics make sense
• Core competency differentiation

When to use platform:

• Speed to market matters
• Small team
• RAG isn’t your product, it enables your product

For most businesses, the platform approach wins until you hit scale limitations.

Security considerations nobody mentioned:

Data concerns:

1. What data are you sending to external embedding APIs?
2. What data goes to LLM providers?
3. Where is your vector DB hosted?

Options for sensitive data:

• Self-hosted embedding models (Sentence Transformers)
• Self-hosted vector DB (Qdrant, Milvus)
• On-premise LLM (Llama, Mixtral)
• VPC-deployed managed services

Compliance checklist:

• Data residency requirements met
• Encryption at rest and in transit
• Access controls and audit logging
• Data retention policies
• PII handling procedures

Don’t assume managed services meet your compliance needs. Check explicitly.

This thread has been incredibly valuable. Here’s my updated plan:

Architecture decision:

Going with managed services for speed and team size constraints:

• Pinecone for vector storage
• OpenAI text-embedding-3 for embeddings
• Cohere reranker
• Claude for generation
• LangChain for orchestration

Key learnings:

1. Chunking strategy matters as much as vector DB choice - Will invest time here
2. Reranking is high-ROI - Adding it from the start
3. Hybrid search for coverage - Will implement vector + BM25
4. Monitoring from day 1 - Building observability in, not bolting on
5. Security review early - Confirming compliance before going to production

Timeline:

• Week 1-2: Data pipeline and chunking
• Week 3-4: Core RAG implementation
• Week 5: Monitoring and optimization
• Week 6: Security review and production prep

Thanks everyone for the detailed insights. This community is gold.