# Long Context vs Vector Retrieval: The Production Trade-off

Category: Retrieval & Pipelines
Stack: Gemini 1.5 Pro, Claude 3.5 Sonnet, Mosaic AI Vector Search

## Overview
Balancing raw context window brute-forcing against chunked vector retrieval based on query latency, token budgets, and cache hit rates.

## The Problem
Dumping complete multi-megabyte documents into raw context windows costs significant token spend and introduces attention degradation ('needle-in-a-haystack' drop-off) in middle tokens.

## Production Rule
Combine prompt caching with targeted retrieval. Use vector or hybrid search to retrieve relevant chapters, then feed structured sections into the cached model prompt.

## Code / Specification
File: `hybrid_context_strategy.md`

```architecture pattern
Production Heuristic:
1. File size < 50 pages: Direct prompt with prompt caching enabled (90% cost reduction on subsequent calls).
2. File size > 50 pages: Hybrid search (BM25 + Dense Vector) -> Top 10 chunks -> Re-ranker -> Filtered context window.
3. Tabular / Structured: SQL extraction first, never raw context dumping.
```