A Vector Database is a specialized database designed to store, index, and query high-dimensional vectors (also called embeddings). These vectors are numerical representations of unstructured data — text, images, audio — generated by AI models. Vector databases power the retrieval layer of AI applications like RAG (Retrieval-Augmented Generation), semantic search, and recommendation engines.
How Vector Databases Work
``
1. EMBED: Convert data to vectors using AI models
"Snowflake Architecture" → [0.23, -0.45, 0.78, ...] (768 dimensions)
2. STORE: Index vectors for fast similarity search
Vector DB stores millions of vectors with metadata
3. QUERY: Find similar vectors using distance metrics
"cloud data warehouse" → Find top 10 most similar documents
``
Why Vectors Matter
Traditional databases search by exact match (WHERE name = 'John'). Vector databases search by similarity — finding items that are conceptually close, even if they share no keywords.
| Traditional Search | Vector Search |
|---|---|
| WHERE title LIKE '%snowflake%' | Find articles about cloud data warehousing |
| Exact keyword matching | Semantic meaning matching |
| Misses synonyms and context | Understands intent and relationships |
Key Concepts
Embeddings
Numerical vectors that capture the semantic meaning of data:
- Text: Generated by models like OpenAI Ada, Cohere, Snowflake Arctic
- Images: Generated by CLIP, ResNet
- Dimensions: Typically 256 to 1536 floating-point numbers
Similarity Metrics
- Cosine Similarity: Angle between vectors (most common for text)
- Euclidean Distance: Straight-line distance
- Dot Product: Combination of direction and magnitude
Indexing Algorithms
- HNSW (Hierarchical Navigable Small World): Fast, accurate, memory-intensive
- IVF (Inverted File Index): Good balance of speed and memory
- PQ (Product Quantization): Compressed vectors for huge datasets
Popular Vector Databases
| Database | Type | Best For |
|----------|------|----------|
| Pinecone | Managed SaaS | Production RAG, zero-ops |
| Weaviate | Open Source | Hybrid search (vector + keyword) |
| Milvus | Open Source | Massive scale (billions of vectors) |
| Qdrant | Open Source | Rust-based performance |
| Chroma | Open Source | Local development, prototyping |
| pgvector | PostgreSQL Extension | Adding vectors to existing Postgres |
| Snowflake Cortex | Integrated | Snowflake-native RAG |
Use Cases
1. RAG (Retrieval-Augmented Generation): Feed relevant context to LLMs
2. Semantic Search: Find results by meaning, not keywords
3. Recommendation Engines: Suggest similar products/content
4. Image Search: Find visually similar images
5. Anomaly Detection: Identify data points far from normal patterns