Knowledge Vault

Semantic Search and Text Embeddings

2 min read

Semantic Search and Text Embeddings

Overview

Text embeddings are high-dimensional vector representations of text generated by encoder-based models like BERT. By converting text into numerical vectors, semantic similarity between documents can be quantified mathematically — enabling search that understands meaning rather than just matching keywords. This approach powers modern information retrieval, recommendation systems, and clustering applications.

Text Embedding Process

Token-Level Embeddings

Encoder models like BERT produce a vector for each token in the input sequence. A sentence of N tokens yields N embedding vectors, each capturing contextual meaning.

Pooling to Sentence Embeddings

Two primary strategies aggregate token embeddings into a single sentence-level vector:

  • CLS token pooling — Use the embedding of the special [CLS] token as the sentence representation
  • Mean pooling — Average all token embeddings, typically producing better results for similarity tasks

Similarity Measurement

Cosine Similarity

The standard measure for comparing embedding vectors. Computes the cosine of the angle between two vectors — values near 1 indicate high semantic similarity, values near 0 indicate low similarity. Works well because it normalizes for vector magnitude, focusing on direction (meaning) rather than length.

For large-scale retrieval, brute-force cosine similarity is too slow. FAISS provides efficient nearest-neighbor search through:

  • Index structures optimized for high-dimensional vectors
  • Approximate nearest neighbor (ANN) algorithms for sub-linear search time
  • GPU acceleration for massive corpora

Integration with Hugging Face Datasets enables indexing document embeddings and querying with a single function call.

Semantic Search Pipeline

  1. Encode corpus — Pass all documents through the encoder model, apply pooling, store embeddings
  2. Build index — Create a FAISS index from document embeddings
  3. Encode query — Pass the search query through the same encoder model
  4. Retrieve — Find nearest neighbors in the index by cosine similarity
  5. Rank — Return top-K most similar documents

This pipeline moves beyond keyword matching to context-aware document comparison — “How do I fix a flat tire?” retrieves documents about tire repair even if they never contain the word “fix.”

See Also

Graph View

Backlinks