Vector Database

June 18, 2024 (2y ago)

Vector Database

Vector databses indexes and stores vector embeddings for fast retrival and similarity search.

My diagram

After the generation of embedding vectors using embedding models, We need to retrive the similar context from the database, For this we can use different similarity search techniques like :

  1. Euclidean Distance

Disadvantages of Euclidean Distance in Similarity Search

  1. Curse of Dimensionality In high-dimensional spaces, all distances tend to become similar, reducing contrast between close and far vectors.

  2. Sensitive to Magnitude Two vectors with the same direction but different lengths are considered far apart, even if semantically similar.

  3. Requires Normalization Without normalizing vectors, distance calculations may be skewed by scale differences.

  4. Can Mislead in Dense Embedding Spaces In tight clusters of vectors (e.g., sentence embeddings), Euclidean distance may overemphasize minor spatial differences.

Cosine Similarity

Cosine similarity measures how similar two vectors are based on the angle between them — not their length or magnitude.

Formula

\text{cosine\_similarity}(A, B) = \frac{A \cdot B}{\|A\| \|B\|}

Where:

* $A \cdot B$ is the dot product of vectors A and B
* $\|A\|$ and $\|B\|$ are the magnitudes (lengths) of vectors A and B

Value Range

For most real-world embeddings (which are non-negative or positive), the range is usually [0, 1].

Advantages of Cosine Similarity

  1. Ignores magnitude Only considers direction — useful when length has no meaning (e.g., TF-IDF or sentence embeddings).

  2. Works well in high dimensions Less affected by the curse of dimensionality compared to Euclidean distance.

  3. Scale invariant Two vectors pointing the same way are 100% similar, even if one is 10x longer.

  4. Computationally efficient Especially when vectors are normalized beforehand.

Limitations of Cosine Similarity

  1. Ignores magnitude Can’t distinguish between strong and weak signals if vector length matters.

  2. Zero vectors cause issues Division by zero occurs if a vector has magnitude 0.

  3. Not suitable when distance matters In applications like clustering where density or compactness is important, Euclidean may be better.

Manhattan Distance

Manhattan distance (also called L1 distance, taxicab distance, or city block distance) measures the total absolute difference between the coordinates of two vectors.

It’s like measuring the distance a taxi would drive in a grid-like city — moving only along axes, not diagonally.

alt text

Formula

For two vectors $A = (a_1, a_2, ..., a_n)$ and $B = (b_1, b_2, ..., b_n)$:


\text{Manhattan Distance} = \sum_{i=1}^{n} |a_i - b_i|

Example

Let:

\text{Manhattan Distance} = |3 - 1| + |5 - 9| = 2 + 4 = 6

Advantages

  1. Works well with high-dimensional sparse data Common in text classification and recommendation systems.

  2. More robust to outliers than Euclidean distance Doesn’t square differences, so large deviations have less impact.

  3. Computationally simpler No need to square or take square roots.

Disadvantages

  1. Less intuitive in geometric space Doesn't correspond to “as-the-crow-flies” distance.

  2. Can be less accurate for embeddings In semantic vector spaces, direction often matters more than raw coordinate differences.

  3. Not rotation-invariant Like Euclidean, it changes if the coordinate axes rotate.

These all techiques along with other multiple techiques for similarity search aren't used for retrival of relevant documents from vector databse. This is because there are millions of embedding with high-dimentions for large database and calucating similarity in these above approach takes O(N) times with computing cost and time. So we use Indexes for fast retrieval.

Common Vector Indexs include:

LinkedIn
X