Modern applications require the ability to respond to complex user queries not just with keywords, but semantically. From product recommendations to document search, chatbots to content filtering, semantic search goes beyond traditional methods to deliver more relevant and rich results. In this guide, we will explore vector databases, key tools that power these AI-driven search capabilities, and their practical applications.

Semantic Search and Vector Representations

At the heart of semantic search lies the idea of creating numerical representations (embeddings) of data (text, images, audio, etc.) in high-dimensional vector spaces. These vectors capture the semantic content of the data; data that is semantically close will have vectors that are also close to each other in this space. The search process involves calculating the similarity between a query's vector representation and other vectors in the database.

Pre-trained deep learning models are typically used to create vector representations. For text, options like SBERT (Sentence-BERT) or OpenAI's embedding models are available. Here's a simple SBERT embedding example with Python:

from sentence_transformers import SentenceTransformer
import numpy as np

# Load a pre-trained SBERT model
model = SentenceTransformer('paraphrase-multilingual-MiniLM-L12-v2')

# Example sentences
sentences = [
    "How are artificial intelligence models trained?",
    "Machine learning algorithms",
    "Deep learning architectures",
    "How computer hardware works?"
]

# Convert sentences to vectors
embeddings = model.encode(sentences)

print("Vector dimension of the first sentence:", embeddings[0].shape)
print("Cosine similarity between the first 2 sentences' vectors:", np.dot(embeddings[0], embeddings[1]) / (np.linalg.norm(embeddings[0]) * np.linalg.norm(embeddings[1])))

Why Vector Databases?

Traditional relational or NoSQL databases are not designed to efficiently perform similarity searches between high-dimensional vectors. Finding the Nearest Neighbor Search (NNS) or Approximate Nearest Neighbor (ANN) among millions or even billions of vectors requires specialized algorithms and indexing structures. Vector databases offer optimized structures specifically designed to solve this challenge.

Local and In-Memory Solutions: Faiss

Faiss (Facebook AI Similarity Search) is a library developed by Facebook AI for high-performance similarity search and clustering. It is specifically optimized for fast approximate nearest neighbor searches on millions of vectors. It operates in-memory and is generally designed for use on a single server or client-side.

Here's a simple example of creating a Faiss index and searching:

import faiss

# Vector dimension and number of data points
d = embeddings.shape[1] # 384
nb = embeddings.shape[0] # 4 (in example)

# Converting to Float32 is a common requirement for Faiss
xb = embeddings.astype('float32')

# Create a Faiss index (e.g., an IndexFlatL2)
# This index stores every vector and performs an exact search (NNS, not ANN)
index = faiss.IndexFlatL2(d) 

print("Is index trained?", index.is_trained) # Should be True as IndexFlatL2 doesn't need training

# Add vectors to the index
index.add(xb)

print("Number of vectors in the index:", index.ntotal)

# Create a query vector (e.g., the vector of the first sentence)
xq = np.array([model.encode("Information about artificial intelligence training")])
xq = xq.astype('float32')

# Search for the 2 nearest neighbors
k = 2
distances, indices = index.search(xq, k)

print("Distances of nearest neighbors (L2):", distances)
print("Indices of nearest neighbors:", indices)
print("Nearest sentences:", [sentences[i] for i in indices[0]])

Faiss offers different index types (IndexFlatL2, IndexIVFFlat, IndexHNSWFlat, etc.) allowing you to balance performance and accuracy. For larger datasets, using ANN algorithms like IndexIVFFlat or IndexHNSWFlat is more efficient.

Distributed and Scalable Solutions: Milvus

When you need to work with billions of vectors or require high availability and scalability on a distributed system, cloud-native vector databases like Milvus come into play. Milvus is an open-source system designed for large-scale vector search and can be deployed on platforms like Kubernetes.

Here's an example of creating a collection, adding vectors, and searching with Milvus (assuming a Milvus instance running on Docker):

from pymilvus import connections, FieldSchema, CollectionSchema, DataType, Collection, utility
import random

# Connect to the Milvus server
connections.connect("default", host="localhost", port="19530")

# Define the collection schema
fields = [
    FieldSchema(name="id", dtype=DataType.INT64, is_primary=True, auto_id=True),
    FieldSchema(name="embedding", dtype=DataType.FLOAT_VECTOR, dim=d), # d = 384 from SBERT
    FieldSchema(name="text", dtype=DataType.VARCHAR, max_length=512)
]
schema = CollectionSchema(fields, "This is an example collection for semantic search.")

# Define the collection name
collection_name = "semantic_search_collection"

# If the collection already exists, drop it (for development)
if utility.has_collection(collection_name):
    utility.drop_collection(collection_name)

# Create the collection
collection = Collection(collection_name, schema)

# Prepare vectors and associated text
data = [
    model.encode(s).tolist() for s in sentences
]
text_data = sentences

# Insert data into Milvus
mr = collection.insert([data, text_data])

# Wait for data to synchronize in Milvus
collection.flush()

print("Number of vectors added:", collection.num_entities)

# Create an index (e.g., IVFFlat)
index_params = {
    "metric_type": "L2",
    "index_type": "IVF_FLAT",
    "params": {"nlist": 128}
}
collection.create_index(field_name="embedding", index_params=index_params)

# Load the collection into memory for searching
collection.load()

# Prepare the query vector
search_vector = model.encode("Artificial intelligence learning methods").tolist()

# Search parameters
search_params = {"metric_type": "L2", "params": {"nprobe": 10}}

# Perform the search
results = collection.search(
    data=[search_vector],
    anns_field="embedding",
    param=search_params,
    limit=2, # Retrieve the 2 nearest results
    expr=None,
    output_fields=["text"]
)

print("Search results:")
for hit in results[0]:
    print(f"Distance (L2): {hit.distance}, Text: {hit.entity.get('text')}")

# Release the collection from memory
collection.release()

In the example above, we saw how to create a semantic search collection, add vectors, and query them using Milvus's core features. Milvus supports various index types such as FLAT, IVF_FLAT, and HNSW, and thanks to its distributed architecture, it can easily manage high volumes of data and query loads.

Best Practices

• Vector Dimension and Quality: The embedding model you use, and thus the vector dimension, directly impacts search accuracy and performance. Choose the most suitable model and vector dimension for your application.
• Index Type Selection: Selecting the correct Faiss or Milvus index type (e.g., IVFFlat, HNSW) is critical based on your data size and accuracy/speed requirements.
• Pre-filtering: Use the filtering capabilities of vector databases (like Milvus's expr parameter) to narrow down semantic searches based on specific metadata. This increases both speed and relevance.
• Hybrid Search: Sometimes a combination of keyword and semantic search yields the best results. You can develop hybrid search capabilities by integrating vector databases with traditional search engines (like Elasticsearch).
• Updates and Deletions: For dynamic datasets, thoroughly understand and effectively use the update and deletion mechanisms of vector databases.

Conclusion

Vector databases are one of the cornerstones of modern artificial intelligence applications. They offer a wide range of tools, from local solutions like Faiss to distributed systems like Milvus, enabling developers to build scalable and intelligent search, recommendation, and data analysis systems. With this practical guide, you've taken the first step to empower your own semantic search capabilities. Continue exploring the power of vector databases in your applications!