> ## Documentation Index
> Fetch the complete documentation index at: https://engramdb.mintlify.site/llms.txt
> Use this file to discover all available pages before exploring further.

# Vector Index API Reference

> Detailed reference for the Vector Index API

# Vector Index API Reference

The Vector Index is responsible for storing and searching vector embeddings in EngramDB. This document provides a detailed reference for the Vector Index API.

EngramDB supports two primary vector index implementations:

1. **Linear Index (`VectorIndex`)**: Simple brute-force search for small to medium datasets
2. **HNSW Index (`HnswIndex`)**: Hierarchical Navigable Small World graph for fast approximate search in large datasets

Both implementations share a common interface through the `VectorSearchIndex` trait, allowing them to be used interchangeably.

## VectorSearchIndex Trait

The `VectorSearchIndex` trait defines the common interface for all vector index implementations.

### Common Methods

All vector indices implement these methods:

* `add(node)`: Adds a memory node to the index
* `remove(id)`: Removes a memory node from the index
* `update(node)`: Updates a memory node in the index
* `search(query, limit, threshold)`: Searches for similar vectors
* `len()`: Returns the number of vectors in the index
* `is_empty()`: Checks if the index is empty
* `get(id)`: Gets a reference to a vector by its ID

## VectorIndex

The `VectorIndex` class provides a simple linear search implementation for vector embeddings.

### Creating a Vector Index

#### `VectorIndex::new()`

Creates a new empty vector index.

**Returns:**

* A new `VectorIndex` instance

**Example:**

```rust theme={null}
let mut vector_index = VectorIndex::new();
```

### Managing Vectors

#### `add(node)`

Adds a memory node to the index.

**Parameters:**

* `node`: `&MemoryNode` - The memory node to add

**Returns:**

* `Result<()>` - Success or an error

**Example:**

```rust theme={null}
let memory = MemoryNode::new(vec![0.1, 0.2, 0.3, 0.4]);
vector_index.add(&memory)?;
```

#### `remove(id)`

Removes a memory node from the index.

**Parameters:**

* `id`: `Uuid` - The UUID of the memory node to remove

**Returns:**

* `Result<()>` - Success or an error

**Example:**

```rust theme={null}
vector_index.remove(memory_id)?;
```

#### `update(node)`

Updates a memory node in the index.

**Parameters:**

* `node`: `&MemoryNode` - The memory node to update

**Returns:**

* `Result<()>` - Success or an error

**Example:**

```rust theme={null}
memory.set_embeddings(vec![0.2, 0.3, 0.4, 0.5]);
vector_index.update(&memory)?;
```

### Searching

#### `search(query, limit, threshold)`

Performs a similarity search to find the most similar vectors.

**Parameters:**

* `query`: `&[f32]` - The query vector
* `limit`: `usize` - Maximum number of results to return
* `threshold`: `f32` - Minimum similarity threshold (0.0 to 1.0)

**Returns:**

* `Result<Vec<(Uuid, f32)>>` - A vector of (UUID, similarity) pairs, sorted by descending similarity

**Example:**

```rust theme={null}
let query_vector = vec![0.15, 0.25, 0.35, 0.45];
let results = vector_index.search(&query_vector, 5, 0.0)?;

for (memory_id, similarity) in results {
    println!("Memory ID: {}, Similarity: {:.4}", memory_id, similarity);
}
```

### Utility Methods

#### `len()`

Returns the number of vectors in the index.

**Returns:**

* `usize` - The number of vectors

**Example:**

```rust theme={null}
let count = vector_index.len();
println!("Index contains {} vectors", count);
```

#### `is_empty()`

Checks if the index is empty.

**Returns:**

* `bool` - True if the index is empty, false otherwise

**Example:**

```rust theme={null}
if vector_index.is_empty() {
    println!("Index is empty");
}
```

#### `get(id)`

Gets a reference to a vector by its ID.

**Parameters:**

* `id`: `Uuid` - The UUID of the memory node

**Returns:**

* `Option<&Vec<f32>>` - A reference to the vector if found, otherwise None

**Example:**

```rust theme={null}
if let Some(vector) = vector_index.get(memory_id) {
    println!("Vector: {:?}", vector);
}
```

## Similarity Functions

EngramDB provides several similarity functions for comparing vectors.

### `cosine_similarity(a, b)`

Computes the cosine similarity between two vectors.

**Parameters:**

* `a`: `&[f32]` - First vector
* `b`: `&[f32]` - Second vector

**Returns:**

* `Option<f32>` - The cosine similarity value between the two vectors (between -1 and 1), or None if either vector is empty

**Example:**

```rust theme={null}
let vec1 = vec![0.1, 0.2, 0.3];
let vec2 = vec![0.2, 0.3, 0.4];

if let Some(similarity) = cosine_similarity(&vec1, &vec2) {
    println!("Cosine similarity: {:.4}", similarity);
}
```

### `dot_product(a, b)`

Computes the dot product between two vectors.

**Parameters:**

* `a`: `&[f32]` - First vector
* `b`: `&[f32]` - Second vector

**Returns:**

* `f32` - The dot product value, or 0.0 if the vectors have different lengths

**Example:**

```rust theme={null}
let vec1 = vec![0.1, 0.2, 0.3];
let vec2 = vec![0.2, 0.3, 0.4];

let dot = dot_product(&vec1, &vec2);
println!("Dot product: {:.4}", dot);
```

### `euclidean_distance(a, b)`

Computes the Euclidean distance between two vectors.

**Parameters:**

* `a`: `&[f32]` - First vector
* `b`: `&[f32]` - Second vector

**Returns:**

* `Option<f32>` - The Euclidean distance between the two vectors, or None if the vectors have different lengths

**Example:**

```rust theme={null}
let vec1 = vec![0.1, 0.2, 0.3];
let vec2 = vec![0.2, 0.3, 0.4];

if let Some(distance) = euclidean_distance(&vec1, &vec2) {
    println!("Euclidean distance: {:.4}", distance);
}
```

## HnswIndex

The `HnswIndex` class provides a Hierarchical Navigable Small World (HNSW) graph implementation for fast approximate nearest neighbor search. This is particularly useful for large datasets where linear search becomes prohibitively expensive.

### Creating an HNSW Index

#### `HnswIndex::new()`

Creates a new HNSW index with default parameters.

**Returns:**

* A new `HnswIndex` instance

**Example:**

```rust theme={null}
let mut hnsw_index = HnswIndex::new();
```

#### `HnswIndex::with_config(config)`

Creates a new HNSW index with custom parameters.

**Parameters:**

* `config`: `HnswConfig` - Configuration parameters for the HNSW algorithm

**Returns:**

* A new `HnswIndex` instance with the specified configuration

**Example:**

```rust theme={null}
let config = HnswConfig {
    m: 16,                  // Maximum connections per node per layer
    ef_construction: 100,   // Size of dynamic candidate list during construction
    ef: 10,                 // Size of dynamic candidate list during search
    level_multiplier: 1.0,  // Base level for multi-layer construction
    max_level: 16,          // Maximum level in the hierarchy
};
let mut hnsw_index = HnswIndex::with_config(config);
```

### HNSW Configuration

The `HnswConfig` struct allows you to configure the HNSW algorithm:

* `m`: Maximum number of connections per node per layer (default: 16)
* `ef_construction`: Size of the dynamic candidate list during index construction (default: 100)
* `ef`: Size of the dynamic candidate list during search (default: 10)
* `level_multiplier`: Base level for the multi-layer construction (default: 1.0/ln(2))
* `max_level`: Maximum level in the hierarchy (default: 16)

Increasing `m` and `ef_construction` improves search quality at the cost of increased memory usage and indexing time. Increasing `ef` improves search quality at the cost of search speed.

## Using Vector Indices with Database

EngramDB's `Database` class supports different vector index algorithms through its configuration.

### Creating a Database with Linear Search (Default)

```rust theme={null}
// Create an in-memory database with the default linear search index
let db = Database::in_memory();
```

### Creating a Database with HNSW Index

```rust theme={null}
// Create an in-memory database with the HNSW index
let db = Database::in_memory_with_hnsw();

// Alternatively, create a file-based database with HNSW
let db = Database::file_based_with_hnsw("./my_database")?;
```

### Custom Vector Index Configuration

```rust theme={null}
// Create a database with custom HNSW parameters
let config = DatabaseConfig {
    storage_type: StorageType::Memory,
    storage_path: None,
    cache_size: 100,
    vector_index_config: VectorIndexConfig {
        algorithm: VectorAlgorithm::HNSW,
        hnsw: Some(HnswConfig {
            m: 32,                  // More connections per node
            ef_construction: 200,   // More neighbors during construction
            ef: 50,                 // More candidates during search
            level_multiplier: 1.0,
            max_level: 16,
        }),
    },
};
let db = Database::new(config)?;
```

## Performance Comparison

The HNSW index provides significant performance improvements over linear search, especially for larger datasets:

| Dataset Size | Linear Search (ms) | HNSW Search (ms) | Speedup |
| ------------ | ------------------ | ---------------- | ------- |
| 1,000        | 5                  | 1                | 5×      |
| 10,000       | 50                 | 2                | 25×     |
| 100,000      | 500                | 5                | 100×    |
| 1,000,000    | 5,000              | 10               | 500×    |

These are approximate values and will vary depending on vector dimensions, hardware, and specific HNSW parameters.

## Example: Vector Search Workflow

Here's a complete example of how to use vector search:

```rust theme={null}
// Create a database with HNSW index for faster search
let mut db = Database::in_memory_with_hnsw();

// Create and add memory nodes
let memory1 = MemoryNode::new(vec![1.0, 0.0, 0.0]); // X-axis
let memory2 = MemoryNode::new(vec![0.7, 0.7, 0.0]); // 45 degrees in XY plane
let memory3 = MemoryNode::new(vec![0.0, 1.0, 0.0]); // Y-axis
let memory4 = MemoryNode::new(vec![0.0, 0.0, 1.0]); // Z-axis

let id1 = db.save(&memory1)?;
let id2 = db.save(&memory2)?;
let id3 = db.save(&memory3)?;
let id4 = db.save(&memory4)?;

// Search for something closer to the X-axis
let query = vec![0.9, 0.1, 0.0];
let results = db.search_similar(&query, 2, 0.0)?;

// Process results
for (id, similarity) in results {
    let memory = db.load(id)?;
    println!("Memory ID: {}, Similarity: {:.4}", id, similarity);
}
```