Database API Reference

Detailed reference for the Database API

Database API Reference

The Database is the main interface for interacting with EngramDB. This document provides a detailed reference for the Database API.

Creating a Database

Database::in_memory()

Creates a new in-memory database with linear vector search (volatile, for testing and development).

Returns:

• A new Database instance using memory storage and linear vector index

Example:

let mut db = Database::in_memory();

Database::in_memory_with_hnsw()

Creates a new in-memory database with HNSW vector search for faster similarity queries.

Returns:

• A new Database instance using memory storage and HNSW vector index

Example:

let mut db = Database::in_memory_with_hnsw();

Database::file_based(dir)

Creates a file-based database at the specified directory with linear vector search.

Parameters:

• dir: &str or Path - Path to the storage directory

Returns:

• A new Database instance using file storage, or an error if initialization failed

Example:

let mut db = Database::file_based("./my_database")?;

Database::file_based_with_hnsw(dir)

Creates a file-based database at the specified directory with HNSW vector search.

Parameters:

• dir: &str or Path - Path to the storage directory

Returns:

• A new Database instance using file storage and HNSW vector index, or an error if initialization failed

Example:

let mut db = Database::file_based_with_hnsw("./my_database")?;

Database::new(config)

Creates a new database with the given configuration.

Parameters:

• config: DatabaseConfig - Configuration options for the database

Returns:

• A new Database instance, or an error if initialization failed

Example:

// Custom HNSW configuration
let config = DatabaseConfig {
    storage_type: StorageType::Memory,
    storage_path: None,
    cache_size: 100,
    vector_index_config: VectorIndexConfig {
        algorithm: VectorAlgorithm::HNSW,
        hnsw: Some(HnswConfig {
            m: 32,                  // More connections per node
            ef_construction: 200,   // More neighbors during construction
            ef: 50,                 // More candidates during search
            level_multiplier: 1.0,
            max_level: 16,
        }),
    },
};

let mut db = Database::new(config)?;

DatabaseConfig

The DatabaseConfig struct provides configuration options for the database:

pub struct DatabaseConfig {
    pub storage_type: StorageType,
    pub storage_path: Option<String>,
    pub cache_size: usize,
    pub vector_index_config: VectorIndexConfig,
}

Fields:

• storage_type: The type of storage to use (StorageType::Memory, StorageType::MultiFile, or StorageType::SingleFile)
• storage_path: Directory path for file storage (ignored if using memory storage)
• cache_size: Size of the query result cache (0 to disable caching)
• vector_index_config: Configuration for the vector index

StorageType

pub enum StorageType {
    Memory,      // In-memory storage (volatile)
    MultiFile,   // Multi-file storage (one file per node)
    SingleFile,  // Single-file storage (all nodes in one file)
}

VectorIndexConfig

pub struct VectorIndexConfig {
    pub algorithm: VectorAlgorithm,
    pub hnsw: Option<HnswConfig>,
}

Fields:

• algorithm: The vector indexing algorithm to use (VectorAlgorithm::Linear or VectorAlgorithm::HNSW)
• hnsw: Configuration for the HNSW algorithm (if used)

Basic Operations

initialize()

Initializes the database by loading existing memories into the vector index.

Returns:

• Result<()> - Success or an error

Example:

db.initialize()?;

save(node)

Saves a memory node to the database.

Parameters:

• node: &MemoryNode - The memory node to save

Returns:

• Result<Uuid> - The ID of the saved memory node, or an error

Example:

let memory = MemoryNode::new(vec![0.1, 0.2, 0.3, 0.4]);
let memory_id = db.save(&memory)?;

load(id)

Loads a memory node by its ID.

Parameters:

• id: Uuid - The ID of the memory node to load

Returns:

• Result<MemoryNode> - The loaded memory node, or an error if not found

Example:

let memory = db.load(memory_id)?;

delete(id)

Deletes a memory node by its ID.

Parameters:

• id: Uuid - The ID of the memory node to delete

Returns:

• Result<()> - Success or an error

Example:

db.delete(memory_id)?;

list_all()

Lists all memory node IDs in the database.

Returns:

• Result<Vec<Uuid>> - A vector of all memory node IDs, or an error

Example:

let all_ids = db.list_all()?;
println!("Database contains {} memories", all_ids.len());

Vector Similarity Search

search_similar(query_vector, limit, threshold)

Searches for memory nodes with similar vector embeddings.

Parameters:

• query_vector: &[f32] - The query vector
• limit: usize - Maximum number of results to return
• threshold: f32 - Minimum similarity threshold (0.0 to 1.0)

Returns:

• Result<Vec<(Uuid, f32)>> - A vector of (ID, similarity) pairs, sorted by descending similarity

Example:

let query_vector = vec![0.15, 0.25, 0.35, 0.45];
let results = db.search_similar(&query_vector, 5, 0.0)?;

for (memory_id, similarity) in results {
    println!("Memory ID: {}, Similarity: {:.4}", memory_id, similarity);
}

Query Builder

query()

Creates a new query builder for complex queries.

Returns:

• A new QueryBuilder instance

Example:

let query_results = db.query()
    .with_vector(query_vector)
    .with_attribute_filter(category_filter)
    .with_attribute_filter(importance_filter)
    .execute()?;

Connection Management

get_connections(id)

Gets all connections for a memory node.

Parameters:

• id: Uuid - The ID of the memory node

Returns:

• Result<Vec<ConnectionInfo>> - A vector of connection information, or an error

Example:

let connections = db.get_connections(memory_id)?;
for connection in connections {
    println!("Connected to: {}, Type: {}, Strength: {:.2}",
        connection.target_id, connection.type_name, connection.strength);
}

add_connection(source_id, target_id, relationship_type, strength)

Adds a connection between two memory nodes.

Parameters:

• source_id: Uuid - The ID of the source memory node
• target_id: Uuid - The ID of the target memory node
• relationship_type: RelationshipType - The type of relationship
• strength: f32 - The strength of the connection (0.0 to 1.0)

Returns:

• Result<()> - Success or an error

Example:

db.add_connection(
    source_id,
    target_id,
    RelationshipType::Association,
    0.75
)?;

remove_connection(source_id, target_id)

Removes a connection between two memory nodes.

Parameters:

• source_id: Uuid - The ID of the source memory node
• target_id: Uuid - The ID of the target memory node

Returns:

• Result<bool> - True if a connection was removed, false otherwise, or an error

Example:

let removed = db.remove_connection(source_id, target_id)?;
if removed {
    println!("Connection removed");
}

Background Processing

EngramDB includes a background processing system that can perform operations during idle periods:

Enabling Background Processing

// Configure background processing
let background_config = BackgroundTaskConfig {
    idle_threshold: 5.0,            // Seconds of inactivity before considering idle
    max_tokens: Some(10000),        // Limit token usage
    max_cost: Some(0.01),           // Cost limit in dollars
    max_concurrent_tasks: 2,        // Number of concurrent tasks
};

// Enable background processing when creating the database
let config = DatabaseConfig {
    // Other configuration...
    background_processing: Some(background_config),
};

let db = Database::new(config)?;

Scheduling Background Tasks

// Schedule a task to summarize related memories
let task_id = db.schedule_background_task(
    TaskType::Summarize { 
        node_ids: vec![node1_id, node2_id], 
        prompt: Some("Summarize these related memories")
    },
    TaskPriority::Normal
)?;

// Check task status
let status = db.get_task_status(task_id)?;
println!("Task status: {:?}", status);

// Get task result when completed
if status == TaskStatus::Completed {
    let result = db.get_task_result(task_id)?;
    match result {
        TaskResult::Summary { node_id } => {
            // Load the summary node
            let summary = db.load(node_id)?;
            println!("Summary: {:?}", summary);
        },
        _ => println!("Other task result: {:?}", result),
    }
}

Available Task Types

EngramDB supports several types of background tasks:

// Create a summary of related memories
TaskType::Summarize {
    node_ids: vec![node1_id, node2_id],
    prompt: Some("Create a concise summary"),
}

// Infer connections between memory nodes
TaskType::InferConnections {
    node_ids: vec![node1_id, node2_id, node3_id],
    prompt: Some("Find relationships"),
}

// Enrich a memory node with additional context
TaskType::EnrichNode {
    node_id: node_id,
    prompt: Some("Add context about locations"),
}

// Predict and pre-compute likely future queries
TaskType::PredictQueries {
    recent_queries: Some(vec!["What did I learn?".to_string()]),
}

Error Handling

The database operations return a Result type that can contain various error types:

pub enum EngramDbError {
    Storage(String),
    Query(String),
    Vector(String),
    Serialization(String),
    Validation(String),
    Background(String),
    Other(String),
}

Example:

match db.load(non_existent_id) {
    Ok(memory) => println!("Found memory: {:?}", memory),
    Err(EngramDbError::Storage(msg)) => println!("Storage error: {}", msg),
    Err(e) => println!("Other error: {:?}", e),
}