Redis RAG Implementation Plan

Overview

A high-performance Redis-based RAG implementation leveraging LangChain Go's native Redis vector store support. This provides true semantic search capabilities with vector embeddings, superior to both JSON and SQLite FTS5 approaches.

Redis RAG Architecture

Performance Advantages

Vector Search vs Traditional Search

Feature

JSON (Current)

SQLite FTS5

Redis Vector

Search Type

Substring match

Keyword search

Semantic similarity

Understanding

None

Basic

Contextual meaning

Query Flexibility

Exact terms

Boolean operators

Natural language

Multilingual

Limited

Yes (embeddings)

Synonym Support

Manual

Automatic

Performance

O(n) linear

O(log n)

O(log n) + semantic

Real-World Benefits

# Traditional search limitations:
Query: "profit margins declining"
JSON/SQLite: Must contain exact words "profit", "margins", "declining"

# Redis vector search power:
Query: "profit margins declining" 
Redis: Finds documents about:
- "revenue decreasing"
- "profitability challenges" 
- "financial performance down"
- "earnings under pressure"

LangChain Go Integration

Native Redis Vector Store

LangChain Go provides first-class Redis support via vectorstores/redisvector:

import (
    "github.com/tmc/langchaingo/vectorstores/redisvector"
    "github.com/tmc/langchaingo/embeddings/openai"
    "github.com/redis/go-redis/v9"
)

// Native LangChain Go Redis vector store
type RedisRAG struct {
    vectorStore  *redisvector.Store
    embeddings   *openai.EmbeddingModel
    redisClient  *redis.Client
}

Key Advantages of LangChain Integration

✅ Native Support - Official Redis vector store implementation
✅ Embedding Models - Built-in OpenAI, Hugging Face, local model support
✅ Similarity Search - Cosine, Euclidean, dot product similarity
✅ Metadata Filtering - Advanced filtering capabilities
✅ Async Operations - Non-blocking vector operations
✅ Production Ready - Battle-tested in enterprise environments

Implementation Architecture

Core Components

// internal/rag/redis.go
package rag

import (
    "context"
    "github.com/tmc/langchaingo/vectorstores/redisvector"
    "github.com/tmc/langchaingo/embeddings/openai"
    "github.com/tmc/langchaingo/schema"
    "github.com/redis/go-redis/v9"
)

type RedisRAG struct {
    vectorStore   *redisvector.Store
    embeddings    *openai.EmbeddingModel
    redisClient   *redis.Client
    indexName     string
}

type RedisConfig struct {
    Addr         string `json:"redis_addr"`
    Password     string `json:"redis_password"`
    DB           int    `json:"redis_db"`
    IndexName    string `json:"redis_index"`
    OpenAIKey    string `json:"openai_api_key"`
    EmbedModel   string `json:"embedding_model"`
    ChunkSize    int    `json:"chunk_size"`
    ChunkOverlap int    `json:"chunk_overlap"`
}

Document Ingestion Pipeline

func (r *RedisRAG) IngestPDF(filePath string) error {
    // 1. Extract text using existing LangChain PDF loader
    documents, err := r.loadPDF(filePath)
    if err != nil {
        return err
    }

    // 2. Chunk documents
    chunker := textsplitter.NewRecursiveCharacter()
    chunker.ChunkSize = r.config.ChunkSize
    chunker.ChunkOverlap = r.config.ChunkOverlap
    
    chunks, err := chunker.SplitDocuments(documents)
    if err != nil {
        return err
    }

    // 3. Generate embeddings and store in Redis
    ctx := context.Background()
    _, err = r.vectorStore.AddDocuments(ctx, chunks)
    return err
}

Semantic Search Implementation

func (r *RedisRAG) Search(query string, limit int) ([]Document, error) {
    ctx := context.Background()
    
    // Perform semantic similarity search
    results, err := r.vectorStore.SimilaritySearch(
        ctx, 
        query,
        limit,
        redisvector.WithScoreThreshold(0.7), // Similarity threshold
        redisvector.WithFilter(map[string]interface{}{
            // Optional metadata filtering
            "file_type": "pdf",
        }),
    )
    
    if err != nil {
        return nil, err
    }
    
    // Convert to our Document format
    documents := make([]Document, len(results))
    for i, result := range results {
        documents[i] = Document{
            Content:  result.PageContent,
            Source:   result.Metadata["source"].(string),
            Score:    result.Score,
            Metadata: result.Metadata,
        }
    }
    
    return documents, nil
}

Redis Configuration & Setup

Redis Stack Installation

# Option 1: Docker (recommended for development)
docker run -d --name redis-stack \
  -p 6379:6379 \
  -p 8001:8001 \
  redis/redis-stack:latest

# Option 2: Redis Cloud (recommended for production)
# Sign up at https://redis.com/try-free/

# Option 3: Local installation
brew install redis-stack
redis-stack-server

Vector Index Configuration

// Redis vector index configuration
func (r *RedisRAG) createIndex() error {
    indexDef := redisvector.IndexDefinition{
        IndexName: r.indexName,
        Schema: []redisvector.FieldSchema{
            {
                FieldName: "content_vector",
                FieldType: "VECTOR",
                VectorArgs: redisvector.VectorArgs{
                    Algorithm: "FLAT", // or "HNSW" for large datasets
                    Attributes: map[string]interface{}{
                        "TYPE":         "FLOAT32",
                        "DIM":          1536, // OpenAI ada-002 dimensions
                        "DISTANCE_METRIC": "COSINE",
                    },
                },
            },
            {
                FieldName: "content",
                FieldType: "TEXT",
            },
            {
                FieldName: "source",
                FieldType: "TEXT",
            },
            {
                FieldName: "file_type",
                FieldType: "TAG",
            },
        },
    }
    
    return r.vectorStore.CreateIndex(context.Background(), indexDef)
}

Configuration Integration

Enhanced Config Structure

{
  "llm_provider": "openai",
  "openai_api_key": "sk-...",
  "rag_enabled": true,
  "rag_provider": "redis",
  "rag_config": {
    "redis_addr": "localhost:6379",
    "redis_password": "",
    "redis_db": 0,
    "redis_index": "documents",
    "openai_api_key": "sk-...",
    "embedding_model": "text-embedding-ada-002",
    "chunk_size": 1000,
    "chunk_overlap": 200,
    "similarity_threshold": 0.7,
    "max_results": 5
  },
  "custom_prompt": "Search the knowledge base first using rag_search before responding...",
  "slack_bot_token": "xoxb-...",
  "slack_app_token": "xapp-..."
}

CLI Commands

Enhanced CLI Support

# Ingest documents with Redis backend
./slack-mcp-client --rag-ingest ./docs --rag-provider redis

# Semantic search
./slack-mcp-client --rag-search "revenue declining profitability" --rag-provider redis

# Index management
./slack-mcp-client --rag-reindex --rag-provider redis
./slack-mcp-client --rag-stats --rag-provider redis

# Migration from JSON/SQLite
./slack-mcp-client --rag-migrate --from json --to redis

Advanced Features

Hybrid Search (Best of Both Worlds)

// Combine vector similarity with keyword filtering
func (r *RedisRAG) HybridSearch(query string, filters map[string]interface{}) ([]Document, error) {
    // 1. Vector similarity search
    vectorResults, err := r.vectorStore.SimilaritySearch(
        context.Background(),
        query,
        20, // Get more candidates
        redisvector.WithFilter(filters),
    )
    
    // 2. Keyword re-ranking for precision
    rerankedResults := r.rerankByKeywords(vectorResults, query)
    
    // 3. Return top N results
    return rerankedResults[:min(len(rerankedResults), 5)], nil
}

// Support for different content types
type ContentType string

const (
    ContentTypePDF        ContentType = "pdf"
    ContentTypeText       ContentType = "text"
    ContentTypeMarkdown   ContentType = "markdown"
    ContentTypeSpreadsheet ContentType = "spreadsheet"
)

func (r *RedisRAG) IngestWithType(filePath string, contentType ContentType) error {
    switch contentType {
    case ContentTypePDF:
        return r.ingestPDF(filePath)
    case ContentTypeSpreadsheet:
        return r.ingestSpreadsheet(filePath)
    case ContentTypeMarkdown:
        return r.ingestMarkdown(filePath)
    default:
        return r.ingestText(filePath)
    }
}

Real-Time Updates

// File watcher for automatic re-ingestion
func (r *RedisRAG) StartFileWatcher(watchDir string) error {
    watcher, err := fsnotify.NewWatcher()
    if err != nil {
        return err
    }
    
    go func() {
        for {
            select {
            case event := <-watcher.Events:
                if event.Op&fsnotify.Write == fsnotify.Write {
                    r.IngestFile(event.Name)
                }
            }
        }
    }()
    
    return watcher.Add(watchDir)
}

Migration Strategy

Phase 1: Parallel Implementation

Implement Redis RAG alongside existing JSON system
Use rag_provider config to switch between backends
Maintain backward compatibility

Phase 2: Performance Testing

A/B test search quality between JSON and Redis
Measure query response times and memory usage
Optimize Redis index configuration

Phase 3: Migration Tools

Automated migration from JSON to Redis
Bulk import with progress tracking
Verification tools to ensure data integrity

Phase 4: Production Deployment

Switch default to Redis backend
Monitor performance and error rates
Gradual rollout with fallback options

Performance Benchmarks

Expected Performance Gains

Metric

JSON (Current)

Redis Vector

Search Quality

Basic substring

Semantic understanding

Query Speed

50-200ms

5-20ms

Memory Usage

350KB+ (linear)

<10MB (indexed)

Scalability

~1K documents

100K+ documents

Multi-language

Yes

Synonym Support

Automatic

Real-World Search Examples

# Query: "How is our European market performing?"
# JSON: Looks for "European" + "market" + "performing" (exact)
# Redis: Understands context, finds:
#   - "EMEA sales declining"
#   - "EU revenue challenges" 
#   - "Continental performance metrics"
#   - "European demand analysis"

Deployment Options

Development

Docker Compose: Redis Stack + application
Local Redis: Simple setup for testing
Mock Mode: In-memory vector store for unit tests

Production

Redis Cloud: Managed service with high availability
Redis Cluster: Self-hosted with sharding
Redis Sentinel: High availability configuration

Cost Analysis

Redis Cloud Pricing (Production Ready)

Free Tier: 30MB RAM, perfect for testing
Small Production: $5-15/month for 100MB-1GB
Enterprise: $50-200/month for 10GB+ with clustering

Infrastructure Costs

Self-hosted: EC2 t3.medium ($25/month) + storage
Managed: Redis Cloud standard plans
Hybrid: Local development + cloud production

Security Considerations

Data Protection

TLS Encryption: All Redis communications encrypted
Authentication: Redis AUTH + ACL controls
Network Security: VPC isolation in production
API Keys: Secure OpenAI key management

Compliance

Data Residency: Choose Redis regions for compliance
Audit Logging: Track all document access and searches
Data Retention: Configurable TTL for sensitive documents

Next Steps

Quick Start (Immediate)

Setup Redis Stack: Local Docker installation
Configure OpenAI: API key for embeddings
Implement Core: internal/rag/redis.go with LangChain Go
Test Migration: Convert existing knowledge base

Production Readiness (Next Phase)

Redis Cloud: Production deployment setup
Monitoring: Performance metrics and alerting
Backup Strategy: Vector index backup and restore
Load Testing: Simulate production query volumes

Advanced Features (Future)

Hybrid Search: Combine vector + keyword search
Multi-Modal: Images, tables, structured data
Real-Time: File watching and auto-updates
Analytics: Search patterns and usage metrics

Comparison Summary

Approach

Implementation Effort

Search Quality

Performance

Scalability

JSON (Current)

✅ Complete

⚠️ Basic

⚠️ Slow

❌ Limited

SQLite FTS5

🔨 Moderate

✅ Good

✅ Fast

✅ Medium