Junior Developer Implementation Guide: Issue #369

RAG Context Compression - Comprehensive Unit Testing

Overview

This guide covers comprehensive unit testing for RAG context compression strategies. Context compression reduces retrieved document size while preserving relevance, enabling more documents to fit within LLM context windows. The implementations exist - your task is thorough testing.

For Beginners: What Is Context Compression?

The Problem: Token Limits

Scenario: You retrieve 20 documents (each 500 words) for a query.

Total: 10,000 words
LLM limit: 4,000 tokens (~3,000 words)
Problem: Can't fit all retrieved documents!

Solutions:

Truncate: Only use first 3,000 words (might lose important info)
Compress: Reduce each document while keeping relevant parts (smart!)

Real-World Analogy

Without Compression:
You're reading a textbook chapter to answer one question. You read every word, every example, every footnote - even parts unrelated to your question.

With Compression:

LLM-based: AI reads chapter, extracts only sentences relevant to your question
Summarization: AI creates condensed summary keeping key points
Selective: AI scores sentences, keeps only high-relevance ones

Compression Strategies in AiDotNet

1. LLMContextCompressor - Uses LLM to extract relevant portions

Document: "The solar system has 8 planets. Mercury is closest to the sun. Jupiter is the largest..."
Query: "How many planets?"
Compressed: "The solar system has 8 planets."

2. DocumentSummarizer - Creates concise summaries

Document: [10 paragraphs about photosynthesis]
Compressed: "Photosynthesis converts sunlight, water, and CO2 into glucose and oxygen in plant cells."

3. SelectiveContextCompressor - Scores and filters sentences

Document: [Mix of relevant and irrelevant sentences]
Query: "machine learning"
Compressed: [Only sentences scoring >0.7 relevance to ML]

4. AutoCompressor - Automatically chooses best strategy

If (long documents): Use Summarizer
Elif (need precision): Use LLM Compressor
Else: Use Selective filter

What EXISTS in the Codebase

Core Infrastructure

Interface: IContextCompressor<T>

Compress(documents, query, options)
Returns compressed documents preserving relevance

Base Class: ContextCompressorBase<T>

Validation logic
Common compression utilities
Integration with INumericOperations<T>

Implementations

1. LLMContextCompressor (LLMContextCompressor.cs)

Uses LLM to extract relevant parts
Query-guided extraction
Preserves semantic meaning

2. DocumentSummarizer (DocumentSummarizer.cs)

Creates concise summaries
Configurable compression ratio
Maintains key information

3. SelectiveContextCompressor (SelectiveContextCompressor.cs)

Sentence-level scoring
Relevance threshold filtering
Embedding-based relevance

4. AutoCompressor (AutoCompressor.cs)

Strategy selection logic
Adaptive compression
Fallback mechanisms

What's MISSING (This Issue)

Comprehensive tests for all compression strategies:

Test Coverage Gaps

Basic Functionality:

Constructor validation
Compress method with valid inputs
Query relevance preservation
Document structure preservation

Compression Quality:

Verify output is shorter than input
Verify relevant information is retained
Verify irrelevant information is removed
Compression ratio targets

Edge Cases:

Empty document list
Documents already smaller than target
Very long documents (100KB+)
Documents with no query-relevant content
Special characters, Unicode, code blocks

Integration:

Compression with different embedding models
Compression with different LLMs
Chaining multiple compressors

Step-by-Step Implementation

Step 1: Base Compressor Tests

// File: tests/RetrievalAugmentedGeneration/ContextCompression/ContextCompressorTestBase.cs

using Xunit;
using AiDotNet.RetrievalAugmentedGeneration.ContextCompression;
using AiDotNet.RetrievalAugmentedGeneration.Models;

namespace AiDotNet.Tests.RetrievalAugmentedGeneration.ContextCompression;

/// <summary>
/// Base class for context compressor tests with shared utilities.
/// </summary>
public abstract class ContextCompressorTestBase
{
    /// <summary>
    /// Creates sample documents for testing.
    /// </summary>
    protected List<Document<double>> CreateSampleDocuments()
    {
        return new List<Document<double>>
        {
            new Document<double>
            {
                Id = "doc1",
                Content = "Machine learning is a subset of artificial intelligence that enables " +
                         "computers to learn from data without being explicitly programmed. " +
                         "It uses algorithms to identify patterns and make decisions. " +
                         "Deep learning is a subset of machine learning using neural networks.",
                Metadata = new Dictionary<string, object> { ["topic"] = "ML" }
            },
            new Document<double>
            {
                Id = "doc2",
                Content = "Natural language processing (NLP) is a field of AI that enables computers " +
                         "to understand and generate human language. " +
                         "It powers chatbots, translation services, and sentiment analysis. " +
                         "NLP combines linguistics with machine learning techniques.",
                Metadata = new Dictionary<string, object> { ["topic"] = "NLP" }
            },
            new Document<double>
            {
                Id = "doc3",
                Content = "Computer vision enables machines to interpret and understand visual information. " +
                         "It uses deep learning to recognize objects in images and videos. " +
                         "Applications include self-driving cars and medical imaging. " +
                         "Convolutional neural networks are commonly used in computer vision.",
                Metadata = new Dictionary<string, object> { ["topic"] = "CV" }
            }
        };
    }

    /// <summary>
    /// Verifies compressed documents are shorter than originals.
    /// </summary>
    protected void AssertCompressed(List<Document<double>> original, List<Document<double>> compressed)
    {
        Assert.Equal(original.Count, compressed.Count);

        for (int i = 0; i < original.Count; i++)
        {
            Assert.True(compressed[i].Content.Length <= original[i].Content.Length,
                $"Compressed doc {i} ({compressed[i].Content.Length} chars) " +
                $"should be <= original ({original[i].Content.Length} chars)");
        }
    }

    /// <summary>
    /// Verifies relevant keywords are preserved in compressed documents.
    /// </summary>
    protected void AssertRelevancePreserved(string query, List<Document<double>> compressed)
    {
        var queryKeywords = query.ToLower().Split(' ', StringSplitOptions.RemoveEmptyEntries);

        foreach (var doc in compressed)
        {
            var docLower = doc.Content.ToLower();
            bool hasRelevantKeyword = queryKeywords.Any(keyword => docLower.Contains(keyword));

            Assert.True(hasRelevantKeyword || doc.Content.Length == 0,
                $"Document '{doc.Id}' should contain at least one query keyword");
        }
    }

    /// <summary>
    /// Calculates compression ratio (0 = fully compressed, 1 = no compression).
    /// </summary>
    protected double CalculateCompressionRatio(List<Document<double>> original, List<Document<double>> compressed)
    {
        int originalLength = original.Sum(d => d.Content.Length);
        int compressedLength = compressed.Sum(d => d.Content.Length);

        if (originalLength == 0) return 1.0;
        return (double)compressedLength / originalLength;
    }
}

Step 2: LLMContextCompressor Tests

// File: tests/RetrievalAugmentedGeneration/ContextCompression/LLMContextCompressorTests.cs

using Xunit;
using AiDotNet.RetrievalAugmentedGeneration.ContextCompression;
using AiDotNet.RetrievalAugmentedGeneration.Generators;

namespace AiDotNet.Tests.RetrievalAugmentedGeneration.ContextCompression;

/// <summary>
/// Tests for LLM-based context compression.
/// </summary>
public class LLMContextCompressorTests : ContextCompressorTestBase
{
    [Fact]
    public void Constructor_WithValidGenerator_Initializes()
    {
        // Arrange
        var generator = new StubGenerator<double>();

        // Act
        var compressor = new LLMContextCompressor<double>(generator);

        // Assert
        Assert.NotNull(compressor);
    }

    [Fact]
    public void Constructor_WithNullGenerator_ThrowsArgumentNullException()
    {
        // Act & Assert
        Assert.Throws<ArgumentNullException>(() =>
            new LLMContextCompressor<double>(generator: null));
    }

    [Fact]
    public void Compress_WithValidDocuments_ReducesContentLength()
    {
        // Arrange
        var generator = new StubGenerator<double>();
        generator.SetResponse("Machine learning enables computers to learn from data.");

        var compressor = new LLMContextCompressor<double>(generator);
        var documents = CreateSampleDocuments();
        var query = "What is machine learning?";

        // Act
        var compressed = compressor.Compress(documents, query);

        // Assert
        AssertCompressed(documents, compressed);
    }

    [Fact]
    public void Compress_WithQueryRelevantContent_PreservesRelevantParts()
    {
        // Arrange
        var generator = new StubGenerator<double>();
        generator.SetResponse("Machine learning is a subset of AI.");

        var compressor = new LLMContextCompressor<double>(generator);
        var documents = CreateSampleDocuments();
        var query = "machine learning";

        // Act
        var compressed = compressor.Compress(documents, query);

        // Assert
        AssertRelevancePreserved(query, compressed);

        // Verify compressed content mentions machine learning
        Assert.Contains("machine learning",
            compressed[0].Content.ToLower());
    }

    [Fact]
    public void Compress_WithEmptyDocuments_ReturnsEmptyList()
    {
        // Arrange
        var generator = new StubGenerator<double>();
        var compressor = new LLMContextCompressor<double>(generator);
        var documents = new List<Document<double>>();

        // Act
        var compressed = compressor.Compress(documents, "test query");

        // Assert
        Assert.Empty(compressed);
    }

    [Fact]
    public void Compress_WithNullDocuments_ThrowsArgumentNullException()
    {
        // Arrange
        var generator = new StubGenerator<double>();
        var compressor = new LLMContextCompressor<double>(generator);

        // Act & Assert
        Assert.Throws<ArgumentNullException>(() =>
            compressor.Compress(documents: null, "test query"));
    }

    [Fact]
    public void Compress_WithNullOrEmptyQuery_ThrowsArgumentException()
    {
        // Arrange
        var generator = new StubGenerator<double>();
        var compressor = new LLMContextCompressor<double>(generator);
        var documents = CreateSampleDocuments();

        // Act & Assert
        Assert.Throws<ArgumentException>(() =>
            compressor.Compress(documents, query: null));

        Assert.Throws<ArgumentException>(() =>
            compressor.Compress(documents, query: ""));

        Assert.Throws<ArgumentException>(() =>
            compressor.Compress(documents, query: "   "));
    }

    [Fact]
    public void Compress_WithLargeDocuments_HandlesEfficiently()
    {
        // Arrange
        var generator = new StubGenerator<double>();
        generator.SetResponse("Compressed content.");

        var compressor = new LLMContextCompressor<double>(generator);

        // Create very large document
        var largeDoc = new Document<double>
        {
            Id = "large1",
            Content = string.Join(" ", Enumerable.Repeat("Some content.", 1000)),  // ~13KB
            Metadata = new Dictionary<string, object>()
        };

        var documents = new List<Document<double>> { largeDoc };

        // Act
        var stopwatch = System.Diagnostics.Stopwatch.StartNew();
        var compressed = compressor.Compress(documents, "test query");
        stopwatch.Stop();

        // Assert
        Assert.True(stopwatch.ElapsedMilliseconds < 5000,
            $"Compression took {stopwatch.ElapsedMilliseconds}ms (should be < 5000ms)");

        AssertCompressed(documents, compressed);
    }

    [Fact]
    public void Compress_WithCompressionRatioOption_RespectsTarget()
    {
        // Arrange
        var generator = new StubGenerator<double>();
        generator.SetResponse("Compressed.");

        var compressor = new LLMContextCompressor<double>(generator);
        var documents = CreateSampleDocuments();

        var options = new Dictionary<string, object>
        {
            ["compressionRatio"] = 0.5  // Target 50% compression
        };

        // Act
        var compressed = compressor.Compress(documents, "test query", options);

        // Assert
        var ratio = CalculateCompressionRatio(documents, compressed);
        Assert.InRange(ratio, 0.3, 0.7);  // Allow some variance
    }

    [Fact]
    public void Compress_PreservesDocumentMetadata()
    {
        // Arrange
        var generator = new StubGenerator<double>();
        generator.SetResponse("Compressed content.");

        var compressor = new LLMContextCompressor<double>(generator);
        var documents = CreateSampleDocuments();

        // Act
        var compressed = compressor.Compress(documents, "test query");

        // Assert
        for (int i = 0; i < documents.Count; i++)
        {
            Assert.Equal(documents[i].Id, compressed[i].Id);
            Assert.Equal(documents[i].Metadata["topic"], compressed[i].Metadata["topic"]);
        }
    }

    [Fact]
    public void Compress_WithUnicodeContent_PreservesCharacters()
    {
        // Arrange
        var generator = new StubGenerator<double>();
        generator.SetResponse("René Descartes était philosophe.");

        var compressor = new LLMContextCompressor<double>(generator);

        var doc = new Document<double>
        {
            Id = "unicode1",
            Content = "René Descartes était un philosophe français très influent. " +
                     "Il a écrit 'Je pense, donc je suis' dans ses Méditations métaphysiques.",
            Metadata = new Dictionary<string, object>()
        };

        var documents = new List<Document<double>> { doc };

        // Act
        var compressed = compressor.Compress(documents, "Descartes");

        // Assert
        Assert.Contains("é", compressed[0].Content);
    }
}

Step 3: DocumentSummarizer Tests

// File: tests/RetrievalAugmentedGeneration/ContextCompression/DocumentSummarizerTests.cs

using Xunit;
using AiDotNet.RetrievalAugmentedGeneration.ContextCompression;
using AiDotNet.RetrievalAugmentedGeneration.Generators;

namespace AiDotNet.Tests.RetrievalAugmentedGeneration.ContextCompression;

/// <summary>
/// Tests for document summarization-based compression.
/// </summary>
public class DocumentSummarizerTests : ContextCompressorTestBase
{
    [Fact]
    public void Constructor_WithValidGenerator_Initializes()
    {
        // Arrange
        var generator = new StubGenerator<double>();

        // Act
        var summarizer = new DocumentSummarizer<double>(generator, summaryRatio: 0.3);

        // Assert
        Assert.NotNull(summarizer);
    }

    [Fact]
    public void Compress_CreatesConcisSummaries()
    {
        // Arrange
        var generator = new StubGenerator<double>();
        generator.SetResponse("ML is AI subset enabling computers to learn from data.");

        var summarizer = new DocumentSummarizer<double>(generator, summaryRatio: 0.3);
        var documents = CreateSampleDocuments();

        // Act
        var compressed = summarizer.Compress(documents, "machine learning");

        // Assert
        AssertCompressed(documents, compressed);

        // Verify summarization (should be significantly shorter)
        var ratio = CalculateCompressionRatio(documents, compressed);
        Assert.InRange(ratio, 0.1, 0.5);  // 10-50% of original
    }

    [Fact]
    public void Compress_WithDifferentRatios_ProducesDifferentLengths()
    {
        // Arrange
        var generator = new StubGenerator<double>();
        generator.SetResponse("Summary.");

        var aggressiveSummarizer = new DocumentSummarizer<double>(generator, summaryRatio: 0.2);
        var conservativeSummarizer = new DocumentSummarizer<double>(generator, summaryRatio: 0.6);

        var documents = CreateSampleDocuments();

        // Act
        var aggressive = aggressiveSummarizer.Compress(documents, "test");
        var conservative = conservativeSummarizer.Compress(documents, "test");

        // Assert
        var aggressiveRatio = CalculateCompressionRatio(documents, aggressive);
        var conservativeRatio = CalculateCompressionRatio(documents, conservative);

        Assert.True(aggressiveRatio < conservativeRatio,
            $"Aggressive ({aggressiveRatio:F2}) should be < Conservative ({conservativeRatio:F2})");
    }

    [Fact]
    public void Compress_MaintainsKeyInformation()
    {
        // Arrange
        var generator = new StubGenerator<double>();
        generator.SetResponse("Machine learning enables computers to learn from data using algorithms.");

        var summarizer = new DocumentSummarizer<double>(generator);
        var documents = CreateSampleDocuments();

        // Act
        var compressed = summarizer.Compress(documents, "machine learning");

        // Assert
        // Key terms should be preserved in summary
        var summary = compressed[0].Content.ToLower();
        Assert.Contains("machine learning", summary);
        Assert.Contains("learn", summary);
    }
}

Step 4: SelectiveContextCompressor Tests

// File: tests/RetrievalAugmentedGeneration/ContextCompression/SelectiveContextCompressorTests.cs

using Xunit;
using AiDotNet.RetrievalAugmentedGeneration.ContextCompression;
using AiDotNet.RetrievalAugmentedGeneration.Embeddings;

namespace AiDotNet.Tests.RetrievalAugmentedGeneration.ContextCompression;

/// <summary>
/// Tests for selective sentence-based context compression.
/// </summary>
public class SelectiveContextCompressorTests : ContextCompressorTestBase
{
    [Fact]
    public void Constructor_WithValidEmbeddingModel_Initializes()
    {
        // Arrange
        var embeddingModel = new StubEmbeddingModel<double>(dimension: 384);

        // Act
        var compressor = new SelectiveContextCompressor<double>(
            embeddingModel,
            relevanceThreshold: 0.7
        );

        // Assert
        Assert.NotNull(compressor);
    }

    [Fact]
    public void Compress_FiltersSentencesByRelevance()
    {
        // Arrange
        var embeddingModel = new StubEmbeddingModel<double>(dimension: 384);
        var compressor = new SelectiveContextCompressor<double>(
            embeddingModel,
            relevanceThreshold: 0.7
        );

        var documents = CreateSampleDocuments();

        // Act
        var compressed = compressor.Compress(documents, "machine learning");

        // Assert
        AssertCompressed(documents, compressed);
        AssertRelevancePreserved("machine learning", compressed);
    }

    [Fact]
    public void Compress_WithHighThreshold_FiltersMoreAggressively()
    {
        // Arrange
        var embeddingModel = new StubEmbeddingModel<double>(dimension: 384);

        var lenient = new SelectiveContextCompressor<double>(embeddingModel, 0.5);
        var strict = new SelectiveContextCompressor<double>(embeddingModel, 0.9);

        var documents = CreateSampleDocuments();

        // Act
        var lenientResult = lenient.Compress(documents, "test");
        var strictResult = strict.Compress(documents, "test");

        // Assert
        var lenientRatio = CalculateCompressionRatio(documents, lenientResult);
        var strictRatio = CalculateCompressionRatio(documents, strictResult);

        Assert.True(strictRatio <= lenientRatio,
            "Strict threshold should compress more");
    }
}

Testing Strategy

Coverage Targets

LLMContextCompressor: 85%+
DocumentSummarizer: 85%+
SelectiveContextCompressor: 80%+
AutoCompressor: 80%+

Performance Benchmarks

[Fact]
public void Compress_10KB_CompletesUnder2Seconds()
{
    // Test compression speed for realistic document sizes
}

[Fact]
public void Compress_100Documents_BatchProcessingEfficient()
{
    // Test batch compression efficiency
}

Common Pitfalls

Pitfall 1: Not Verifying Compression Quality

Wrong:

var compressed = compressor.Compress(docs, query);
Assert.NotEmpty(compressed);  // Too vague!

Correct:

var compressed = compressor.Compress(docs, query);

// Verify actual compression occurred
Assert.True(compressed[0].Content.Length < docs[0].Content.Length);

// Verify relevance preserved
Assert.Contains(queryKeyword, compressed[0].Content);

Pitfall 2: Ignoring Edge Cases

Wrong:

// Only test with medium-sized documents

Correct:

[Theory]
[InlineData(10)]      // Very short
[InlineData(500)]     // Medium
[InlineData(10000)]   // Very long
public void Compress_VariousDocumentSizes_HandlesCorrectly(int docLength)
{
    var doc = CreateDocument(length: docLength);
    var compressed = compressor.Compress(new[] { doc }, "query");
    // Verify compression works for all sizes
}

Testing Checklist

For Each Compressor

Integration Tests

Compression → Retrieval pipeline
Multiple compressors chained
Compression with different LLMs
Real-world document sets

Next Steps

Implement all tests (80+ test methods across 4 compressors)
Achieve 80%+ code coverage
Performance benchmark tests
Integration tests
Move to Issue [Test Coverage] Implement Tests for RAG Embedding Management #370 (Embedding Management)

Resources

Compression Techniques

Extractive: Select relevant sentences
Abstractive: Generate new summary
Hybrid: Combine both approaches

Key Metrics

Compression ratio: compressed_length / original_length
Relevance preservation: % of query terms retained
Information retention: ROUGE/BLEU scores

Good luck! Context compression is crucial for working within LLM token limits while maximizing information density.

Uh oh!

[Test Coverage] Implement Tests for RAG Context Compression #369

Description

Metadata

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

Issue actions