Benchmarks

contextdb includes multiple benchmark suites for evaluating retrieval quality, adversarial resistance, and long-term memory performance.

Running benchmarks

# Full benchmark suite (writes HTML report)
make bench

# MTEB retrieval quality
make bench-mteb

# Adversarial resistance
make bench-adversarial

# LongMemEval (requires dataset download)
make longmemeval

# Fitness evaluation suite
make fitness

# Representative corpus ranking snapshot
contextdb eval ranking --out ranking-eval.json --report

# Human-readable ranking release recap
contextdb eval ranking --markdown-out ranking-eval.md

# Release-to-release ranking diff
contextdb eval ranking --compare previous-ranking-eval.json --diff-markdown

# Write versioned JSON and Markdown ranking baselines
contextdb eval ranking --baseline-dir .contextdb/ranking-baselines

# Compare against the latest previous baseline in that directory
contextdb eval ranking --compare-baseline-dir .contextdb/ranking-baselines --diff-markdown

# Inspect retained and pruneable ranking baselines
contextdb eval ranking --baseline-retention-dir .contextdb/ranking-baselines --baseline-retention-keep 5

# Emit a reviewable shell script for pruneable baseline artifacts
contextdb eval ranking --baseline-retention-dir .contextdb/ranking-baselines --baseline-retention-keep 5 --emit-delete-script

# Write a machine-readable artifact inventory with bytes and hashes
contextdb eval ranking --baseline-retention-dir .contextdb/ranking-baselines --baseline-retention-keep 5 --baseline-manifest-out ranking-baseline-manifest.json

Ranking Eval Snapshots

contextdb eval ranking emits a JSON snapshot for the representative corpus in testdata/corpus.go. The report includes top-k results for each labelled query, expected rank, reciprocal rank, MRR, raw score components, weighted score breakdowns, and retrieval source. Use it before and after ranking changes to review score drift even when the golden corpus still passes.

Use --markdown or --markdown-out when you want a compact release-review recap with pass/fail totals, MRR, failure details, top results, and score breakdowns.

Use --compare with a previous JSON snapshot when you want a historical diff. Add --diff-report, --diff-out, --diff-markdown, or --diff-markdown-out to capture MRR deltas, pass-count changes, pass flips, largest rank movements, and largest top-score movements.

Use --baseline-dir when you want release-friendly baseline artifacts. The command writes both ranking-eval-vX.Y.Z.json and ranking-eval-vX.Y.Z.md for the current contextdb version. Use --compare-baseline-dir to resolve the latest previous ranking-eval-vX.Y.Z.json in that directory and compare the current ranking run against it. This keeps release ranking reviews consistent without hand-picking a baseline filename each time.

Use --baseline-retention-dir when you want a read-only retention report for versioned ranking baselines. The report marks the newest baseline as current, retains the newest --baseline-retention-keep versions, and lists older versions as pruneable without deleting files. Add --emit-delete-script to print a shell script containing only rm -- ... commands for existing pruneable JSON and Markdown artifacts; review the script before running it. Add --baseline-manifest-out to write a JSON artifact inventory with bytes and SHA-256 hashes for CI retention evidence. Later, run contextdb eval ranking baseline manifest verify --manifest ranking-baseline-manifest.json --report to verify inventory paths, byte sizes, and hashes. Add --bundle-dir when CI should keep JSON, Markdown, and annotation artifacts with stable names, then run contextdb eval ranking baseline manifest bundle verify --index ranking-baseline-verification/ranking-baseline-manifest-verification-index.json --report when an archived bundle needs to be re-checked. Add --markdown or --markdown-out when CI needs a compact release recap for manifest verification, and --annotations or --annotations-out when CI should highlight failed artifact checks directly. See the ranking baseline retention cookbook for keep-count and CI artifact recipes.

P0 retrieval evaluation workload

The separate TestP0RetrievalEvaluation workload uses stable node IDs, a fixed UTC clock, and a small synthetic vector corpus. It exercises an exact identifier whose vector deliberately points elsewhere; valid-time before and after a change; transaction-time before and after a late correction; a contradiction; and a source dispute. Each case logs the expected node's rank and hit-at-5. The test logs aggregate MRR@5 and local p50/p95 retrieval durations, and requires matching ranks from memory and BadgerDB. An intended quality miss is reported rather than silently removed from the corpus.

go test ./internal/retrieval -run TestP0RetrievalEvaluation -v -count=1
go test ./internal/retrieval -run '^$' -bench BenchmarkP0Retrieval -benchmem

This is a regression fixture, not a benchmark of natural-language understanding. The query vectors are prescribed, the source-dispute case uses node confidence as a stand-in for already assessed source trust, and the contradiction case measures retrieval ordering rather than conflict explanation. Durations from one local run depend on host load and corpus size; compare repeated runs on the same host before drawing a latency conclusion. The test covers memory and BadgerDB; Postgres and remote backend parity still require an integration environment with the matching schema and vector extension. The older contextdb eval ranking snapshots use a different corpus and remain comparable to their own historical baselines.

MTEB retrieval quality

The MTEB benchmark (bench/mteb/) evaluates retrieval quality using standard information retrieval metrics:

Metric	Description
NDCG@10	Normalized Discounted Cumulative Gain at rank 10
Recall@K	Fraction of relevant documents retrieved at various K values
MRR	Mean Reciprocal Rank of the first relevant result

The benchmark tests contextdb's hybrid retrieval (vector + graph fusion) against pure vector search baselines. It exercises the full pipeline: embedding, vector indexing, graph traversal, fusion, and scoring.

Adversarial resistance

The adversarial benchmark (bench/adversarial/) tests contextdb's resilience to:

Poisoning attacks

• Flood writes from low-credibility sources
• Near-duplicate injection with contradicting content
• Measures how many poison nodes pass the admission gate
• Validates that source credibility correctly throttles adversarial writes

Temporal consistency

• Writes facts with overlapping validity windows
• Queries at specific points in time
• Validates that AsOf filtering correctly excludes expired or future facts
• Tests that bi-temporal ordering is preserved across updates

LongMemEval

The LongMemEval benchmark (bench/longmemeval/) evaluates long-session memory:

• Loads or downloads the LongMemEval dataset from HuggingFace
• Simulates multi-turn conversations across extended sessions
• Measures recall of facts mentioned in earlier turns
• Tests the interaction between memory decay, consolidation, and retrieval

Fitness suite

The fitness evaluation (bench/fitness_test.go) is a comprehensive end-to-end test:

• Loads the test corpus from testdata/corpus.go
• Writes all items through the admission gate
• Runs a battery of retrieval queries
• Validates correctness of results (expected items present, ordering reasonable)
• Measures latency percentiles for write and retrieval operations
• Tests scoring function behaviour across namespace modes

Score benchmarks

bench/score_bench_test.go microbenchmarks the composite scoring function:

go test -bench=. ./bench/

This measures raw scoring throughput independent of storage backend.