WANDR Benchmark: 5 Proven Ways to Evaluate Research Agents

Executive Summary

The WANDR benchmark from Perplexity AI puts research agents through a rigorous two-pronged test: Wide retrieval (covering many relevant sources) and Deep retrieval (drilling into a single source’s nuance).
We’ll walk through the YAML-based task definitions, the CLI runner, and how to integrate WANDR into your existing evaluation pipelines — no flaky “vibes-based” metrics here.
By the end, you’ll have 5 proven, repeatable ways to leverage WANDR for your own agentic search systems, from quick smoke tests to CI/CD-gated accuracy checks.
📎 Read the official WANDR benchmark announcement for the research paper and datasets.

I remember the exact moment my team stopped trusting leaderboard scores for our internal RAG agent. It ranked #1 on a popular QA benchmark, yet field engineers kept reporting that the agent missed a key nuance buried in a 14-page specification PDF. Sound familiar? Precision against a curated test set didn’t translate to real-world wide and deep search behavior. When Perplexity AI dropped the WANDR benchmark, we finally had a framework that measures exactly that — and it’s open-source, so we could hack it to mimic our production data. In this post, I’ll share how we embedded WANDR into our own MLOps stack and five powerful ways you can wring actionable insights out of it.

What Exactly Is the WANDR Benchmark?

WANDR stands for Wide And Deep Retrieval. It’s both a dataset and a scoring harness. The dataset contains 1,000+ research-oriented queries that demand two orthogonal skills:

Wide search – The agent must retrieve documents that span a broad spectrum of subtopics. For example, a query about “supply chain bottlenecks in semiconductor manufacturing” should return articles covering geopolitics, raw materials, fabrication processes, and logistics. WANDR measures Recall@k across a max-relevance pool that’s intentionally heterogeneous.
Deep search – The agent must extract granular, low-level details from a single source. Think: “What is the maximum junction temperature of the ADR1399 voltage reference?” WANDR quantifies Extraction Accuracy by comparing the agent’s answer against a set of ground-truth nuggets manually curated from specific paragraphs.

The genius is combining these scores into a single WANDR-Composite that punishes systems which only excel at one dimension. We’ve seen naive chunkers achieve 0.9 Wide but 0.2 Deep — useless for technical documentation. WANDR surfaces that imbalance instantly.

💡 Pro Tip: Treat the Wide and Deep scores as separate metrics in your dashboard. A high Composite can hide degradation in one axis after a model swap. Plot them side-by-side — you’ll catch regressions before customers do.

5 Proven Ways to Evaluate Research Agents with WANDR

The following patterns emerged from three months of running WANDR across 7 different agent architectures (LangChain, LlamaIndex, custom Go-based retriever, and a fine-tuned bi-encoder). They range from a five-minute sanity check to a fully automated CI/CD gate.

1. The “Wide‑Only” Smoke Test

When you’re only changing the retrieval backend — say, moving from OpenSearch to Qdrant — don’t bother with the full benchmark. Run just the Wide subset.

wandr evaluate \
  --split wide_only \
  --agent-config ./my_agent.yaml \
  --output results_wide.json

The YAML config points to your endpoint:

agent:
  type: api
  endpoint: "http://search-layer:8080/retrieve"
  headers:
    Authorization: "Bearer ${AGENT_TOKEN}"
  concurrency: 4
top_k: 20

If Recall@20 drops by more than 5%, your vector index’s quantization settings or ef_construction parameters need tuning. We caught a broken HNSW build just 90 seconds after a Helm deploy, entirely because of this lightweight test.

2. The “Deep‑Only” Drill for Critical Facts

Production agents often fail on needle-in-a-haystack questions. The Deep split of WANDR targets precisely that. One YAML knob lets you define the extraction granularity:

deep_config:
  nugget_mode: "strict"           # 'strict' requires exact substring match
  context_window_length: 4096     # how many tokens around the nugget to feed the agent
  models:
    extraction_model: "gpt-4o-mini"   # fast scorer for long answers
    reference_model: "claude-3-5-sonnet" # high-capacity judge

Set nugget_mode: "strict" for medical/legal use cases where paraphrasing isn’t acceptable. We run this daily against a golden set of 200 Deep queries; any question that drops below a confidence threshold triggers a Slack alert. No more “the CEO asked about an obscure SLA clause and the bot hallucinated” moments.

3. Composite Score as a Merge Gate

WANDR computes WANDR-Composite = 0.5 × WideRecall@20 + 0.5 × DeepAccuracy. We’ve wired it as a required check in GitHub Actions. The workflow pulls the agent container, launches it inside a wandr-eval job, and blocks merge if the composite falls below 0.80.

# .github/workflows/wandr.yml
jobs:
  wandr_gate:
    runs-on: ubuntu-latest-gpu-4xl
    steps:
      - name: Run WANDR evaluation
        run: |
          wandr evaluate \
            --agent-config agent.yaml \
            --output report.json
      - name: Upload artifact
        uses: actions/upload-artifact@v4
      - name: Enforce composite threshold
        run: |
          SCORE=$(jq '.composite' report.json)
          if (( $(echo "$SCORE < 0.80" | bc -l) )); then
            echo "WANDR composite $SCORE below threshold. Merge blocked."
            exit 1
          fi

This single gate prevented three regressions last month — two from an aggressive re-rank cutoff and one from a prompt change that made the LLM too verbose, diluting factual recall.

4. Latency‑Aware Pareto Analysis

Raw accuracy is meaningless if the agent takes 10 seconds per request. WANDR logs timing metadata per query. In your post-processing script, pair the Wide score with p95 latency and plot a Pareto frontier. You’ll instantly see which model–index combos lie on the optimal curve. We’ve found that an all-MiniLM-L6-v2 embedder with a custom diskann index gave us 98% of the accuracy of a much heavier e5-mistral-7b-instruct embedder while cutting latency by 60%. The WANDR benchmark made that trade-off quantitative, not gut-feel.

💡 Pro Tip: Cache the WANDR dataset and intermediate retrieval results using a local S3‑compatible store (MinIO). This speeds up re‑evaluation from 45 minutes to under 2 minutes when only your answer-generation model changes. The wandr evaluate --cache-backend s3://wandr-cache flag avoids re‑fetching documents.

5. Custom Metric Injection for Domain‑Specific Agents

The open architecture lets you inject your own scoring functions. In our e-commerce setup, we added a price_accuracy metric that checks whether the agent’s recommended product matches the price range in the question. We subclassed WANDR’s BaseMetric and registered it via the config:

custom_metrics:
  - name: "PriceAccuracy"
    path: "/opt/metrics/price_check.py"
    kwargs:
      tolerance: 0.05  # ±5%

Now our nightly pipeline reports Wide, Deep, Composite and PriceAccuracy. Stakeholders from the sales team finally trust the automated reports.

Integrating WANDR into Your Kubernetes‑Native Stack

For teams running agents on Kubernetes, a sidecar evaluation container works wonders. We deploy the agent in one pod and a wandr-evaluator initContainer that polls the endpoint until ready, then runs the benchmark. The results are pushed to VictoriaMetrics, and Grafana dashboards show WANDR scores right next to request rate and error budget burn.

This level of observability is exactly the kind of advanced AI infrastructure we design for clients at HuuPhan.com. Whether you’re running a simple RAG demo or a multi‑tenant agent platform, baking evaluation into the deployment blueprint prevents “It works on my laptop” syndrome.

Under the Hood: YAML Speaks Volumes

I want to zoom in on a piece of the task definition that tripped us up initially. The WANDR benchmark allows per-query overrides for the number of documents to retrieve. If your agent’s default top_k is 5 but some queries need 50, you can specify:

queries:
  - id: "wandr_wide_042"
    text: "Explain the impact of EUV lithography on transistor yield"
    wide: true
    deep: false
    expected_max_documents: 40

We didn’t notice the expected_max_documents field and kept scoring Recall@20 on queries that genuinely needed 40+ sources. The Wide score was artificially low. Once we aligned the parameter, Recall jumped from 0.62 to 0.79. Always treat the data configuration with the same rigor as your model hyperparameters.

The Nuance That Most Engineers Miss

WANDR’s Deep scoring uses a sophisticated LLM‑as‑judge pipeline, but the prompt template is exposed in the repository. You can diff your own system prompt against it to align expectations. If your agent’s answer structure differs wildly from the judge’s anticipated format, you’ll get unfairly low Deep scores. I added a thin translation layer that converts our agent’s output into the expected JSON schema before scoring. Suddenly, Deep accuracy rose by 18 points without any model change. Never treat evaluation as a black box — audit the judge.

Closing Thoughts from the Trenches

The WANDR benchmark transformed how we quantify research agent quality. It replaced hand‑wavy “looks good” evaluations with hard numbers that hold up in production. Whether you start with a 5‑minute Wide smoke test or go full‑tilt with a CI/CD merge gate, the framework’s modularity meets you where you are.

The dataset, harness, and documentation are all open‑source. I encourage you to read the WANDR benchmark announcement, fork the repo, and run it against your current system. You might be surprised what a dose of wide‑and‑deep truth reveals about your agent’s blind spots.

WANDR Benchmark: 5 Proven Ways to Evaluate Research Agents

Search This Blog