Benchmarks

Evaluates whether deep-research agents can discover and reason over relevant structured datasets in a million-scale data lake, then produce verifiable answers with explicit provenance.

Continuously draws from newly reported Linux kernel crashes to test whether agents can diagnose and resolve real, post-cutoff failures without relying on memorized fixes.

Measures whether branchable databases can support the creation, exploration, and pruning patterns of agentic workloads across configurable branch shapes, schemas, and operations.

Pairs a curated dataset of real Linux kernel bugs with deterministic reproduction scripts and a sandboxed experimentation platform for evaluating agent-driven crash diagnosis and patching.

A benchmark dataset for building and evaluating digital twins based on real people, designed to study how faithfully agents capture individual behavior and variation across domains.

A versatile user-simulation toolkit for evaluating interactive agents through realistic multi-turn behavior, reducing dependence on costly human evaluation.

Generates diverse, realistic user queries specifically designed to expose QA-agent failures, supporting more rigorous evaluation than static, average-case question sets.

Evaluates whether coding agents can migrate enterprise Java applications across frameworks while preserving behavior and handling changes that span a real application.

Diagnoses why coding agents can still fail after reaching the correct region of a codebase, exposing breakdowns in maintaining coherent reasoning and edits over time.

Tests whether AI agents can understand software without source code—a common reality in malware, firmware, and proprietary-software analysis. Every program is built from scratch to prevent data contamination and hardened with modern anti-analysis protections.

Tests whether agents can find, reproduce, and repair real concurrency bugs whose failures depend on particular interactions among threads, timing, and shared state. Each bug includes a specific oracle for validating both reproduction and repair.

Evaluates whether agents can prepare 2025 tax returns accurately, request missing taxpayer information, and apply the correct rule when multiple provisions govern the same field. The benchmark is built from a structured, deterministic tax-rule system and realistic taxpayer scenarios.