Benchmax

Benchmax documentation

From a human expectation to a test an agent can run.

Learn the shared product model once, then use each benchmark's documentation for its domain-specific inputs, rubrics, and contribution rules.

01 · Core model

One environment, many tasks and benchmarks.

An engineer integrates an agent with an environment version—not with individual tasks. Every compatible task can then run through the same integration mapping without task-specific logic.

Environment

The versioned input, tool, state, and result contract an agent connects to once.

Benchmark

A public, versioned dataset repository containing tasks compatible with one environment.

Task

One concrete situation, its agent-visible input, and human-reviewed evaluation criteria.

Run

One immutable benchmark version executed against one named agent version.

environment → benchmark version → task versions → agent run → semantic verdicts

02 · Run a benchmark

Keep the agent in your environment.

  1. 1
    Inspect the benchmark

    Read its tasks, inputs, behavior contracts, and status without an account.

  2. 2
    Create a private fork

    Benchmax pins the public version and gives your team a private working copy.

  3. 3
    Connect once

    Give the environment page to a coding agent; it creates one task-agnostic Run API integration.

  4. 4
    Run and diagnose

    Your runner fetches synthetic input, invokes the real agent, and submits native output for grading.

03 · Contribute a task

Make domain knowledge executable.

Start from a compatible task, change the synthetic evidence and behavior boundary, define required and prohibited meaning, cite the decisive evidence, and publish a new immutable dataset version.

Available now

Private forks, task authoring, human approval, structured change review, immutable versions, and replay.

Public collaboration layer

Upstream contribution proposals, public review, contributor identity, and benchmark-maintainer approval are the next layer—not yet implied by the current prototype.

04 · Trust and grading

Humans can inspect what the agent cannot see.

Task titles, expected behavior, prohibited behavior, decisive evidence annotations, and grader instructions are withheld from the agent during execution. They remain readable to human reviewers.

  • Agent-visible inputs and hidden evaluation criteria are separated.
  • Semantic verdicts cite evidence available in the task.
  • Integration, agent, and grader errors do not become behavior failures.
  • Task, benchmark, environment, agent, and grader versions are pinned.

05 · Benchmark guides

Domain rules live with each benchmark.

Benchmark 001Evidence-grounded call scoring

Input contract, evidence boundaries, task design, evaluation, and contribution guidance.