EVALUATION REPORT PASS RATE: 84.6% SAMPLE: 133 EDIT SCENARIOS (AIDER SUITE) DETERMINISTIC SUITE

Aider Code Editing Benchmark: MinTok Performance & Cost Analysis

The Aider benchmark evaluates a model's ability to read existing code, understand unified diff formats, and return valid search/replace blocks without corrupting surrounding files. MinTok-1-Max achieved an 84.6% success rate on first attempt.

TEST ON LIVE WORKLOADS VIEW PARETO FRONTIER RUN SWE-BENCH REPRODUCER CLI →

Aider Code Editing Benchmark Leaderboard & Unit Economics

Standardized task cost and resolution success rate compared against frontier closed APIs.

Model / System	Score / Pass Rate	Cost Per Task	Arena Score / Tier	P95 Latency
MinTok-1-Max	84.6%	$0.00436	Diff Match	0.42s
MinTok-1-Pro	81.2%	$0.00084	Diff Match	0.38s
Claude 5.5 Sonnet	82.7%	$0.02700	Diff Match	1.45s
OpenAI o3-mini	79.4%	$0.01500	Whole File	2.10s

SCIENTIFIC PROTOCOL

Evaluation Methodology & Contamination Controls

The Aider Code Editing Benchmark evaluation protocol runs in an isolated, sandboxed execution environment. To ensure strict scientific validity and zero data contamination:
1. Fresh Docker Isolate: Every task instance runs in an isolated ephemeral container with pinned language runtimes and exact dependency lockfiles.
2. Deterministic Verification Gate: Solutions are evaluated against hidden test suites and ground-truth unit assertions.
3. Fixed Sampling Temperature: All MinTok models are evaluated at a fixed temperature of 0.2 with nucleus sampling p=0.95.
4. Pass@1 Metric: Scores reflect first-attempt resolution without cherry-picking or post-hoc retries.

ERROR TAXONOMY & FAILURE MODES

Why Standard Frontier Models Fail on SWE Tasks

When evaluating baseline frontier models on large codebases, over 68% of failures stem from context drift and syntax hallucinations rather than algorithmic deficiency:
- Context Window Saturation: As agent conversations grow beyond 80k tokens, models suffer from attention attenuation ('lost in the middle').
- Missing Indentation & Braces: Raw string tokenization frequently causes off-by-one indentation errors in multi-line edits.
- Import Desynchronization: Models hallucinate non-existent module paths when flooded with hundreds of unreferenced imports.

MinTok eliminates these three primary failure modes by replacing uncompressed file dumps with deterministic AST Intermediate Representation.

Key Findings & Production Insights

First-attempt diff format adherence exceeded 99.4%.

Zero multi-turn search/replace desynchronization errors.

Sub-cent cost per successfully merged git commit.

INDEPENDENT VERIFICATION PROTOCOL

Reproduce These Benchmark Results Locally

Execute deterministic local verification with the official MinTok benchmark harness.

1 Install the official MinTok CLI harness: `curl -fsSL https://mintok.adstim.net/install | bash`

2 Run the benchmark harness: `mintok benchmark run --suite aider-benchmark --model mintok-1-max --output results.json`

3 Generate the verified report: `mintok benchmark verify results.json`

4 Compare Pareto frontier metrics: `mintok benchmark plot results.json`

FREQUENTLY ASKED QUESTIONS

Benchmark & Methodology Questions

How does MinTok achieve a 84.6% score on Aider Code Editing Benchmark?

MinTok's context compiler eliminates repetitive token noise, allowing the model's self-attention layers to focus exclusively on the bug's active call sites and type contracts.

Can these benchmark results be independently reproduced?

Yes. The complete benchmark harness, dataset seeds, and verification commands are publicly accessible via the MinTok CLI.

How does task cost compare to closed proprietary models?

MinTok-1-Max resolves tasks at an average cost of $0.00436 per SWE task—over 6.2x cheaper than Claude 5.5 Sonnet ($0.02700).

What model tier should I choose for my workloads?

For complex algorithmic refactoring, use `mintok-1-max` (1658 Elo). For standard feature delivery and test generation, `mintok-1-pro` (1638 Elo) provides exceptional value.