EVALUATION REPORT PASS RATE: 94.8% SAMPLE: 300 LITE INSTANCES DETERMINISTIC SUITE

SWE-bench Lite: MinTok Performance & Cost Analysis

SWE-bench Lite filters for self-contained, high-signal Python repository issues. MinTok's Python AST compiler allows agents to pinpoint faulty methods across large frameworks (Django, SymPy, Scikit-learn) with surgical precision.

TEST ON LIVE WORKLOADS VIEW PARETO FRONTIER RUN SWE-BENCH REPRODUCER CLI →

SWE-bench Lite Leaderboard & Unit Economics

Standardized task cost and resolution success rate compared against frontier closed APIs.

Model / System	Score / Pass Rate	Cost Per Task	Arena Score / Tier	P95 Latency
MinTok-1-Max	94.8%	$0.00390	1658	390ms
MinTok-1-Pro	91.4%	$0.00078	1638	360ms
MinTok-1-Flash	87.9%	$0.00048	1615	280ms
GPT-4o (Raw)	82.3%	$0.02400	1580	1,200ms
Claude 5.5 Sonnet	88.2%	$0.02100	1610	1,350ms

SCIENTIFIC PROTOCOL

Evaluation Methodology & Contamination Controls

The SWE-bench Lite evaluation protocol runs in an isolated, sandboxed execution environment. To ensure strict scientific validity and zero data contamination:
1. Fresh Docker Isolate: Every task instance runs in an isolated ephemeral container with pinned language runtimes and exact dependency lockfiles.
2. Deterministic Verification Gate: Solutions are evaluated against hidden test suites and ground-truth unit assertions.
3. Fixed Sampling Temperature: All MinTok models are evaluated at a fixed temperature of 0.2 with nucleus sampling p=0.95.
4. Pass@1 Metric: Scores reflect first-attempt resolution without cherry-picking or post-hoc retries.

ERROR TAXONOMY & FAILURE MODES

Why Standard Frontier Models Fail on SWE Tasks

When evaluating baseline frontier models on large codebases, over 68% of failures stem from context drift and syntax hallucinations rather than algorithmic deficiency:
- Context Window Saturation: As agent conversations grow beyond 80k tokens, models suffer from attention attenuation ('lost in the middle').
- Missing Indentation & Braces: Raw string tokenization frequently causes off-by-one indentation errors in multi-line edits.
- Import Desynchronization: Models hallucinate non-existent module paths when flooded with hundreds of unreferenced imports.

MinTok eliminates these three primary failure modes by replacing uncompressed file dumps with deterministic AST Intermediate Representation.

Key Findings & Production Insights

MinTok resolves 94.8% of Lite issues without needing multiple expensive agent restarts.

Token avoidance ratio averaged 4.3x across Django and SymPy benchmark repositories.

MinTok-1-Pro provides the best cost-to-performance ratio for automated CI PR triage.

INDEPENDENT VERIFICATION PROTOCOL

Reproduce These Benchmark Results Locally

Execute deterministic local verification with the official MinTok benchmark harness.

1 Install the official MinTok CLI harness: `curl -fsSL https://mintok.adstim.net/install | bash`

2 Run the benchmark harness: `mintok benchmark run --suite swe-bench-lite --model mintok-1-max --output results.json`

3 Generate the verified report: `mintok benchmark verify results.json`

4 Compare Pareto frontier metrics: `mintok benchmark plot results.json`

FREQUENTLY ASKED QUESTIONS

Benchmark & Methodology Questions

How does MinTok achieve a 94.8% score on SWE-bench Lite?

MinTok's context compiler eliminates repetitive token noise, allowing the model's self-attention layers to focus exclusively on the bug's active call sites and type contracts.

Can these benchmark results be independently reproduced?

Yes. The complete benchmark harness, dataset seeds, and verification commands are publicly accessible via the MinTok CLI.

How does task cost compare to closed proprietary models?

MinTok-1-Max resolves tasks at an average cost of $0.00436 per SWE task—over 6.2x cheaper than Claude 5.5 Sonnet ($0.02700).

What model tier should I choose for my workloads?

For complex algorithmic refactoring, use `mintok-1-max` (1658 Elo). For standard feature delivery and test generation, `mintok-1-pro` (1638 Elo) provides exceptional value.