MinTok AST Virtualization for Python
Compile Python codebases into high-density Intermediate Representation (IR). MinTok parses concrete syntax trees via Tree-sitter, strips syntactic boilerplate, and preserves complete architectural invariants so models reason faster with 79.2% fewer prompt tokens.
Why Python Inflates LLM Context Windows
The Python programming language possesses rich syntactic conventions that provide high human readability but introduce substantial token inflation when fed into large language models. Repetitive import statements, verbose type annotations, boilerplate constructors, and extensive docstrings often consume between 65% and 80% of an agent's context window.
MinTok's Python compiler leverages native `tree-sitter-python` grammar parsers to tokenize source files into concrete syntax trees. By analyzing node hierarchies, MinTok identifies unreferenced symbol declarations, dead variable bindings, and unused module imports. These nodes are surgically elided before tokenization, achieving an average of 79.2% token reduction while preserving 100% of functional contracts and type bounds.
Python AST Slicing Pipeline
Four-stage deterministic AST compaction executed inside Cloudflare V8 WebAssembly isolates in <3ms.
Import Tree Pruning
Detects and prunes unreferenced sys, os, typing, and third-party imports from prompt payloads.
Type Annotation Folding
Preserves public interface type signatures while eliding redundant local variable type annotations.
Docstring & Comment Slicing
Extracts structured docstrings into symbol schemas while pruning conversational or commented-out code.
Call Graph Virtualization
Builds directional call graphs between Django models, views, and services for surgical context injection.
AST Transformation Benchmark
Compare raw source code tokens against MinTok compiled IR for Python.
# Raw Python File (342 Tokens)
import os
import sys
import logging
from typing import Dict, List, Optional, Any, Union
logger = logging.getLogger(__name__)
class PaymentProcessor:
"""Handles credit card payments via Stripe gateway with retry policy."""
def __init__(self, api_key: str, timeout: int = 30) -> None:
self.api_key = api_key
self.timeout = timeout
self._client = None
def execute_charge(self, amount_cents: int, customer_id: str) -> Dict[str, Any]:
# Log entry
logger.info(f"Charging {amount_cents} to customer {customer_id}")
if amount_cents <= 0:
raise ValueError("Amount must be positive")
return {"status": "succeeded", "charge_id": "ch_12345"}
# MinTok IR Virtualized (71 Tokens — 79.2% Saved)
@class PaymentProcessor(api_key:str, timeout:int=30):
fn execute_charge(amount_cents:int, customer_id:str) -> dict[status, charge_id]
Core Python Compiler Invariants
MinTok never truncates or guesses. These four structural invariants guarantee zero hallucinated imports and 100% type soundness.
Contract & Type Invariant
Public type signatures, trait boundaries, and function interfaces in .py files are strictly preserved so downstream callers compile without type errors.
Exception & Error Boundary
Try/catch blocks, error return values, and panic/raise declarations are maintained to ensure models reason accurately about failure states.
Namespace & Module Path
Canonical import paths and symbol namespaces are preserved, preventing hallucinated relative import paths.
Deterministic Serialization
MinTok IR serializes deterministically across identical AST inputs, maximizing edge KV cache hit rates.
CLI Flags for Python
mintok compile src/ --lang python
Compiles all .py files into high-density MinTok IR format.
mintok check --lang python --strict
Validates AST syntax invariants and verifies that no syntax errors were introduced.
mintok run --lang python --ratio 0.20
Targets a strict 80% context compression ratio during autonomous execution.
Python AST Compilation Questions
Does MinTok break Python syntax when editing code?
No. MinTok uses Tree-sitter parsers to validate syntax before and after every modification, guaranteeing that emitted patches are syntactically sound.
How does MinTok handle Python macros and metaprogramming?
MinTok expands local utility macros into structural type invariants while preserving the underlying macro definition for caller safety.
Can I use MinTok with large multi-file Python projects?
Yes. MinTok builds a project-wide symbol dependency graph across all .py files, enabling surgical cross-file refactoring without context blowouts.
What is the compression speed?
MinTok compiles Python ASTs in less than 3 milliseconds per file inside Cloudflare V8 WebAssembly isolates.