Skillware · EMPIRICAL EVALUATION · 23 MODELS · 736 TEST CONDITIONS
Skillware-Bench: The Transparent Evaluation Framework for Governed Agent Skills
Skillware-Bench evaluates 23 frontier and sovereign models across an empirical 16-layer Progressive Tiered Ablation Benchmark (4 crucibles × 4 difficulty tiers = 16 layers, 736 conditions). It rigorously contrasts unconditioned code generation (Vanilla baseline) against governed execution (Skillware equipped) to quantify agent reliability, economic efficiency, and determinism.
Grand Aggregate Fleet Results
- 91.6% Fleet Pass Rate: 337/368 vs 300/368 unassisted (+10.1% net lift across all 23 models)
- -51.4% Token Consumption: 1,586,162 prompt and completion tokens saved via typed schema preflight
- 1.60x Execution Speedup: 72.2m vs 115.2m cumulative execution wall-clock latency
- 464 → 0 Debug Script Loops: 100% of workspace script debugging traps eliminated to zero
- 5.1x Tighter Determinism: Inter-run coefficient of variation compressed from CV = 0.50 to CV = 0.10
- 92.0% Mean Defense Rate: Elevating adversarial security across prompt injection, unicode smuggling, and statutory entity spoofing
- ECE 14.5% → 2.9%: Expected Calibration Error deflated by -79.7%, grounding verbal model confidence in empirical tool execution
Ten Flagship Publication Exhibits
- Exhibit 01 · Figure 2: Anatomy of a Skill Bundle — Five first-class primitives: Contract, Effect, Directive, Assurance, and Corpus.
- Exhibit 02 · Figure 4: The Benchmark Cost Cascade — Same task, up to -97.3% enterprise spend reduction.
- Exhibit 03 · Figure 5: The Parameter Paradox — Compact open-weight models matching frontier APIs at 1/30th cost.
- Exhibit 04 · Figure 7: Deterministic Execution & Contract Gating — 100% deterministic execution in 1 turn with 0 retries.
- Exhibit 05 · Figure 33: 368-Layer Call Survival Flow — 71.2% → 94.1% clean first-pass completion; aborts cut from 57 to 7.
- Exhibit 06 · Figure 11: Latency Anatomy of Complex Workflows — 27.5s → 12.2s (-55.6% wall-clock latency acceleration).
- Exhibit 07 · Figure 14: Run-to-Run Determinism & Variance Contraction — CV compressed from 0.50 to 0.10 (5.1x tighter).
- Exhibit 08 · Figure 17: Governance & Security Radar Profile — 84.1% → 92.0% mean adversarial defense across 6 attack vectors.
- Exhibit 09 · Figure 27: Calibration & Overconfidence Diagram — Expected Calibration Error reduced from 14.5% to 2.9%.
- Exhibit 10 · Table 7: Four-Axis Master Operational Summary — Empirical standings across 23 models and 736 conditions.
23 Evaluated Frontier & Sovereign Models
Frontier Cloud APIs: Claude Sonnet 5, Claude Haiku 4.5, Claude Opus 4, GPT-6 Luna, GPT-5 Turbo, GPT-5 Mini, Gemini 3.8 Pro, Gemini 3.8 Flash, Command R+.
Sovereign Heavyweights: Llama 4 Maverick 70B, Llama 3.3 70B, Qwen 3.8 Max, Qwen 2.5 72B, DeepSeek V3, DeepSeek R1, Mistral Large 2.
Compact Workhorses: Qwen 3.8 Flash 16B, Gemma 4 31B, Gemma 3 27B, Llama 4 Scout 17B, Mistral NeMo 12B, GLM-4 9B, Yi-1.5 34B.
Open interactive benchmark overview →