ICML 2026 · Execution-based benchmark

Rule2DRC Leaderboard

Synthesizing KLayout DRC scripts from natural-language chip design rules, graded by execution on held-out layouts. 1,000 tasks, 13,921 evaluation layouts.

		Public set · 1,000 tasks		Private set · 100 tasks
		Docs in context	No context	Docs in context	No context
#	Model
	GPT-OSS-120B paper reasoning effort medium; no-context is a single run via OpenRouter · 2026-05
	GPT-OSS-20B paper reasoning effort medium; no-context is a single run via OpenRouter · 2026-05
	Qwen3-30B-A3B-Instruct-2507 paper instruct (non-reasoning); no-context is a single run via OpenRouter · 2026-05
	Qwen3-235B-A22B-Thinking-2507 thinking · 2026-07
	Qwen3-235B-A22B-Instruct-2507 instruct (non-reasoning) · 2026-07
	Qwen3-30B-A3B-Thinking-2507 thinking; with-docs output capped at 9.8K (provider limit) · 2026-07
	GLM-5.2 thinking · 2026-07
	MiniMax-M3 thinking · 2026-07
	MiMo-v2.5 thinking · 2026-07
	DeepSeek-V4-Pro thinking · 2026-07
	Hy3 2026-07

Metrics

Success rate (higher is better): fraction of tasks where the generated DRC script agrees with the ground-truth script on every evaluation layout.
Error rate (lower is better): fraction of tasks where the generated script triggers a compile-time or runtime error.
Docs in context: the prompt includes the 60K-token KLayout DRC API documentation; no context means the model sees only the rule.

Protocol

Single sample per task (Sample-1), no Best-of-N selection and no tester agents.
Rows tagged paper are Sample-1 results from the ICML 2026 paper, averaged over 30 runs; their no-context cells are single runs via OpenRouter. Other rows are single runs via OpenRouter.
Private set: 100 held-out tasks (70 cascade, 30 niche syntax), built with the same pipeline as the public set but never released, so scores are free of data contamination. It is also deliberately harder: success rates run well below the public set. Private cells are single runs via OpenRouter.

Submit a model

Run scripts/leaderboard/run_eval.sh from the Rule2DRC repository against any OpenAI-compatible endpoint, then open an issue or pull request with the printed result row.