Before a chip layout can be manufactured, it must satisfy thousands of geometric design rules. Design rule checking, or DRC, enforces them by running executable scripts on the layout and reporting violations. Translating each natural language rule into a correct script is labor intensive, requires specialized expertise, and must be repeated for every process node.
This makes DRC script synthesis a natural target for LLM agents, but progress has been hard to measure. We introduce Rule2DRC, a large, fully open benchmark for translating natural language design rules into executable DRC scripts, and SplitTester, an execution-guided test generation agent that improves Best-of-N selection by generating test layouts that separate candidate scripts.
Summary
Rule2DRC has 1,000 rule-to-script tasks and 13,921 evaluation layouts, far more than prior DRC test sets, and all of it is open source. A script counts as correct only when its violation outputs match the ground truth on every evaluation layout, and the agent never sees those layouts.
We highlight two results. Putting a 60K-token KLayout API document in context raises pass@1 by over 20 points and pass@20 by over 40 points on GPT-OSS-120B. SplitTester, which generates tests aimed at candidates the current tests cannot tell apart, improves on one-shot inference through inference-time scaling. On GPT-OSS-20B, Best-of-N selection with SplitTester at N = 20 raises the success rate from 16.9% for one-shot inference to 44.4%.
The Task
An agent receives only the rule and must produce a short script in a domain specific language, run by a DRC engine. We use KLayout, an open source engine, with layouts in the GDSII format. A rule such as “Pad must be inside M4 and enclosed by M4 by at least 0.850 µm” maps to a script like the one below.
pad.enclosed(m4, 0.850.um)
.output("PAD_M4_ENC", "pad must be enclosed by m4 by at least 0.850um")
(pad - m4).output("PAD_M4_ENC", "pad not fully contained by m4")
We run the result on a private set of evaluation layouts and compare its violation outputs with those of a ground truth script. The task counts as solved only when the two agree on every layout.
Why Execution-Based Evaluation
Prior evaluations in this area share a few limitations. Test sets are small, often under 200 rule and script pairs. Scripts are compared by surface level code similarity. And the benchmarks are rarely released, since rules and layouts are proprietary.
Code similarity is especially unreliable here, because the same rule can be implemented in many syntactically different but functionally equivalent ways. Execution feedback is also underused by existing methods. One line of work never runs the DRC engine. DRC-Coder does, but it requires evaluation layouts with ground truth violation labels as agent input, which presumes exactly the expert effort we want to remove.
Rule2DRC is designed to remove each of these limitations. It is far larger than prior test sets. It grades by execution instead of code similarity. It is fully open source. And it keeps the evaluation layouts hidden from the agent, so the agent must generate its own tests.
Inside the Benchmark
The 1,000 tasks come in three groups.
- SkyWater derived rules (310 tasks). Extracted from the public SkyWater130 PDK, with hand built scripts and layouts. Each task includes threshold corner cases at the 1 nm resolution of the rule and topological corner cases such as partial overlap and containment.
- Synthetic multi-constraint rules (490 tasks). Multi layer rules with chained geometric operations, in the style of advanced process nodes. Drafted with a frontier model, then manually reviewed and corrected.
- Syntax coverage rules (200 tasks). Rules that exercise rarely used grammar, bringing the benchmark to 184 unique DRC methods overall.
We report success rate, the fraction of tasks where the generated script matches the ground truth on every layout, and error rate, the fraction of tasks where the script fails to compile or run. Oracle@N gives the upper bound of any Best-of-N selector.
Documentation in Context
The KLayout DRC language is niche. We crawled the official documentation into a single 60K token document and include it in the prompt. On GPT-OSS-120B this raises pass@1 by over 20 points and pass@20 by over 40 points, a larger effect than the gap between any two selection methods. For domain specific languages, in-context documentation is a prerequisite rather than an optimization, so all remaining experiments include the document.
SplitTester
The DRC engine gives the agent an executable environment, so a natural recipe is Best-of-N. Sample N candidate scripts, generate test layouts with expected outcomes, and select the candidate that scores best. The challenge is selection. Models rarely produce the corner cases that separate a correct script from a nearly correct one, so many candidates behave identically under weak tests.
SplitTester targets these indistinguishable candidates directly.
- Generate initial tests with expected outcomes, run all candidates, and score each by agreement with the expectations.
- Cluster candidates that produce identical outputs on all current tests.
- Pick the cluster with the largest product of score and size, the most likely place where correct and nearly correct scripts are still mixed.
- Prompt the LLM to generate a test that splits three representatives of that cluster, run it, recluster, and repeat until the budget runs out.
- Let a judge LLM pick among the top three candidates, given the tests on which they disagree.
The expected labels are noisy, with error rates from about 16% to 38% depending on the model. SplitTester compensates with its clustering-based approach, which groups candidates by their actual outputs rather than trusting the labels. The ablation shows that the components are complementary. On GPT-OSS-120B with ten candidates, removing the judge drops success from 58.0% to 55.5%, removing the labels drops it to 57.1%, and pruning to the top three drops it to 57.4%.
Results
We evaluate Qwen3-30B-A3B-Instruct-2507, GPT-OSS-20B, and GPT-OSS-120B with 10, 15, and 20 candidates, giving every tester agent the same budget of 16 test layouts.
SplitTester improves on one-shot inference through inference-time scaling. On GPT-OSS-20B, one-shot inference succeeds on 16.9% of tasks, and Best-of-N selection with SplitTester at N = 20 raises this to 44.4%. One-shot success is 14.1% on Qwen3-30B and 32.5% on GPT-OSS-120B, and SplitTester improves on both. It also achieves the best success rate among the selection methods in every setting. With 20 candidates it reaches 44.4% on GPT-OSS-20B against 41.1% for CodeMonkey, 18.0% against 17.2% on Qwen3-30B, and 63.8% against 62.7% on GPT-OSS-120B, where Oracle@20 is 72.1%. Error rates drop as well.
SplitTester also lies on the runtime Pareto frontier for all three models, so its higher success rate does not come from spending more compute. The gains hold with half the test budget, across all rule categories, under a partial credit F1 metric, and when the selected script is further revised against the collected tests.
Conclusion
Functionally equivalent DRC scripts rarely look alike, so they have to be graded by execution. For a niche language like KLayout’s, putting the documentation in context helped more than any choice of selection method. When selecting among candidates, where the tests are aimed matters more than how many there are, and noisy self-generated labels are still useful as long as a final judge looks at the tests that separate the candidates.
Rule2DRC measures inference-time capability. A natural next step is training, for example with reinforcement learning on workflows where an agent writes, runs, and revises scripts against the engine. The benchmark, the evaluation pipeline, and all implemented methods are on GitHub, and the paper is on arXiv.