LLM Rules

LLM rules send the git diff to a language model with a custom prompt and evaluate the response.

When to use LLM rules

Use llm when the check requires understanding intent — something that can't be expressed as a pattern or structure. Examples:

• "Does this change include a test for the new behavior?"
• "Is the PR description accurate relative to the actual diff?"
• "Does this function have a clear single responsibility?"

Full example

rules:
  - name: has-test-coverage
    description: New features should include tests
    type: llm
    prompt: |
      Review this git diff. If new functions or classes are added,
      check whether corresponding tests are also present in the diff.
      Answer YES if tests are present or no new code was added.
      Answer NO if new code was added without tests.
    model: qwen/qwen-2.5-72b-instruct
    on_fail: warn

Field reference

Field	Required	Default	Description
name	Yes	—	Unique rule identifier
description	Yes	—	Shown in CLI output
type	Yes	—	Must be llm
prompt	Yes	—	Instruction sent to the model with the diff appended
model	No	qwen/qwen-2.5-72b-instruct	Any OpenRouter model string
on_fail	No	warn	warn or fail
depends_on	No	—	Rule name; skip this rule if that rule did not pass

How it works

1. Comply assembles the diff with your prompt prepended
2. The combined message is sent to the specified model via OpenRouter
3. The model response is evaluated: YES → PASS, NO → FAIL/WARN
4. Partial matches (e.g. "Yes, but...") are treated as PASS

Prompt writing tips

• Be explicit about the expected response format: "Answer YES or NO."
• Give the model enough context: describe what "good" looks like
• Avoid compound questions — split them into separate rules
• Test with comply check on a representative diff before committing

Model selection

Any model available on OpenRouter can be used. The default qwen/qwen-2.5-72b-instruct is fast and cost-effective for most checks.

For complex semantic reasoning, consider a larger model:

model: anthropic/claude-3-5-sonnet

Next steps

Regex rules for fast, local pattern matching.