From a rule to a useful test.

Reading your results

• Removal detected: the tests reliably caught a missing instruction. Keep the test and rerun it when your setup changes.
• Removal missed: the tests still passed without the instruction. Review the test; the AI may also keep the behavior without being told.
• No test linked: no test was confidently matched to this rule. Add one before checking rule removal.

The example uses saved, repeatable results. It demonstrates the method, not the performance of a live model.

Developer setup

Ready to test your own AI? This part uses a terminal, your prompt, and your test suite. A developer can help connect your project.

Try the interactive demo immediately with no setup or API key. For the CLI, npm publication is pending: use a source checkout with Git and Node.js 22+.

git clone https://github.com/aravinda-1402/causeval.git
cd causeval
npx --yes pnpm@10.17.1 install --frozen-lockfile
npx --yes pnpm@10.17.1 build

It runs the bundled support-agent example end to end and prints one causally covered rule, one pseudo-covered rule and one uncovered rule, with the baseline and mutant runs behind each.

Your own project

npx --yes pnpm@10.17.1 causeval init --dir my-agent
npx --yes pnpm@10.17.1 causeval scan --config my-agent/causeval.config.ts
npx --yes pnpm@10.17.1 causeval verify --config my-agent/causeval.config.ts

init writes a config, prompt and eval suite pointed at the bundled fixture, so both commands work immediately. Change provider and set CAUSEVAL_MODEL to analyse your own prompt with your own model. Run commands from the checkout and use --config to point at your project.

No eval suite yet

That is a supported starting point. scan reports the behavioral contract and its severity breakdown instead of failing, then:

npx --yes pnpm@10.17.1 causeval init --dir prompt-only --prompt-only
npx --yes pnpm@10.17.1 causeval scan --config prompt-only/causeval.config.ts
npx --yes pnpm@10.17.1 causeval generate --config prompt-only/causeval.config.ts
npx --yes pnpm@10.17.1 causeval review --config prompt-only/causeval.config.ts --list

Generated cases are written as GENERATED — UNREVIEWED and are excluded from every coverage metric until you review and accept them. Use review --accept <id> with the same config. Generated or custom evals require your own provider or runner for verification; the fixture executes only bundled evals.

Two metrics. One important gap.

Verification repeats each mapped eval three times by default, and at least twice always. Baseline pass rate must be at least 0.80, per eval as well as in aggregate. Detection effect (baseline minus mutant pass rate) must be at least 0.50. Unstable baselines are flaky; ambiguous results stay indeterminate; a scan never claims causal coverage.

Pseudo-coverage means one precise thing: under the tested model and configuration, this eval did not detect removal of this instruction. It is not proof the rule is untested. A base model may keep the behavior without being told, and another rule may still enforce it. Both confounds are attached to every pseudo-covered result.

Configuration

// causeval.config.ts
export default {
  prompt: './prompts/system.md',
  evals: ['./evals/**/*.yaml'],
  provider: { type: 'openai', model: process.env.CAUSEVAL_MODEL },
  thresholds: {
    mappingConfidence: 0.70,
    minimumTraceCoverage: 0.70,
    minimumCausalCoverage: 0.50,
    maximumHighRiskUncovered: 0,
  },
  causal: {
    runsPerEval: 3,
    minimumBaselinePassRate: 0.80,
    minimumDetectionEffect: 0.50,
    strictMutationValidation: false,
  },
  runnerTimeoutMs: 30000,
};

--strict re-extracts mutants to verify unrelated behaviors remain. --no-cache bypasses extraction and mapping cache. Execution outcomes are never reused across repetitions.

Native evals

Use versioned YAML or JSON. Each eval needs a unique ID, input or messages, and an expected behavior. Literal assertions and semantic judging must both pass.

version: 1
evals:
  - id: confirmation-before-email
    input: Email my manager that I will be late.
    expected:
      behavior: Ask for explicit confirmation before sending.
      mustContain:
        - confirm
    tags:
      - external-action
      - confirmation

Deterministic assertions run first and cannot be overruled by a judge: mustContain, mustNotContain, mustMatch and mustNotMatch (regular expressions), json and jsonSchema. An eval can skip LLM judging entirely with judge: false when it declares at least one of them. Prefer deterministic checks: they are free, reproducible, and cannot be talked out of a verdict by the output they score.

Assertions are case-sensitive. A separate judge provider uses the same options as the analysis provider and never sees the system prompt, so it cannot tell a baseline run from a mutant run. Native evaluation executes no tools; use a custom runner for sandboxed tools.

Bring your existing eval suite

A local runner can wrap Promptfoo, DeepEval, pytest, or an internal agent. CausEval sends JSON through stdin and expects JSON on stdout.

causeval verify --runner "node ./scripts/run-evals.js"

// stdin
{
  "promptPath": "/temporary/prompt.md",
  "evalIds": ["confirmation-before-email"],
  "runId": "unique-run-id",
  "dryRun": true
}

// stdout
{
  "results": [
    { "id": "confirmation-before-email", "passed": false }
  ]
}

Return exactly one result per requested ID. Execution errors are indeterminate analysis, not failed evals. Runners are terminated on timeout and temporary files are removed.

Your model. Your infrastructure.

No model name is hard-coded in the core. Provider contracts are tested with mock HTTP responses. Live model behavior requires your credentials or running endpoint.

Inspect and share the evidence

causeval verify --suggest
causeval report --open
causeval badge
causeval diff before-report.json after-report.json
causeval diff origin/main HEAD
causeval verify --fail-on threshold

Versioned JSON and offline HTML include source clauses, mappings, mutation diffs, repeated outcomes, classifications, warnings, and suggested tests. The SVG badge is a local file you can commit.

Git comparison analyzes both prompts against the current eval suite. Stable keys match normalized semantics. Uncertain changes are reported as added and removed rules.

Protect the contract in CI

The composite action lives at packages/action. Install dependencies and build the CLI before invoking it. It produces outputs, a job summary, an artifact, and an optional updatable PR comment.

- uses: ./packages/action
  with:
    config: examples/support-agent/causeval.config.ts
    mode: verify
    fail-on: threshold
    comment: 'false'

Cloud credentials must be repository secrets. Normal CI uses fixtures with no paid API. Comments require opt-in and pull-requests: write permission. See docs/github-action.md for full setup.

Evidence with clear boundaries

• No telemetry, accounts, database, or API keys on the website.
• Only relevant prompt and eval content goes to the configured provider.
• Local cache and reports may contain sensitive prompt content. Keep private data out of public Git history.
• Secret redaction is best-effort; it cannot detect every secret.
• Small samples, imperfect extraction, overlapping instructions, model priors, and weak judges limit causal conclusions.

CausEval provides testing evidence, not proof of correctness, security, safety, compliance, or absence of harmful behavior.