> ## Documentation Index
> Fetch the complete documentation index at: https://docs.promptlayer.com/llms.txt
> Use this file to discover all available pages before exploring further.

# Scorers

> Add scorecard checks to evaluate(...) with typed helpers.

Pass scorers into `evaluate(...)` to grade each case. On the experiment sheet they show up as scorecard evaluators (Contains, Trajectory, and the rest).

Thresholds, weights, and the overall score work the same as [Scorecards](/features/tables/scorecards) in Tables. The underlying check types are documented in [Column Types](/features/tables/column-types).

## Scorers at a glance

| Scorer | Python | JavaScript | Checks |
| - | - | - | - |
| [Compare](#compare) | `compare_scorer` | `compareScorer` | A column matches a literal or another column (default `Output` vs `expected`) |
| [Contains](#contains) | `contains_scorer` | `containsScorer` | A column contains a literal or another column's value |
| [Regex](#regex) | `regex_scorer` | `regexScorer` | A column matches a regular expression |
| [Count](#count) | `count_scorer` | `countScorer` | Character / word / sentence / paragraph count within bounds |
| [Assert Valid](#assert-valid) | `assert_valid_scorer` | `assertValidScorer` | A column is a valid object, number, or SQL |
| [LLM Assertion](#llm-assertion) | `llm_assertion_scorer` | `llmAssertionScorer` | An LLM judge grades a column |
| [Trajectory](#trajectory) | `trajectory_scorer` | `trajectoryScorer` | Tool-call sequence matches accepted scenarios |
| [Custom Code](#custom-code) | named fn / `code_execution_column` | named fn / `codeExecutionColumn` | Your own scoring logic |

Every typed helper also accepts [shared scoring options](#shared-scoring-options) — weight, required, and thresholds.

## Shared scoring options

Every scorer produces a score in `[0, 1]`. These three options — accepted by every typed helper — control how much a scorer counts and where its pass/fail line sits. They behave exactly like [Scorecards](/features/tables/scorecards) in Tables.

* **Weight** — how much this scorer contributes to the overall score. The scorecard is a weighted mean, so `weight=2` pulls twice as hard as the default `weight=1`. Use it to make the checks you care about most dominate the result.
* **Required** — if `true`, this scorer failing fails the whole scorecard, no matter how well the weighted average comes out. Use it for must-pass checks (e.g. the agent called the right tools).
* **Thresholds** — the per-scorer cutoffs that decide pass vs. warn vs. fail, each in `[0, 1]` (default pass `0.8`, warn `0.6`). Raise the pass threshold for a stricter bar on that check.

| Option | Python | JavaScript | Meaning |
| - | - | - | - |
| Weight | `weight` | `weight` | How much this step counts (default `1`) |
| Required | `required` | `required` | If `true`, failing this step can fail the scorecard |
| Thresholds | `thresholds` | `thresholds` | Per-step pass / warn cutoffs in `[0, 1]` |
| Pass alias | `pass_threshold` | `passThreshold` | Sets `thresholds.pass` |
| Warn alias | `failure_threshold` | `failureThreshold` | Sets `thresholds.warn` |

This support agent example weights the trajectory check higher, marks it required, and sets a pass threshold:

<CodeGroup>
  ```python Python theme={null}
  from agents import Agent, Runner, function_tool
  from promptlayer import evaluate, contains_scorer, trajectory_scorer
  from promptlayer.integrations.openai_agents import instrument_openai_agents

  instrument_openai_agents()

  @function_tool
  def lookup_user(email: str) -> str:
      """Look up a user account by email."""
      return f"Found user for {email}."

  @function_tool
  def send_reset(email: str) -> str:
      """Send a password reset email."""
      return f"Reset link sent to {email}."

  agent = Agent(
      name="Support agent",
      instructions=(
          "Help with password resets. Call lookup_user, then send_reset, "
          "then apologize briefly and confirm the reset was sent."
      ),
      model="gpt-5.6",
      tools=[lookup_user, send_reset],
  )

  def run_agent(user_message: str) -> str:
      return str(Runner.run_sync(agent, user_message).final_output)

  evaluate(
      "support-bot",
      dataset=[{"input": "Please reset my password for user@example.com"}],
      runner=run_agent,
      scorers=[
          contains_scorer(
              "Has apology",
              source_column="Output",
              expected="sorry",
              weight=1.0,
          ),
          trajectory_scorer(
              expected=[["lookup_user", "send_reset"]],
              weight=2.5,
              required=True,
              pass_threshold=0.9,
          ),
      ],
  )
  ```

  ```javascript JavaScript theme={null}
  import { Agent, run, tool } from "@openai/agents";
  import { instrumentOpenAIAgents } from "promptlayer/openai-agents";
  import { evaluate, containsScorer, trajectoryScorer } from "promptlayer";
  import { z } from "zod";

  await instrumentOpenAIAgents();

  const lookupUser = tool({
    name: "lookup_user",
    description: "Look up a user account by email.",
    parameters: z.object({ email: z.string() }),
    execute: async ({ email }) => `Found user for ${email}.`,
  });

  const sendReset = tool({
    name: "send_reset",
    description: "Send a password reset email.",
    parameters: z.object({ email: z.string() }),
    execute: async ({ email }) => `Reset link sent to ${email}.`,
  });

  const agent = new Agent({
    name: "Support agent",
    instructions:
      "Help with password resets. Call lookup_user, then send_reset, then apologize briefly and confirm the reset was sent.",
    model: "gpt-5.6",
    tools: [lookupUser, sendReset],
  });

  async function runAgent(userMessage) {
    return String((await run(agent, userMessage)).finalOutput);
  }

  await evaluate("support-bot", {
    dataset: [{ input: "Please reset my password for user@example.com" }],
    runner: runAgent,
    scorers: [
      containsScorer({
        title: "Has apology",
        sourceColumn: "Output",
        expected: "sorry",
        weight: 1.0,
      }),
      trajectoryScorer({
        expected: [["lookup_user", "send_reset"]],
        weight: 2.5,
        required: true,
        passThreshold: 0.9,
      }),
    ],
  });
  ```
</CodeGroup>

## Scorer types

Each helper below maps to a scorecard check type in [Column Types](/features/tables/column-types). All of them accept the [shared scoring options](#shared-scoring-options) above.

### Compare

Compare a column to a literal or another column. By default that is `Output` vs the `expected` column as strings. Pass the left column as `source_column` / `sourceColumn`, then exactly one of `expected` (literal) or `expected_column` / `expectedColumn`. If you omit both expected options, the SDK uses column `expected`. Behavior matches the Compare type in [Column Types](/features/tables/column-types).

| Key | Required | Default |
| - | - | - |
| `source_column` / `sourceColumn` | no | `"Output"` |
| `expected` | one of | literal value |
| `expected_column` / `expectedColumn` | one of | `"expected"` when both omitted |
| `comparison_type` / `comparisonType` | no | `"STRING"` |

Set how they compare with `comparison_type` / `comparisonType`:

| Mode | What it does | Example |
| - | - | - |
| `STRING` (default) | Exact string match | `"STRING"` |
| `JSON` | Compare JSON values; optional path | `{"type": "JSON", "json_path": "$.id"}` |
| `NUMBER` | Compare numbers with an operator (`EQ` by default) | `{"type": "NUMBER", "operator": "GTE"}` |

Number operators: `EQ`, `NE`, `GT`, `GTE`, `LT`, `LTE`.

<CodeGroup>
  ```python Python theme={null}
  from openai import OpenAI
  from promptlayer import evaluate, compare_scorer

  client = OpenAI()

  def run_llm(user_message: str) -> str:
      completion = client.chat.completions.create(
          model="gpt-5.6",
          messages=[
              {
                  "role": "system",
                  "content": "Reply with exactly one word: hello",
              },
              {"role": "user", "content": str(user_message)},
          ],
      )
      return completion.choices[0].message.content or ""

  evaluate(
      "compare-demo",
      dataset=[{"input": "Greet the user.", "expected": "hello"}],
      runner=run_llm,
      scorers=[
          compare_scorer(
              "Exact match",
              source_column="Output",
              expected_column="expected",
              comparison_type="STRING",
          )
      ],
  )
  ```

  ```javascript JavaScript theme={null}
  import OpenAI from "openai";
  import { evaluate, compareScorer } from "promptlayer";

  const client = new OpenAI();

  async function runLlm(userMessage) {
    const completion = await client.chat.completions.create({
      model: "gpt-5.6",
      messages: [
        { role: "system", content: "Reply with exactly one word: hello" },
        { role: "user", content: String(userMessage) },
      ],
    });
    return completion.choices[0].message.content ?? "";
  }

  await evaluate("compare-demo", {
    dataset: [{ input: "Greet the user.", expected: "hello" }],
    runner: runLlm,
    scorers: [
      compareScorer({
        title: "Exact match",
        sourceColumn: "Output",
        expectedColumn: "expected",
        comparisonType: "STRING",
      }),
    ],
  });
  ```
</CodeGroup>

### Contains

Check whether a column contains a literal or another column's value. Pass exactly one of `expected` or `expected_column` / `expectedColumn`.

| Key | Required | Default |
| - | - | - |
| `source_column` / `sourceColumn` | no | `"Output"` |
| `expected` | one of | literal substring |
| `expected_column` / `expectedColumn` | one of | column title |

<CodeGroup>
  ```python Python theme={null}
  from openai import OpenAI
  from promptlayer import evaluate, contains_scorer

  client = OpenAI()

  def run_llm(user_message: str) -> str:
      completion = client.chat.completions.create(
          model="gpt-5.6",
          messages=[
              {
                  "role": "system",
                  "content": "You are a support agent. Apologize and help in one short sentence.",
              },
              {"role": "user", "content": str(user_message)},
          ],
      )
      return completion.choices[0].message.content or ""

  evaluate(
      "contains-demo",
      dataset=[{"input": "I cannot log in to my account."}],
      runner=run_llm,
      scorers=[contains_scorer(source_column="Output", expected="sorry")],
  )
  ```

  ```javascript JavaScript theme={null}
  import OpenAI from "openai";
  import { evaluate, containsScorer } from "promptlayer";

  const client = new OpenAI();

  async function runLlm(userMessage) {
    const completion = await client.chat.completions.create({
      model: "gpt-5.6",
      messages: [
        {
          role: "system",
          content: "You are a support agent. Apologize and help in one short sentence.",
        },
        { role: "user", content: String(userMessage) },
      ],
    });
    return completion.choices[0].message.content ?? "";
  }

  await evaluate("contains-demo", {
    dataset: [{ input: "I cannot log in to my account." }],
    runner: runLlm,
    scorers: [containsScorer({ sourceColumn: "Output", expected: "sorry" })],
  });
  ```
</CodeGroup>

### Regex

Match a column against a regular expression.

| Key | Required | Default |
| - | - | - |
| `source_column` / `sourceColumn` | no | `"Output"` |
| `regex_pattern` / `regexPattern` | yes | — |

<CodeGroup>
  ```python Python theme={null}
  from openai import OpenAI
  from promptlayer import evaluate, regex_scorer

  client = OpenAI()

  def run_llm(user_message: str) -> str:
      completion = client.chat.completions.create(
          model="gpt-5.6",
          messages=[
              {
                  "role": "system",
                  "content": (
                      "You are an order lookup bot. Reply with only an order id "
                      "like order-123."
                  ),
              },
              {"role": "user", "content": str(user_message)},
          ],
      )
      return completion.choices[0].message.content or ""

  evaluate(
      "regex-demo",
      dataset=[{"input": "What is the order id for my headphones?"}],
      runner=run_llm,
      scorers=[regex_scorer(source_column="Output", regex_pattern=r"order-\d+")],
  )
  ```

  ```javascript JavaScript theme={null}
  import OpenAI from "openai";
  import { evaluate, regexScorer } from "promptlayer";

  const client = new OpenAI();

  async function runLlm(userMessage) {
    const completion = await client.chat.completions.create({
      model: "gpt-5.6",
      messages: [
        {
          role: "system",
          content:
            "You are an order lookup bot. Reply with only an order id like order-123.",
        },
        { role: "user", content: String(userMessage) },
      ],
    });
    return completion.choices[0].message.content ?? "";
  }

  await evaluate("regex-demo", {
    dataset: [{ input: "What is the order id for my headphones?" }],
    runner: runLlm,
    scorers: [regexScorer({ sourceColumn: "Output", regexPattern: "order-\\d+" })],
  });
  ```
</CodeGroup>

### Count

Score by character, word, sentence, or paragraph count bounds. Require at least one of `min_count` / `minCount` or `max_count` / `maxCount`. Backend `type` values: `chars`, `words`, `sentences`, `paragraphs`.

| Key | Required | Default |
| - | - | - |
| `source_column` / `sourceColumn` | no | `"Output"` |
| `type` | no | `"chars"` |
| `min_count` / `minCount` | one of min/max | — |
| `max_count` / `maxCount` | one of min/max | — |

<CodeGroup>
  ```python Python theme={null}
  from openai import OpenAI
  from promptlayer import evaluate, count_scorer

  client = OpenAI()

  def run_llm(user_message: str) -> str:
      completion = client.chat.completions.create(
          model="gpt-5.6",
          messages=[
              {
                  "role": "system",
                  "content": "Answer in one short sentence of at most five words.",
              },
              {"role": "user", "content": str(user_message)},
          ],
      )
      return completion.choices[0].message.content or ""

  evaluate(
      "count-demo",
      dataset=[{"input": "How do I reset my password?"}],
      runner=run_llm,
      scorers=[count_scorer(source_column="Output", type="words", min_count=1, max_count=5)],
  )
  ```

  ```javascript JavaScript theme={null}
  import OpenAI from "openai";
  import { evaluate, countScorer } from "promptlayer";

  const client = new OpenAI();

  async function runLlm(userMessage) {
    const completion = await client.chat.completions.create({
      model: "gpt-5.6",
      messages: [
        {
          role: "system",
          content: "Answer in one short sentence of at most five words.",
        },
        { role: "user", content: String(userMessage) },
      ],
    });
    return completion.choices[0].message.content ?? "";
  }

  await evaluate("count-demo", {
    dataset: [{ input: "How do I reset my password?" }],
    runner: runLlm,
    scorers: [
      countScorer({ sourceColumn: "Output", type: "words", minCount: 1, maxCount: 5 }),
    ],
  });
  ```
</CodeGroup>

### Assert Valid

Validate a column as object, number, or SQL. There is no `json` / `boolean` / `string` assert type.

| Key | Required | Default | Allowed |
| - | - | - | - |
| `source_column` / `sourceColumn` | no | `"Output"` | column title |
| `type` | yes | `"object"` | `object` \| `number` \| `sql` |

<CodeGroup>
  ```python Python theme={null}
  from openai import OpenAI
  from promptlayer import evaluate, assert_valid_scorer

  client = OpenAI()

  def run_llm(user_message: str) -> str:
      completion = client.chat.completions.create(
          model="gpt-5.6",
          messages=[
              {
                  "role": "system",
                  "content": (
                      "Reply with a JSON object only, no markdown. "
                      'Example: {"status":"ok","ticket_id":101}'
                  ),
              },
              {"role": "user", "content": str(user_message)},
          ],
      )
      return completion.choices[0].message.content or ""

  evaluate(
      "assert-valid-demo",
      dataset=[{"input": "Create a support ticket for a billing issue."}],
      runner=run_llm,
      scorers=[assert_valid_scorer(source_column="Output", type="object")],
  )
  ```

  ```javascript JavaScript theme={null}
  import OpenAI from "openai";
  import { evaluate, assertValidScorer } from "promptlayer";

  const client = new OpenAI();

  async function runLlm(userMessage) {
    const completion = await client.chat.completions.create({
      model: "gpt-5.6",
      messages: [
        {
          role: "system",
          content:
            'Reply with a JSON object only, no markdown. Example: {"status":"ok","ticket_id":101}',
        },
        { role: "user", content: String(userMessage) },
      ],
    });
    return completion.choices[0].message.content ?? "";
  }

  await evaluate("assert-valid-demo", {
    dataset: [{ input: "Create a support ticket for a billing issue." }],
    runner: runLlm,
    scorers: [assertValidScorer({ sourceColumn: "Output", type: "object" })],
  });
  ```
</CodeGroup>

### LLM Assertion

Score a column with an LLM judge prompt. Pass exactly one of `prompt` or `prompt_source` / `promptSource`.

| Key | Required | Default |
| - | - | - |
| `source_column` / `sourceColumn` | no | `"Output"` |
| `prompt` | one of | literal prompt text |
| `prompt_source` / `promptSource` | one of | column title |
| `variable_mappings` / `variableMappings` | no | Prompt variable → column title |

<CodeGroup>
  ```python Python theme={null}
  from openai import OpenAI
  from promptlayer import evaluate, llm_assertion_scorer

  client = OpenAI()

  def run_llm(user_message: str) -> str:
      completion = client.chat.completions.create(
          model="gpt-5.6",
          messages=[
              {
                  "role": "system",
                  "content": "Answer the user's question in one short factual sentence.",
              },
              {"role": "user", "content": str(user_message)},
          ],
      )
      return completion.choices[0].message.content or ""

  evaluate(
      "llm-assert-demo",
      dataset=[{"input": "Why is the sky blue?"}],
      runner=run_llm,
      scorers=[
          llm_assertion_scorer(
              source_column="Output",
              prompt="Is this factually reasonable? Reply yes or no.",
          )
      ],
  )
  ```

  ```javascript JavaScript theme={null}
  import OpenAI from "openai";
  import { evaluate, llmAssertionScorer } from "promptlayer";

  const client = new OpenAI();

  async function runLlm(userMessage) {
    const completion = await client.chat.completions.create({
      model: "gpt-5.6",
      messages: [
        {
          role: "system",
          content: "Answer the user's question in one short factual sentence.",
        },
        { role: "user", content: String(userMessage) },
      ],
    });
    return completion.choices[0].message.content ?? "";
  }

  await evaluate("llm-assert-demo", {
    dataset: [{ input: "Why is the sky blue?" }],
    runner: runLlm,
    scorers: [
      llmAssertionScorer({
        sourceColumn: "Output",
        prompt: "Is this factually reasonable? Reply yes or no.",
      }),
    ],
  });
  ```
</CodeGroup>

### Trajectory

Scores `Tool: <name>` spans from the imported `Trace` against accepted scenarios. Any `runner` works if those spans exist — emit them with a framework helper or [`traceTool`](/features/observability/traces/manual-tracing#trace-tools) (see [Runner](/sdks/evals/agent-tracing)), or copy a harness from the [Quickstart](/sdks/evals/quickstart).

Pass **exactly one of** inline `expected` (list of tool-name lists), or `expected_column` / `expectedColumn` (usually `"expected_trace"` / `"expectedTrace"`).

| Key | Required | Default |
| - | - | - |
| `source_column` / `sourceColumn` | no | `"Trace"` |
| `expected` | one of | list of non-empty tool-name lists |
| `expected_column` / `expectedColumn` | one of | column title |
| `mode` | no | `"strict"` |

| Mode | Behavior |
| - | - |
| `strict` (default) | Observed tools must equal the scenario exactly |
| `non_strict` | Required tools appear in order as a subsequence |

Inline `expected`:

<CodeGroup>
  ```python Python theme={null}
  from agents import Agent, Runner, function_tool
  from promptlayer import evaluate, trajectory_scorer, contains_scorer
  from promptlayer.integrations.openai_agents import instrument_openai_agents

  instrument_openai_agents()

  @function_tool
  def get_weather(city: str) -> str:
      """Return demo weather for a city."""
      return f"{city} is 72F and sunny."

  agent = Agent(
      name="Weather agent",
      instructions="Use get_weather, then answer in one short sentence.",
      model="gpt-5.6",
      tools=[get_weather],
  )

  evaluate(
      "trajectory-demo",
      dataset=[{"input": "What is the weather in Tokyo?"}],
      runner=lambda message: str(Runner.run_sync(agent, message).final_output),
      scorers=[
          trajectory_scorer(expected=[["get_weather"]], mode="non_strict"),
          contains_scorer(source_column="Output", expected="72"),
      ],
  )
  ```

  ```javascript JavaScript theme={null}
  import { Agent, run, tool } from "@openai/agents";
  import { instrumentOpenAIAgents } from "promptlayer/openai-agents";
  import { evaluate, trajectoryScorer, containsScorer } from "promptlayer";
  import { z } from "zod";

  await instrumentOpenAIAgents();

  const getWeather = tool({
    name: "get_weather",
    description: "Return demo weather for a city.",
    parameters: z.object({ city: z.string() }),
    execute: async ({ city }) => `${city} is 72F and sunny.`,
  });

  const agent = new Agent({
    name: "Weather agent",
    instructions: "Use get_weather, then answer in one short sentence.",
    model: "gpt-5.6",
    tools: [getWeather],
  });

  await evaluate("trajectory-demo", {
    dataset: [{ input: "What is the weather in Tokyo?" }],
    runner: async (userMessage) => String((await run(agent, userMessage)).finalOutput),
    scorers: [
      trajectoryScorer({ expected: [["get_weather"]], mode: "non_strict" }),
      containsScorer({ sourceColumn: "Output", expected: "72" }),
    ],
  });
  ```
</CodeGroup>

Or read scenarios from the `expected_trace` / `expectedTrace` column. Cell JSON:

```json theme={null}
{
  "accepted_scenarios": [
    { "required_tools": ["get_weather"] }
  ]
}
```

<CodeGroup>
  ```python Python theme={null}
  trajectory_scorer(
      expected_column="expected_trace",
      mode="non_strict",
  )
  ```

  ```javascript JavaScript theme={null}
  trajectoryScorer({
    expectedColumn: "expectedTrace",
    mode: "non_strict",
  })
  ```
</CodeGroup>

Put that JSON on each case as `expected_trace` / `expectedTrace` (or on a dashboard dataset sheet).

Local helpers: `score_trajectory` / `scoreTrajectory`, `diagnose_trajectory_failure` / `diagnoseTrajectoryFailure`, and JS-only `extractTrajectoryToolNames`.

### Custom Code

Prefer passing a **named function by reference** in `scorers`. The SDK serializes it to a `CODE_EXECUTION` scorecard step.

Parameter names map to columns:

| Param | Column |
| - | - |
| `input` | `input` |
| `output` | `Output` |
| `expected` | `expected` |
| `expected_trace` | `expected_trace` (Python) |
| `trace` | `Trace` |

Any other param name is looked up as a column title of the same name. Return a number (or `{"score": x}`, which is normalized to `x`).

* Use a **named** function — no lambdas / anonymous functions.
* Sync only — async scorers are rejected.
* Default title is the function name with `_` → spaces. Override with `scorer_from_function(fn, title=...)` / `scorerFromFunction(fn, { title })` when needed.

<CodeGroup>
  ```python Python theme={null}
  from openai import OpenAI
  from promptlayer import evaluate

  client = OpenAI()

  def run_llm(user_message: str) -> str:
      completion = client.chat.completions.create(
          model="gpt-5.6",
          messages=[
              {
                  "role": "system",
                  "content": "You are a support agent. Answer in one short sentence.",
              },
              {"role": "user", "content": str(user_message)},
          ],
      )
      return completion.choices[0].message.content or ""

  def exact_match(output, expected):
      return 1 if expected.lower() in (output or "").lower() else 0

  evaluate(
      "fn-scorer-demo",
      dataset=[
          {
              "input": "How do I reset my password?",
              "expected": "password",
          }
      ],
      runner=run_llm,
      scorers=[exact_match],
  )
  ```

  ```javascript JavaScript theme={null}
  import OpenAI from "openai";
  import { evaluate } from "promptlayer";

  const client = new OpenAI();

  async function runLlm(userMessage) {
    const completion = await client.chat.completions.create({
      model: "gpt-5.6",
      messages: [
        {
          role: "system",
          content: "You are a support agent. Answer in one short sentence.",
        },
        { role: "user", content: String(userMessage) },
      ],
    });
    return completion.choices[0].message.content ?? "";
  }

  function exactMatch(output, expected) {
    return String(output || "")
      .toLowerCase()
      .includes(String(expected).toLowerCase())
      ? 1
      : 0;
  }

  await evaluate("fn-scorer-demo", {
    dataset: [
      {
        input: "How do I reset my password?",
        expected: "password",
      },
    ],
    runner: runLlm,
    scorers: [exactMatch],
  });
  ```
</CodeGroup>

#### `code_execution_column` / `codeExecutionColumn`

Use this when you want a raw code string instead of a function. `code` is required. Default `language` is `PYTHON` (Python helper) or `JAVASCRIPT` (JS helper). Row values are available as `data`.

<CodeGroup>
  ```python Python theme={null}
  from openai import OpenAI
  from promptlayer import evaluate, code_execution_column

  client = OpenAI()

  def run_llm(user_message: str) -> str:
      completion = client.chat.completions.create(
          model="gpt-5.6",
          messages=[
              {
                  "role": "system",
                  "content": "You are a support agent. Answer in one short sentence.",
              },
              {"role": "user", "content": str(user_message)},
          ],
      )
      return completion.choices[0].message.content or ""

  evaluate(
      "code-scorer-demo",
      dataset=[
          {
              "input": "How do I reset my password?",
              "expected": "password",
          }
      ],
      runner=run_llm,
      scorers=[
          code_execution_column(
              "exact",
              code=(
                  'return 1 if str(data.get("expected", "")).lower() '
                  'in str(data.get("Output", "")).lower() else 0'
              ),
          )
      ],
  )
  ```

  ```javascript JavaScript theme={null}
  import OpenAI from "openai";
  import { evaluate, codeExecutionColumn } from "promptlayer";

  const client = new OpenAI();

  async function runLlm(userMessage) {
    const completion = await client.chat.completions.create({
      model: "gpt-5.6",
      messages: [
        {
          role: "system",
          content: "You are a support agent. Answer in one short sentence.",
        },
        { role: "user", content: String(userMessage) },
      ],
    });
    return completion.choices[0].message.content ?? "";
  }

  await evaluate("code-scorer-demo", {
    dataset: [
      {
        input: "How do I reset my password?",
        expected: "password",
      },
    ],
    runner: runLlm,
    scorers: [
      codeExecutionColumn("exact", {
        code: 'return String(data["Output"] || "").toLowerCase().includes(String(data["expected"] || "").toLowerCase()) ? 1 : 0;',
      }),
    ],
  });
  ```
</CodeGroup>


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.