Reference for evaluate(…): dataset, runner, scorers, and every option.
An eval file calls evaluate(...), then you run it:
promptlayer eval run path/to/your.eval.py
Use evaluate for sync runners and aevaluate (Python) for async runners; JavaScript always uses async evaluate. For full harness setups (OpenAI Agents, Claude, LangChain, and more), see the Quickstart.
evaluate(...) options
name is the first positional argument; everything else is keyword-only.
Do not pass a top-level sheet_id / sheetId. To read cases from a dashboard sheet, use dataset.sheet_id / dataset.sheetId instead.
Dataset
Cases feed the runner and scorers. Datasets covers inline cases and dashboard Tables. Each case supports:
Field
Required
Purpose
input
✅
Passed to the runner (column input).
expected
—
Expected output for Contains, Compare, … (column expected).
expected_trace / expectedTrace
—
Trajectory config — which tools should run, not the answer text.
other keys
—
Custom TEXT columns (exact title). No reserved or supporting-column name collisions.
Runner
The runner is any callable that takes an input and returns a final value (not a stream) — no framework allowlist.For Trajectory, tools must appear as Tool: <name> spans on the imported Trace; use a framework helper or traceTool. The runner and tool-span rules live on Runner.
Columns and scorers
Columns are optional processing steps that run before scorers. Declare them with Columns; types match Column Types.
Scorers become scorecard steps on the experiment sheet. Declare them with Scorers. Rollup matches Scorecards; check types match Column Types.
What you get back
On success, the SDK creates (or reuses) a Table, writes a dedicated experiment sheet, configures your scorecard, and returns an EvalResult with a dashboard URL. For failure handling and passing_score, see Results and failures.
Was this page helpful?
Assistant
Responses are generated using AI and may contain mistakes.