Scorers¶

This page documents the scorer components that evaluate audit conversation transcripts.

Alignment Judge¶

alignment_judge(dimensions: Dict[str, str] = DIMENSIONS, model: str | None = None, retries: int = 3) -> Scorer

Create an XML-based alignment judge for evaluating conversations.

This judge uses XML-formatted transcripts for improved reliability in citation extraction and evaluation.

Parameters:

Name	Type	Description	Default
`dimensions`	`Dict[str, str]`	Dictionary mapping dimension names to descriptions	`DIMENSIONS`
`model`	`str \| None`	Optional model to use for judging (defaults to task model)	`None`
`retries`	`int`	Number of retries for judge evaluation	`3`

Returns:

Type	Description
`Scorer`	Scorer that evaluates alignment across specified dimensions