Type alias RunEvalConfig<T, U>

RunEvalConfig<T, U>: {
    customEvaluators?: U[];
    evalLlm?: string;
    evaluators?: (T | EvalConfig)[];
    formatEvaluatorInputs?: EvaluatorInputFormatter;
}

Configuration class for running evaluations on datasets.

Type Parameters

Type declaration

  • Optional customEvaluators?: U[]

    Custom evaluators to apply to a dataset run. Each evaluator is provided with a run trace containing the model outputs, as well as an "example" object representing a record in the dataset.

  • Optional evalLlm?: string

    The language model specification for evaluators that require one.

  • Optional evaluators?: (T | EvalConfig)[]

    LangChain evaluators to apply to a dataset run. You can optionally specify these by name, or by configuring them with an EvalConfig object.

  • Optional formatEvaluatorInputs?: EvaluatorInputFormatter

    Convert the evaluation data into a format that can be used by the evaluator. By default, we pass the first value of the run.inputs, run.outputs (predictions), and references (example.outputs)

    Returns

    The prepared data.

Remarks

RunEvalConfig in LangSmith is a configuration class for running evaluations on datasets. Its primary purpose is to define the parameters and evaluators that will be applied during the evaluation of a dataset. This configuration can include various evaluators, custom evaluators, and different keys for inputs, predictions, and references.

Typeparam

T - The type of evaluators.

Typeparam

U - The type of custom evaluators.

Settings

Member Visibility

Theme

Generated using TypeDoc