​
Evaluator

class Evaluator:

Evaluator class to evaluate messages using a reward model and filter data based on the scores.

Parameters:

  • reward_model (BaseRewardModel): A reward model to evaluate messages.

​
init

def __init__(self, reward_model: BaseRewardModel):

​
evaluate

def evaluate(self, messages: List[Dict[str, str]]):

Evaluate the messages using the reward model.

Parameters:

  • messages (List[Dict[str, str]]): A list of messages where each message is a dictionary with ‘role’ and ‘content’.

Returns:

Dict[str, float]: A dictionary mapping score types to their values.

​
filter_data

def filter_data(
    self,
    messages: List[Dict[str, str]],
    thresholds: Dict[str, float]
):

Filter messages based on the scores.

Parameters:

  • messages (List[Dict[str, str]]): A list of messages where each message is a dictionary with ‘role’ and ‘content’.
  • thresholds (Dict[str, float]): A dictionary mapping score types to their values.

Returns:

bool: A boolean indicating whether the messages pass the filter.