​
Evaluator

class Evaluator:
Evaluator class to evaluate messages using a reward model and filter data based on the scores. Parameters:
  • reward_model (BaseRewardModel): A reward model to evaluate messages.

​
init

def __init__(self, reward_model: BaseRewardModel):

​
evaluate

def evaluate(self, messages: List[Dict[str, str]]):
Evaluate the messages using the reward model. Parameters:
  • messages (List[Dict[str, str]]): A list of messages where each message is a dictionary with ‘role’ and ‘content’.
Returns: Dict[str, float]: A dictionary mapping score types to their values.

​
filter_data

def filter_data(
    self,
    messages: List[Dict[str, str]],
    thresholds: Dict[str, float]
):
Filter messages based on the scores. Parameters:
  • messages (List[Dict[str, str]]): A list of messages where each message is a dictionary with ‘role’ and ‘content’.
  • thresholds (Dict[str, float]): A dictionary mapping score types to their values.
Returns: bool: A boolean indicating whether the messages pass the filter.