Skip to main content

ModelProcessingError

Raised when an error occurs during model processing.

ModelManager

ModelManager choosing a model from provided list. Models are picked according to defined strategy. Parameters:
  • models (Union[BaseModelBackend, List[BaseModelBackend]]): model backend or list of model backends (e.g., model instances, APIs)
  • scheduling_strategy (str): name of function that defines how to select the next model. (default: :str:round_robin)

init

model_type

Returns: Union[ModelType, str]: Current model type.

model_config_dict

Returns: Dict[str, Any]: Config dictionary of the current model.

model_config_dict

Set model_config_dict to the current model. Parameters:
  • model_config_dict (Dict[str, Any]): Config dictionary to be set at current model.

current_model_index

Returns: int: index of current model in given list of models.

num_models

Returns: int: The number of models available in the model manager.

token_limit

Returns: int: The maximum token limit for the given model.

token_counter

Returns: BaseTokenCounter: The token counter following the model’s tokenization style.

add_strategy

Add a scheduling strategy method provided by user in case when none of existent strategies fits. When custom strategy is provided, it will be set as “self.scheduling_strategy” attribute. Parameters:
  • name (str): The name of the strategy.
  • strategy_fn (Callable): The scheduling strategy function.

round_robin

Returns: BaseModelBackend for processing incoming messages.

always_first

Returns: BaseModelBackend for processing incoming messages.

random_model

Returns: BaseModelBackend for processing incoming messages.

run

Process a list of messages by selecting a model based on the scheduling strategy. Sends the entire list of messages to the selected model, and returns a single response. Parameters:
  • messages (List[OpenAIMessage]): Message list with the chat history in OpenAI API format.
Returns: Union[ChatCompletion, Stream[ChatCompletionChunk], ChatCompletionStreamManager[BaseModel]]: ChatCompletion in the non-stream mode, or Stream[ChatCompletionChunk] in the stream mode, or ChatCompletionStreamManager[BaseModel] for structured-output stream.