Skip to main content

​
ImageAnalysisToolkit

class ImageAnalysisToolkit(BaseToolkit):
A toolkit for comprehensive image analysis and understanding. The toolkit uses vision-capable language models to perform these tasks.

​
init

def __init__(
    self,
    model: Optional[BaseModelBackend] = None,
    timeout: Optional[float] = None
):
Initialize the ImageAnalysisToolkit. Parameters:
  • model (Optional[BaseModelBackend]): The model backend to use for image analysis tasks. This model should support processing images for tasks like image description and visual question answering. If None, a default model will be created using ModelFactory. (default: :obj:None)
  • timeout (Optional[float]): The timeout value for API requests in seconds. If None, no timeout is applied. (default: :obj:None)

​
image_to_text

def image_to_text(self, image_path: str, sys_prompt: Optional[str] = None):
Generates textual description of an image with optional custom prompt. Parameters:
  • image_path (str): Local path or URL to an image file.
  • sys_prompt (Optional[str]): Custom system prompt for the analysis. (default: :obj:None)
Returns: str: Natural language description of the image.

​
ask_question_about_image

def ask_question_about_image(
    self,
    image_path: str,
    question: str,
    sys_prompt: Optional[str] = None
):
Answers image questions with optional custom instructions. Parameters:
  • image_path (str): Local path or URL to an image file.
  • question (str): Query about the image content.
  • sys_prompt (Optional[str]): Custom system prompt for the analysis. (default: :obj:None)
Returns: str: Detailed answer based on visual understanding

​
_load_image

def _load_image(self, image_path: str):
Loads an image from either local path or URL. Parameters:
  • image_path (str): Local path or URL to image.
Returns: Image.Image: Loaded PIL Image object.

​
_analyze_image

def _analyze_image(
    self,
    image_path: str,
    prompt: str,
    system_message: BaseMessage
):
Core analysis method handling image loading and processing. Parameters:
  • image_path (str): Image location.
  • prompt (str): Analysis query/instructions.
  • system_message (BaseMessage): Custom system prompt for the analysis.
Returns: str: Analysis result or error message.

​
get_tools

def get_tools(self):
Returns: List[FunctionTool]: A list of FunctionTool objects representing the functions in the toolkit.
⌘I