> ## Documentation Index
> Fetch the complete documentation index at: https://docs.camel-ai.org/llms.txt
> Use this file to discover all available pages before exploring further.

# Camel.benchmarks.gaia

<a id="camel.benchmarks.gaia" />

<a id="camel.benchmarks.gaia.RetrieverProtocol" />

## RetrieverProtocol

```python theme={"system"}
class RetrieverProtocol(Protocol):
```

Protocol for the retriever class. Any retriever class implementing
this protocol can be used in the benchmark class.

<a id="camel.benchmarks.gaia.RetrieverProtocol.retrieve" />

### retrieve

```python theme={"system"}
def retrieve(
    self,
    query: str,
    contents: List[str],
    **kwargs: Dict[str, Any]
):
```

Retrieve the relevant content for the query.

**Parameters:**

* **query** (str): The query to retrieve the content for.
* **contents** (List\[str]): The list of contents to search in. \*\*kwargs (Dict\[str, Any]): Additional keyword arguments.

**Returns:**

Dict\[str, Any]: The relevant content for the query.

<a id="camel.benchmarks.gaia.RetrieverProtocol.reset" />

### reset

```python theme={"system"}
def reset(self, **kwargs):
```

Reset the retriever.
Some benchmarks may require resetting the retriever
after each query.

**Returns:**

bool: True if the reset was successful, False otherwise.

<a id="camel.benchmarks.gaia.DefaultGAIARetriever" />

## DefaultGAIARetriever

```python theme={"system"}
class DefaultGAIARetriever(AutoRetriever):
```

Default retriever for the GAIA benchmark.
This retriever uses AutoRetriever in camel to retrieve the content based on
the query.

<a id="camel.benchmarks.gaia.DefaultGAIARetriever.retrieve" />

### retrieve

```python theme={"system"}
def retrieve(
    self,
    query: str,
    contents: List[str],
    **kwargs: Any
):
```

Retrieve the content based on the query.

**Parameters:**

* **query** (str): The query to search for.
* **contents** (List\[str]): The list of contents to search from. \*\*kwargs (Any): The keyword arguments to pass to the retriever.

**Returns:**

Dict\[str, Any]: The retrieved content.

<a id="camel.benchmarks.gaia.DefaultGAIARetriever.reset" />

### reset

```python theme={"system"}
def reset(self, **kwargs: Any):
```

Reset the retriever.

**Returns:**

bool: Whether the reset was successful.

<a id="camel.benchmarks.gaia.GAIABenchmark" />

## GAIABenchmark

```python theme={"system"}
class GAIABenchmark(BaseBenchmark):
```

GAIA Benchmark adapted from ["GAIA: a benchmark for General AI
Assistants"](https://huggingface.co/datasets/gaia-benchmark/GAIA).

**Parameters:**

* **data\_dir** (str): The directory to save the data.
* **save\_to** (str): The file to save the results.
* **retriever** (Optional\[RetrieverProtocol]): The retriever to use. (default: :obj:`None`)
* **processes** (int, optional): The number of processes to use. (default: :obj:`1`)

<a id="camel.benchmarks.gaia.GAIABenchmark.__init__" />

### **init**

```python theme={"system"}
def __init__(
    self,
    data_dir: str,
    save_to: str,
    retriever: Optional[RetrieverProtocol] = None,
    processes: int = 1
):
```

Initialize the GAIA benchmark.

**Parameters:**

* **data\_dir** (str): The directory to save the data.
* **save\_to** (str): The file to save the results.
* **retriever** (Optional\[RetrieverProtocol], optional): The retriever to use. (default: :obj:`None`)
* **processes** (int, optional): The number of processes to use for parallel processing. (default: :obj:`1`)

<a id="camel.benchmarks.gaia.GAIABenchmark.download" />

### download

```python theme={"system"}
def download(self):
```

Download the GAIA dataset.

<a id="camel.benchmarks.gaia.GAIABenchmark.load" />

### load

```python theme={"system"}
def load(self, force_download = False):
```

Load the GAIA dataset.

**Parameters:**

* **force\_download** (bool, optional): Whether to force download the data.

<a id="camel.benchmarks.gaia.GAIABenchmark.train" />

### train

```python theme={"system"}
def train(self):
```

Get the training set.

<a id="camel.benchmarks.gaia.GAIABenchmark.run" />

### run

```python theme={"system"}
def run(
    self,
    agent: ChatAgent,
    on: Literal['train', 'valid', 'test'],
    level: Union[int, List[int], Literal['all']],
    randomize: bool = False,
    subset: Optional[int] = None
):
```

Run the benchmark.

**Parameters:**

* **agent** (ChatAgent): The agent to run the benchmark.
* **on** (`Literal["valid", "test"]`): The set to run the benchmark.
* **level** (`Union[int, List[int], Literal["all"]]`): The level to run the benchmark.
* **randomize** (bool, optional): Whether to randomize the data. (default: :obj:`False`)
* **subset** (Optional\[int], optional): The subset of data to run. (default: :obj:`None`)

**Returns:**

Dict\[str, Any]: The results of the benchmark.

<a id="camel.benchmarks.gaia.GAIABenchmark._prepare_task" />

### \_prepare\_task

```python theme={"system"}
def _prepare_task(self, task: Dict[str, Any]):
```

Prepare the task by validating and enriching its data.

<a id="camel.benchmarks.gaia.GAIABenchmark._create_user_message" />

### \_create\_user\_message

```python theme={"system"}
def _create_user_message(self, task: Dict[str, Any]):
```

Create a user message from a task.

<a id="camel.benchmarks.gaia.GAIABenchmark._process_result" />

### \_process\_result

```python theme={"system"}
def _process_result(
    self,
    agent: ChatAgent,
    task: Dict[str, Any],
    result: Any,
    file_obj: Any
):
```

Process and store the result of a task.

<a id="camel.benchmarks.gaia.GAIABenchmark._handle_error" />

### \_handle\_error

```python theme={"system"}
def _handle_error(
    self,
    task: Dict[str, Any],
    error: Exception,
    file_obj: Any
):
```

Handle errors encountered during task processing.

<a id="camel.benchmarks.gaia.GAIABenchmark._generate_summary" />

### \_generate\_summary

```python theme={"system"}
def _generate_summary(self):
```

Generate and return a summary of the benchmark results.

<a id="camel.benchmarks.gaia.GAIABenchmark.question_scorer" />

### question\_scorer

```python theme={"system"}
def question_scorer(self, model_answer: str, ground_truth: str):
```

Scorer for the GAIA benchmark.
[https://huggingface.co/spaces/gaia-benchmark/leaderboard/blob/main/](https://huggingface.co/spaces/gaia-benchmark/leaderboard/blob/main/)
scorer.py

**Parameters:**

* **model\_answer** (str): The model answer.
* **ground\_truth** (str): The ground truth answer.

**Returns:**

bool: The score of the model

<a id="camel.benchmarks.gaia.GAIABenchmark.normalize_number_str" />

### normalize\_number\_str

```python theme={"system"}
def normalize_number_str(self, number_str: str):
```

<a id="camel.benchmarks.gaia.GAIABenchmark.split_string" />

### split\_string

```python theme={"system"}
def split_string(self, s: str, char_list: Optional[List[str]] = None):
```

Split a string based on a list of characters.

**Parameters:**

* **s** (str): The string to split.
* **char\_list** (Optional\[List\[str]], optional): T he list of characters to split on. (default: :obj:`None`)

<a id="camel.benchmarks.gaia.GAIABenchmark.normalize_str" />

### normalize\_str

```python theme={"system"}
def normalize_str(self, input_str, remove_punct = True):
```

Normalize a string.

**Parameters:**

* **input\_str**: The input string to normalize.
* **remove\_punct**: Whether to remove punctuation.

**Returns:**

str: The normalized string.

<a id="camel.benchmarks.gaia.GAIABenchmark.get_final_answer" />

### get\_final\_answer

```python theme={"system"}
def get_final_answer(self, content: str):
```

Get the final answer from the content.

**Parameters:**

* **content** (str): The content to extract the final answer from.

**Returns:**

str: The final answer.