​
NexusSample

class NexusSample:

Nexus benchmark dataset sample.

​
NexusBenchmark

class NexusBenchmark(BaseBenchmark):

Nexus Function Calling Benchmark adapted from NexusRaven V2 Function Calling Benchmark <https://huggingface.co/collections/Nexusflow/nexusraven-v2-function-calling-benchmark-657a597fb84dbe7a09ebfc3e>.

Parameters:

  • data_dir (str): The directory to save the data.
  • save_to (str): The file to save the results.
  • processes (int, optional): The number of processes to use. (default: :obj:1)

​
init

def __init__(
    self,
    data_dir: str,
    save_to: str,
    processes: int = 1
):

Initialize the Nexus Function Calling benchmark.

Parameters:

  • data_dir (str): The directory to save the data.
  • save_to (str): The file to save the results.
  • processes (int, optional): The number of processes to use for parallel processing. (default: :obj:1)

​
download

def download(self):

Download the Nexus Functional Calling Benchmark dataset.

​
load

def load(self, dataset_name: str, force_download: bool = False):

Load the Nexus Benchmark dataset.

Parameters:

  • dataset_name (str): Name of the specific dataset to be loaded.
  • force_download (bool): Whether to force download the data.

​
train

def train(self):

Get the training set.

​
run

def run(
    self,
    agent: ChatAgent,
    task: Literal['NVDLibrary', 'VirusTotal', 'OTX', 'PlacesAPI', 'ClimateAPI', 'VirusTotal-ParallelCalls', 'VirusTotal-NestedCalls', 'NVDLibrary-NestedCalls'],
    randomize: bool = False,
    subset: Optional[int] = None
):

Run the benchmark.

Parameters:

  • agent (ChatAgent): The agent to run the benchmark. task (Literal[“NVDLibrary”, “VirusTotal”, “OTX”, “PlacesAPI”, “ClimateAPI”, “VirusTotal-ParallelCalls”, “VirusTotal-NestedCalls”, “NVDLibrary-NestedCalls”]): The task to run the benchmark.
  • randomize (bool, optional): Whether to randomize the data. (default: :obj:False)
  • subset (Optional[int], optional): The subset of data to run. (default: :obj:None)

Returns:

Dict[str, Any]: The results of the benchmark.

​
construct_tool_descriptions

def construct_tool_descriptions(dataset_name: str):

Construct tool descriptions from function definitions and descriptions.

​
construct_prompt

def construct_prompt(input: str, tools: str):

Construct prompt from tools and input.

​
parse_function_call

def parse_function_call(call: str):

Parse a function call string to extract the function name, positional arguments, and keyword arguments, including nested function calls.

Parameters:

  • call (str): A string in the format func(arg1, arg2, kwarg=value).

Returns:

tuple: (function_name (str), positional_args (list), keyword_args (dict)) or (None, None, None).

​
compare_function_calls

def compare_function_calls(agent_call: str, ground_truth_call: str):

Compare the function name and arguments of agent_call and ground_truth_call.

Parameters:

  • agent_call (str): Function call by agent.
  • ground_truth_call (str): Ground truth function call.

Returns:

  • True if the function names and arguments match.
  • False otherwise.