This is a multi-turn benchmark dataset developed by benchmarking the BFCL (Berkeley Function Calling Leaderboard) to evaluate the action execution capabilities of domain-specific agents across finance, legal, medical, manufacturing, and defense domains.
Each data instance consists of a paired structure: tools_catalog_multiturn.json, which defines the tools available for each domain, and benchmark_scenarios_multiturn.json, which defines the tasks and scenarios to be performed. This structure enables the evaluation of an LLM’s ability to understand user requests and conversational context, select and invoke appropriate domain-specific tools, and accurately execute scenarios across multiple turns.
The dataset focuses on domains that require specialized knowledge and tool usage. Beyond simple question-answering performance, it is designed to assess an agent’s practical task execution capabilities, including tool selection, function calling, multi-turn context retention, and complex task execution.
| Query ID | Domain | Tool Name | Key Parameters |
|---|---|---|---|
| DEF-00001 | 국방 | diagnose_network_status | network_id, region_id, detailed_log |
| DEF-00002 | 국방 | generate_operation_scenario | target_region_id, mission_objective, duration_days |
A Tool Catalog is first created for each domain, defining the functions available to the agent, including function names, descriptions, parameters, and input requirements. Based on the defined tools, multi-turn benchmark scenarios are then designed to connect user requests, reasoning processes, tool calls, input arguments, and tool execution results across multiple turns.
The data is authored using a predefined spreadsheet template, with user messages, agent reasoning, and tool-call information structured and recorded for each turn. After authoring and validation, the data is converted into paired JSON files: tools_catalog_multiturn.json, which defines the available tools for each domain, and benchmark_scenarios_multiturn.json, which contains the benchmark tasks and their expected execution flows.
Each record is reviewed for structural and logical consistency between benchmark_scenarios and tool_catalog. Reviewers verify that each scenario's sub-domain and task align with the corresponding Tool Catalog, and that Functionally Similar Tools and Decoy Tools are used as intended. They also check that tool name and description fields are correctly mapped and that the tools defined in the scenario match the actual tool_calls.name.
For tool calls, reviewers verify that tools.parameters are consistent with tool_calls.arguments and that the tool selected in messages.think matches the tool actually invoked. They also ensure that tools not selected during reasoning are not incorrectly invoked.
Parameter validation checks that all required parameters are present and that optional values such as start_date and end_date are included when needed. Parameter values in the scenario are checked against those described in messages.think, and identical parameter names used across different tools are reviewed to ensure they are clearly distinguishable.
Reviewers also verify that metadata fields and query_id values are correctly assigned. For time- or period-related parameters, required date values are checked for completeness, and relative expressions such as “today” or “yesterday” are verified against explicit dates. Parameter data types are also checked for consistency with the Tool Catalog definitions.
Records containing incorrect tool selections, missing or inconsistent parameters, mismatches between the Tool Catalog and scenarios, or metadata errors are revised and re-reviewed before inclusion in the final dataset.