This dataset contains AI agent execution trajectories and review data generated from open-source vulnerability reproduction tasks in the CyberGym security benchmark. It consists of step-level tool calls and execution results, review labels, reviewer evaluations, and a subset of rerun records, and can be used for research on supervised fine-tuning (SFT), process reward modeling (PRM), and agent execution process evaluation.
Use Cases
- Agent Training: Provides SFT training examples and step-level PRM labels for research on training security AI agents.
- Shortcut Detection: Distinguishes runs that were scored as successful but referenced public patches or reproduction files, supporting research on shortcut behavior that outcome scores alone do not reveal.
- Evaluation Reliability Analysis: Includes judgments from reviewers and two AI evaluation models, enabling identification of points where judgments diverge and labels that need re-examination.
- Trajectory Comparison Analysis: Provides original runs alongside reruns from reviewer-marked points, enabling comparison of task execution paths and outcomes.