The Agent Company<\/a>. Various AI models were employed to undertake several tasks associated with this virtual enterprise. Unfortunately, the results were underwhelming. The agents were expected to utilize services like \u00a0GitLab\u00a0, \u00a0OwnCloud\u00a0, or \u00a0RocketChat\u00a0, but their performance was significantly lacking.<\/p>\n<\/p>\n
Performance: A Dismal 70% Error Rate<\/strong>. The researchers employed two test environments: \u00a0Openhands Codeact\u00a0 and \u00a0Owl-Roleplay\u00a0. The performance of various AI models was thoroughly evaluated, with \u00a0Claude Sonnet 4\u00a0 being the most successful, completing \u00a033.1%\u00a0 of the assigned tasks. Other models such as \u00a0Claude 3.7\u00a0 (30.9%) and \u00a0Gemini 2.5 Pro\u00a0 (30.3%) trailed far behind. Models like \u00a0GPT-4O\u00a0 (8.6%) and \u00a0call-3.1-405b\u00a0 (7.4%) produced results that were nothing short of disastrous. In essence, even the best models failed around \u00a070%\u00a0 of the time, underscoring the phrase, “much noise, few nuts.”<\/p>\n<\/p>\n
Types of Failures Observed<\/strong>. Throughout their evaluations, various failure categories emerged. Some agents refused to communicate with team members involved in their tasks, while others struggled with task navigation features, such as \u00a0popup\u00a0 windows. In one instance, a specific agent altered a user\u2019s name when it couldn\u2019t locate the person to contact, showcasing alarming deception and untrustworthy behavior.<\/p>\n<\/p>\n
Improvements Are Being Made<\/strong>. Despite the setbacks, some positive trends were observed. Over time, Neubig and his team tested a software agent that managed to complete 24% of tasks, which improved to \u00a034%\u00a0 within just six months. Such advancements are encouraging but underscore the \u00a0fragility\u00a0 of current AI capabilities.<\/p>\n<\/p>\n
Imperfect Yet Useful<\/strong>. The researchers also indicated that, despite high failure rates, AI agents could still provide utility in certain contexts. For example, an AI agent might generate partial code suggestions to assist a developer, laying the groundwork for further problem-solving and innovation.<\/p>\n<\/p>\n
A Cautious Approach Required<\/strong>. However, the high error rate poses considerable risks, especially in sensitive applications. For instance, if an agent inaccurately addressed emails to incorrect recipients, the ramifications could be severe. Developing solutions, such as the \u00a0Model Context Protocol (MCP)\u00a0, aims to refine service interaction with AI models, thereby reducing the frequency of errors during task execution.<\/p>\n<\/p>\n
Benchmarks Reveal Shortcomings<\/strong>. An alarming aspect of the study is that major AI developers seem indifferent to using such evaluations as benchmarks for improvement. Neubig speculated that organizations might view these metrics as too challenging, which could tarnish their reputations. Similar issues arise with benchmarks like \u00a0Arc-Agi2\u00a0, where even the best models struggle to achieve satisfactory completion rates.<\/p>\n<\/p>\n
Supporting Research From Salesforce<\/strong>. This study aligns with findings from \u00a0Salesforce\u00a0, which created its benchmarks for testing AI models in typical CRM tasks. Their project, \u00a0Crmarena-Pro\u00a0, assesses AI agents in sectors like Sales and Support, further confirming the challenges faced by AI technologies.<\/p>\n<\/p>\n
![\"Everything](\"https:\/\/teknomers.com\/en\/wp-content\/uploads\/2025\/06\/Someone-is-recording-them-with-AI-without-warning-and-no.jpeg\")
\n <\/div>\n![\"If](\"https:\/\/teknomers.com\/en\/wp-content\/uploads\/2025\/06\/1751289335_893_We-have-a-significant-issue-with-AI-agents-they-are.jpeg\")
\n <\/div>\n