# ITBench-AA: Frontier Models Score Below 50% on the First Benchmark for Agentic Enterprise IT Tasks — by Artificial Analysis and IBM

Published: 2026-05-27 · Updated: 2026-05-27 · Category: Benchmark

Source page: https://www.sipply.dev/story/itbench-aa-frontier-models-score-below-50-on-the-first-2bf58466 · 中文: https://www.sipply.dev/zh/story/itbench-aa-frontier-models-score-below-50-on-the-first-2bf58466

## Summary

Artificial Analysis and IBM have introduced ITBench-AA, a pioneering benchmark designed to assess frontier AI models in the context of agentic enterprise IT tasks, particularly in Site Reliability Engineering (SRE). The benchmark aims to provide a standardized evaluation framework for AI models' performance in real-world IT scenarios. Initial results from this benchmark reveal that all tested models, including Claude Opus 4.7, scored below 50%, with the highest score being 47%. This indicates significant room for improvement in the capabilities of current AI models when applied to complex IT tasks.

## Key takeaways

- The ITBench-AA benchmark is the first of its kind for evaluating AI models in enterprise IT tasks.
- All evaluated models scored below 50%, highlighting the current limitations of AI in handling agentic tasks.
- Claude Opus 4.7 achieved the highest score at 47%, but still falls short of effective performance.

## Why it matters

The introduction of ITBench-AA marks a critical step in understanding and improving AI capabilities in enterprise IT environments. The low scores indicate that while AI is advancing, it still struggles with complex, real-world applications.

- **Product managers:** For product managers, this benchmark highlights the importance of setting realistic expectations for AI capabilities in enterprise applications and the need for continuous improvement and innovation.
- **Developers:** Developers should take note of the benchmark results as they reflect the current limitations of AI models, guiding them to focus on enhancing model performance in specific enterprise IT tasks.
- **Students and job seekers:** For students and job-seekers, the findings from ITBench-AA underscore the evolving landscape of AI in IT, suggesting opportunities for research and development in improving AI applications for enterprise solutions.

## Sources

- [ITBench-AA: Frontier Models Score Below 50% on the First Benchmark for Agentic Enterprise IT Tasks — by Artificial Analysis and IBM](https://huggingface.co/blog/ibm-research/itbench-aa) (Hugging Face Blog, 2026-05-27)

## 中文

### ITBench-AA：前沿模型在代理企业IT任务的首个基准测试中得分低于50% — 来自Artificial Analysis和IBM

Artificial Analysis和IBM推出了ITBench-AA，这是一个开创性的基准测试，旨在评估前沿AI模型在代理企业IT任务中的表现，特别是在网站可靠性工程（SRE）方面。该基准旨在为AI模型在实际IT场景中的表现提供标准化评估框架。初步结果显示，所有测试模型，包括Claude Opus 4.7，得分均低于50%，最高得分为47%。这表明当前AI模型在处理复杂IT任务时仍有显著的改进空间。

**要点：**

- ITBench-AA基准是评估企业IT任务中AI模型的首个基准。
- 所有评估模型得分均低于50%，突显了AI在处理代理任务中的当前局限性。
- Claude Opus 4.7以47%的得分获得最高，但仍未达到有效性能。

**为什么重要：** ITBench-AA的推出标志着理解和提升AI在企业IT环境中能力的重要一步。低得分表明，尽管AI在进步，但在复杂的实际应用中仍面临挑战。

---
Sipply: Sip AI News Simply. https://www.sipply.dev