Turing Community

EnterpriseOps-Gym is taking off! 2K downloads in 3 days, trending #6 dataset + #3 paper of the day! Let's keep going!

Turing is featured in @ServiceNowRSRCH's Enterprise Ops Gym paper. We built the task and evaluation backbone: -1,000 prompts -7 single domain plus 1 hybrid workflow -7 to 30 step planning horizons -Expert reference executions with logged tool calls -Deterministic validation for success and side effect control Enabling structured comparison of enterprise agent performance across domains and complexity tiers. Dataset -> Paper -> Website -> Code. Below.

Dataset: huggingface.co/datasets/Servi…

Paper: arxiv.org/abs/2603.13594 Website: enterpriseops-gym.github.io Code: github.com/ServiceNow/Ent…

Most AI models can pass standard benchmarks. But real-world risk does not live at the surface. As frontier models improve, many STEM benchmarks are saturating. High pass rates on academic datasets reduce evaluation signal and can mask weaknesses in advanced reasoning. That’s why we built HLE++.

HLE++ is a calibrated STEM evaluation framework designed to preserve measurable pass@k separation beyond baseline benchmarks like Humanity’s Last Exam (HLE). It measures how leading large language models perform on graduate-to-PhD-level, multi-step math and science tasks under strict structural constraints.

Why this matters: -Advanced STEM reasoning underpins high-impact AI use cases across finance, life sciences, manufacturing, energy, and defense.  -Small reasoning errors compound at scale in automated decision systems.  Evaluation design, ambiguity control, and rubric structure materially affect reported model performance.

What we’re seeing: -Performance gaps widen on structured, multi-step domain tasks.  -Models that perform well on general benchmarks degrade under domain-specific stress.  -Calibrated difficulty bands reveal weaknesses leaderboard scores often hide.

Explore HLE++ and see how today’s frontier models perform under domain-grade scrutiny: go.turing.com/llm-stem-evalu…

NVIDIA just open-sourced OpenShell (Apache 2.0), a dedicated runtime environment designed to address the security risks associated with autonomous AI agents. As agents move from simple chat interfaces to executing code and accessing local/remote tools, they require a secure execution layer that prevents unauthorized system access or data exfiltration. OpenShell provides this infrastructure through three primary technical pillars: 1️⃣ Sandboxed Execution Using kernel-level isolation (Landlock LSM), OpenShell creates an ephemeral environment for agent tasks. This ensures that any shell commands or scripts generated by an LLM are contained, protecting the host system from unintended modifications or destructive commands. 2️⃣ Policy-Enforced Access Control Rather than providing broad permissions, OpenShell utilizes a granular policy engine. Developers can define restrictions at multiple levels: → Per-binary: Explicitly allow or deny specific executables (e.g., git, python). → Per-endpoint: Restrict network traffic to authorized domains or IP addresses. → Per-method: Control specific API calls or L7 protocols. → Audit Logging: Every action is recorded for debugging and compliance. 3️⃣ Private Inference Routing To manage privacy and costs, OpenShell includes a routing layer that intercepts model traffic. This allows organizations to enforce data-handling rules and route inference requests between local and cloud models without changing the agent's code. OpenShell is currently in alpha....... Read our full analysis on OpenShell: marktechpost.com/2026/03/18/nvi… GitHub: github.com/NVIDIA/OpenShe… Docs: docs.nvidia.com/openshell/late… Technical details: developer.nvidia.com/blog/run-auton… @nvidia @NVIDIAAI @NVIDIAAIDev

Excited to share that Turing contributed to Enterprise Ops Gym, @ServiceNowRSRCH's new enterprise agent benchmark submitted to ICML. Enterprise Ops Gym moves beyond short-horizon tool calls and evaluates end-to-end enterprise operations across realistic, multi-system workflows. The paper, website, dataset, and code is below.

Website: enterpriseops-gym.github.io

Dataset: huggingface.co/datasets/Servi…

Code: github.com/ServiceNow/Ent…

CC @ShivaMalay @PShravannayak @TheJishnuNair @sagardavasam @SathwikTejaswi @tscholak @NVIDIAAI @ServiceNowNews @Mila @ServiceNowRSRCH

🧵 Introducing ENTERPRISEOPS-GYM — a new stateful benchmark for enterprise AI agents. Best model: 37.4% success — with oracle tool access. Give a tiny model a human plan? It catches up to giants. The bottleneck isn’t tools. It’s planning. 👇

Benchmarking frontier models takes more than bigger datasets. Turing built 5,000+ HLE-grade STEM problems to stress-test deep reasoning across physics, chemistry, biology, and math. 100% accepted. 40+ subdomains. Built for real model differentiation. This is what next-gen AI evaluation looks like. Review the case study below.

Most AI agents today are failing the enterprise 'vibe check.' ServiceNow Research just released EnterpriseOps-Gym, and it’s a massive reality check for anyone expecting autonomous agents to take over IT and HR tomorrow. We’re moving past simple benchmarks. This is a containerized sandbox with 164 database tables and 512 functional tools. It’s designed to see if agents can actually handle long-horizon planning amidst persistent state changes and strict access protocols. The Brutal Numbers: → Claude Opus 4.5 (the top performer) only achieved a 37.4% success rate. → Gemini-3-Flash followed at 31.9%. → DeepSeek-V3.2 (High) leads the open-source pack at 24.5%. Why the low scores? The research study found that strategic reasoning, not tool invocation, is the primary bottleneck. When the research team provided agents with a human-authored plan, performance jumped by 14-35 percentage points. Strikingly, with a good plan, tiny models like Qwen3-4B actually become competitive with the giants. The TL;DR for AI Devs: ✅ Planning > Scale: We can’t just scale our way to reliability; we need better constraint-aware plan generation. ✅ MAS isn't a Silver Bullet: Decomposing tasks into subtasks often regressed performance because it broke sequential state dependencies. ✅ Sandbox Everything: If you aren't testing your agents in stateful environments, you aren't testing them for the real world. Read our full analysis here: marktechpost.com/2026/03/18/ser… Check out the benchmark: enterpriseops-gym.github.io Paper: arxiv.org/pdf/2603.13594 Codes: github.com/ServiceNow/Ent… @ServiceNow @ServiceNowRSRCH @RajeswarSai @ShivaMalay @PShravannayak @TheJishnuNair @sagardavasam @SathwikTejaswi @tscholak @NVIDIAAI @turingcom @ServiceNowNews @jonsidd

Turing contributed to Enterprise Ops Gym, ServiceNow’s new enterprise agent benchmark. We designed 1,000 prompts across 8 enterprise scenarios spanning HR, CSM, ITSM, Email, Calendar, Drive, Teams, and hybrid cross domain workflows. Tasks range from 7 to 30 steps with stateful system updates, expert reference traces, and deterministic verification scripts to evaluate correctness and policy compliance. A rigorous step forward for long horizon enterprise agent evaluation. Learn more: Paper: arxiv.org/abs/2603.13594 Data Set: huggingface.co/datasets/Servi… Website: enterpriseops-gym.github.io Code: github.com/ServiceNow/Ent…