Barr Moses

Everyone's going agent-first. Not everyone's running agents in production. There's a difference.

I'm speaking at the AI Agent Conference in NYC on May 5th about the agent reliability crisis. Be there! agentconference.com/schedule #AIagent #AgentReliability

I speak to data + AI leaders every week that tell me: "We have agents running in production. They're doing things. We just don't know exactly what." We built the Troubleshooting Agent to solve this across the agentic stack: context, behavior, performance, outputs. Check it out.

Really enjoyed the Data Exchange podcast episode with @lgavish (CTO & Co-Founder with @BM_DataDowntime of @montecarlodata). One of the clearest takes I’ve heard on how data observability is evolving as AI becomes embedded in real production systems. @bigdata's highly recommended @GradientFlowR blog post that instigated this conversation: Beyond Black Boxes: A Guide to Observability for Agentic AI. Lior made two points that resonated: 1- Observability is going to be about ensuring trust in AI workflows. The opportunity is huge. 2- It is hard to extract insight from telemetry (~23:30). gradientflow.substack.com/p/are-your-ai-…

AI didn’t eliminate data quality problems. It simply reduced the easy ones.  The remaining failures (semantic drift, schema volatility, system assumptions) are harder, subtler, and increasingly common. And the cost is higher than ever. More here: montecarlodata.com/blog-has-ai-as…

@montecarlodata @Snowflake More here: montecarlodata.com/blog-agent-obs…

Today, @montecarlodata's Agent Observability is natively integrated with @Snowflake Intelligence so every Cortex Agent is automatically discoverable, traceable, and monitored. No code, no SQL. Teams can manage their full agent fleet with the same rigor they apply to data systems.

All of your data doesn’t need to be “AI-ready.” Start by making some of it “agent-ready. Agent-ready data: - Datasets agents can reliably use, with clear metadata, governance, observability - Visibility into the entire system, from trusted inputs to verifiable outputs Thoughts?

According to @IDC, 90% of data is unstructured. The question has always been—how do we make it reliable for production? Well, now we have an answer. I’m thrilled to OFFICIALLY ANNOUNCE Monte Carlo’s support for Unstructured Data Monitoring. As part of an ongoing commitment to accelerate AI-ready data at scale, Monte Carlo users can now leverage all of Monte Carlo’s power—including our NEW Observability Agents—to monitor both structured and unstructured data in production. As my cofounder Lior Gavish puts it: “Enterprises aren’t just building AI—they’re racing to build AI they can trust…High-quality unstructured data—like customer feedback, support tickets, or internal documentation—isn’t just important; it’s foundational to building powerful, reliable AI… That’s why we designed our monitoring capabilities to proactively detect issues before they impact the business.” And we’re not stopping there. This latest AI-ready data release ALSO includes new integrations to support AI-ready data for Snowflake Cortex Agents and Databricks AI/BI as well. You can check out the coverage from SiliconANGLE & theCUBE here: lnkd.in/gZ6bBRaA At Monte Carlo, we’re committed to defining the future of reliable AI for enterprise data teams. That means extending coverage to every layer, resource, and integration that powers it. That’s what makes a true data + AI observability solution. And that’s exactly what we’re building. Want to learn more? Stop by one of our booths and check it out for yourself! Snowflake Summit: lnkd.in/gDWFa7bj Databricks Data + AI Summit: lnkd.in/gp-JxMbD

Think model evaluation will replace the need for high quality data? Think again. Bad data has been eroding great data pipelines for years—but in an agentic workflow, those risks can cascade into all kinds of systemic problems. And the worst part? Model evaluation is leading too many teams to take their eye off the ball. In his latest, Monte Carlo's very own Elor Arieli discusses how his team discovered a breaking data gap during testing for their own troubleshooting agent—and why they spent 2 days investigating the model before they understood the real root cause of the problem: bad data. According to Elor, here are the five steps to reliable AI: Implement comprehensive observability across your data + AI pipelines Observe model outputs (but not in a silo) Establish cross-functional incident response processes Test your AI with synthetic data gaps Document critical data dependencies. If you’re looking for real-world examples of how teams are managing AI reliability, give this one a read.

Agents to drive reliable data + AI applications are here. When we launched Monte Carlo in 2019, we wanted to help every team own the quality and trust of their data—and this is an enormous leap towards that reality.

This might be the biggest news we’ve shared all year—and I’m so freakin’ excited about it. Monte Carlo just announced our first-ever Observability Agents to accelerate reliability workflows for enterprise teams—beginning with our Monitoring and Troubleshooting Agents to drastically accelerate monitor creation and incident resolution. You may have seen automated monitor suggestions before, but not like this. This is the first AI agent that makes recommendations based on: - a data profile AND - Metadata for the larger contextual meaning AND - Query logs to understand how the data is used. The result is more sophisticated, helpful suggestions—and a 60% acceptance rate. But don’t take my word for it. You can take a demo or a self-guided tour of our Monitoring Agent today. Please check this one out and let me know what you think: lnkd.in/gSDPaAQ2

Have you seen Stanford University's new AI Index Report? There's a ton to digest, but this takeaway stands out to me the most: “The responsible AI ecosystem evolves—unevenly.” In the report, the editors highlight that AI-related incidents are on the rise, but standardized evaluations for validating response quality (and safety) leave MUCH to be desired. From a corporate perspective, there’s an observable gap between understanding the RISKS of errant model responses— and actually taking meaningful ACTION (at the data, system, code, and model response levels) to mitigate it. The primary thrust of the AI movement seems to be this: Build, build, and build some more…then process the consequences later. I think we need to take a step back as data leaders and ask if that’s really an acceptable approach. Should it be? A couple of bright spots highlighted in the report included a few new benchmarks which all offer promising steps toward assessing the factuality and safety of model responses: - HELM Safety - AIR-Bench - and FACTS Governments are also taking notice. “In 2024, global cooperation on AI governance intensified, with organizations including the OECD, EU, U.N., and African Union releasing frameworks focused on transparency, trustworthiness, and other core responsible AI principles.” When it comes to AI, it’s not just our customers who suffer when things go awry. Our stakeholders, our teammates, and our reputations are all on the hook for generative missteps. In short, rewards may still be high—but the risks have never been higher. What do you think? Let me know in the comments!

I don't care how big your context window is. While it’s true that running a single complex action on a model with a large context window would lead to a more favorable output than a smaller model all other things being equal, that assumes that you actually need to run that complex action as a single task. The reality is, twenty smaller models running in parallel and outputting a smaller number of tokens will almost always be faster than a single large model running on all the data all at once. And I’ll do you one better—small models can even improve the performance of your AI agents too. Strategies like horizontal task splitting minimize the number of input tokens, output tokens, and model size required to complete an operation—reducing runtime, maintaining (or even reducing) costs, and delivering more deterministic responses for their respective tasks. The secret? Curating the right high quality data to make it work.

Question—Is it possible to drive differentiation through better governance? Governance has historically been viewed as a cost-center for most teams—a pair of administrative handcuffs to mitigate regulatory or compliance risk (among other things). But is it possible that it’s more than that? Monte Carlo recently hosted Lance Schafer from Lotlinx to share his team’s data + AI journey and how they’ve been managing their data to get “AI-ready” (more on that in the future). During the discussion, the conversation quickly shifted to the topics of data quality and governance (naturally), and Lance had some very interesting things to say. When it comes to AI, there’s no doubt that the value of the deliverable is determined by the value of the data that’s powering it. Discussing his team’s latest AI projects, Lance noted that much of his team’s recent success has been due (in large part) to the access they’ve been able to cultivate not just to their customers’ data but also their partners’. According to Lance, Lotlinx rigorous quality and governance standards have created an environment of trust that’s encouraged partners and industry adjacencies to pass through data they wouldn’t otherwise consider. In short, greater trust created greater opportunities, particularly with AI. And I think that’s a lesson every data leader needs to learn—limited trust creates limited value. Your reputation doesn’t simply dictate the value you can deliver to customers right now—it might also determine the use cases you can support in the future. Governance isn’t handcuffs. It’s a blank check. How much are you writing in?

Data and AI are no longer two separate technologies. It’s time we stopped treating them that way. That’s why I’m ecstatic to announce that Monte Carlo will be extending our partnership with Databricks to bring our vision for data + AI observability to the Databricks’ Data Intelligence Platform. Databricks SVP of Product Adam Conway says it well: “We’re incredibly excited to see Monte Carlo expanding their data + AI observability capabilities to support unstructured data pipelines in Databricks' Data Intelligence Platform. This collaboration empowers our customers to gain deeper insights and trust in their AI-driven workloads, accelerating innovation with reliable, high-quality data.” If you're a Databricks customer, you can expect: - Coverage for structured and unstructured data pipelines - AI-powered alerts and detection - Lineage tracking to identify incident impact - And root cause within AI agents The first step to adopting any new technology is trust. If you can't trust the technology, you can't depend on it. End-to-end data + AI observability is no longer a nice-to-have; it's essential for the reliable operation of AI agents and applications. This partnership is the next big step to get us there. Link to the full announcement: montecarlodata.com/blog-monte-car…

Explore the convergence of structured/unstructured data, AI, and SaaS stacks, and why end-to-end observability is crucial for enterprise-grade reliability in 2026. Read the full article by @BM_DataDowntime free now. towardsdatascience.com/2026-will-be-t…

The internet was powering AOL chatrooms before site reliability engineering delivered the fabled “5 9s of reliability.” Data warehouses were creating printed charts to ignore in board rooms before data observability was protecting critical cloud-based data products like ML models and marketing dashboards. Data + AI will deliver exponentially more unique value for enterprise organizations than data alone…but where there’s increased impact, there’s destined to be increased reliability requirements.

Bad data is coming for your AI models. Poor data quality has wreaked havoc on dashboards and ML models for years. But at the scale of AI, the data quality challenge is bigger than ever. Relying on manual data quality checks to effectively cover the massive volumes of data feeding AI models is like tossing a log into the Amazon and hoping it makes a dam. Without automation and comprehensive root-cause insights to make alerts immediately actionable, we can't hope to protect our data at the dizzying pace of AI development.

Here are 6 things I think every CDO needs to hear about AI-readiness this year: 1. If you’re not in the cloud, you need to be. 2. Your first-party data is your ONLY moat. 3. It’s not enough to make your data available for AI if it isn’t also understandable—so invest in semantic programs. 4. AI is a security nightmare—govern it that way. 5. I know you think you’re close to the business. You need to get even closer. 6. And most importantly—garbage in will always mean garbage out, so invest in data quality. Invest in data quality. INVEST IN DATA QUALITY.