MLflow

Eval-driven dev in MLflow 👇 🔹 Find failures with eval 🔹 Fix + rerun same suite 🔹 Ship with tracked deltas 📕 Start here: mlflow.org/cookbook/eval-… #MLflow #LLMOps

📣 REMINDER: This Wednesday — Join us for a Deep Dive into MLflow 3.12! Curious about what’s next for the ecosystem? We’ll go under the hood of 3.12 and walk through what’s new, including: 🔹 Coding agent tracing + OpenClaw 🔹 Multimodal attachment support in traces 🔹 Guardrails in AI Gateway 🔹 Testing & refining Claude Skills 📅 Wed, May 6 @ 4 PM PT 🎥 Live on LinkedIn, YouTube, & X ➡️ RSVP and join us live: luma.com/mlflow312 #opensource #mlflow #genai #aiops #aigovernance #llmops

LangGraph + MLflow traces 👇 🔹 Tool-calling travel agent 🔹 Full-step tracing 🔹 Tool selection accuracy eval 📕 Read the cookbook: mlflow.org/cookbook/langg… #MLflow #Agents

Want to know what’s coming in MLflow 3.12? Don't miss the next MLflow Community Meetup! 🎉 We are diving into all of the awesome features in MLflow 3.12, including: 🔹Coding agent tracing + OpenClaw 🔹Multimodal Attachment Support in Traces 🔹Guardrails in AI Gateway 🔹Testing & Refining Claude Skills 🗓️ May 6 🕜 4PM PT Register 👉 luma.com/mlflow312 #opensource #mlflow #genai

Traces shouldn't be walls of base64. MLflow multimodal tracing pulls binary out of spans into your artifact store (S3, Azure Blob, GCS, DBFS, local)—spans keep a small reference URI, and the UI renders images, audio, and PDFs inline when you open a trace. ⚡ Zero config if you already autolog OpenAI, Anthropic, Gemini, Bedrock, or LangChain (8 multimodal patterns). For PDFs or paths autolog misses: Attachment.from_file. Multimodal tracing is available in MLflow 3.11+. Learn more 👉 mlflow.org/blog/multimoda… #MLflow #tracing #GenAI

Long support isn’t a “long file + needle” eval — it’s messy, tool-heavy chat. @parloa_ai ran stretched CS sims: verbose users, extra clarifications, and 0/3/20 extra metadata in tool outputs. GPT-4.1 vs 4.1 mini: the bigger model had no clear task-success drop at ~2–3× length; mini struggled more with indirection and noisy tool JSON. High-noise case: ~60–70% of tokens = tool responses (~20–30% user, ~10–15% agent). Runs were logged on an internal stack built on MLflow. Takeaway 👉 optimize what APIs return, not how people talk. 🔗 Learn more: parloa.com/labs/research/… #Parloa #MLfloe

Teams run LLM evals. The follow-up is harder: Did the model actually get better? Which examples regressed? And is the improvement real or inside the noise? It was standing room only at @_odsc East in Boston this week for a workshop on exactly that. 🚀 Debu Sinha (@databricks) led “Evaluating LLM Applications with MLflow”, a hands-on session on closing the loop with MLflow 3.x (unified scorers, reproducible judges, run-to-run compare, and gates you can put in front of production). 🔗 Check out the setup guide and notebook: github.com/debu-sinha/mlf… #MLflow #LLMEvaluation #MLOps #ODSC

Custom judges in MLflow 👇 🔹 Guideline + programmatic scorers 🔹 LLM judges for fuzzy rules 🔹 Combine instead of one brittle rubric 📕 Read the cookbook: mlflow.org/cookbook/custo… #MLflow #Evaluation

Congrats to the @MLflow  maintainers, contributors, and community for this massive celebratory milestone and achievement: 1 billion PyPI downloads since it was released! 🎆 🎉 🎈 🙌 🚀

Cost vs quality with traces in MLflow 👇 🔹 Same eval across providers 🔹 Latency + tokens in traces 🔹 Back routing with evidence ➡️ Open the guide: mlflow.org/cookbook/cost-… #MLflow #GenAI

MLflow Skills close the measurement loop 👇 🔹 npx skills add mlflow/skills + stack tracing 🔹 Traces → analyze → judges → fix → verify 🔹 Claude Code, Codex, Gemini CLI, OpenCode 🔗 Learn more: mlflow.org/blog/self-impr… #MLflow #LLMOps

Agent improvement loop in MLflow 👇 🔹 Tool agent + domain judges 🔹 Expert alignment + GEPA on prompts 🔹 Trace-linked evaluation scores ➡️ Check out the cookbook: mlflow.org/cookbook/agent… #MLflow #LLMOps

At the latest Euro Databricks Community hangout, MLflow Ambassador Thor Steen Larsen joined Pål de Vibe to demonstrate how to move beyond messy local files to a structured, enterprise-ready machine learning environment using MLflow. 🚀 In this webinar, Thor explains how MLflow provides essential tools for experiment tracking, model evaluation, and various deployment strategies suitable for both Data Scientists and ML Engineers. Watch the full replay here ➡️ youtube.com/watch?v=3wxkCs… #mlflow #opensource #oss #tracking #genai

MLflow Assistant: Claude Code in the MLflow UI. 👇 🔹 Less copy-paste of IDs, error stack traces, evaluation results 🔹 Trace→code; suggest fixes; tracing/eval pipelines 🔹 No extra API keys; no added cost; local, full transparency 🔗 Learn more: mlflow.org/docs/latest/ge… #MLflow #ClaudeCode

Our latest blog post walks through eval-driven development with MLflow: a three-phase cycle of tracing, evaluation with human feedback and LLM judges, and prompt versioning that moves you from guessing to measuring. 🔍 Trace every LLM call with mlflow.openai.autolog() 🧪 Layer built-in + custom judges via make_judge 📝 Version and optimize prompts with GEPA Learn more 👉 mlflow.org/blog/structure… #MLflow #AIObservability #LLMOps #GenAI #Evaluation

📣 𝗗𝗲𝗲𝗽 𝗗𝗶𝘃𝗲 𝗶𝗻𝘁𝗼 𝗠𝗟𝗳𝗹𝗼𝘄 𝟯.𝟭𝟮! Want to know what’s coming in MLflow 3.12? Join us as we dive into all of the awesome features, including: 🔹Coding agent tracing + OpenClaw 🔹Multimodal Attachment Support in Traces 🔹Guardrails in AI Gateway 🔹Testing & Refining Claude Skills 🗓️ May 6 🕣4:00 PM PT Register 👉 luma.com/mlflow312 #opensource #oss #mlflow #genai

MLflow AI Gateway now supports guardrails! 🚀 🛡️ Safety, PII Detection, and Custom guardrail types ⚙️ Block or Sanitize flagged content 🔄 Gateway-level — no app code changes Learn more 👉 mlflow.org/blog/gateway-g… #MLflow #LLMOps #GenAI

In the latest episode of our "Mastering GenAI with MLflow" series, Jules Damji explores how MLflow integrates with over 30 open-source frameworks to provide automatic tracing and evaluation. 🚀 Learn about three open source frameworks integrated with MLflow  and how you to trace them effortlessly: 🔗 𝗟𝗮𝗻𝗴𝗖𝗵𝗮𝗶𝗻 🗂️ 𝗟𝗹𝗮𝗺𝗮𝗜𝗻𝗱𝗲𝘅 🕸️ 𝗟𝗮𝗻𝗴𝗚𝗿𝗮𝗽𝗵 📺 Watch the full tutorial: youtube.com/watch?v=VgSRCG… #opensource #oss #mlflow #genai #langchain #llamaindex #langgraph