MLflow

📣 Final call: MLflow Ambassador applications close Monday, July 13 at 11:59 PM PT. Building, teaching, or leading in the MLflow community? Know someone who is? Ambassadors help grow the community by: 🔹 Meetups & events: local advocacy and outreach 🔹 Content: talks, demos, and tutorials for practitioners 🔹 Mentorship: support for contributors and connections across OSS 🔗 Apply or nominate: docs.google.com/forms/d/e/1FAI… #MLflow #OpenSource #GenAI #LLMOps

You call an LLM, get a response back. What was sent? How long did it take? What came back? Notebook 1.3 of Mastering MLflow for GenAI: one line of mlflow.openai.autolog() captures model, messages, tokens, and latency. 👇 📹 Full video: youtu.be/npiKufwkyoo?si… 📓 Notebook 1.3: github.com/dmatrix/mlflow… #MLflow #GenAI #LLMOps @2twitme

What's new in MLflow 3.14 + 3.15? Join us for the next MLflow virtual webinar on July 29, 4PM PT 👇 🔹 @omnigent_ai Tracing 🔹 Review Queue: flag traces for review 🔹 uvx MLflow Agent Setup + Assistant improvements 🔹 Q&A RSVP: luma.com/mlflow-x51h #MLflow #GenAI #LLMOps

Price alone won't pick your LLM. Compare gpt-4o-mini vs gpt-4o with MLflow eval and traced token costs. 👇 🔹 Correctness + Completeness 🔹 mlflow.openai.autolog() 🔹 Cost vs quality in one table 🔗 Cookbook: mlflow.org/cookbook/cost-… #MLflow #GenAI #LLMOps

Multi-harness agents create an observability hole. @omnigent_ai + MLflow Tracing standardizes traces across harnesses. 👇 🔹 Agent turns, tools, tokens 🔹 Automatic tracing, no code changes 🔹 Before/after model swaps in MLflow 🔗 mlflow.org/blog/omnigent-… #MLflow #GenAI #Agents

Price alone won't pick your LLM. Compare gpt-4o-mini vs gpt-4o with MLflow eval and traced token costs. 👇 🔹 Correctness + Completeness 🔹 mlflow.openai.autolog() 🔹 Cost vs quality in one table 🔗 Cookbook: mlflow.org/cookbook/cost-… #MLflow #GenAI #LLMOps

Same model, 3 runs, 3 configs. Compare output + cost across LLM experiments with MLflow. Notebook 1.2 of Mastering MLflow for GenAI: track params, metrics, and tokens with mlflow.openai.autolog(). One call, less cluttered code. 👇 📹 Full video: youtu.be/ykjYM3r0X8o?si… 📓 Notebook 1.2: github.com/dmatrix/mlflow… #MLflow #GenAI #LLMOps @2twitme

Hand-tweaking agent prompts doesn't scale. 👇 🔹 make_judge domain rubrics 🔹 MemAlign + mlflow.log_feedback() 🔹 optimize_prompts (GEPA) 🔗 Cookbook: mlflow.org/cookbook/agent… #MLflow #Agents #GenAI

Are you actively building, teaching, or leading in the MLflow community, or know someone who is? Applications are now open for the MLflow Ambassador Program! Ambassadors help grow the community by: 🌍 Promoting MLflow projects in their communities 📚 Educating practitioners on the latest MLflow features 🤝 Engaging through meetups, content, mentorship, and advocacy Apply or nominate someone in your network. Applications close July 13 at PM PT ⏰ Learn about the program: mlflow.org/ambassadors/ Application: docs.google.com/forms/d/e/1FAI… #MLflow #OpenSource #AIEngineering #LLMOps #Community

There are four pillars in GenAI. In Notebook 1.1 of the Mastering MLflow for GenAI series, Jules Damji breaks down how each pillar fits together. 👇 📺 Full video: youtu.be/IzUDKJlDo7Q?si… 📓 Notebook: github.com/dmatrix/mlflow… #MLflow #GenAI #LLMOps @2twitme

🔹 Usage Dashboard: volume, errors, latency percentiles, cost by model/provider 🔹 Encrypted credentials, traffic splitting, fallback chains One platform for gateway, tracing, evaluation, and observability. 🔗mlflow.org/blog/mlflow-ai… #MLflow #GenAI #LLMOps

Runs as part of the MLflow Tracking Server alongside tracing and evaluation. Governed LLM access, no extra infra. 🔹 One OpenAI-compatible endpoint (OpenAI, Anthropic, Gemini, Bedrock+) 🔹 Native in MLflow 3.11.0rc1, no LiteLLM 🔹 Auto traces when usage tracking is on

Scaling GenAI means scattered API keys and no visibility into token use or cost. Sensitive data can reach third-party LLM APIs without centralized governance. MLflow AI Gateway is built for this. 👇 Learn more: mlflow.org/blog/mlflow-ai… #MLflow #GenAI #LLMOps

🔗 Notebook 1.9: github.com/dmatrix/mlflow…

New in the Mastering GenAI with MLflow series: Notebook 1.9. Build + trace a complete RAG app with MLflow. 🔹 6 steps under a CHAIN span (PARSER, EMBEDDING, RETRIEVER, LLM) 🔹 Per-step latency, tokens & cost in the UI 🔹 RAGAS Faithfulness + ContextRelevance on traces 📺 New video: youtu.be/_fKBGr5_Gdw #MLflow #GenAI #RAG #LLMOps

In this video, Joana Mesquita evaluates a RAG agent end-to-end. 👇 School assistant agent. LangChain + FAISS + Amazon Bedrock. One MLflow interface for the full eval loop. 📋 Prompt Registry with production aliases 🔍 Traces with SME ground truth ⚖️ Custom judges, Ragas, Arize Phoenix, deterministic scorers Watch : youtube.com/watch?v=4wqkHr… #MLflow #LLMOps #GenAI #RAG #OpenSource

inspect-mlflow logs Inspect AI evals to MLflow. Set MLFLOW_TRACKING_URI + MLFLOW_INSPECT_TRACING=true. 👇 📊 Hierarchical runs + metrics 🔍 Span-level traces 📈 Assessments in Traces table Learn more: mlflow.org/blog/inspect-m… #MLflow #InspectAI #GenAI