Apache Hamilton (incubating)

Things I hear in private circles: "even the CEOs of @langchain and @llama_index say there is little value in abstracting over LLM API calls". If you haven't been following Apache @hamilton_os or Apache @burr_framework that's exactly what we've been saying for years -- you need to own the LLM call -- why? because customization here is what is going to deliver value, not the POC you threw together with those overly abstract VC backed frameworks. Plug (1): building on Apache Hamilton &/or Apache Burr is a better investment because you can drive the direction. Plug (2): @hugobowne and I cover this our @MavenHQ course -maven.com/s/course/67c6a…

Big update on FlowerPower! 🚀 I’ve completed a full refactoring for cleaner, more efficient code and added comprehensive docs: tutorials, API refs, & examples. Now built on @hamilton_os & RQ Docs: legout.github.io/flowerpower #Python #OpenSource #DataEngineering

Quick update: 1. New repo location github.com/apache/hamilton 2. Please bear with us as we transition a few things with this change.

@jeremyphoward 🤷, 😊

Well that's sad (Sonnet 3.7)...

If you want something simpler than #Langgraph, and want to try more closely modeling @AnthropicAI workflow best practices - give me a go - see below: Source: emeka.ca/thoughts/to-ag… #python #opensource #llmops github.com/dagworks-inc/h…

Check out the latest article in my newsletter: February Updates linkedin.com/pulse/february… via @LinkedIn @hamilton_os: - crosses 2K⭐️ - installs in @pyodide envs - mutlithreaded parallel DAG execution @burr_framework: - crosses 1.5K ⭐️ - async persister implementations - tag aliases for actions

@arundsharma @samuelcolvin @rrherr @pydantic No need to create a dataclass - just implement a validator and specify the type it is for, e.g. see `@ check_output_custom` section here - blog.dagworks.io/p/data-quality… You could also do this check as "lifecycle hook" blog.dagworks.io/p/customizing-…)

@hamilton_os @samuelcolvin @rrherr @pydantic Do you have an example of creating a dataclass and then generating a pydantic/pandas-df type out of it via Hamilton?

We've just released @Pydantic AI v0.0.19. This comes with the biggest new feature since we announced PydanticAI — graph support! I was originally cynical about graphs, but I'm now really excited about their use cases, both with GenAI and in general purpose development. Our approach to graphs — using type hints to define nodes and edges, makes graphs less work to implement and much much safer to develop and extend. I honestly think this graph API might be the single most innovative thing I've ever built in Python! If you have complex logic flows, or logic that is distributed over compute and time, I think graphs will be a massive win. It's still early, but I think the potential is obvious. ai.pydantic.dev/graph/

@samuelcolvin @rrherr @pydantic Yes. Jury still out how coupled you want graphs to be. That's why we take a different edge definition strategy. Our approach makes it easy to add new nodes without having to perform surgery on existing code. Trade-offs of graph scale and maintenance.

@hamilton_os @rrherr @pydantic Pydantic Graph nodes can be used in any graph with the same state and dependency types. But they're designed to be type checked too. And since it's just standard type checking, you can use anything that has proper type hints: dataclasses, pydantic models, pandas, polars etc.

@arundsharma @samuelcolvin @rrherr @pydantic You can turn that on too with Hamilton, and do it for any type you want, not just Pydantic. E.g. pandas data frames, or extend it for your own custom object types...

@samuelcolvin @rrherr @pydantic @hamilton_os So pydantic's main strength is that it does runtime data validation. For that people pay a 4x construction cost. For SQLModel it's more like 10x. #issuecomment-2593933826" target="_blank" rel="nofollow noopener">github.com/bespokelabsai/…

@samuelcolvin @rrherr @pydantic Hamilton's functions are independent of each other and could be part of many DAGs. So type checkers wouldn't be able to handle things anyway. This is why Hamilton checks things at graph construction time. Optionally one can also enforce runtime object inspection.

@rrherr @pydantic @hamilton_os Right, but because of how it's defined, stactic type checkers like mypy and pyright can't enforce that nodes are connected correctly, unlike in pydantic graph.

Check out the latest article in my newsletter: Last week of 2024 / first week of 2025 @hamilton_os @burr_framework stats: > 35M+ telemetry events (10x growth), > 100K+ unique IPs (10x growth) from 1000+ companies, > crossed 1M+ total downloads. More in the newsletter! linkedin.com/pulse/last-wee… via @LinkedIn

@jeremyphoward "Real" is the ambiguous term here. Good news is that it doesn't matter if you use Hamilton, as it has @ProjectJupyter magic that enables notebook development that then transfers out of a notebook very easily... github.com/DAGWorks-Inc/h…

SW eng manager: No real work gets done in Jupyter notebooks. Alex Radford: I invented GPT and CLIP in Jupyter notebooks.

Super excited for this HOW-TO on running the Hamilton UI on @SnowflakeDB : "Observability of Python Code and Application Logic with @hamilton_os UI on @SnowflakeDB Containers" Services by Greg Przemyslaw Kantyka link.medium.com/FIwGhJqzsPb

Tired of the deployment headaches with #LLM agents? #BentoML takes care of the heavy lifting - from REST endpoints to scaling. Great post by Thierry Jean from @DagWorks showing how to combine #Burr and BentoML for streamlined agent deployment! Read the blog post to learn more: blog.dagworks.io/p/burr-bentoml…

Super excited for this post by @thierryjean that provides a well laid out golden path for folks iterating in the #GenAI space. This follows up from my tweet yesterday about framework coupling. If the framework allows loose coupling, then graduating out of it isn't a thing you'll run into. In this post Thierry lays out to use Burr and how it doesn't get in your way as you build more complex and sophisticated workflows and agents. blog.dagworks.io/p/the-graduati…

A @burr_framework user shared this video youtube.com/watch?v=evmZTh… with me on the "problem with frameworks". Worth a watch. I 💯% agree with the take home. The "coupling problem" is the reason why people graduate out of frameworks like @langchain & @llama_index. You become overly coupled and then find yourself overtime asking why am I using this? It's not that the framework authors did this maliciously, it's just that they optimized for their needs, in this case for POCs, which is not the same as long lived maintainable code. That's why we see people either build their own frameworks, or they hopefully discover @hamilton_os & @burr_framework and realize that there are frameworks that allow "loose coupling" that don't get in your way (as a platform person and framework author we optimize for maintainable code). A good example of this is @FastAPI - very easy to not couple your business logic and iterate quickly no matter the age of the code. So how can you tell where a framework lies on the spectrum of tight coupling versus not? Some tests: (1) How many "objects" do I need to use from the framework to couple the logic I want to happen? The fewer the better. E.g. with @hamilton_os by default it's 0. With @burr_framework it's just 1 and that's due to state for edge transitions, but that's easy to decouple from. (2) How many framework imports do you need to get something to run? The less the better - consider @burr_framework vs @langchain using #langgraph (see images) If you're feeling framework pains, or are curious on how it can be done right, checkout: github.com/DAGWorks-Inc/b… (agents) github.com/DAGWorks-Inc/h… (pipelines/tools) or subscribe to our blog - blog.dagworks.io #python #opensource #genai #llmops

Woo Hamilton Star in the wild (yes I'm an @OpenLineage producer). #dataforai #meetup

Excited to be running a course on @MavenHQ with @hugobowne around building GenAI products using "first principles". But the course material isn't set yet. We'd love your feedback please! If you can spare a few minutes, we'd love your input here - maven.com/forms/b83ce4 #genai #llmops #genaiops #SoftwareEngineering

FlowerPower - Python workflow framework Based on @hamilton_os and APScheduler 🔄 DAG-based pipelines ⏰ Scheduling ⚙️ Simple YAML-based setup 📊 Tracking 📡 Distributed execution github.com/legout/flowerp…

In ep. 22 of Generationship, @rachelchalmers speaks with Elijah Ben Izzy (@elijahbenizzy) of @DagWorks about the importance of streaming in chatbot interactions, highlighting how partial, real-time responses can significantly enhance user engagement. hubs.ly/Q02T9RdP0