Matthew Halliday

Latest @dbt_labs #ADEBench Leaderboard #1 OSS altimate-code: 74.4% pass with Sonnet 4.6 #2 @Snowflake Cortex: 65% pass with Opus 4.6 #3 @claudeai Code: 58% pass with Opus 4.6 Try it out: altimate.sh #DataEngineering #Pipelines #ADEBench #ADE

@DatisAgent @sspaeti Compaction prunes old tool outputs into fingerprints, preserves schema/lineage/dbt context in a structured summary and replays the last user message to resume. Memory injection runs in parallel so learned context survives compression. Circuit breaker caps retries at 3.

compaction strategy matters a lot — protecting schema context from eviction is exactly the right call. the tricky part is deciding what counts as load-bearing memory vs ephemeral state. curious how you handle retrieval when the harness needs to reconstruct context mid-run after a compaction cycle

Altimate-code: a new open-source code editor for data engineering based on opencode. Easter comes early for every developer this year. Altimate-code is an OSS agentic code editor that works in the terminal, based on the admired OpenCode AI editor, with dedicated data engineering features. The difference to general AI coding agents such as Claude or Codex is that Altimate is focused on SQL and data engineering tasks, with an understanding of the data stack and integration in tooling. Altimate gives any LLM a deterministic data engineering intelligence layer, avoiding guessing at schemas. Some capabilities that altimate-code adds compared to general coding agents: →SQL anti-pattern detection with 19 rules, confidence-scored →Column-level lineage, automatic from SQL, any dialect →Schema-aware autocomplete based on live-indexed warehouse metadata →Cross-dialect SQL translation for popular DWHs →FinOps & cost analysis, which shows credits & expensive queries →dbt integration with parsing manifest, model scaffolding, and lineage →Observability, with local-first tracing capabilities Under the hood, altimate-code runs a Rust-native SQL analysis engine (altimate-core) that does AST (Abstract Syntax Tree)-based parsing. Very similar to SDF (now dbt Fusion) and SQLMesh, altimate-code has #deterministic SQL linter capabilities, not based on scanning the full project via bash that might take minutes, but in seconds and sub-seconds. Data Engineering Companion Altimate is backed by DuckDB for local SQL validation and Zod for runtime type-safe parameter validation across all tool integrations. It works with dbt, DuckDB, Airflow, Dagster, and all your data tools from the data OSS data ecosystem. It integrates with any DE stack and any LLM provider you know of. JIRA, GitHub, and soon Kubernetes and ServiceNow, so you have an agentic coding partner for all things data engineering, plus ticket management with Infra. Security is a big part of agents, and altimate-code includes a permission guard system. If you have sensitive PII data, Altimate can run entirely locally, pointing it at DuckDB or hooking it up to Ollama. PII detection uses regex patterns, and the schema is cached in an embedded SQLite database locally. The altimate-code check command requires no API key — but you can add to unlock LLM-powered suggestions. It uses Tree-sitter (behind VS Code, Zed, Neovim) to parse and analyze shell commands for safety before execution to detect redirects, file patterns, and potential working directory risks. But try it out yourself. Run a `/discover` or a `/trace`, or just check out the command palette, it's fully open source and available at github.com/AltimateAI/alt…. I personally love the three modes, besides `Plan` and `Builder`, it also has an `Analyst` role, where it explores data, selects queries, and makes analyses - like a real analyst. #genai #dataengineering #paidpartnership

Open-source agentic data engineering harness for dbt, SQL, and cloud warehouses. 100+ tools, 10 warehouses, AI-powered. github.com/AltimateAI/alt…

@sspaeti A couple of things I think everyone should try is ask altimate-code to do a /review of your code changes. Also /trace is pretty impressive to understand exactly what is going on.

@DatisAgent @sspaeti Our harness handles memory and compaction to ensure the correct pieces stay in place for example we don't allow eviction of schemas from context.

Column-level lineage in an agentic terminal editor is interesting. The missing piece I keep running into with agentic data tools is memory — the agent forgets schema context between sessions. Storing column metadata as searchable memory chunks (plain text, BM25 retrieval) works better than re-reading DDL every time.

Check out my latest article: Help me celebrate with a star! linkedin.com/pulse/help-me-… via @LinkedIn

Excited for this webinar by @intjesus on why @databricks would pay $1B+ for @tabulario, and why #Substrait is the next critical step after @TheASF #Arrow, #Iceberg in the quest for data democratization! Register here: us06web.zoom.us/webinar/regist…

April 8th, 2024. Austin. #Eclipse2024

I think @comcast you have a problem. I have no idea how to convey this to your support / chat. They cannot even pull up my account. Thankfully @PingPlotter can show you exactly why I am getting high latency and massive package loss. Can anyone fix this? #highspeedinternet

MyPOV: Event Report: Rebuilding Trust - The 2024 Davos Agenda raywang.org/blog/2024-01/e… The Bottom Line: Rebuilding Trust Must Move From Words To Actions #WEF24 #Davos

As a Muslim CEO in tech, let me just say this: Hamas is an evil terrorist organization that needs to be wiped off the face of the Earth. Murdering innocent civilians (children, babies, families asleep in their homes) goes against every single teaching of the Qur'an. May they all rot in Hell. I pray for peace and innocent lives to be spared as Israel completes this task. But complete it, I hope they do.

What has happened? Have we been transported to Nazi Germany in the 1940s? If you don’t stand up to this type of sentiment don’t be surprised when they come after you and no one is left.

@HenMazzig 💔 Lost for words. I literally sat here looking at the keyboard not knowing what to say. Anyone on earth who does these kinds of acts is demonic.

Mor’s grandmother was slaughtered by Hamas terrorists. She found out about it when the Hamas terrorist used her grandma’s Facebook to livestream his barbaric act 💔

In the light of what’s happened in Israel and Gaza, a song about non-violence seems somewhat ridiculous, even laughable, but our prayers have always been for peace and for non-violence… But our hearts and our anger, you know where that’s pointed. So sing with us… and those beautiful kids at that music festival…

@heyalexfriedman If it was day 1 weren’t they co-founders? What about those from day 2? I could read this as being just the two of them became instant millionaires. Most people who join a startup even early on won’t make the money they think even if the company sells for $500M

My dad and his cofounder sold their company for over $1B a few years ago. I asked them, “What was the best part of selling?” They said, “Seeing all the people who worked with us from day 1 become millionaires.”

I had the privilege of being at opening night for @u2 at the @SphereVegas in #LasVegas. Amazing audio and visuals. The drone captures the size of the sphere but nothing fully captures the feeling of being there. #U2 #Drone youtube.com/watch?v=vAnHTl…

@incent2IMPACT @Apple @BestBuy Thanks @incent2IMPACT it’s been too long. Looks like you’re doing well!

@layereddelay @Apple @BestBuy Congratulations @layereddelay! Sounds very exciting!

Started my next role. Co-creator of a new kind of product. Working with some of the smartest people I’ve ever had the pleasure of working with. Started out right with an @Apple #Macbook from @BestBuy. Stay tuned for news of innovation in the #data space. #startup early stage.

MyPOV: Will Generative AI Drive Humanity Into The Dark Ages of Knowledge? #AI @reinventnet @peteleyden @BoggioJoe @Jerry_Kaplan @elichen @fionama @danielerasmus @gbolles @dazzagreenwood @brianbehlendorf @WhiteCase Earl Comstock raywang.org/blog/2023-08/m…

"To write is human, to edit is Devine." - @StephenKing In the world of #GenerativeAI: "To write is AI. To edit is human." Do you want to be human or devine in the work environment?