DataEngDude

I got the Agate hound out.

Your Financial Life Is Too Complicated. Do This Instead. open.substack.com/pub/firefinanc… #money #finance #culture

@Triamus1 Well, like, whatever man.

@dataenggdude I don't understand these kind of posts. The user interface is just different and it's not a replacement, it's a developer interface change. Just a different thing imo.

Why I'm replacing Polars with DuckDB open.substack.com/pub/dataengine…

Solo River Hike + Steak Cooked on a Bluff (Pure Peace) youtu.be/j9rdPaQw9n8?si… via @YouTube #outdoors #nature #bushcraft

Lunch. Suckers.

@derekdfulton Self

@dataenggdude Shots fired. Self hosted or Mother Duck?

When my five closest friends were addicts in 2011, I was an addict When my five closest friends were big tech engineers in 2016, I was a big tech engineer When my five closest friends were content creators in 2023, I was a content creator The fastest way to change your destiny is to change your friend group. You cannot change your friends to be better. But you can always find better friends! Stop thinking you need to learn yet another AI tool. If your friend group isn’t inspiring, you’ll stay stuck!

On observability If you can’t see lag, throughput, and error rate for your pipelines, you’re not operating a platform; you’re hoping, and hope is not a runbook.

On schema change Schema drift isn’t proof you planned badly, it’s what happens when the business, the apps, and the vendors all keep moving; the goal isn’t to freeze schemas forever, it’s to handle change without breaking everything downstream.

On CDC CDC is powerful, but it isn’t automatically the right choice; don’t implement it just because it’s modern. Implement it when full refreshes are too slow, deletes matter, or the business actually pays for near-real-time sync.

We don't care if it's Monday night or not.

Don't Be That Guy open.substack.com/pub/firefinanc… #finance #money

On vendor noise When you pick batch, micro-batch, or streaming, this SaaS ... that Saas, you’re choosing cost, complexity, latency, and ops burden; and the vendors’ marketing is not going to spell those trade-offs out for you. Use common sense, question everything.

Architectural Foundations & Infrastructure - Part 1 Get it right … or else. #data #software #architecture open.substack.com/pub/dataengine…

On batch Batch ingestion is the workhorse of most data platforms, and it’s probably what you should use most of the time, when the business question is “how fresh does this really need to be,” the honest answer is often batch, not streaming dressed up as innovation.

Artemis II crew just sent this back… round earthers you good?

tools vs architecture Choosing Spark, Kafka, or DuckDB isn’t a magic pill that fixes every problem or delivers a platform immune to failure, and when someone answers every platform question by blaming a single pipeline tool, it usually means the real issues run deeper than the framework.

The Most Disturbing Pioneer Tale Ever Documented youtu.be/Jv8DEkOOJR8?si… via @YouTube

Bury me here.