Ryanne Dolan

We've been doing cool stuff with SQL @LinkedInEng: engineering.linkedin.com/blog/2023/decl…

@gunnarmorling This is why I write design docs, even tho no one ever reads my design docs.

Never underestimate the power of rubber-ducking. Explaining a problem in a thorough way to someone else (what are you trying to achieve, what have you done, what did you expect to happen vs. what actually happened) will lead you to finding the answer yourself surprisingly often.

@gunnarmorling This is something I am super not good at.

A true super-power to have as a software engineer is finding the right level of detail and depth for a given conversation, depending on its context, participants, goals, available time, etc. It's a rare skill, so you often end up with either too much information, or not enough.

@gunnarmorling That's what we did at Twitter with Avro. With proto, it's the only way that makes any sense.

Question for you Kafka users out there: is there a serializer which resolves Avro/ProtoBuf schemas statically? I.e. instead of connecting to a schema registry at runtime, I'd like to provide schemas and their known ids at build time to my app. Is there such a thing?

@ibuildthecloud Simple systems are always complicated under the hood. You just think your systems are complicated cuz you know how they work.

I love simple systems, but I create ridiculously complicated systems. But I enjoy the pursuit of trying to make them simple. I think I've gotten better at it over time, I used to make insanely complicated systems.

Yo Gemini call screening is magic. Why is no one talking about this. Also how do I get the nice man who answers my phone to just handle everything for me.

@MatthiasJSax @gunnarmorling But all the rest is garbage.

@gunnarmorling Just saying: hook up Kafka as your ingest to your “streaming DB” and use “incremental materialized view maintenance” is a very incomplete picture… (just as an example; I did hear exactly this recently in a podcast about the topic).

There is a lot a material about "streaming" out there. Is it just me, or does a lot of content reduce the problem to "incremental processing"? I am wondering why? "Incremental processing" is an important but non-functional requirement... it seems to miss half of the story?

If there is an AI bubble, it's this: every company needs to be an AI company --> only your OS really needs AI.

Why are all these pro golfers showering together.

I don't trust people who use "non functional requirements". I see you have considered how the system behaves, but have you considered how it should perform? Oh shit 🤯never thought of that wow thanks

@PervisTime Feels good to finally put a name to this disease.

a Pervis book is comic in 4 days! insiders.skybound.com/the-book-of-sh…

@julianhyde If by data you also mean URIs that point at the data, that's basically what we do with Flink connectors. Smth like CREATE TABLE...WITH ('uri'=...). I've always considered this an abuse of DDL, cuz you aren't really CREATing anything that persists anywhere.

@DolanRyanne And that was the case with Hadoop jobs too. Because each job was just a program (jar file), and there was no schema except that baked into the program. But I'm saying you should pass each job not just the schema but also the data it needs.

Serious proposal: Make SQL a stateless language. In every SQL query, you pass in the whole database that you want to query. I already do this when I use sqlfiddle.com, like this:

Who's gonna be the first CAIO?

@julianhyde We do something along these lines with our Flink SQL jobs. Each job includes all the DDL it needs to run, and otherwise has no a priori metadata.

So why not pass in the URI of an immutable database? This syntax makes it evident that SQL just an engine, not a bookkeeper. And with some caching it can return quick results.

Limelight is back in the two spot 💪.

Fair weather in TX means I can barely maneuver my oversized truck without rolling over a pedestrian or two.

spooooky

I don't really need ChatGPT cuz my wife is a speed reader. I regularly send her PDFs so I can ask her questions. She has a separate email address for this purpose. No joke.

Such a bad take. If you give a bunch of money to a bunch of engineers, they'll end up in space whether or not you wanted them to.

@nbuesing @coltmcnealy Technically it follows Scala constructs. It was ported from Scala, which has always done this.

@coltmcnealy Another reason I love the kafka-clients API. consumerRecoed.topic() not getTopic(). Best of both worlds - encapsulation but less boilerplate. Follows the Java Record constructs; not the JavaBeans constructs.

One of the engineers I admire the most told me today "public fields are fine," and you don't need to use getters and setters. Just sayin' 😂

@gunnarmorling I think the difference between lib and app is at best murky. At scale, every app ends up being a library. I've even imported entire CLIs.

@DolanRyanne That's primarily from the PoV of library authors though, right? As app dev, you can (and should) use the latest and greatest, assuming all your dependencies play well with it of course (though I feel that's less and less an issue after the big changes in 9-11).

The biggest problem of #Java is poor perception. It's technically super-solid, but too often folks discard it based on misconceptions or information outdated years ago.