Jack Vanlightly

1.6K posts

Jack Vanlightly banner
Jack Vanlightly

Jack Vanlightly

@vanlightly

@confluentinc thinking about event streaming. Ex @Splunk, @VMware https://t.co/3axXZezyy4, https://t.co/voJWmL4KM6 Credit: ESO/B. Tafreshi

Barcelona, Spain Katılım Kasım 2016
234 Takip Edilen4.7K Takipçiler
Jack Vanlightly
Jack Vanlightly@vanlightly·
Seen some discussion of unit tests vs end-to-end/integrations tests in the age of AI. A few years ago I proposed the "testing cafetiere" to replace the "test pyramid". These days I think the randomized system testing could be replaced by, or least augmented with deterministic simulation testing.
Jack Vanlightly@vanlightly

In my #heisenbug talk on #distributedsystems testing I introduced the "Testing Cafetiere" as a distributed systems specific test strategy. - Use randomness to find your edge cases --> add your edge cases to your regression tests - Test your implementation - Verify your design

English
0
0
3
504
Jack Vanlightly
Jack Vanlightly@vanlightly·
@thegeeknarrator @almoggavra For example, you might have a leaderless system which is on the stateless end, which is more efficient if you do some soft affinity (shifting it a little on the spectrum).
English
1
0
1
34
Jack Vanlightly
Jack Vanlightly@vanlightly·
@thegeeknarrator @almoggavra Right, in general terms. On the stateless end of the spectrum it's very responsive. I call it a continuum because there are degrees of state-less/full-ness.
English
1
0
2
32
Almog Gavra
Almog Gavra@almoggavra·
"object store native" gets tossed around too lightly. As I see it, there are 3 gens of systems: (a) col/tiered storage, (b) disaggregated compute and (c) stateless (?) "we use object storage for durability" feels substantially different from "nodes are just caches"
English
3
1
7
3.6K
Jack Vanlightly
Jack Vanlightly@vanlightly·
@thegeeknarrator @almoggavra I agree (may be not the terms) but the stateless vs stateful side. There is a stateless<-->stateful continuum even on systems which separate compute from storage which dictates how disruptive change is, how responsive a system is to change, whether failures or scale-out/in.
English
1
0
2
47
Kaivalya Apte - The Geek Narrator
The way I see it (which I think aligns with your view I guess) - Tiered Storage: local first, object storage for cold durability - Disaggregated compute: Object storage is for durability. Still fundamentally stateful workers. - Object-native: Object storage is the truth. Workers are disposable caches.
English
2
0
4
453
Kaivalya Apte - The Geek Narrator
I HAVE JOINED @confluentinc 🥳 Thrilled to be joining the Flink engine team as a Software Engineer, which works at the intersection of Distributed systems, Databases and Stream processing. All the topics I am very excited about in general. This is the first company where it feels like I already know a lot of folks and have talked to them about nerdy topics on The GeekNarrator show. Looking forward to be working closely with them all and building something amazing.
Kaivalya Apte - The Geek Narrator tweet mediaKaivalya Apte - The Geek Narrator tweet media
English
27
4
239
13K
Jack Vanlightly
Jack Vanlightly@vanlightly·
@TylerFCloutier I am so glad I bought this thing before the worst of the RAMpocalyse! I *only* had to pay 500 euros per 64GB stick.
English
0
0
0
31
Jack Vanlightly
Jack Vanlightly@vanlightly·
I love my Threadripper 9980X with 256GB of RAM plus a couple of Samsung 9100 PRO 8TB drives. I run some decently sized multi-node Kafka experiments on it and it hums quite nicely. But damn, TLA+ TLC is memory bandwidth hungry. With only 4 memory channels on the Threadripper, TLC reaches my 170GB/s actual memory bandwidth quite easily. My memory is often the bottleneck. The random walk nature of TLC means L3 cache is pretty much useless.
English
3
0
12
1.8K
Jack Vanlightly
Jack Vanlightly@vanlightly·
@criccomini I'm too busy making Kafka burn out my expensive SSDs 😂 Here's hoping I can expense that one
English
0
0
1
97
Chris
Chris@criccomini·
@vanlightly Aren’t you supposed to be running open weight models on a machine like this? 😀
English
1
0
1
203
James Cowling
James Cowling@jamesacowling·
Don't know how I got this far in my career without realizing that us-east-1a/b/c are just random identifiers and 1a for one account might be 1c for another
English
32
7
567
81.4K
Jack Vanlightly
Jack Vanlightly@vanlightly·
I think of that quote every time I shoot in a game of basketball, but unfortunately, there's only so far my skill can take me 😂
English
0
0
0
235
Jack Vanlightly retweetledi
Gunnar Morling 🌍
Gunnar Morling 🌍@gunnarmorling·
📝 "Hardwood Reaches Beta: S3, Predicate Push-Down, CLI, and More" Happy to announce Hardwood 1.0.0.Beta1, adding S3 support, predicate push-down for both local and remote files, Avro bindings, a CLI for inspecting Parquet files, a new website, and much more. Links below 👇!
Gunnar Morling 🌍 tweet media
English
2
5
22
3K
Jack Vanlightly retweetledi
Gunnar Morling 🌍
Gunnar Morling 🌍@gunnarmorling·
Hello, hardwood-cli 🖥️! Happy to share the latest addition to the #Hardwood project: a command line tool for inspecting Parquet files. Think of it as a Swiss Army knife for Parquet; use it to get general file info, schema, encodings, compression ratios, page details, and more. You can take a quick look at some records, convert to JSON, etc. Early access builds for Linux/macOS/Windows available from the GitHub repo (link below); a first tagged release of the CLI is planned for the next week. A big, big shout-out to @brbrown for spearheading this effort. While I had planned this for only after the 1.0 release, he didn't hesitate and made it happen right now. That's the magic of open-source in action!
Gunnar Morling 🌍 tweet media
English
1
5
37
3K
Jack Vanlightly
Jack Vanlightly@vanlightly·
Yeah, before bed, I 've got to set up my agents to keep running overnight. It used to be I would tell my wife "I just need to kick off a benchmark, or TLC to run overnight so I have results in the morning." Now it's "I've just got to tell claude what to do overnight so I have a new feature in the morning"
English
1
0
1
160
Jack Vanlightly
Jack Vanlightly@vanlightly·
@gunnarmorling So basically, it's like talking to a chat bot. Mix of decent, some incredible and some truly bizarre screw ups!
English
1
0
1
417
Gunnar Morling 🌍
Gunnar Morling 🌍@gunnarmorling·
Working with Claude Code is such an up and down. One moment it identifies and fixes issues with unbounded state growth in a Flink job (noice!), the next minute it does really bizarre mistakes like inventing "as" imports for Java.
Gunnar Morling 🌍 tweet media
English
6
4
54
6.5K
Jack Vanlightly
Jack Vanlightly@vanlightly·
It blows my mind sometimes the nuanced edges cases that TLA+ finds, edge cases that I had never considered. The foundations of what I thought I knew about the leaderless shared log recovery protocol I am designing have been so wrong!
English
3
2
43
3.2K
Conor McCarter
Conor McCarter@ConorMcCarter·
@vanlightly i'd love to understand TLA+ and learn how to implement it. any recommendations or resources for ramping up?
English
1
0
0
121