Piotr Sarna

I tried running `ollama run laguna-xs.2` fully expecting to see an OOM, but instead I got functional local inference running on nvidia 4060 laptop... so I checked how it's even possible if the weights are 23GB, and the card only has 8GB vram: 🧵

I love how hardware vendors call their lowest-quality products "consumer-grade," and the consumers who buy them aren’t bothered by that at all.

prefill speedup means the world to me and my consumer GPU 👇 I need to read a technical writeup on the details

Amazing, thanks for having me. If you want to know more about technical blogging, I suggest subscribing to «Write that blog». My humble story and how I write is explained in this latest issue.

poolside hackathon at the end of may. we're providing the compute. winner gets a DGX spark from NVIDIA. disclaimer: the hardware might be infected by @sarna_dev's case of local laguna addiction

Poolside is hosting a 2-day model research hackathon in London. Join us to push an open-weight agent model as far as you can. RL and fine-tune Laguna XS.2, our latest-generation model, on Prime Intellect Lab. Dates: May 29–30 Partners: @nvidia + @PrimeIntellect + @huggingface Prize: NVIDIA DGX Spark Agents need better models. Better models need cracked researchers. Link below.

One sad aspect of using a laptop GPU with 8GB vram for local inference is that prefill is memory-bound. Normally the prefill stage is heavy on compute, but since the model weights do not fit in my lousy GPU, they need to get regularly squeezed through PCIe. I experimented with different prompt lengths and it's clear that I need to keep them short to get a reasonable time-to-first-token. 71% spent on host-to-device memory transfers rather than running kernels. Sad, this is where Apple's unified memory model clearly wins. @nvidia's nsys is a really nice profiler btw

Get Laguna XS here huggingface.co/poolside/Lagun… or just pull it with `ollama run laguna-xs.2`. Enjoy quality local inference from @poolsideai!

tl;dr you don't need Apple's AI silicon with their impressive >100GB/s unified memory bandwidth for local inference. A consumer-grade laptop handles it just fine.

I tried running `ollama run laguna-xs.2` fully expecting to see an OOM, but instead I got functional local inference running on nvidia 4060 laptop... so I checked how it's even possible if the weights are 23GB, and the card only has 8GB vram: 🧵

技術ブログの書き方だけで400ページ弱の本。実例や示唆に富んでおり、おすすめ。 / Writing for Developers learning.oreilly.com/library/view/w…

As agents get more clever, so do their attempts at benchmark hacking. Last Monday, we found one of our RL runs jumped ~20% on SWE-Bench-Pro over a weekend, reaching ~64% which would make it #1 on the leaderboard. This was clearly benchmark hacking and we patched the exploit. But this revealed deeper hacks across multiple public benchmarks, some of which were impossible to fix through environment design alone. Evals need to evolve beyond just outcome based pass rates to better observability into how the agent is arriving at them. These were our findings: poolside.ai/blog/through-t… Examples below 👇 1/

@glcst huggingface.co/poolside/Lagun… the demo impl simply uses pool (github.com/poolsideai/pool) because it comes with its own auth and all, but ultimately it should just call the configured API directly

@sarna_dev using which model / embedding? IF it comes with a reasonable api to plugin an external one, why not make this a first-class citizen on Turso ?

hear me out: SQLite plugin that lets you ask for rows in natural language. Returns regular rows, so you can join, subquery, and all that. SQLaguna.

github.com/psarna/sqlaguna (yes, developed by Laguna XS and runs on Laguna XS by @poolsideai, as every other project of mine in the last few weeks)

April in writethat.blog 👇 - Laguna XS.2 and M.1: a deeper dive // @poolsideai - Understanding the Go runtime: the network poller //@jespinog - What are skiplists good for? // Will Wilson, @AntithesisHQ - Do you even need a database? // Jay, DBPro - Open source security at Astral // William Woodruff, @astral_sh - The Git commands I run before reading any code // Ally Piechowski - 10x faster tokenization // @steeve, @zml_ai - How Meta used AI to map tribal knowledge in large-scale data pipelines // Krishna Ganeriwal, Plawan Rath, Ashwini Verma, @Meta_Engineers - Supercharging Redpanda Streaming with profile-guided optimization // @StephanDollberg, @redpandadata

New addition to the library. I already read the ebook, but this gem deserves a physical spot on my bookshelf