@finkd If you don't open source nobody will care about your models
English
nonsensinator
460 posts
nonsensinator
@nonsensinator
not everything needs to make sense
Katılım Şubat 2026
27 Takip Edilen10 Takipçiler
Elon was wrong
I got it wrong too, expected September but way too fast.
Only took a month from this picture.
Jun Song@jun_song
Prediction: By the end of Q3, 2026 Fable class on your own computer.
English
Day 11/90 of Inference Engineering
How does vLLM work and how is it used in production?
Before we discuss how vLLM works internally, it helps to understand what vLLM is. At a high level, vLLM is an inference engine that is designed to serve LLMs to thousands of concurrent users efficiently while managing scarce compute and memory. The goal for vLLM is to maximize throughput and minimize latency; optimizing for the best inference economics and experience for end users.
With every request from the end user, it eventually ends up in the engine core, gets scheduled alongside other requests from other concurrent users, executes on the GPU, and updates the KV cache with the new key and value vectors, and streams the tokens back to the user.
The Scheduler decides what requests should execute next while continuously batching requests together to maximize GPU utilization. Continuous batching is an inference optimization that allows new requests to join a running batch as other requests finish generating tokens. This helps with keeping the GPU utilization high instead of letting it sit idle waiting for an entire batch to complete generating.
After the scheduler dispatches the selected batch to the Model Executor, the Model Executor prepares the tensors and metadata required for inference, retrieves each request’s block table from KV Cache Manager, launches the optimized transformer forward pass on the GPU, computes the logits, updates the KV cache with the new key and value vectors, and finally returns the results for sampling and streaming.
The KV Cache Manager uses the PagedAttention memory layout to allocate fixed-size cache blocks on demand and maintains a Free Block Queue on the CPU that tracks which blocks in the GPU’s Paged KV Cache are currently free. When a request needs additional KV cache space, the KV Cache manager takes a free block from the queue and assigns it to that request, thus avoiding an expensive search through GPU memory for available cache blocks.
All of these components form the core of vLLM’s inference engine. The Scheduler determines what requests are executed, the Model Executor determines how those requests are executed, the KV Cache Manager determines where each request’s KV cache lives using the PagedAttention Memory Layout. This architecture enables vLLM to serve thousands of concurrent requests with high throughput, low latency, and efficient GPU memory utilization.
Heres a little animation that visualizes everything!
-
I've also completed the forward pass for my mnist.c project. I had a nice chat with @shreybirmiwal, such a knowledgeable guy. Excited to learn more about vLLM and implement a tiny-vLLM one day.
English
@LLMJunky that's like anthropic 20$ plan then
pretty useless
thanks for the info
English
The rate limits on Kimi Code on the $19 plan are really bad imo.
I only got through about ~85% of building a browser game before hitting the 5hr quota. This burned 20% of my weekly, so I only get 4 more quotas for the week.
That was one prompt, and it didnt even finish. This is not a large project.
Fable one shot this for me on the $20 plan without running out at all.
English
@elonmusk You uploaded user data to the servers and we need to trust it ??🤣🤣🤣
English
Try Grok
thehype.@thehypedotnews
kimi k3 vs gpt 5.6 sol vs fable 5 vs grok 4.5 @Kimi_Moonshot just dropped kimi k3 – a 2.8t param native multimodal model, the first open 3t-class release. key facts: • 1m token context. stable latentmoe activating 16 of 896 experts, built on kimi delta attention (kda) and attention residuals • quantization-aware training from the sft stage onward – mxfp4 weights, mxfp8 activations. moonshot claims ~2.5x scaling efficiency over k2 • max thinking effort by default. low- and high-effort modes are "coming in updates" – there is no way to turn the thinking down today, and you feel it in every run • pricing: $0.30/mtok cache-hit input, $3.00/mtok cache-miss, $15.00/mtok output. claims >90% cache hit rate on coding workloads • benchmarks: swe marathon 42.0 (1st – fable 5: 35.0, sol: 39.0, opus 4.8: 40.0), terminal bench 2.1 88.3, browsecomp 91.2 (1st), program bench 77.8 (1st), gpqa-diamond 93.5. loses frontierswe 81.2 vs fable's 86.6, and deepswe 67.5 vs sol's 73.0 our test – 3 prompts, single-file html, @threejs, fully procedural, no assets: 1. photorealistic european roulette wheel – 37 pockets in the real sequence, mahogany clearcoat bowl, chrome turret, diamond deflectors, flick-to-spin, ball that spirals inward and settles on a mathematically real number 2. las vegas slot machine – 3 reels behind transmissive glass, drag the chrome lever to play, mechanical odometer counters modelled in 3d, coin physics on win 3. full pinball table – 6.5° tilted playfield, flipper impulse physics, spline ramps, drop targets, 6 bumpers, mechanical score reels in the backbox we ran the test on @aimlapi platform results: - cost #1 grok 4.5 – $0.30 #2 kimi k3 – $0.71 #3 gpt 5.6 sol – $2.05 #4 fable 5 – $7.69 - tokens #1 grok 4.5 – 34,241 #2 gpt 5.6 sol – 51,748 #3 fable 5 – 144,126 #4 kimi k3 – 157,999 - lines of code #1 gpt 5.6 sol – 3,054 #2 grok 4.5 – 3,047 #3 kimi k3 – 2,255 #4 fable 5 – 1,950 - generation time #1 grok 4.5 – 5.1 min #2 gpt 5.6 sol – 22.0 min #3 fable 5 – 31.5 min #4 kimi k3 – 75.6 min observations: • kimi k3 is cheap and it is slow. 75.6 minutes across three prompts against grok's 5.1. it is 2.4x grok's price and 15x grok's wall clock. the roulette took 15 min, the slot 18, the pinball 42 • it failed 2 of 3. only the roulette works. the slot machine has reel cutouts on both faces of the cabinet and the symbols face backwards – you can only read your spin by walking around to the rear of the machine. the pinball table stands vertically on its edge with the legs floating detached beside it. • 81% of kimi's output tokens are reasoning, not code. grok: 22%. you are not paying for a bigger answer, you are paying for a longer argument with itself • price per 100 shipped lines – grok $0.010, kimi $0.031, sol $0.067, fable $0.394. a 39x spread for the same three files kimi k3's code quality: upsides: • the roulette is genuinely good – procedural wood grain with real specular breakup, correct european sequence (0-32-15-19-4...), chrome turret, diamond deflectors, clean console • the pinball artwork is the best in the test – a synthwave "nova strike / deep space" field with six individually coloured neon bumper rings, a retro sun on a grid horizon, a nova burst, and a scoring legend printed on the apron. no other model printed the rules on the machine. it is a beautiful texture on a broken object • physics reasoning is real – it derived a 480hz substep for the collider, worked out ball settle conditions and termination guarantees, and checked every ramp exit vector by hand before writing any of it • it is the only model that saw the importmap trap coming. sol shipped a blank white page twice because three.js addons import the bare specifier 'three' and die without an import map downsides: • it dodged that trap on the slot by loading three.js r128 through classic script tags – a 2021 build with no working transmission. its slot glass rendered fully opaque and buried all three reels behind a white pane. the code asks for transmission: 0.93, ior: 1.5 – correct, and silently ignored by a renderer that predates the feature • after 42 minutes and 212k characters of reasoning, the pinball cabinet is not assembled. the table stands vertically on its edge like a wardrobe – the prompt asked for 6.5° from horizontal, it delivered 90°. the legs float detached in the void beside it. head-on it photographs beautifully; orbit ten degrees and it is a painted slab with four chrome rods hovering nearby • the playfield z-fights with the glass – hard black banding across the whole field as soon as you pull the camera back a note on the pinball, in fairness to kimi: nobody passed it. every model shipped broken ball physics and controls you cannot trust. it is the hardest prompt we have run and the whole field failed it, each in its own way kimi k3 reasons better than anything else here and it shows exactly where reasoning pays – physics constants, sequences, edge cases, traps the others walked into follow @thehypedotnews for 24/7 ai news, analysis and breakdowns
English
Kimi founder Zhilin Yang studied in the US and loves hippie music and prog rock enough to name his office rooms after them. Opennessmaxxing indeed brings out the best creativity in the high IQ East Asian... congrats on K3
Russ Salakhutdinov@rsalakhu
Congratulations to Zhilin Yang, founder and CEO of @Kimi_Moonshot, on the latest Kimi release. What a huge win for the open-source community! It feels like just yesterday Zhilin was graduating from my lab at CMU, jointly co-advised with William Cohen. Not only did he complete his Ph.D. in just four years, but he also made truly fundamental contributions to ML during his time at CMU. What a spectacular career! Congrats again Zhilin, and thank you and the entire Kimi team for everything you're doing for the open-source community.
English
YouTube
English
Tracking a lot of stuff from qwen locally, could add numbers and graphs, but it's not the way.
Now I'll be moving toward setting up experiments and seeing how the model moves across time, differnt prompts, manipulating the inference, etc...
Want to see comparisons and patterns
English
@alemannoEU A meeting to talk about the meeting to decide about when to put the meeting about making decisions about meetings.
English
Really? This is no joke. It’s more ‘simplification’
English
dear Apple,
please bring back the Touch Bar. with a design like this.
sincerely, every MacBook user ever.
English
@livsmayfield The fact the players don't give a shit about this match
English
11 hours ago kimi update --> v0.25.0
2 hours ago kimi update --> v0.26.0
And K3 is here!!!!!!
English
Kimi k3 is god
I made a farm with real animals, now it's building solar panels.
I will soon retire
English
Our first model, Inkling. Trained from scratch, weights are open, fine-tunable on Tinker today.
Thinking Machines@thinkymachines
Today, we are introducing Inkling. Inkling reasons efficiently across text, image, and audio modalities. We are making the full weights available. thinkingmachines.ai/news/introduci… Available today for fine-tuning on Tinker. Play with it in the Inkling Playground. 🧵
English
@BraedendotTECH Had the same feeling some months ago.
Got back into reading books, exercising a bit and playing music and I feel normal again.
English
I'm 33 and I think Claude Code is melting my brain.
For 6 months straight I've had 5-6 terminals open at once, waiting on responses just to smash "enter" 90% of the time. That's the whole job now.
And it's doing something to me. A few friends and I keep circling back to the same thing in conversations: none of us feel as sharp as we used to.
Maybe it's just us. But I keep wondering how many other people in their 30s feel it too.
(And yeah: this is a me problem, how I lean on the tool, not the tool itself. Doesn't make the effect any less real.)
English