ทวีตที่ปักหมุด
Pruthviraj P
817 posts
Pruthviraj P
@spidernvdev
Deep Learning SDE @nvidia · ex-nobody cuda guy
California, USA เข้าร่วม Ocak 2017
188 กำลังติดตาม301 ผู้ติดตาม
simple mental model:
large model training is like moving a huge machine across multiple rooms.
each gpu holds part of the work.
but the hard part is not only splitting it.
the hard part is making all parts communicate fast enough.
English
morning from the green matrix.
green matrix note 01:
why big models need parallelism
a 70b parameter model needs ~140gb just for fp16 weights.
that is before activations, gradients, optimizer states, and training data.
one gpu is not enough.
English
@spidernvdev Indeed, products only need imagination, but engineers have a ton of other stuff to handle.
English
most ai advice online is about prompts.
but if you want to build real ai products, you also need to understand:
latency
memory
cost
evaluation
deployment
prompting helps.
systems knowledge compounds.
English
we're starting rollout of GPT-5.5-Cyber, a frontier cybersecurity model, to critical cyber defenders in the next few days.
we will work with the entire ecosystem and the government to figure out trusted access for cyber; we want to rapidly help secure companies/infrastructure.
English
GPT-5.5 is going to have a party for itself. it chose 5/5 at 5:55 pm for the date and time.
if you'd like to come, let us know here: luma.com/5.5
codex will help the team pick people from the replies. 5.5 had some good ideas/requests for the party, which we'll do.
English
one thing i underestimated early in ai:
the model is only one layer.
behind every good ai product there is data loading, preprocessing, gpu memory, serving, monitoring, and debugging.
that full stack is where engineering gets interesting.
English
@seunosewa @hiarun02 It can, but total cost isn’t just per-token, its also how many passes it takes. Fewer retries can offset higher pricing.
English
Most cs grads studying rn would not even know all these terms including me.... Still learning.....long way to go.....
maharshi@maharshii
triton, gluon, cutedsl, hopper, blackwell, tensorcores, layouts, composition, local_tile, partitionS, partitionD, wgmma, tcgen05, TMA, block scaling, coalesced access, ampere, ada lovelace, cutlass, cublas, cudnn, flash attention, gemm, sgemm, fp16, bf16, mxfp8, nvfp4, int4, quantization, mixed precision, occupancy, reductions, warp divergence, bank conflicts, memory coalescing, shared memory, global memory, texture memory, constant memory, unified memory, epilogues, kernel fusion, graph optimization, tensorrt, torch compile, dynamo, inductor, graph capture, thread blocks, warps, SIMT, streaming multiprocessors, L1 cache, L2 cache, register spilling, thread divergence, memory bandwidth, compute capability, CUDA cores, ldg, stg, ncu, nsys, atomic operations, syncthreads, cooperative groups, dynamic parallelism, persistent kernels, vectorized loads, static quantization, tensors, swizzling, predication, instruction throughput, memory latency hiding...
English
anthropic adding claude connectors for tools like blender, adobe, autodesk, and splice is the right direction.
the next ai jump may not be only smarter models.
it may be models that understand the tools people already use.
English
sometimes i wonder if i'm learning fast enough or just keeping up with noise
English
@PengmingWang this took me a while to learn
you can have a great model and terrible evals,
and you won't know until something breaks in production
English
My updated take on "The 'it' in AI models is the dataset." is that the 'it' is the evals you're using.
I rather have bad scores but great benchmarks than good scores but bad benchmarks.
English
@nvidia blackwell costs 2x more than hopper on paper
delivers 35x lower cost per million tokens in practice
that gap is the what needs to be understood
English
groot-x has 24,000 simulated teleoperation runs across humanoids and manipulators
mixing this synthetic data with real improved model performance by 40%
n1.7 now runs with a 3.3B param backbone on ~6GB vram
open source physical ai is moving faster than anyone expected
NVIDIA Robotics@NVIDIARobotics
The Physical AI Robotics GR00T‑X Embodiment Sim dataset has surpassed 10 million downloads on @HuggingFace. 🥳 A huge shoutout to the global research and developer community exploring the future of embodied AI and robotics with this open dataset — you made this milestone possible. 📥 Try it on Hugging Face 👉 nvda.ws/3Qv64Ul
English
@ivanfioravanti @kernelpool curious to see how it holds up in the 8-bit tests.
English
Look at the amazing performance boost after applying a Metal Kernel as suggested by @kernelpool 🤩
Night & Day in Prefill that is now much faster!
Gonna publish an 8bit comparison test soon and then testing this super model with coding soon!
Ivan Fioravanti ᯅ@ivanfioravanti
MLX Ling-2.6-flash support added! 💪 Here my (preliminary, because I bet @angeloskath will improve performance) context benchmark for the 4bit version running on M3 Ultra (cooking a new version) I created the PR with the amazing transformer_to_mlx skill by @pcuenq and Opus 4.7. Few iterations and it seems 😂 working 100%! Ultra fast model created by @TheInclusionAI. Can't wait to test it with a code harness! Raw results: Ling-2.6-flash-mlx-4bit MLX Benchmark Results Hardware: Apple M3 Ultra, 512.0GB RAM, 32 CPU cores, 80 GPU cores 0.5k pp 632 tg 79 t/s mem 59.4GB kv 0.03GB 1k pp 676 tg 79 t/s mem 59.9GB kv 0.04GB 2k pp 693 tg 79 t/s mem 61.1GB kv 0.04GB 4k pp 704 tg 79 t/s mem 61.1GB kv 0.05GB 8k pp 708 tg 78 t/s mem 61.2GB kv 0.07GB 16k pp 700 tg 77 t/s mem 61.9GB kv 0.11GB 32k pp 678 tg 74 t/s mem 64.5GB kv 0.18GB 64k pp 637 tg 70 t/s mem 69.5GB kv 0.33GB 128k pp 564 tg 63 t/s mem 79.6GB kv 0.64GB Total generated tokens: 1135 Batch TPS: b1 78 b2 123 b4 164 b8 218 b16 307 b32 418 Batch KV : b1 0.04GB b2 0.08GB b4 0.16GB b8 0.32GB b16 0.63GB b32 1.26GB
English