Phil Howes

We've launched the fastest GLM 5 API available at 190 TPS and 0.79 sec TTFT with the Baseten Inference Stack. Ready for your coding and agentic workflows. baseten.co/blog/how-we-bu…

so much potential in this model and @aqaderb coming out of the gates just ripping the landscape on perf

speculation, in this case a eagle-3, remains one of the biggest levers to go from good to great. amazing job to leapfrog the market and get the most out of our GPUs

💪🫡 still plenty of juice to squeeze out of this one

you can just do things faster

@jxmnop if you read this and still want to learn cuda anyway, we’re hiring for this at @baseten to get more brrrr/dollar. dms open

in 2025, if you want to become a successful AI engineer or researcher, you should NOT learn CUDA furthermore – i'd guess that 80% of successful ML researchers have never written a CUDA kernel practical ML is about training models and using them to make predictions. this has nothing to do with CUDA CUDA is necessary in two cases: (a) you are developing a radically new model that isn't easily expressible in PyTorch or Jax (i.e. Mamba) (b) you are running into performance bottlenecks from current CUDA code and need to make it faster i doubt that either case applies to you chances are you aren't building the next Mamba, and the bottlenecks you'll run into in practice are different you should work on finding the right data or hardware or setting things up properly or distributing efficiently across hardware or researching new efficient ways to run models that other people are working on (like vLLM and SGLang) or better than that, work on your eval pipeline. find ways to measure your model's performance that are more realistic, comprehensive, efficient, fair, etc. TLDR: want to learn? spend your time tinkering with models in PyTorch and Jax. not writing matrix multiplications

New Qwen-QWQ running at 90tokens/s generation speed on a single H100 @baseten using a new spec-dec stack. Around 2x more than the rest of the leaderboard (artificialanalysis.ai/leaderboards/p…).

hit new peak demand today, 3 million RPS. thanks for stress testing our infra anon internet friend

@tuhinone what were your <think> tokens?

The models are available at the following links: Llama 3 8B Instruct: baseten.co/library/llama-… Llama 3 70B Instruct: baseten.co/library/llama-…

Congrats to Conviction and Embed companies @baseten @Figure_robot @harvey__ai @langchain @MistralAI @sierraplatform @pika_labs (and our many pioneering friends) for making the #ForbesAI50 list! Ground floor of the revolution that will lead to many massive companies.

2 things. 1. i have loved working on this team. model performance is so much fun and so rewarding. 2. persistence is key. we started working on model performance end of 2023 and watching us slowly become better and better has been an incredible experience.

@saranormous i strongly recommend @Fall_of_Civ_Pod for you both, great production quality long form history

My 6yo daughter is really into archaeology so I’ve been learning — I get more excited about ancient civilizations than about dinosaurs, and archaeology x tech is a cool intersection. A couple sites I’ve been scoping for an expedition:

when i tell people working in infra is like being a plumber people assume it’s because of lots of pipe connecting, when in fact it’s because i spend most of my day digging through shit

@thomasschiavone @saranormous @awscloud @baseten gotta get gpus somewhere. welcome aboard, happy to bear the brunt of the pain

@aqaderb couldn't do it without you friend

enduring businesses are 10x better and cheaper than incumbents. it's hard to believe that there isn't a world where AI powers 10x better products. but it's unclear if those products are cheaper. Baseten has helped, and will continue to help, builders and enterprises build those enduring businesses. we will make it cheap to run these models, fast to make your experiences magical and reliable so you can focus on building.

every day i get to work with a world class team supporting customers with world class products. today we get to dream a little bigger

Ready to try open source LLMs? Switch from GPT to Mistral 7B in the smallest refactor you'll ever ship: just 3 tiny code changes. If you're making the jump, DM us for $1,000 in free credits. baseten.co/blog/gpt-vs-mi…

Repurposing @tuhinone's Llama v2 truss, got FreeWilly 2 up in under a minute. `:s/meta-llama\/Llama-2-70b-chat-hf/stabilityai\/FreeWilly2`. 275GB of weights later we're running at 23 tok/s out of the box.

We keep getting asked by users if they can use the 70B parameter model in production. We're serving the chat variant of Llama-2 70B on 2xA100 and getting pretty great throughput — it's cooking!