Mike Bilodeau

@AliesTaha @philipkiely @AliesTaha i am almost certain there's some b10 ice cream in the eng side freezer

@philipkiely what is inference? how does it work? @philipkiely can i come to learn (and also maybe get ice-cream)?

Ice cream and books were a hit yesterday. ICYMI we're doing another, this time at the Ferry Building. Thursday 4/2 from 2-4 PM: luma.com/khxc93ju

@aishwarya_08 i also like when they tell me what i'm doing

@mj_bilodeau What a milestone 🤣

the AI sdr urge to start off every email by congratulating someone for the exact dollar amount and valuation of their last fundraise

richard hendricks still undefeated it seems :/

What is AI inference engineering, why is it such an in-demand skill, and how do you break into the field? With author of Inference Engineering @philipkiely and head of training at Baseten @oneill_c 0:00: What is inference? 2:47: History of inference 4:59: Downstream effects of AI research on inference 13:54: What you'll learn from Inference Engineering 16:14: Advice for engineers transitioning into AI 19:00: Open source models driving inference growth 20:55: Specialization vs. frontier closed models 23:51: "Big Token" and the importance of open source AI 27:18: Where to get Inference Engineering

@Madisonkanna i am thrilled there are no wood slats in this recording

@mon__lim @baseten just you wait

addicted to @baseten ice cream. hope they make this a staple. inference has never tasted so good

@FintechKristen @evancharles @baseten higher rations of ice cream for sure than a billboard

@evancharles @baseten Is this Humphry Slocombe? Higher or lower ROI than an SF billboard?

Would love to know which genius in growth at @baseten is responsible for sponsoring an ice cream flavor

getting to work with teams like the one at @zeddotdev is what makes this job fun

We love working with Nathan, Conrad and the Zed team, and we're proud to power inference for their edit prediction for users around the world!

@zeddotdev @baseten Zeta speed 🤝 b10 speed

Your AI code completions in Zed show up in ~200ms. That's Zeta, our Edit Prediction model, running on @baseten. We love partnering with companies who keep the bar high — Baseten is one of them.

@tuhinone great product and people

@rapprach @baseten ayy no lanyards

Had a little too much caffeine this morning Come say hi at KubeCon! Booth 585 💚

We are thrilled to welcome Sameer Paranjpye to lead our engineering organization. Welcome, Sameer! baseten.co/blog/welcome-s…

@Austen inredible open rate

New logo wall for our website what do you think?

@ad0rnai @netcapgirl never meet your heroes

finally met my favorite tech egirl @netcapgirl

Special thanks to @maxricodecastro for designing the beautiful website! And huge thanks to @alanaagoyal ,@amiruci, and @baseten team for providing us the inference engine to deploy the model bioreason.net

@oneill_c dang and here i thought karpathy just oneshotted you guys out of a job

Thoughts on what makes autoresearch work and where you shouldn't expect magic Once you have a clearly defined metric and a way to normalise experiments ie usually wall clock time; you can't do steps or flops or tokens or whatever. But this is important, because no matter how the model decides to try and reward hack, every experiment is directly comparable. For example, if you fix the steps and our autoresearch agent tries increasing the size of the model you get less gradient updates per second; same with tokens. So basically you need to know what to fix ie with this hardware and this constraint, what's the best result we can get? (this is also why karpathy chooses bits per byte instead of cross entropy loss, as you can change CE by changing the vocab size) It then seems like everything else is a degree of freedom but really you've actually fixed the hardest part of research: not the steps, but the eval. 98% of good research is coming up with the right questions to ask in the first place. Of course, hillclimbing a metric/eval is useful, but autoresearch to me in its current form is a more general hyperparameter optimiser, where you kind of implicitly define hyperparameters that include things like architecture and design decisions, not just what you can specify a grid search over in ints or floats. On this point, the way I personally use these sort of loops is to have a running list of ideas I want to try or hypotheses to test. The former are optimising for a certain metric, and the latter are often trying to figure out the contours of the problem i'm working on. Models tend to degenerate/collapse to really small niche changes without intervention from a good human researcher, as even seen in Karpathy's stuff, and lack the creativity to continually drive new ideas forward based on previous results. It's like their value function is too vanilla I also think you basically need to constrain it to be single-file; the agent gets confused and creates a lot of mess if you don't do this. This is part of the reason why having truss train push (our Baseten training product) as a constraint, even though it seems trivially the same as just sshing into a node, is important. It creates focus for the agent. Finally, most people I know who have been taking advantage of LLMs in their work and research already run some sort of autoresearch loop and have been doing so for ages. Things tend to go viral when karpathy posts them, and he has figured out the minimal abstractions to run this, but I also think it needs to not be overhyped and interpreted in the context of previous prompt-based optimisation loops like GEPA

the creativity required to get this type of gain is what has always made me love infra (and our infra team). end-users of applications will never see it or interact with it, they'll just feel it when the products they love get better.