Luis Capelo

Fun conversation with @swyx on our journey building the cloud for true elastic inference, sandboxes, and more. And of course, how we're evolving Modal's dev experience to be better for agents.

Rates are not costs! Serverless GPUs can cost more per hour but in many practical cases they cost less in aggregate. The key statistic is the workload's peak-to-average demand ratio. I wrote a lil article for the @modal blog (and vibed up a lil widget) demonstrating this.

I didn't have wifi on my flight from SF to Seoul for ICML, so instead of coding, I wrote this article on my favorite answer to a common question: "What is Modal?"

Turns out that speculative decoding is one of the most effective ideas for achieving great perf running your own LLM. This only really works if you get to run everything: inference, training, proxy, etc. x.com/modal/status/2…

Modal Auto Endpoints provide state-of-the-art open source inference perf with a click. Learn how we developed our low latency inference playbook with @DecagonAI, delivering responses 60ms faster than the best proprietary provider. modal.com/blog/achieve-s…

Huge congrats to @modal on Modal Auto Endpoints! 🎉 Owning your inference means owning the code that runs it, and we love seeing that philosophy in action🫡Honored that SGLang powers part of the stack, alongside the work on DFlash speculative decoding 🧡

You no longer have to pick between the performance of a black box API and the flexibility and control of @modal. Auto Endpoints give you both. We're unlocking frontier performance for everyone without having to talk to sales or an FDE. More cooking here, stay tuned.

We spent a lot of time at @modal working with customers to figure out the best way to deploy frontier open-source models. I think we have a pretty good idea now. x.com/modal/status/2…

It is not too late to _actually_ own your inference. Introducing: Modal Auto Endpoints.

@diptanu fwiw snapshotting a sandbox greatly depends on the use case. In many agent-based use-cases you don't know which dependencies the agent might decide to install nor when, making it a bit of a harder problem.

I agree with the problem but the solution is not great. You can do a lot better today than running warm pools of sandboxes and pay for compute when you are not using them. If an app takes multiple seconds to start, snapshot the sandbox after it’s ready and boot new sandboxes from snapshots with the process already initialized. There are some instances where snapshots don’t work - such as warming up connection pools and so on but these are generally online applications, and you want predictive autoscaling for those class of applications.

.wait_until_ready(), set, go Building performant sandbox systems goes way beyond the initial container boot. We're unpacking what that means, and breaking down some tools to help you manage the entire lifecycle.

@diptanu @bernhardsson We are happy contributors to a lot of the open-source optimizations—and will continue to do so.

@bernhardsson The main problem with open source models is that inference is not straight forward unless the community puts in significant effort in making sure vllm or sglang has all the optimizations to run them as efficiently as the labs creating the models do.

If you want to use LLMs in production for serious stuff, you should probably use open source models

Speculation Is All You Need. In this blog post, we announce the co-release (w/ Z Lab) of six more state-of-the-art DFlash speculators for @Alibaba_Qwen 3.x. Over 1k output tps for 3.5 122B-A10B on a B200. Read the blog for why we're all-in on spec dec. modal.com/blog/spec-is-a…

@erikdunteman @charles_irl couldn't agree more. treasure.

I could go my entire career working in proximity to @charles_irl and that'd be a career well spent

d'Modici modal.art

Reinforcement learning has exploded on Modal, and we've been cooking. Here's a review of lessons learned helping teams train at scale, the patterns we kept seeing, and an open-source library to get started with RL on Modal quickly.

@botirkhaltaevv @modal glad to have you!

personal update: i'm joining @modal modal unequivocally is the best AI infra company. i've been a customer for about a year now. i'm in london for two months, if any startup seed and above is willing and able to move to modal. DM me. I will help out personally

@BEBischof @modal @BEBischof can you message support@modal.com with your username and email? We’ll take a look.

Is @modal down? Status page says nothing but seeing issues

Day 0 support for Step 3.7 Flash on Modal. - 198B parameter MoE with 11B active - 256K context - 3 reasoning levels - Native image & video understanding Great to work with @StepFun_ai and @sgl_project on this one.