We wrote @tensorlake’s cluster scheduler to build a serverless durable execution engine where the functions in an application are transparently distributed across a cluster, and a call to a function uses S3 for RPC. We got free checkpointing out of it, good developer experience and latency was acceptable for high-throughput data ingestion applications.
We’re using the same scheduler to run sandboxes now, since we pivoted to making sandboxes the main product instead of the distributed function runtime.
One key design decision that isn’t working anymore is marking a host dead when there’s a transient network blip. Sandboxes run much longer than small functions, and you don’t want them terminated because of a brief network partition due to faulty proxies or other network hops.
We’re moving towards a model where hosts running sandboxes register with beacons in and across AZs. These beacons manage liveness checks and inform the scheduler about real dataplane loss only after multiple levels of checks.
English
