Javier

The most important technical concept in AI in recent years was introduced by group comprised mostly of immigrants. The most important technology in the 21st century is largely coming out of the US today due to immigration. America should not take this for granted.

Cloudflare serves around 20% of the web with 46 million requests a second. Surely they must have a lot of data. Where do they store it? Plain old PostgreSQL. 🐘 Around 15-20 clusters of them. Each cluster consists of 3 servers split into two regions. The primary region is where the writes go, and the secondary is the region which replicates this data asynchronously and serves reads. Within those servers exist many databases owned by different people - they essentially offer multi tenancy. And as you know, multi-tenancy comes with a lot of challenges: 🤨 How do you decide which physical cluster a new tenant goes into? As they say in CloudFlare - it’s more of an art than a science! 🎨👨‍🎨 Think twice. Deploy once. 👌 But it depends on a lot of questions that get asked, like: • Is your data the source of truth, or can it be repopulated from elsewhere? • How sensitive is the data? (PII, etc) • What’s the expected traffic pattern - read heavy or write heavy? • How long do you need to store the data for? • How are your apps opening connections? (connection hungry apps can take a lot out of postgresql) • What is the growth projected to be? 🐙 Well-Connected DB Each connection in PostgreSQL is a new OS process. This makes connections expensive! As such, Cloudflare has to gatekeep the number of connections - and it does so, via PgBouncer. 🛡 🐬 PgBouncer pools a maximum server-side connections which it then allocates across tenants. From there, it forwards queries to HAProxy, which load balances across Postgres’ primary and read replicas. 🆙 High Availability The latency of an offline database is infinite. To keep high availability, CloudFlare uses the Stolon cluster manager to replicate data across Postgres instances and elect leaders/failover under high load scenarios. Health is tracked via a local health check on each node that periodically heartbeats to a distributed store like etcd. 🦏 Thundering Herd When apps get redeployed, they re-initialize all their state and connect to the database at once. This can be costly, as they compete with other tenants for the same shared underlying physical resources. Cloudflare implemented a way to handle this in their own fork of PgBouncer - it supports granular load shedding by either throttling or outright killing existing user connections. 🌞 No Clouds CloudFlare does NOT run on the cloud. They deploy on bare metal instances in their own data centres without any virtualisation whatsoever. As such, they hit some unique challenges here too: • 🔥 natural disasters / cooling issues e.g if your data centre in Portland overheats due to hot weather, your performance slows down. You can say your “db is running hot”, but bad puns aside - your data infrastructure should be resilient to that. • ⚡️ network partitions It’s a lot of work to get to the bottom of diagnosing a network partition. You have to test connectivity between a lot of components. Instead of being reactive, CloudFlare is proactive and runs chaos tests to test their distributed system’s behaviour in the presence of such partitions. 👌