Ionut Toader

The latest network stress test put serious pressure on our BoN infrastructure, especially on the gateway-facing observers and the indexers. A few important details on the current setup: - one observer per shard behind the gateway - well below spec for sustained stress testing; on mainnet we use multiple observers per shard for load balancing - a single indexing cluster accessed by all services - on mainnet, different services use different clusters - a single observer per shard (one squad), with notifier, serving internal services - on mainnet we run multiple squads, and these roles are split across different machines. Regular API requests and VM queries are also served by different machine types there We also found a misconfiguration on BoN for VM queries. Under stress, it led to request buildup and out-of-memory issues, which affected both the API and the gateway. We are fixing this. Over the last two days of testing, we also uncovered optimizations needed for Supernova when the network is hit with very high transaction volume continuously over long periods. These issues only surface under sustained heavy load. The optimization has already been prepared and tested internally, and we will likely deploy it on BoN as well. One clear lesson: running stress tests like this while also keeping infra costs low is a challenge on its own. That said, the network is still up and running, and more optimizations are on the way to improve behavior under heavy load. And finally, a big thank you to everyone involved - these findings were possible because of your efforts.