Xdrops Oracle ☀️

Network incident post-mortem: Validator sampling bug I. Summary On June 26, 2025 at 2:40 AM UTC, Flare Mainnet experienced a brief outage, causing a temporary halt in block production. The issue was identified as a known and since-patched bug in the validator sampling mechanism inherited from an upstream dependency. A pre-planned, audited network upgrade containing the fix was deployed ahead of schedule, successfully restoring network functionality by 11.15 AM UTC on June 26, 2025. II. Root cause The outage was triggered by a bug in the go-flare v1.10 client software, originating from an upstream Avalanche dependency (avalanchego). The bug was located in the validator sampling algorithm, which is critical for block production and peer selection. Under specific network conditions, the cumulative weight of all validators surpassed the maximum value for a 64-bit signed integer (MaxInt64). This resulted in an errOutOfRange error, causing validator nodes to shut down. The loss of these validators interrupted the network's ability to produce new blocks. III. Resolution The underlying bug had already been rectified in avalanchego v1.11.0, which updated the sampling logic to support the larger uint64 data type. Flare had proactively audited and prepared its corresponding v1.11 release ahead of its original schedule. In response to the incident, this planned upgrade was deployed as an emergency patch. The new software was successfully rolled out, resolving the error and restoring network stability and operations. IV. Next steps The successful deployment of this upgrade has enhanced the network's resilience and stability, positioning it to handle future growth. We have reviewed our incident response process to ensure rapid deployment of critical patches if similar situations arise. We extend our gratitude to our infrastructure providers and the wider community for their swift response and support, which were instrumental in the speedy resolution of this incident.

Network incident post-mortem: Validator sampling bug I. Summary On June 26, 2025 at 2:40 AM UTC, Flare Mainnet experienced a brief outage, causing a temporary halt in block production. The issue was identified as a known and since-patched bug in the validator sampling mechanism inherited from an upstream dependency. A pre-planned, audited network upgrade containing the fix was deployed ahead of schedule, successfully restoring network functionality by 11.15 AM UTC on June 26, 2025. II. Root cause The outage was triggered by a bug in the go-flare v1.10 client software, originating from an upstream Avalanche dependency (avalanchego). The bug was located in the validator sampling algorithm, which is critical for block production and peer selection. Under specific network conditions, the cumulative weight of all validators surpassed the maximum value for a 64-bit signed integer (MaxInt64). This resulted in an errOutOfRange error, causing validator nodes to shut down. The loss of these validators interrupted the network's ability to produce new blocks. III. Resolution The underlying bug had already been rectified in avalanchego v1.11.0, which updated the sampling logic to support the larger uint64 data type. Flare had proactively audited and prepared its corresponding v1.11 release ahead of its original schedule. In response to the incident, this planned upgrade was deployed as an emergency patch. The new software was successfully rolled out, resolving the error and restoring network stability and operations. IV. Next steps The successful deployment of this upgrade has enhanced the network's resilience and stability, positioning it to handle future growth. We have reviewed our incident response process to ensure rapid deployment of critical patches if similar situations arise. We extend our gratitude to our infrastructure providers and the wider community for their swift response and support, which were instrumental in the speedy resolution of this incident.

Flare has resumed block production following a brief outage. Validators representing over 85% of stake have successfully updated, and the network is now stable and consistently producing blocks. The FTSO system has resumed price updates and has already self-corrected, thanks to its robustness and the swift response of FTSO providers. We ask that remaining validators update promptly to support full network stability (see link below). A clear explanation of the changes will be shared. Importantly, the outage did not compromise ledger integrity. All user assets remain secure and fully under user control. Thank you for your continued support.