GitLab.com Status

The combination of infrastructure and application improvements made has sustained stability during peak hours yesterday. This incident has been moved to resolved following our monitoring. Details: gitlab.com/gitlab-com/gl-… status.gitlab.com

The team has continued to make improvements to the underlying infrastructure to prevent further impact. We have not seen any recent occurrences of background job delays. We will keep this incident open through peak load times over the next 20 hours to m... status.gitlab.com

We observed an increased delay in background job processing and CI job queue times, both of which have now recovered. We're closely monitoring the situation as we work on additional improvements. Details: gitlab.com/gitlab-com/gl-… status.gitlab.com

We are currently investigating performance degradation and our team is currently actively working on a mitigation. More details in gitlab.com/gitlab-com/gl-… status.gitlab.com

The team has continued to make improvements since yesterday's update to further reduce impact during peak load. We are actively monitoring the situation. Details: gitlab.com/gitlab-com/gl-… status.gitlab.com

Background job processing has recovered. Additional changes have been implemented which helped to reduce impact during peak load. We will continue to monitor this as we move through peak load times tomorrow. Details: gitlab.com/gitlab-com/gl-… status.gitlab.com

We are investigating delay in background job processing. After a monitoring period that followed yesterday fixes, and expanding infra resources, we are investigating further improvements, more details in gitlab.com/gitlab-com/gl-… status.gitlab.com

Our metrics indicate this is now fixed. We will now be marking this as resolved. Details in: gitlab.com/gitlab-com/gl-… status.gitlab.com

Background job processing has recovered as peak load has passed. Additional changes are under review to address this going forward. This incident will remain in monitoring as we deploy these changes and observe metrics through tomorrow's peak. Details: ... status.gitlab.com

Improvements to help address this issue are currently under review. We are still observing a delay in background job processing. Users may still see delays in CI job status updates. Details in: gitlab.com/gitlab-com/gl-… status.gitlab.com

We are investigating delay in background job processing. We are working on further improvements, more details in gitlab.com/gitlab-com/gl-… status.gitlab.com

The fix has reached production and users should not experience 500 errors on API calls. We will be monitoring the fix before marking this incident as Resolved. More details gitlab.com/gitlab-com/gl-… status.gitlab.com

The fix has now reached Canary environment, users can enable it by clicking "Next" at next.gitlab.com until the fix reaches production. More details: gitlab.com/gitlab-com/gl-… status.gitlab.com

After monitoring the CI pipelines, we are marking the incident as resolved. More details gitlab.com/gitlab-com/gl-… status.gitlab.com

Monitoring of CI pipelines shows metrics have returned to normal. We will keep monitoring the fix for the next hour. More details: gitlab.com/gitlab-com/gl-… status.gitlab.com

The revert change has been merged, users will still experience 500 errors until the fix reaches production. We estimate it will be fully deployed at the end of EMEA day. More details in gitlab.com/gitlab-com/gl-… status.gitlab.com

We have identified a code change that provokes 500 errors for API requests. We are currently reverting the change. Workaround in: gitlab.com/gitlab-com/gl-… status.gitlab.com

We are observing improved system metrics and reduced user impact. Mitigations are pending deployment with the goal of preventing future recurrence. This incident will remain in monitoring status as we observe system behavior. Deta... status.gitlab.com

The primary fix has been merged and deployment is underway. Full rollout typically takes several hours, so we're pursuing additional mitigations and implementation changes in parallel. Details: gitlab.com/gitlab-com/gl-… status.gitlab.com

We've identified a potential fix and are working to deploy it. We'll update you once we have a production deployment timeline. Details: gitlab.com/gitlab-com/gl-… status.gitlab.com