The capacity upgrade in GCP Europe West 4 is complete, and error rates have stayed at normal levels. Between 13:40 and 15:25 UTC, some repository creation requests failed with 503 errors. Between 15:18 and 15:25 UTC, some file reads and Git fetches also failed. Requests that failed during these windows can be retried. We are adding capacity monitoring so we can act before load reaches this limit again.
Resolved
The capacity upgrade in GCP Europe West 4 is complete, and error rates have stayed at normal levels. Between 13:40 and 15:25 UTC, some repository creation requests failed with 503 errors. Between 15:18 and 15:25 UTC, some file reads and Git fetches also failed. Requests that failed during these windows can be retried. We are adding capacity monitoring so we can act before load reaches this limit again.
Monitoring
We have identified the cause of the elevated error rates. Since 13:40 UTC, some repository creation requests in GCP Europe West 4 have failed with 503 errors or taken up to two minutes to fail. The cause was a capacity limit in the storage layer for this region. We have moved storage to larger machines. While the move was in progress, between 15:18 and 15:25 UTC, some file reads and Git fetches also returned 503 errors. Error rates have returned to normal since 15:25 UTC. The final part of the move is still in progress, and we are watching error rates until it completes.