Get email notifications whenever GitHubcreates,updatesorresolvesan incident.
Get text message notifications whenever GitHubcreatesorresolvesan incident.
Get incident updates and maintenance status messages in Slack.
Subscribe via Slack
By subscribing you acknowledge ourPrivacy Policy.In addition, you agree to the AtlassianCloud Terms of Serviceand acknowledge Atlassian'sPrivacy Policy.
Get webhook notifications whenever GitHubcreatesan incident,updatesan incident,resolvesan incident orchangesa component status.
Visit oursupport site.
All Systems Operational
Git Operations
?
Operational
API Requests
?
Operational
Webhooks
?
Operational
Visit www.githubstatus.com for more information
Operational
Issues
?
Operational
Pull Requests
?
Operational
Actions
?
Operational
Packages
?
Operational
Pages
?
Operational
Codespaces
?
Operational
Copilot
Operational
Operational
Degraded Performance
Partial Outage
Major Outage
Maintenance
Past Incidents
Jul12,2024
No incidents reported today.
Jul11,2024
Resolved-
This incident has been resolved.
Jul11,15:21UTC
Jul11,15:21UTC
Update-
Copilot is operating normally.
Jul11,15:21UTC
Jul11,15:21UTC
Update-
We have mitigated the intermittent timeout errors impacting Copilot’s Chat functionality and expect the incident to be resolved shortly.
Jul11,15:19UTC
Jul11,15:19UTC
Update-
We continue to investigate the cause of intermittent timeouts impacting Copilot’s Chat functionality. This is impacting a small fraction of customers. The timeout errors we are seeing has reduced back to healthy levels for the last 60 minutes but we are monitoring closely.
Jul11,15:04UTC
Jul11,15:04UTC
Update-
We continue to investigate the cause of intermittent timeouts impacting Copilot’s Chat functionality. This is impacting a small fraction of customers. We will provide further updates as we continue resolving the issue.
Jul11,14:14UTC
Jul11,14:14UTC
Update-
We continue to investigate the cause of intermittent timeouts impacting Copilot’s Chat functionality. This is impacting a small fraction of customers. We will provide further updates as we continue resolving the issue.
Jul11,13:32UTC
Jul11,13:32UTC
Update-
Copilot's Chat functionality is experiencing intermittent timeouts, we are investigating the issue.
Jul11,13:02UTC
Jul11,13:02UTC
Investigating-
We are investigating reports of degraded performance for Copilot
Jul11,13:02UTC
Jul11,13:02UTC
Jul10,2024
No incidents reported.
Jul9,2024
No incidents reported.
Jul8,2024
Resolved-
On July 8th, 2024, between 18:18 UTC and 19:11 UTC, various services relying on static assets were degraded, including user uploaded content on github.com, access to docs.github.com and Pages sites, and downloads of Release assets and Packages.
The outage primarily affected users in the vicinity of New York City, USA, due to a local CDN disruption.
Service was restored without our intervention.
We are working to improve our external monitoring, which failed to detect the issue and will be evaluating a backup mechanism to keep critical services available, such as being able to load assets on GitHub.com, in the event of an outage with our CDN.
Jul8,19:45UTC
The outage primarily affected users in the vicinity of New York City, USA, due to a local CDN disruption.
Service was restored without our intervention.
We are working to improve our external monitoring, which failed to detect the issue and will be evaluating a backup mechanism to keep critical services available, such as being able to load assets on GitHub.com, in the event of an outage with our CDN.
Jul8,19:45UTC
Update-
Issues and Pages are operating normally.
Jul8,19:44UTC
Jul8,19:44UTC
Update-
Issues and Pages are experiencing degraded performance. We are continuing to investigate.
Jul8,19:44UTC
Jul8,19:44UTC
Update-
Issues is operating normally.
Jul8,19:44UTC
Jul8,19:44UTC
Update-
Issues is experiencing degraded performance. We are continuing to investigate.
Jul8,19:44UTC
Jul8,19:44UTC
Update-
Pages and Issues are operating normally.
Jul8,19:44UTC
Jul8,19:44UTC
Update-
Our assets are serving normally again and all impact is resolved.
Jul8,19:44UTC
Jul8,19:44UTC
Update-
We are beginning to see recovery of our assets and are monitoring for additional impact.
Jul8,19:16UTC
Jul8,19:16UTC
Update-
githubstatus.com may not be available or may be degraded for some users in some regions.
Jul8,19:12UTC
Jul8,19:12UTC
Update-
We are investigating issues with loading assets, including JavaScript assets, on various parts of the site for some users.
Jul8,19:02UTC
Jul8,19:02UTC
Investigating-
We are investigating reports of degraded performance for Issues and Pages
Jul8,19:01UTC
Jul8,19:01UTC
Jul7,2024
No incidents reported.
Jul6,2024
No incidents reported.
Jul5,2024
Resolved-
On July 5, 2024, between 16:31 UTC and 18:08 UTC, the Webhooks service was degraded, with customer impact of delays to all webhook delivery. On average, delivery delays were 24 minutes, with a maximum of 71 minutes. This was caused by a configuration change to the Webhooks service, which led to unauthenticated requests sent to the background job cluster. The configuration error was repaired and re-deploying the service solved the issue. However, this created a thundering herd effect which overloaded the background job queue cluster which put its API layer at max capacity, resulting in timeouts for other job clients, which presented as increased latency for API calls.
Shortly after resolving the authentication misconfiguration, we had a separate issue in the background job processing service where health probes were failing, leading to reduced capacity in the background job API layer which magnified the effects of the thundering herd. From 18:21 UTC to 21:14 UTC, Actions runs on PRs experienced approximately 2 minutes delay and maximum of 12 minutes delay. A deployment of the background job processing service remediated the issue.
To reduce our time to detection, we have streamlined our dashboards and added alerting for this specific runtime behavior. Additionally, we are working to reduce the blast radius of background job incidents through better workload isolation.
Jul5,20:57UTC
Shortly after resolving the authentication misconfiguration, we had a separate issue in the background job processing service where health probes were failing, leading to reduced capacity in the background job API layer which magnified the effects of the thundering herd. From 18:21 UTC to 21:14 UTC, Actions runs on PRs experienced approximately 2 minutes delay and maximum of 12 minutes delay. A deployment of the background job processing service remediated the issue.
To reduce our time to detection, we have streamlined our dashboards and added alerting for this specific runtime behavior. Additionally, we are working to reduce the blast radius of background job incidents through better workload isolation.
Jul5,20:57UTC
Update-
We are seeing recovery in Actions start times and are observing for any further impact.
Jul5,20:44UTC
Jul5,20:44UTC
Update-
We are still seeing about 5% of Actions runs taking longer than 5 minutes to start. We are scaling and shifting resources to encourage recovery of the problem.
Jul5,20:32UTC
Jul5,20:32UTC
Update-
We are still seeing about 5% of Actions runs taking longer than 5 minutes to start. We are evaluating mitigations to increase capacity to decrease latency.
Jul5,19:58UTC
Jul5,19:58UTC
Update-
We are seeing about 5% of Actions runs not starting within 5 minutes. We are continuing investigation.
Jul5,19:19UTC
Jul5,19:19UTC
Update-
We have seen recovery of Actions run delays. Keeping the incident open to monitor for full recovery.
Jul5,18:40UTC
Jul5,18:40UTC
Update-
Webhooks is operating normally.
Jul5,18:10UTC
Jul5,18:10UTC
Update-
We are seeing delays in Actions runs due to the recovery with webhook deliveries. We expect this to resolve with the recovery of webhooks.
Jul5,18:09UTC
Jul5,18:09UTC
Update-
Actions is experiencing degraded performance. We are continuing to investigate.
Jul5,18:07UTC
Jul5,18:07UTC
Update-
We are seeing recovery as webhooks are being delivered again. We are burning down our queue of events. No events have been lost. New webhook deliveries will be delayed while this process recovers.
Jul5,17:57UTC
Jul5,17:57UTC
Update-
Webhooks is experiencing degraded performance. We are continuing to investigate.
Jul5,17:55UTC
Jul5,17:55UTC
Update-
We are reverting a configuration change that is suspected to contribute to the problem with webhook deliveries.
Jul5,17:42UTC
Jul5,17:42UTC
Update-
Our telemetry shows that most webhooks are failing to be delivered. We are queueing all undelivered webhooks and are working to remediate the problem.
Jul5,17:20UTC
Jul5,17:20UTC
Update-
Webhooks is experiencing degraded availability. We are continuing to investigate.
Jul5,17:17UTC
Jul5,17:17UTC
Investigating-
We are investigating reports of degraded performance for Webhooks
Jul5,17:04UTC
Jul5,17:04UTC
Jul4,2024
No incidents reported.
Jul3,2024
Resolved-
On July 3, 2024, between 1:34 PM UTC and 4:42 PM UTC the GitHub documentation was degraded and showed a 500 on non-cached pages. On average, the error rate was 2-5% and peaked at 5% of requests to the service. This was due to an observability misconfiguration. We mitigated the incident by updating the observability configuration and redeploying. We are working to reduce our time to detection and mitigation of issues like this one in the future.
Jul3,16:40UTC
Jul3,16:40UTC
Update-
Mitigation measures have been rolled out and we're seeing errors disappear in our telemetry. We'll continue to monitor our services closely to ensure the docs site is fully healthy.
Jul3,16:37UTC
Jul3,16:37UTC
Update-
We have identified a likely cause of the errors with GitHub Docs and are working on a mitigation.
Jul3,15:59UTC
Jul3,15:59UTC
Investigating-
We are currently investigating this issue.
Jul3,15:24UTC
Jul3,15:24UTC
Jul2,2024
Resolved-
This incident has been resolved.
Jul2,19:24UTC
Jul2,19:24UTC
Update-
API Requests is operating normally.
Jul2,19:24UTC
Jul2,19:24UTC
Update-
The fix has been rolled out and our telemetry indicates that the errors with code search have resolved.
Jul2,19:22UTC
Jul2,19:22UTC
Update-
An issue with faulty data in an in-memory storage causes around one third of code search requests to fail. The team has identified the issue and is working on rolling out a fix.
Jul2,19:18UTC
Jul2,19:18UTC
Update-
API Requests is experiencing degraded performance. We are continuing to investigate.
Jul2,18:47UTC
Jul2,18:47UTC
Investigating-
We are currently investigating this issue.
Jul2,18:45UTC
Jul2,18:45UTC
Resolved-
This incident has been resolved.
Jul2,01:14UTC
Jul2,01:14UTC
Update-
As a result of rerouting traffic, we have seen overall network link health return to normal.
Jul2,01:14UTC
Jul2,01:14UTC
Update-
We are investigating connection issues with one of our network links. We are working to reroute traffic.
Jul2,00:52UTC
Jul2,00:52UTC
Update-
We are investigating intermittent network connection issues. These issues appear to be limited to customers hosted on AWS that are connecting to GitHub's network.
Jul2,00:15UTC
Jul2,00:15UTC
Update-
We're investigating reports of intermittent timeouts and connection errors for git clone operations.
Jul1,23:32UTC
Jul1,23:32UTC
Investigating-
We are investigating reports of degraded performance for Git Operations
Jul1,22:59UTC
Jul1,22:59UTC
Jul1,2024
Jun30,2024
No incidents reported.
Jun29,2024
No incidents reported.
Jun28,2024
Resolved-
On June 28th, 2024, at 16:06 UTC, a backend update by GitHub triggered a significant number of long-running Organization membership update jobs in our job processing system. The job queue depth rose as these update jobs consumed most of our job worker capacity. This resulted in delays for other jobs across services such as Pull Requests and PR-related Actions workflows. We mitigated the impact to Pull Requests and Actions at 19:32 UTC by pausing all Organization membership update jobs. We deployed a code change at 22:30 UTC to skip over the jobs queued by the backend change and re-enabled Organization membership update jobs. We restored the Organization membership update functionality at 22:52 UTC, including all membership changes queued during the incident.
During the incident, about 15% of Action workflow runs experienced a delay of more than five minutes. In addition, Pull Requests had delays in determining merge eligibility and starting associated Action workflows for the duration of the incident. Organization membership updates saw delays for upwards of five hours.
To prevent a similar event in the future from impacting our users, we are working to: improve our job management system to better manage our job worker capacity; add more precise monitoring for job delays; and strengthen our testing practices to prevent future recurrences.
Jun28,22:51UTC
During the incident, about 15% of Action workflow runs experienced a delay of more than five minutes. In addition, Pull Requests had delays in determining merge eligibility and starting associated Action workflows for the duration of the incident. Organization membership updates saw delays for upwards of five hours.
To prevent a similar event in the future from impacting our users, we are working to: improve our job management system to better manage our job worker capacity; add more precise monitoring for job delays; and strengthen our testing practices to prevent future recurrences.
Jun28,22:51UTC
Update-
We are continuing to work to mitigate delays in organization membership changes.
Jun28,22:18UTC
Jun28,22:18UTC
Update-
We are still actively working to mitigate delays in organization membership changes.
Jun28,21:45UTC
Jun28,21:45UTC
Update-
We are actively working to mitigate delays in organization membership changes. Actions and Pull Requests are both functioning normally now.
Jun28,20:46UTC
Jun28,20:46UTC
Update-
Actions is operating normally.
Jun28,20:00UTC
Jun28,20:00UTC
Update-
Pull Requests is operating normally.
Jun28,19:59UTC
Jun28,19:59UTC
Update-
We are continuing to apply mitigations and are seeing improvement in creating pull request merge commits and Actions runs for pull request events. Applying changes to organization members remains delayed.
Jun28,19:51UTC
Jun28,19:51UTC
Update-
We are continuing to work on mitigating delays creating pull request merge commits, Actions runs for pull request events, and changes to organization members.
Jun28,19:03UTC
Jun28,19:03UTC
Update-
Actions runs triggered by pull requests are experiencing start delays. We have engaged the appropriate teams and are investigating the issue.
Jun28,17:59UTC
Jun28,17:59UTC
Update-
Pull Requests is experiencing degraded performance. We are continuing to investigate.
Jun28,17:58UTC
Jun28,17:58UTC
Investigating-
We are investigating reports of degraded performance for Actions
Jun28,17:34UTC
Jun28,17:34UTC
