Incident with all GitHub services
- Started
- 2026-03-03 18:59 UTC
- Resolved
- 2026-03-03 20:09 UTC
- Duration
- 69 minutes
- Date
- 2026-03-03
Incident Timeline
We are investigating reports of degraded availability for Actions, Copilot and Issues
We are investigating reports of degraded availability for Actions, Copilot and Issues
We are investigating reports of degraded availability for Actions, Copilot and Issues
We are investigating reports of degraded availability for Actions, Copilot and Issues
We are investigating reports of degraded availability for Actions, Copilot and Issues
We are investigating reports of degraded availability for Actions, Copilot and Issues
We are investigating reports of degraded availability for Actions, Copilot and Issues
We are investigating reports of degraded availability for Actions, Copilot and Issues
We are investigating reports of degraded availability for Actions, Copilot and Issues
We are investigating reports of degraded availability for Actions, Copilot and Issues
We are investigating reports of degraded availability for Actions, Copilot and Issues
API Requests is experiencing degraded availability. We are continuing to investigate.
API Requests is experiencing degraded availability. We are continuing to investigate.
API Requests is experiencing degraded availability. We are continuing to investigate.
API Requests is experiencing degraded availability. We are continuing to investigate.
API Requests is experiencing degraded availability. We are continuing to investigate.
API Requests is experiencing degraded availability. We are continuing to investigate.
API Requests is experiencing degraded availability. We are continuing to investigate.
API Requests is experiencing degraded availability. We are continuing to investigate.
API Requests is experiencing degraded availability. We are continuing to investigate.
API Requests is experiencing degraded availability. We are continuing to investigate.
API Requests is experiencing degraded availability. We are continuing to investigate.
Pull Requests is experiencing degraded performance. We are continuing to investigate.
Pull Requests is experiencing degraded performance. We are continuing to investigate.
Pull Requests is experiencing degraded performance. We are continuing to investigate.
Pull Requests is experiencing degraded performance. We are continuing to investigate.
Pull Requests is experiencing degraded performance. We are continuing to investigate.
Pull Requests is experiencing degraded performance. We are continuing to investigate.
Pull Requests is experiencing degraded performance. We are continuing to investigate.
Pull Requests is experiencing degraded performance. We are continuing to investigate.
Pull Requests is experiencing degraded performance. We are continuing to investigate.
Pull Requests is experiencing degraded performance. We are continuing to investigate.
Pull Requests is experiencing degraded performance. We are continuing to investigate.
Webhooks is experiencing degraded performance. We are continuing to investigate.
Webhooks is experiencing degraded performance. We are continuing to investigate.
Webhooks is experiencing degraded performance. We are continuing to investigate.
Webhooks is experiencing degraded performance. We are continuing to investigate.
Webhooks is experiencing degraded performance. We are continuing to investigate.
Webhooks is experiencing degraded performance. We are continuing to investigate.
Webhooks is experiencing degraded performance. We are continuing to investigate.
Webhooks is experiencing degraded performance. We are continuing to investigate.
Webhooks is experiencing degraded performance. We are continuing to investigate.
Webhooks is experiencing degraded performance. We are continuing to investigate.
Webhooks is experiencing degraded performance. We are continuing to investigate.
We're seeing some service degradation across GitHub services. We're currently investigating impact.
We're seeing some service degradation across GitHub services. We're currently investigating impact.
We're seeing some service degradation across GitHub services. We're currently investigating impact.
We're seeing some service degradation across GitHub services. We're currently investigating impact.
We're seeing some service degradation across GitHub services. We're currently investigating impact.
We're seeing some service degradation across GitHub services. We're currently investigating impact.
We're seeing some service degradation across GitHub services. We're currently investigating impact.
We're seeing some service degradation across GitHub services. We're currently investigating impact.
We're seeing some service degradation across GitHub services. We're currently investigating impact.
We're seeing some service degradation across GitHub services. We're currently investigating impact.
We're seeing some service degradation across GitHub services. We're currently investigating impact.
Webhooks is experiencing degraded availability. We are continuing to investigate.
Webhooks is experiencing degraded availability. We are continuing to investigate.
Webhooks is experiencing degraded availability. We are continuing to investigate.
Webhooks is experiencing degraded availability. We are continuing to investigate.
Webhooks is experiencing degraded availability. We are continuing to investigate.
Webhooks is experiencing degraded availability. We are continuing to investigate.
Webhooks is experiencing degraded availability. We are continuing to investigate.
Webhooks is experiencing degraded availability. We are continuing to investigate.
Webhooks is experiencing degraded availability. We are continuing to investigate.
Webhooks is experiencing degraded availability. We are continuing to investigate.
Webhooks is experiencing degraded availability. We are continuing to investigate.
Pull Requests is experiencing degraded availability. We are continuing to investigate.
Pull Requests is experiencing degraded availability. We are continuing to investigate.
Pull Requests is experiencing degraded availability. We are continuing to investigate.
Pull Requests is experiencing degraded availability. We are continuing to investigate.
Pull Requests is experiencing degraded availability. We are continuing to investigate.
Pull Requests is experiencing degraded availability. We are continuing to investigate.
Pull Requests is experiencing degraded availability. We are continuing to investigate.
Pull Requests is experiencing degraded availability. We are continuing to investigate.
Pull Requests is experiencing degraded availability. We are continuing to investigate.
Pull Requests is experiencing degraded availability. We are continuing to investigate.
Pull Requests is experiencing degraded availability. We are continuing to investigate.
Codespaces is experiencing degraded performance. We are continuing to investigate.
Codespaces is experiencing degraded performance. We are continuing to investigate.
Codespaces is experiencing degraded performance. We are continuing to investigate.
Codespaces is experiencing degraded performance. We are continuing to investigate.
Codespaces is experiencing degraded performance. We are continuing to investigate.
Codespaces is experiencing degraded performance. We are continuing to investigate.
Codespaces is experiencing degraded performance. We are continuing to investigate.
Codespaces is experiencing degraded performance. We are continuing to investigate.
Codespaces is experiencing degraded performance. We are continuing to investigate.
Codespaces is experiencing degraded performance. We are continuing to investigate.
Codespaces is experiencing degraded performance. We are continuing to investigate.
API Requests is experiencing degraded performance. We are continuing to investigate.
API Requests is experiencing degraded performance. We are continuing to investigate.
API Requests is experiencing degraded performance. We are continuing to investigate.
API Requests is experiencing degraded performance. We are continuing to investigate.
API Requests is experiencing degraded performance. We are continuing to investigate.
API Requests is experiencing degraded performance. We are continuing to investigate.
API Requests is experiencing degraded performance. We are continuing to investigate.
API Requests is experiencing degraded performance. We are continuing to investigate.
API Requests is experiencing degraded performance. We are continuing to investigate.
API Requests is experiencing degraded performance. We are continuing to investigate.
API Requests is experiencing degraded performance. We are continuing to investigate.
API Requests is operating normally.
API Requests is operating normally.
API Requests is operating normally.
API Requests is operating normally.
API Requests is operating normally.
API Requests is operating normally.
API Requests is operating normally.
API Requests is operating normally.
API Requests is operating normally.
API Requests is operating normally.
API Requests is operating normally.
We've identified the issue and have applied a mitigation. We're seeing recovery of services. We continue to montitor for full recovery.
We've identified the issue and have applied a mitigation. We're seeing recovery of services. We continue to montitor for full recovery.
We've identified the issue and have applied a mitigation. We're seeing recovery of services. We continue to montitor for full recovery.
We've identified the issue and have applied a mitigation. We're seeing recovery of services. We continue to montitor for full recovery.
We've identified the issue and have applied a mitigation. We're seeing recovery of services. We continue to montitor for full recovery.
We've identified the issue and have applied a mitigation. We're seeing recovery of services. We continue to montitor for full recovery.
We've identified the issue and have applied a mitigation. We're seeing recovery of services. We continue to montitor for full recovery.
We've identified the issue and have applied a mitigation. We're seeing recovery of services. We continue to montitor for full recovery.
We've identified the issue and have applied a mitigation. We're seeing recovery of services. We continue to montitor for full recovery.
We've identified the issue and have applied a mitigation. We're seeing recovery of services. We continue to montitor for full recovery.
We've identified the issue and have applied a mitigation. We're seeing recovery of services. We continue to montitor for full recovery.
Issues is experiencing degraded performance. We are continuing to investigate.
Issues is experiencing degraded performance. We are continuing to investigate.
Issues is experiencing degraded performance. We are continuing to investigate.
Issues is experiencing degraded performance. We are continuing to investigate.
Issues is experiencing degraded performance. We are continuing to investigate.
Issues is experiencing degraded performance. We are continuing to investigate.
Issues is experiencing degraded performance. We are continuing to investigate.
Issues is experiencing degraded performance. We are continuing to investigate.
Issues is experiencing degraded performance. We are continuing to investigate.
Issues is experiencing degraded performance. We are continuing to investigate.
Issues is experiencing degraded performance. We are continuing to investigate.
Webhooks is experiencing degraded performance. We are continuing to investigate.
Webhooks is experiencing degraded performance. We are continuing to investigate.
Webhooks is experiencing degraded performance. We are continuing to investigate.
Webhooks is experiencing degraded performance. We are continuing to investigate.
Webhooks is experiencing degraded performance. We are continuing to investigate.
Webhooks is experiencing degraded performance. We are continuing to investigate.
Webhooks is experiencing degraded performance. We are continuing to investigate.
Webhooks is experiencing degraded performance. We are continuing to investigate.
Webhooks is experiencing degraded performance. We are continuing to investigate.
Webhooks is experiencing degraded performance. We are continuing to investigate.
Webhooks is experiencing degraded performance. We are continuing to investigate.
Codespaces is operating normally.
Codespaces is operating normally.
Codespaces is operating normally.
Codespaces is operating normally.
Codespaces is operating normally.
Codespaces is operating normally.
Codespaces is operating normally.
Codespaces is operating normally.
Codespaces is operating normally.
Codespaces is operating normally.
Codespaces is operating normally.
Webhooks is operating normally.
Webhooks is operating normally.
Webhooks is operating normally.
Webhooks is operating normally.
Webhooks is operating normally.
Webhooks is operating normally.
Webhooks is operating normally.
Webhooks is operating normally.
Webhooks is operating normally.
Webhooks is operating normally.
Webhooks is operating normally.
Issues is operating normally.
Issues is operating normally.
Issues is operating normally.
Issues is operating normally.
Issues is operating normally.
Issues is operating normally.
Issues is operating normally.
Issues is operating normally.
Issues is operating normally.
Issues is operating normally.
Issues is operating normally.
Pull Requests is experiencing degraded performance. We are continuing to investigate.
Pull Requests is experiencing degraded performance. We are continuing to investigate.
Pull Requests is experiencing degraded performance. We are continuing to investigate.
Pull Requests is experiencing degraded performance. We are continuing to investigate.
Pull Requests is experiencing degraded performance. We are continuing to investigate.
Pull Requests is experiencing degraded performance. We are continuing to investigate.
Pull Requests is experiencing degraded performance. We are continuing to investigate.
Pull Requests is experiencing degraded performance. We are continuing to investigate.
Pull Requests is experiencing degraded performance. We are continuing to investigate.
Pull Requests is experiencing degraded performance. We are continuing to investigate.
Pull Requests is experiencing degraded performance. We are continuing to investigate.
Pull Requests is operating normally.
Pull Requests is operating normally.
Pull Requests is operating normally.
Pull Requests is operating normally.
Pull Requests is operating normally.
Pull Requests is operating normally.
Pull Requests is operating normally.
Pull Requests is operating normally.
Pull Requests is operating normally.
Pull Requests is operating normally.
Pull Requests is operating normally.
Copilot is operating normally.
Copilot is operating normally.
Copilot is operating normally.
Copilot is operating normally.
Copilot is operating normally.
Copilot is operating normally.
Copilot is operating normally.
Copilot is operating normally.
Copilot is operating normally.
Copilot is operating normally.
Copilot is operating normally.
We are seeing recovery across multiple services. Impact is mostly isolated to git operations at this point, we continue to investigate
We are seeing recovery across multiple services. Impact is mostly isolated to git operations at this point, we continue to investigate
We are seeing recovery across multiple services. Impact is mostly isolated to git operations at this point, we continue to investigate
We are seeing recovery across multiple services. Impact is mostly isolated to git operations at this point, we continue to investigate
We are seeing recovery across multiple services. Impact is mostly isolated to git operations at this point, we continue to investigate
We are seeing recovery across multiple services. Impact is mostly isolated to git operations at this point, we continue to investigate
We are seeing recovery across multiple services. Impact is mostly isolated to git operations at this point, we continue to investigate
We are seeing recovery across multiple services. Impact is mostly isolated to git operations at this point, we continue to investigate
We are seeing recovery across multiple services. Impact is mostly isolated to git operations at this point, we continue to investigate
We are seeing recovery across multiple services. Impact is mostly isolated to git operations at this point, we continue to investigate
We are seeing recovery across multiple services. Impact is mostly isolated to git operations at this point, we continue to investigate
Git Operations is experiencing degraded availability. We are continuing to investigate.
Git Operations is experiencing degraded availability. We are continuing to investigate.
Git Operations is experiencing degraded availability. We are continuing to investigate.
Git Operations is experiencing degraded availability. We are continuing to investigate.
Git Operations is experiencing degraded availability. We are continuing to investigate.
Git Operations is experiencing degraded availability. We are continuing to investigate.
Git Operations is experiencing degraded availability. We are continuing to investigate.
Git Operations is experiencing degraded availability. We are continuing to investigate.
Git Operations is experiencing degraded availability. We are continuing to investigate.
Git Operations is experiencing degraded availability. We are continuing to investigate.
Git Operations is experiencing degraded availability. We are continuing to investigate.
Git Operations is operating normally.
Git Operations is operating normally.
Git Operations is operating normally.
Git Operations is operating normally.
Git Operations is operating normally.
Git Operations is operating normally.
Git Operations is operating normally.
Git Operations is operating normally.
Git Operations is operating normally.
Git Operations is operating normally.
Git Operations is operating normally.
Actions is operating normally.
Actions is operating normally.
Actions is operating normally.
Actions is operating normally.
Actions is operating normally.
Actions is operating normally.
Actions is operating normally.
Actions is operating normally.
Actions is operating normally.
Actions is operating normally.
Actions is operating normally.
We're seeing recovery across all services. We're continuing to monitor for full recovery.
We're seeing recovery across all services. We're continuing to monitor for full recovery.
We're seeing recovery across all services. We're continuing to monitor for full recovery.
We're seeing recovery across all services. We're continuing to monitor for full recovery.
We're seeing recovery across all services. We're continuing to monitor for full recovery.
We're seeing recovery across all services. We're continuing to monitor for full recovery.
We're seeing recovery across all services. We're continuing to monitor for full recovery.
We're seeing recovery across all services. We're continuing to monitor for full recovery.
We're seeing recovery across all services. We're continuing to monitor for full recovery.
We're seeing recovery across all services. We're continuing to monitor for full recovery.
We're seeing recovery across all services. We're continuing to monitor for full recovery.
On March 3, 2026, between 18:46 UTC and 20:09 UTC, GitHub experienced a period of degraded availability impacting GitHub.com, the GitHub API, GitHub Actions, Git operations, GitHub Copilot, and other dependent services. At the peak of the incident, GitHub.com request failures reached approximately 40%. During the same period, approximately 43% of GitHub API requests failed. Git operations over HTTP had an error rate of approximately 6%, while SSH was not impacted. GitHub Copilot requests had an error rate of approximately 21%. GitHub Actions experienced less than 1% impact. <br /><br />This incident shared the same underlying cause as an incident in early February where we saw a large volume of writes to the user settings caching mechanism. While deploying a change to reduce the burden of these writes, a bug caused every user’s cache to expire, get recalculated, and get rewritten. The increased load caused replication delays that cascaded down to all affected services. We mitigated this issue by immediately rolling back the faulty deployment. <br /><br />We understand these incidents disrupted the workflows of developers. While we have made substantial, long-term investments in how GitHub is built and operated to improve resilience, we acknowledge we have more work to do. Getting there requires deep architectural work that is already underway, as well as urgent, targeted improvements. We are taking the following immediate steps: <br /><br />- We have added a killswitch and improved monitoring to the caching mechanism to ensure we are notified before there is user impact and can respond swiftly. <br />- We are moving the cache mechanism to a dedicated host, ensuring that any future issues will solely affect services that rely on it.
On March 3, 2026, between 18:46 UTC and 20:09 UTC, GitHub experienced a period of degraded availability impacting GitHub.com, the GitHub API, GitHub Actions, Git operations, GitHub Copilot, and other dependent services. At the peak of the incident, GitHub.com request failures reached approximately 40%. During the same period, approximately 43% of GitHub API requests failed. Git operations over HTTP had an error rate of approximately 6%, while SSH was not impacted. GitHub Copilot requests had an error rate of approximately 21%. GitHub Actions experienced less than 1% impact. <br /><br />This incident shared the same underlying cause as an incident in early February where we saw a large volume of writes to the user settings caching mechanism. While deploying a change to reduce the burden of these writes, a bug caused every user’s cache to expire, get recalculated, and get rewritten. The increased load caused replication delays that cascaded down to all affected services. We mitigated this issue by immediately rolling back the faulty deployment. <br /><br />We understand these incidents disrupted the workflows of developers. While we have made substantial, long-term investments in how GitHub is built and operated to improve resilience, we acknowledge we have more work to do. Getting there requires deep architectural work that is already underway, as well as urgent, targeted improvements. We are taking the following immediate steps: <br /><br />- We have added a killswitch and improved monitoring to the caching mechanism to ensure we are notified before there is user impact and can respond swiftly. <br />- We are moving the cache mechanism to a dedicated host, ensuring that any future issues will solely affect services that rely on it.
On March 3, 2026, between 18:46 UTC and 20:09 UTC, GitHub experienced a period of degraded availability impacting GitHub.com, the GitHub API, GitHub Actions, Git operations, GitHub Copilot, and other dependent services. At the peak of the incident, GitHub.com request failures reached approximately 40%. During the same period, approximately 43% of GitHub API requests failed. Git operations over HTTP had an error rate of approximately 6%, while SSH was not impacted. GitHub Copilot requests had an error rate of approximately 21%. GitHub Actions experienced less than 1% impact. <br /><br />This incident shared the same underlying cause as an incident in early February where we saw a large volume of writes to the user settings caching mechanism. While deploying a change to reduce the burden of these writes, a bug caused every user’s cache to expire, get recalculated, and get rewritten. The increased load caused replication delays that cascaded down to all affected services. We mitigated this issue by immediately rolling back the faulty deployment. <br /><br />We understand these incidents disrupted the workflows of developers. While we have made substantial, long-term investments in how GitHub is built and operated to improve resilience, we acknowledge we have more work to do. Getting there requires deep architectural work that is already underway, as well as urgent, targeted improvements. We are taking the following immediate steps: <br /><br />- We have added a killswitch and improved monitoring to the caching mechanism to ensure we are notified before there is user impact and can respond swiftly. <br />- We are moving the cache mechanism to a dedicated host, ensuring that any future issues will solely affect services that rely on it.
On March 3, 2026, between 18:46 UTC and 20:09 UTC, GitHub experienced a period of degraded availability impacting GitHub.com, the GitHub API, GitHub Actions, Git operations, GitHub Copilot, and other dependent services. At the peak of the incident, GitHub.com request failures reached approximately 40%. During the same period, approximately 43% of GitHub API requests failed. Git operations over HTTP had an error rate of approximately 6%, while SSH was not impacted. GitHub Copilot requests had an error rate of approximately 21%. GitHub Actions experienced less than 1% impact. <br /><br />This incident shared the same underlying cause as an incident in early February where we saw a large volume of writes to the user settings caching mechanism. While deploying a change to reduce the burden of these writes, a bug caused every user’s cache to expire, get recalculated, and get rewritten. The increased load caused replication delays that cascaded down to all affected services. We mitigated this issue by immediately rolling back the faulty deployment. <br /><br />We understand these incidents disrupted the workflows of developers. While we have made substantial, long-term investments in how GitHub is built and operated to improve resilience, we acknowledge we have more work to do. Getting there requires deep architectural work that is already underway, as well as urgent, targeted improvements. We are taking the following immediate steps: <br /><br />- We have added a killswitch and improved monitoring to the caching mechanism to ensure we are notified before there is user impact and can respond swiftly. <br />- We are moving the cache mechanism to a dedicated host, ensuring that any future issues will solely affect services that rely on it.
On March 3, 2026, between 18:46 UTC and 20:09 UTC, GitHub experienced a period of degraded availability impacting GitHub.com, the GitHub API, GitHub Actions, Git operations, GitHub Copilot, and other dependent services. At the peak of the incident, GitHub.com request failures reached approximately 40%. During the same period, approximately 43% of GitHub API requests failed. Git operations over HTTP had an error rate of approximately 6%, while SSH was not impacted. GitHub Copilot requests had an error rate of approximately 21%. GitHub Actions experienced less than 1% impact. <br /><br />This incident shared the same underlying cause as an incident in early February where we saw a large volume of writes to the user settings caching mechanism. While deploying a change to reduce the burden of these writes, a bug caused every user’s cache to expire, get recalculated, and get rewritten. The increased load caused replication delays that cascaded down to all affected services. We mitigated this issue by immediately rolling back the faulty deployment. <br /><br />We understand these incidents disrupted the workflows of developers. While we have made substantial, long-term investments in how GitHub is built and operated to improve resilience, we acknowledge we have more work to do. Getting there requires deep architectural work that is already underway, as well as urgent, targeted improvements. We are taking the following immediate steps: <br /><br />- We have added a killswitch and improved monitoring to the caching mechanism to ensure we are notified before there is user impact and can respond swiftly. <br />- We are moving the cache mechanism to a dedicated host, ensuring that any future issues will solely affect services that rely on it.
On March 3, 2026, between 18:46 UTC and 20:09 UTC, GitHub experienced a period of degraded availability impacting GitHub.com, the GitHub API, GitHub Actions, Git operations, GitHub Copilot, and other dependent services. At the peak of the incident, GitHub.com request failures reached approximately 40%. During the same period, approximately 43% of GitHub API requests failed. Git operations over HTTP had an error rate of approximately 6%, while SSH was not impacted. GitHub Copilot requests had an error rate of approximately 21%. GitHub Actions experienced less than 1% impact. <br /><br />This incident shared the same underlying cause as an incident in early February where we saw a large volume of writes to the user settings caching mechanism. While deploying a change to reduce the burden of these writes, a bug caused every user’s cache to expire, get recalculated, and get rewritten. The increased load caused replication delays that cascaded down to all affected services. We mitigated this issue by immediately rolling back the faulty deployment. <br /><br />We understand these incidents disrupted the workflows of developers. While we have made substantial, long-term investments in how GitHub is built and operated to improve resilience, we acknowledge we have more work to do. Getting there requires deep architectural work that is already underway, as well as urgent, targeted improvements. We are taking the following immediate steps: <br /><br />- We have added a killswitch and improved monitoring to the caching mechanism to ensure we are notified before there is user impact and can respond swiftly. <br />- We are moving the cache mechanism to a dedicated host, ensuring that any future issues will solely affect services that rely on it.
On March 3, 2026, between 18:46 UTC and 20:09 UTC, GitHub experienced a period of degraded availability impacting GitHub.com, the GitHub API, GitHub Actions, Git operations, GitHub Copilot, and other dependent services. At the peak of the incident, GitHub.com request failures reached approximately 40%. During the same period, approximately 43% of GitHub API requests failed. Git operations over HTTP had an error rate of approximately 6%, while SSH was not impacted. GitHub Copilot requests had an error rate of approximately 21%. GitHub Actions experienced less than 1% impact. <br /><br />This incident shared the same underlying cause as an incident in early February where we saw a large volume of writes to the user settings caching mechanism. While deploying a change to reduce the burden of these writes, a bug caused every user’s cache to expire, get recalculated, and get rewritten. The increased load caused replication delays that cascaded down to all affected services. We mitigated this issue by immediately rolling back the faulty deployment. <br /><br />We understand these incidents disrupted the workflows of developers. While we have made substantial, long-term investments in how GitHub is built and operated to improve resilience, we acknowledge we have more work to do. Getting there requires deep architectural work that is already underway, as well as urgent, targeted improvements. We are taking the following immediate steps: <br /><br />- We have added a killswitch and improved monitoring to the caching mechanism to ensure we are notified before there is user impact and can respond swiftly. <br />- We are moving the cache mechanism to a dedicated host, ensuring that any future issues will solely affect services that rely on it.
On March 3, 2026, between 18:46 UTC and 20:09 UTC, GitHub experienced a period of degraded availability impacting GitHub.com, the GitHub API, GitHub Actions, Git operations, GitHub Copilot, and other dependent services. At the peak of the incident, GitHub.com request failures reached approximately 40%. During the same period, approximately 43% of GitHub API requests failed. Git operations over HTTP had an error rate of approximately 6%, while SSH was not impacted. GitHub Copilot requests had an error rate of approximately 21%. GitHub Actions experienced less than 1% impact. <br /><br />This incident shared the same underlying cause as an incident in early February where we saw a large volume of writes to the user settings caching mechanism. While deploying a change to reduce the burden of these writes, a bug caused every user’s cache to expire, get recalculated, and get rewritten. The increased load caused replication delays that cascaded down to all affected services. We mitigated this issue by immediately rolling back the faulty deployment. <br /><br />We understand these incidents disrupted the workflows of developers. While we have made substantial, long-term investments in how GitHub is built and operated to improve resilience, we acknowledge we have more work to do. Getting there requires deep architectural work that is already underway, as well as urgent, targeted improvements. We are taking the following immediate steps: <br /><br />- We have added a killswitch and improved monitoring to the caching mechanism to ensure we are notified before there is user impact and can respond swiftly. <br />- We are moving the cache mechanism to a dedicated host, ensuring that any future issues will solely affect services that rely on it.
On March 3, 2026, between 18:46 UTC and 20:09 UTC, GitHub experienced a period of degraded availability impacting GitHub.com, the GitHub API, GitHub Actions, Git operations, GitHub Copilot, and other dependent services. At the peak of the incident, GitHub.com request failures reached approximately 40%. During the same period, approximately 43% of GitHub API requests failed. Git operations over HTTP had an error rate of approximately 6%, while SSH was not impacted. GitHub Copilot requests had an error rate of approximately 21%. GitHub Actions experienced less than 1% impact. <br /><br />This incident shared the same underlying cause as an incident in early February where we saw a large volume of writes to the user settings caching mechanism. While deploying a change to reduce the burden of these writes, a bug caused every user’s cache to expire, get recalculated, and get rewritten. The increased load caused replication delays that cascaded down to all affected services. We mitigated this issue by immediately rolling back the faulty deployment. <br /><br />We understand these incidents disrupted the workflows of developers. While we have made substantial, long-term investments in how GitHub is built and operated to improve resilience, we acknowledge we have more work to do. Getting there requires deep architectural work that is already underway, as well as urgent, targeted improvements. We are taking the following immediate steps: <br /><br />- We have added a killswitch and improved monitoring to the caching mechanism to ensure we are notified before there is user impact and can respond swiftly. <br />- We are moving the cache mechanism to a dedicated host, ensuring that any future issues will solely affect services that rely on it.
On March 3, 2026, between 18:46 UTC and 20:09 UTC, GitHub experienced a period of degraded availability impacting GitHub.com, the GitHub API, GitHub Actions, Git operations, GitHub Copilot, and other dependent services. At the peak of the incident, GitHub.com request failures reached approximately 40%. During the same period, approximately 43% of GitHub API requests failed. Git operations over HTTP had an error rate of approximately 6%, while SSH was not impacted. GitHub Copilot requests had an error rate of approximately 21%. GitHub Actions experienced less than 1% impact. <br /><br />This incident shared the same underlying cause as an incident in early February where we saw a large volume of writes to the user settings caching mechanism. While deploying a change to reduce the burden of these writes, a bug caused every user’s cache to expire, get recalculated, and get rewritten. The increased load caused replication delays that cascaded down to all affected services. We mitigated this issue by immediately rolling back the faulty deployment. <br /><br />We understand these incidents disrupted the workflows of developers. While we have made substantial, long-term investments in how GitHub is built and operated to improve resilience, we acknowledge we have more work to do. Getting there requires deep architectural work that is already underway, as well as urgent, targeted improvements. We are taking the following immediate steps: <br /><br />- We have added a killswitch and improved monitoring to the caching mechanism to ensure we are notified before there is user impact and can respond swiftly. <br />- We are moving the cache mechanism to a dedicated host, ensuring that any future issues will solely affect services that rely on it.
On March 3, 2026, between 18:46 UTC and 20:09 UTC, GitHub experienced a period of degraded availability impacting GitHub.com, the GitHub API, GitHub Actions, Git operations, GitHub Copilot, and other dependent services. At the peak of the incident, GitHub.com request failures reached approximately 40%. During the same period, approximately 43% of GitHub API requests failed. Git operations over HTTP had an error rate of approximately 6%, while SSH was not impacted. GitHub Copilot requests had an error rate of approximately 21%. GitHub Actions experienced less than 1% impact. <br /><br />This incident shared the same underlying cause as an incident in early February where we saw a large volume of writes to the user settings caching mechanism. While deploying a change to reduce the burden of these writes, a bug caused every user’s cache to expire, get recalculated, and get rewritten. The increased load caused replication delays that cascaded down to all affected services. We mitigated this issue by immediately rolling back the faulty deployment. <br /><br />We understand these incidents disrupted the workflows of developers. While we have made substantial, long-term investments in how GitHub is built and operated to improve resilience, we acknowledge we have more work to do. Getting there requires deep architectural work that is already underway, as well as urgent, targeted improvements. We are taking the following immediate steps: <br /><br />- We have added a killswitch and improved monitoring to the caching mechanism to ensure we are notified before there is user impact and can respond swiftly. <br />- We are moving the cache mechanism to a dedicated host, ensuring that any future issues will solely affect services that rely on it.