minor impact GitHub โœ“ Resolved

Incident with Webhooks

Started
2026-03-09 15:50 UTC
Resolved
2026-03-09 17:03 UTC
Duration
73 minutes
Date
2026-03-09

Incident Timeline

investigating 2026-03-09 15:50 UTC

We are investigating reports of degraded performance for Webhooks

investigating 2026-03-09 15:50 UTC

We are investigating reports of degraded performance for Webhooks

investigating 2026-03-09 15:50 UTC

We are investigating reports of degraded performance for Webhooks

investigating 2026-03-09 15:50 UTC

We are investigating reports of degraded performance for Webhooks

investigating 2026-03-09 15:50 UTC

We are investigating reports of degraded performance for Webhooks

investigating 2026-03-09 15:50 UTC

We are investigating reports of degraded performance for Webhooks

investigating 2026-03-09 15:50 UTC

We are investigating reports of degraded performance for Webhooks

investigating 2026-03-09 15:50 UTC

We are investigating reports of degraded performance for Webhooks

investigating 2026-03-09 15:50 UTC

We are investigating reports of degraded performance for Webhooks

investigating 2026-03-09 15:50 UTC

We are investigating reports of degraded performance for Webhooks

investigating 2026-03-09 15:50 UTC

We are investigating reports of degraded performance for Webhooks

investigating 2026-03-09 15:56 UTC

We are experiencing latency on the API and UI endpoints. We are working to resolve the issue.

investigating 2026-03-09 15:56 UTC

We are experiencing latency on the API and UI endpoints. We are working to resolve the issue.

investigating 2026-03-09 15:56 UTC

We are experiencing latency on the API and UI endpoints. We are working to resolve the issue.

investigating 2026-03-09 15:56 UTC

We are experiencing latency on the API and UI endpoints. We are working to resolve the issue.

investigating 2026-03-09 15:56 UTC

We are experiencing latency on the API and UI endpoints. We are working to resolve the issue.

investigating 2026-03-09 15:56 UTC

We are experiencing latency on the API and UI endpoints. We are working to resolve the issue.

investigating 2026-03-09 15:56 UTC

We are experiencing latency on the API and UI endpoints. We are working to resolve the issue.

investigating 2026-03-09 15:56 UTC

We are experiencing latency on the API and UI endpoints. We are working to resolve the issue.

investigating 2026-03-09 15:56 UTC

We are experiencing latency on the API and UI endpoints. We are working to resolve the issue.

investigating 2026-03-09 15:56 UTC

We are experiencing latency on the API and UI endpoints. We are working to resolve the issue.

investigating 2026-03-09 15:56 UTC

We are experiencing latency on the API and UI endpoints. We are working to resolve the issue.

investigating 2026-03-09 17:03 UTC

Webhooks is operating normally.

investigating 2026-03-09 17:03 UTC

Webhooks is operating normally.

investigating 2026-03-09 17:03 UTC

Webhooks is operating normally.

investigating 2026-03-09 17:03 UTC

Webhooks is operating normally.

investigating 2026-03-09 17:03 UTC

Webhooks is operating normally.

investigating 2026-03-09 17:03 UTC

Webhooks is operating normally.

investigating 2026-03-09 17:03 UTC

Webhooks is operating normally.

investigating 2026-03-09 17:03 UTC

Webhooks is operating normally.

investigating 2026-03-09 17:03 UTC

Webhooks is operating normally.

investigating 2026-03-09 17:03 UTC

Webhooks is operating normally.

investigating 2026-03-09 17:03 UTC

Webhooks is operating normally.

resolved 2026-03-09 17:03 UTC

On March 9, 2026, between 15:03 and 20:52 UTC, the Webhooks API experienced was degraded, resulted in higher average latency on requests and in certain cases error responses. Approximately 0.6% of total requests exceeded the normal latency threshold of 3s, while 0.4% of requests resulted in 500 errors. At peak, 2.0% experienced latency greater than 3 seconds and 2.8% of requests returned 500 errors.<br /><br />The issue was caused by a noisy actor that led to resource contention on the Webhooks API service. We mitigated the issue initially by increasing CPU resources for the Webhooks API service, and ultimately applied lower rate limiting thresholds to the noisy actor to prevent further impact to other users.<br /><br />We are working to improve monitoring to more quickly ascertain noisy traffic and will continue to improve our rate-limiting mechanisms to help prevent similar issues in the future.

resolved 2026-03-09 17:03 UTC

On March 9, 2026, between 15:03 and 20:52 UTC, the Webhooks API experienced was degraded, resulted in higher average latency on requests and in certain cases error responses. Approximately 0.6% of total requests exceeded the normal latency threshold of 3s, while 0.4% of requests resulted in 500 errors. At peak, 2.0% experienced latency greater than 3 seconds and 2.8% of requests returned 500 errors.<br /><br />The issue was caused by a noisy actor that led to resource contention on the Webhooks API service. We mitigated the issue initially by increasing CPU resources for the Webhooks API service, and ultimately applied lower rate limiting thresholds to the noisy actor to prevent further impact to other users.<br /><br />We are working to improve monitoring to more quickly ascertain noisy traffic and will continue to improve our rate-limiting mechanisms to help prevent similar issues in the future.

resolved 2026-03-09 17:03 UTC

On March 9, 2026, between 15:03 and 20:52 UTC, the Webhooks API experienced was degraded, resulted in higher average latency on requests and in certain cases error responses. Approximately 0.6% of total requests exceeded the normal latency threshold of 3s, while 0.4% of requests resulted in 500 errors. At peak, 2.0% experienced latency greater than 3 seconds and 2.8% of requests returned 500 errors.<br /><br />The issue was caused by a noisy actor that led to resource contention on the Webhooks API service. We mitigated the issue initially by increasing CPU resources for the Webhooks API service, and ultimately applied lower rate limiting thresholds to the noisy actor to prevent further impact to other users.<br /><br />We are working to improve monitoring to more quickly ascertain noisy traffic and will continue to improve our rate-limiting mechanisms to help prevent similar issues in the future.

resolved 2026-03-09 17:03 UTC

On March 9, 2026, between 15:03 and 20:52 UTC, the Webhooks API experienced was degraded, resulted in higher average latency on requests and in certain cases error responses. Approximately 0.6% of total requests exceeded the normal latency threshold of 3s, while 0.4% of requests resulted in 500 errors. At peak, 2.0% experienced latency greater than 3 seconds and 2.8% of requests returned 500 errors.<br /><br />The issue was caused by a noisy actor that led to resource contention on the Webhooks API service. We mitigated the issue initially by increasing CPU resources for the Webhooks API service, and ultimately applied lower rate limiting thresholds to the noisy actor to prevent further impact to other users.<br /><br />We are working to improve monitoring to more quickly ascertain noisy traffic and will continue to improve our rate-limiting mechanisms to help prevent similar issues in the future.

resolved 2026-03-09 17:03 UTC

On March 9, 2026, between 15:03 and 20:52 UTC, the Webhooks API experienced was degraded, resulted in higher average latency on requests and in certain cases error responses. Approximately 0.6% of total requests exceeded the normal latency threshold of 3s, while 0.4% of requests resulted in 500 errors. At peak, 2.0% experienced latency greater than 3 seconds and 2.8% of requests returned 500 errors.<br /><br />The issue was caused by a noisy actor that led to resource contention on the Webhooks API service. We mitigated the issue initially by increasing CPU resources for the Webhooks API service, and ultimately applied lower rate limiting thresholds to the noisy actor to prevent further impact to other users.<br /><br />We are working to improve monitoring to more quickly ascertain noisy traffic and will continue to improve our rate-limiting mechanisms to help prevent similar issues in the future.

resolved 2026-03-09 17:03 UTC

On March 9, 2026, between 15:03 and 20:52 UTC, the Webhooks API experienced was degraded, resulted in higher average latency on requests and in certain cases error responses. Approximately 0.6% of total requests exceeded the normal latency threshold of 3s, while 0.4% of requests resulted in 500 errors. At peak, 2.0% experienced latency greater than 3 seconds and 2.8% of requests returned 500 errors.<br /><br />The issue was caused by a noisy actor that led to resource contention on the Webhooks API service. We mitigated the issue initially by increasing CPU resources for the Webhooks API service, and ultimately applied lower rate limiting thresholds to the noisy actor to prevent further impact to other users.<br /><br />We are working to improve monitoring to more quickly ascertain noisy traffic and will continue to improve our rate-limiting mechanisms to help prevent similar issues in the future.

resolved 2026-03-09 17:03 UTC

On March 9, 2026, between 15:03 and 20:52 UTC, the Webhooks API experienced was degraded, resulted in higher average latency on requests and in certain cases error responses. Approximately 0.6% of total requests exceeded the normal latency threshold of 3s, while 0.4% of requests resulted in 500 errors. At peak, 2.0% experienced latency greater than 3 seconds and 2.8% of requests returned 500 errors.<br /><br />The issue was caused by a noisy actor that led to resource contention on the Webhooks API service. We mitigated the issue initially by increasing CPU resources for the Webhooks API service, and ultimately applied lower rate limiting thresholds to the noisy actor to prevent further impact to other users.<br /><br />We are working to improve monitoring to more quickly ascertain noisy traffic and will continue to improve our rate-limiting mechanisms to help prevent similar issues in the future.

resolved 2026-03-09 17:03 UTC

On March 9, 2026, between 15:03 and 20:52 UTC, the Webhooks API experienced was degraded, resulted in higher average latency on requests and in certain cases error responses. Approximately 0.6% of total requests exceeded the normal latency threshold of 3s, while 0.4% of requests resulted in 500 errors. At peak, 2.0% experienced latency greater than 3 seconds and 2.8% of requests returned 500 errors.<br /><br />The issue was caused by a noisy actor that led to resource contention on the Webhooks API service. We mitigated the issue initially by increasing CPU resources for the Webhooks API service, and ultimately applied lower rate limiting thresholds to the noisy actor to prevent further impact to other users.<br /><br />We are working to improve monitoring to more quickly ascertain noisy traffic and will continue to improve our rate-limiting mechanisms to help prevent similar issues in the future.

resolved 2026-03-09 17:03 UTC

On March 9, 2026, between 15:03 and 20:52 UTC, the Webhooks API experienced was degraded, resulted in higher average latency on requests and in certain cases error responses. Approximately 0.6% of total requests exceeded the normal latency threshold of 3s, while 0.4% of requests resulted in 500 errors. At peak, 2.0% experienced latency greater than 3 seconds and 2.8% of requests returned 500 errors.<br /><br />The issue was caused by a noisy actor that led to resource contention on the Webhooks API service. We mitigated the issue initially by increasing CPU resources for the Webhooks API service, and ultimately applied lower rate limiting thresholds to the noisy actor to prevent further impact to other users.<br /><br />We are working to improve monitoring to more quickly ascertain noisy traffic and will continue to improve our rate-limiting mechanisms to help prevent similar issues in the future.

resolved 2026-03-09 17:03 UTC

On March 9, 2026, between 15:03 and 20:52 UTC, the Webhooks API experienced was degraded, resulted in higher average latency on requests and in certain cases error responses. Approximately 0.6% of total requests exceeded the normal latency threshold of 3s, while 0.4% of requests resulted in 500 errors. At peak, 2.0% experienced latency greater than 3 seconds and 2.8% of requests returned 500 errors.<br /><br />The issue was caused by a noisy actor that led to resource contention on the Webhooks API service. We mitigated the issue initially by increasing CPU resources for the Webhooks API service, and ultimately applied lower rate limiting thresholds to the noisy actor to prevent further impact to other users.<br /><br />We are working to improve monitoring to more quickly ascertain noisy traffic and will continue to improve our rate-limiting mechanisms to help prevent similar issues in the future.

resolved 2026-03-09 17:03 UTC

On March 9, 2026, between 15:03 and 20:52 UTC, the Webhooks API experienced was degraded, resulted in higher average latency on requests and in certain cases error responses. Approximately 0.6% of total requests exceeded the normal latency threshold of 3s, while 0.4% of requests resulted in 500 errors. At peak, 2.0% experienced latency greater than 3 seconds and 2.8% of requests returned 500 errors.<br /><br />The issue was caused by a noisy actor that led to resource contention on the Webhooks API service. We mitigated the issue initially by increasing CPU resources for the Webhooks API service, and ultimately applied lower rate limiting thresholds to the noisy actor to prevent further impact to other users.<br /><br />We are working to improve monitoring to more quickly ascertain noisy traffic and will continue to improve our rate-limiting mechanisms to help prevent similar issues in the future.

โ† All GitHub incidents Other incidents on 2026-03-09