Forge Storage API degraded experience
Incident Report for Atlassian Developer
Resolved
This incident has been resolved.
Posted Dec 05, 2022 - 01:15 UTC
Update
We are continuing to monitor for any further issues.
Posted Dec 05, 2022 - 01:13 UTC
Update
The scheduled maintenance for app installations was not performed and will be rescheduled at a later date. There has not been any impact to the previously listed operations.

A new announcement post in the developer community will be posted with next scheduled maintenance window when this becomes available, and a statuspage will be created with the same components.
Posted Dec 05, 2022 - 01:10 UTC
Monitoring
For 15 minutes between 2022-11-28T16:14 and 2022-11-28T16:31 UTC some Forge apps failed to load due to degradation of Forge Storage API. Users may have noticed timeout errors while loading / interacting with Forge apps that utilise Storage API.

Timeline:
2022-11-28T16:14 UTC impact started
Storage API requests became slow due to blocked Node.js event loop resulting in many requests exceeding the maximum HTTP response time.
This saturated all EC2 instances with a high volume of concurrent HTTP requests.
These instances were automatically recycled after which normal operation resumed.
2022-11-28T16:31 UTC impact stopped

Current status: monitoring
As a precaution, we have temporarily increased the service capacity to mitigate further impact. We will continue to monitor the system to ensure actions taken have addressed the root cause.
Posted Nov 30, 2022 - 03:38 UTC
This incident affected: Developer (Forge Function Invocation, Hosted storage).