In progress - We have started the rollout of data residency for Forge hosted storage, where data will be moving to the same location as the host product. We expect the rollout to be completed by Apr 30, 2024 00:00 UTC.
Scheduled - We will be commencing rollout of data residency for Forge hosted storage. As part of this rollout, we will be moving data to the location of the host product (e.g. if an app is installed in a Jira instance whose data is pinned to Singapore, Forge Hosted Storage data will be pinned to Singapore).
This will begin on Mar 28, 2024 00:00 UTC and we expect to be completed by Apr 30, 2024 00:00 UTC. There is no expected downtime during this migration. Once the migration is complete, Forge Hosted Storage data will stored in the same location as the host product for all new and existing Forge apps.
Monitoring - Forge apps may have received intermittent errors when calling Forge Storage. This impacted approximately 0.5% of Storage read or write requests. In Forge logs, the error would read, "Request principal is not authorized to use storage resource". Forge apps that implement retries likely detect this as an increase in retries.
We observed an increase in errors beginning March 11.
The team believe they have identified the cause and have deployed a fix. Due to the intermittent nature and low volume of errors, we will continue to monitor to ensure the issue has been fully resolved.
We will update this issue with further information once we are confident the issue is fixed.
Mar 25, 2024 - 06:50 UTC
Update - We continue to investigate the issue with a small number of requests to Forge Storage failing. We will provide another update in 3 hours or earlier if we have more information to share.
Mar 25, 2024 - 02:34 UTC
Update - We continue to investigate the intermittent Forge Storage failures impacting some Forge apps. We will post updates within the next hour.
Mar 25, 2024 - 01:10 UTC
Investigating - In response to some support tickets raised we have identified that since 2024-03-12 there has been a significant increase in failures for Forge apps calling the Forge Storage API. These failures are affecting multiple apps and appear to be inconsistant in nature. This impacts Forge Apps ability to retrieve and/or persist data using Forge Storage.
We do not yet have any expected resolution time or mitigation for apps or customers.
We will continue to investigate the cause of this incident and provide another update by 2024-03-25 01:00.
Mar 24, 2024 - 23:39 UTC
Resolved -
We have identified and resolved a problem with Forge hosted storage, where some paginated queries kept repeating the last page. The incident was detected by our internal monitoring and was resolved quickly after detection by reverting the deployment.
Activating changes recently made to the query cursors for paginated queries introduced a bug that impacted some apps. A small number of requests were impacted over a 16-minute window, while the incident lasted.
Timeline: - 25/Mar/24 10:08 p.m. UTC - Impact started, when the changes were deployed to production - 25/Mar/24 10:09 p.m. UTC - Incident was detected - 25/Mar/24 10:24 p.m. UTC - Incident was resolved and impact ended
The impact of this incident has been completely mitigated and our monitoring tools confirm that query operations are back to the pre-incident behaviour. We have also resolved the underlying bug and deployed the fix to production, completely eliminating the cause of this incident.
We apologise for any inconvenience this may have caused to our customers and our developer community.
Mar 26, 03:17 UTC
Mar 25, 2024
Unresolved incident: Forge Storage Intermittently Blocking Some Apps to Forge Storage.
Resolved -
This incident has been resolved.
Mar 20, 06:36 UTC
Monitoring -
Reports are up-to-date now and we will continue to monitor
Mar 20, 05:34 UTC
Investigating -
We are noticing delays in licenses and transactions reporting data and investigating the issue. Will share next update by 6 AM UTC.
Mar 20, 02:52 UTC
Completed -
The scheduled maintenance has been completed.
Mar 20, 05:24 UTC
Update -
The migration has been completed. All apps are now on the new platform backing Forge hosted storage.
Mar 20, 05:23 UTC
Update -
Migration has been resumed. This migration will be progressively rolled out per app and Forge environment. As remaining apps are progressively onboarded you will see the format of the cursor change, in line with the format for apps already on the new platform. We will provide updates as necessary.
Feb 29, 23:30 UTC
In progress -
Edit: This message was unintentionally posted. The migration has not started yet and a message will be posted here when it resumes. Scheduled maintenance is currently in progress. We will provide updates as necessary.
Feb 29, 00:00 UTC
Scheduled -
The migration to the new backing platform for Forge hosted storage is resuming. The majority of apps have already been migrated to the new platform.
Timeline
Starting on 2024-02-29 00:00 UTC, the remaining apps will be migrated to the new data store in a progressive way, by app and environment. The progressive migration is expected expected to finish no later than 2024-03-22 00:00 UTC.
Cursor format change during the migration
Once an app is migrated to the new platform you will notice the query cursors format has changed. The query cursor is an opaque string generated by the underlying data store, which is used to fetch next pages in a paginated query when using the query API [https://developer.atlassian.com/platform/forge/runtime-reference/storage-api-query/]. When the migration is complete, the new data store will reject cursors created by the old data store. For a smooth transition and minimising the impact to the apps, the platform will accept both cursor formats for a short period of time (approximately 15 minutes after the cutover). You don’t need to take any action as this is managed by the Forge team. It is advised that these cursors are not persisted and only used within the context of a single paginated query.
Resolved -
The incident has been resolved after monitoring the fix.
Mar 18, 03:28 UTC
Monitoring -
The fix for the issue has been deployed and verified to be working as expected. Team continue to monitor the progress.
Mar 15, 05:45 UTC
Update -
The fix for the issue has been identified and is getting deployed. ETA for the fix to be deployed is max 1 hour (05:10 AM UTC).
Mar 15, 04:11 UTC
Identified -
The issue has been identified. The team is working on the fix. We will keep this thread updated on the progress.
Mar 15, 00:03 UTC
Investigating -
We are currently investigating the issue where the private listings access tokens are not working for the Marketplace apps.
Mar 14, 09:39 UTC