Update - We have remediated the custom entity data store for the remaining identified sites. The read access to the custom entities data store has been restored to a state that no longer returns an empty string. We are monitoring and cross-checking the errors recorded on our platform to ensure the integrity of the data and the migration process. We expect to have an update within the next 24-48 hours.
May 07, 2024 - 06:11 UTC
Monitoring - We have restored read access to custom entity data stores for 99.5% of identified sites.
We have identified a small number of sites where the impacted data requires further remediation steps. Our team is currently looking into this and will investigate these sites on a case by case basis to see what additional steps are required.
We expect to have an update on these sites within the next 24 hours.
May 06, 2024 - 07:51 UTC
Identified - We have identified an issue with Forge apps that integrate the custom entity data store. While reading historical data that has not been updated since April 5th, the service may return an empty string instead of the value stored in the database. This would have appeared to end users as though the application data was missing. This was due to a logic error resulting in incomplete migration of data during our current data residency rollout.
Timeline
2024-04-30 06:15 UTC: Rollout of feature flag to read custom entity data from regional data. stores as per data residency setting for the workspace. 2024-05-05 23:00 UTC: Incident raised after partners reported errors. 2024-05-06 05:00 UTC: Root cause detected. 2024-05-06 06:00 UTC: Remediation steps approved.
We expect to resolve approximately 99.5% of impacted data to be available within the next 10 minutes.
We have identified a small number of sites where the impacted data requires further remediation steps. Our team is currently looking into this and will investigate these sites on a case by case basis to see what additional steps are required.
We expect to have an update on these sites within the next 24 hours.
May 06, 2024 - 07:40 UTC
Update - We are continuing to investigate this issue.
May 06, 2024 - 04:09 UTC
Investigating - We have identified a problem with querying custom entities where a query returns an empty string for some keys instead of the value stored in the database. At this time we believe that the issue is limited to query operations in a small number of apps and all the other operations are not impacted. We are investigating the root cause and will provide another update by 08:00 UTC.
May 06, 2024 - 04:09 UTC
Update - The move of the data for Forge hosted storage to the same location as the host product is taking longer than expected and will not be completed by 30 Apr 2024 00:00 UTC, as previously announced. After careful consideration, the Forge hosted storage decided that the best path forward is to extend the completion date from 30 Apr 2024 to 17 May 2024 00:00 UTC.
Apr 24, 2024 - 08:08 UTC
In progress - We have started the rollout of data residency for Forge hosted storage, where data will be moving to the same location as the host product. We expect the rollout to be completed by Apr 30, 2024 00:00 UTC.
Scheduled - We will be commencing rollout of data residency for Forge hosted storage. As part of this rollout, we will be moving data to the location of the host product (e.g. if an app is installed in a Jira instance whose data is pinned to Singapore, Forge Hosted Storage data will be pinned to Singapore).
This will begin on Mar 28, 2024 00:00 UTC and we expect to be completed by Apr 30, 2024 00:00 UTC. There is no expected downtime during this migration. Once the migration is complete, Forge Hosted Storage data will stored in the same location as the host product for all new and existing Forge apps.
Resolved -
Pipeline run has completed and reporting data is up-to-date. Incident is resolved
May 8, 15:40 UTC
Identified -
The issue has been identified and a fix is implemented. Pipeline run is in progress. We will share next update by 6 PM UTC.
May 8, 14:36 UTC
Investigating -
There is a delay in marketplace reports and we are investigating the issue. We will share further details in next update once we identify the root cause.
May 8, 12:41 UTC
Resolved -
Between 2024-05-07 23:12 UTC and 2024-05-08 04:19 UTC, Confluence did not deliver the search-performed event via webhooks. The issue has been resolved and the service is operating normally.
May 8, 05:51 UTC
Investigating -
We are investigating reports of failure to deliver the webhooks related to Confluence Search-Performed events. We will provide more details once we identify the root cause.
May 8, 05:41 UTC
May 7, 2024
Unresolved incident: Query sometimes returns an empty string for custom entities.
Resolved -
Dev Console is back to normal and stable for the last one hour and no new issues are noted. This is the final update on this incident, no further comms will be published.
Apr 25, 05:16 UTC
Monitoring -
A root cause has been identified. Marketplace is seeing unusual traffic, and as a preventive measure, couple of offending user-agents were blocked. Rate limits rejections have reached a healthy state and Dev Console is back to normal. No failures in Dev Console loading are observed at this point.
Next update will be posted in 1 hour.
Apr 25, 04:36 UTC
Investigating -
We have identified a problem with the Developer Console where apps intermittently fail to load on the landing page. We are investigating the root cause and will provide another update in the next 2 hours.
Apr 25, 01:47 UTC
Resolved -
Pipeline run is complete and Marketplace reports have up-to-date data.
Apr 25, 03:25 UTC
Identified -
The issue is identified and a fix has been deployed. Pipeline run is in progress. We will share next update around 4 am UTC
Apr 25, 01:11 UTC
Investigating -
There is delay in Marketplace reporting data due to a pipeline failure and we are investigating. We will share more findings in next update.
Apr 24, 22:46 UTC
Apr 24, 2024
Unresolved incident: Forge Hosted Storage Scheduled Data Residency Rollout.