Query sometimes returns an empty string for custom entities
Incident
Report for Atlassian Developer
Resolved
We have successfully re-run the migration of custom entity data and have re-enabled data residency for all of the affected sites. We are now marking this incident as resolved.
We apologize for any inconveniences this may have caused you, your team, and our mutual customers. We are committed to finding and preventing the root cause of the issue.
We apologize for any inconveniences this may have caused you, your team, and our mutual customers. We are committed to finding and preventing the root cause of the issue.
Update
We have verified that our remedial actions have been effective in restoring the operations of apps that were affected by this incident. During our cross-examination, we haven't found any evidence of errors impacting the consistency of the data stored in our database. Later today we will re-run the migration of custom entities data to ensure no entry contains an empty value. This process is estimated to finish in the next 2-3 days. Once this is done we will close the incident.
We expect to have an update within the next 3-4 days.
We expect to have an update within the next 3-4 days.
Update
We have remediated the custom entity data store for the remaining identified sites. The read access to the custom entities data store has been restored to a state that no longer returns an empty string. We are monitoring and cross-checking the errors recorded on our platform to ensure the integrity of the data and the migration process.
We expect to have an update within the next 24-48 hours.
We expect to have an update within the next 24-48 hours.
Monitoring
We have restored read access to custom entity data stores for 99.5% of identified sites.
We have identified a small number of sites where the impacted data requires further remediation steps. Our team is currently looking into this and will investigate these sites on a case by case basis to see what additional steps are required.
We expect to have an update on these sites within the next 24 hours.
We have identified a small number of sites where the impacted data requires further remediation steps. Our team is currently looking into this and will investigate these sites on a case by case basis to see what additional steps are required.
We expect to have an update on these sites within the next 24 hours.
Identified
We have identified an issue with Forge apps that integrate the custom entity data store. While reading historical data that has not been updated since April 5th, the service may return an empty string instead of the value stored in the database. This would have appeared to end users as though the application data was missing. This was due to a logic error resulting in incomplete migration of data during our current data residency rollout.
Timeline
2024-04-30 06:15 UTC: Rollout of feature flag to read custom entity data from regional data. stores as per data residency setting for the workspace.
2024-05-05 23:00 UTC: Incident raised after partners reported errors.
2024-05-06 05:00 UTC: Root cause detected.
2024-05-06 06:00 UTC: Remediation steps approved.
We expect to resolve approximately 99.5% of impacted data to be available within the next 10 minutes.
We have identified a small number of sites where the impacted data requires further remediation steps. Our team is currently looking into this and will investigate these sites on a case by case basis to see what additional steps are required.
We expect to have an update on these sites within the next 24 hours.
Timeline
2024-04-30 06:15 UTC: Rollout of feature flag to read custom entity data from regional data. stores as per data residency setting for the workspace.
2024-05-05 23:00 UTC: Incident raised after partners reported errors.
2024-05-06 05:00 UTC: Root cause detected.
2024-05-06 06:00 UTC: Remediation steps approved.
We expect to resolve approximately 99.5% of impacted data to be available within the next 10 minutes.
We have identified a small number of sites where the impacted data requires further remediation steps. Our team is currently looking into this and will investigate these sites on a case by case basis to see what additional steps are required.
We expect to have an update on these sites within the next 24 hours.
Update
We are continuing to investigate this issue.
Investigating
We have identified a problem with querying custom entities where a query returns an empty string for some keys instead of the value stored in the database. At this time we believe that the issue is limited to query operations in a small number of apps and all the other operations are not impacted. We are investigating the root cause and will provide another update by 08:00 UTC.
This incident affected: Developer (Hosted storage).