install webhooks for Connect apps are sent to the wrong region
Incident
Report for Atlassian Developer
Resolved
Between 07/03/2022 11:08 PM UTC to 10/03/2022 11:17 AM UTC, 16 Connect apps with enabled data residency setting could receive reinstallation hooks in the wrong region. The root cause was a bug in the code that didn't use the values from 'regionBaseUrls' and used 'baseUrl' instead. We disabled the feature that controlled this behaviour and deployed a hotfix. During the next two days after disabling the feature, we were sending reinstallation events to the affected apps' tenants. All events have now been sent, the issue is resolved and the service is operating normally.
Monitoring
We have sent reinstallation events to all but two apps' tenants and got confirmation that the fix is working fine. We have turned on the process of sending reinstallation events for the rest of the apps and it should finish within the next 24 hours.
We will keep monitoring the progress and send the next comms once the job is finished.
We will keep monitoring the progress and send the next comms once the job is finished.
Update
We have released fixes to Jira and Confluence for the underlying issue that caused this incident. The consequence of which is that all future installs from this point onwards will respect the region-based routing found in App descriptors. This means that there should be no new instances of installation events being sent to the wrong routes.
Starting shortly, in the next hour, we intend to start sending out reinstallation events to all of the Jira or Confluence products that had Apps installed that were affected by this incident. App developers should notice these reinstallation events hitting their Apps. This change will roll out over a 24 hour time period as per our standard Cloud rollout strategy.
Starting shortly, in the next hour, we intend to start sending out reinstallation events to all of the Jira or Confluence products that had Apps installed that were affected by this incident. App developers should notice these reinstallation events hitting their Apps. This change will roll out over a 24 hour time period as per our standard Cloud rollout strategy.
Identified
We have confirmed that some of the aforementioned 16 apps with enabled data residency were affected. We're working on deploying a hotfix right now. After the hotfix is deployed, we will trigger the installation sync process that will send installation hooks to all of the affected tenants to the correct region. This will happen in the next 6-12 hours.
We will provide more details once the fix is deployed.
We will provide more details once the fix is deployed.
Monitoring
We have identified 16 potentially affected apps that could possibly be affected. This issue affects the apps with data residency setting enabled. We couldn't identify the broken installations based on our research. We have identified the root cause and the feature affecting the behaviour is turned off. No new installations will be affected by this.
We will trigger the installation sync process tomorrow: this should fix the potentially broken installations in approximately 24 hours. We will also keep monitoring the support requests.
We will provide more details once the fix is delivered.
We will trigger the installation sync process tomorrow: this should fix the potentially broken installations in approximately 24 hours. We will also keep monitoring the support requests.
We will provide more details once the fix is delivered.
Investigating
We are investigating cases of Connect apps install webhooks that were sent to the wrong region. Between 07/03/2022 11:08 PM UTC to 10/03/2022 11:19 AM UTC a number of Connect apps could receive install webhooks in the wrong region that could lead to the broken installations. We are aware of 24 number of apps being affected. We have identified the root cause and the feature affecting the behaviour is turned off. No new installations will be affected by this.
We're trying to understand if we can trigger re-installations for affected apps. If this is not possible, we will reach out to the affected app owners.
We will provide more details within the next hour.
We're trying to understand if we can trigger re-installations for affected apps. If this is not possible, we will reach out to the affected app owners.
We will provide more details within the next hour.
This incident affected: Marketplace (In-product Marketplace and app installation (Cloud)).