Unauthorized Users Added to Marketplace Vendor Accounts from Bug caused by Database Failover
Incident
Report for Atlassian Developer
Resolved
We have now completed the reversing of all user mapping swaps, and can mark this incident as resolved.
The "Create new vendor" and "update vendor contacts" functionality has now been restored on Marketplace. All Marketplace access has now been restored with full functionality.
Thank you for your patience as we worked through this incident.
We have already started the Post Incident Review process and will be seeking input from Marketplace partners as part of this process.
The "Create new vendor" and "update vendor contacts" functionality has now been restored on Marketplace. All Marketplace access has now been restored with full functionality.
Thank you for your patience as we worked through this incident.
We have already started the Post Incident Review process and will be seeking input from Marketplace partners as part of this process.
Monitoring
User swaps for affected partner accounts were reversed today, and partners should now have access to their vendor accounts with the right privileges. Please raise a ticket at AMKTHELP if you still have a problem with the contacts in your vendor accounts.
Create new vendor and modify vendor contacts functionality is still blocked while we complete a final set of checks. We plan to re-enable this functionality tomorrow.
Tomorrow, we will also look to complete the reversal of user swaps for non-partner accounts.
Thanks you for your patience through this entire process.
Create new vendor and modify vendor contacts functionality is still blocked while we complete a final set of checks. We plan to re-enable this functionality tomorrow.
Tomorrow, we will also look to complete the reversal of user swaps for non-partner accounts.
Thanks you for your patience through this entire process.
Update
On Saturday and Monday, we have undid the Atlassian id swaps for all partners who had requested urgent access to their account via the support Helpdesk.
We aim to complete the user swaps for all affected partners tomorrow.
We do understand that the remaining affected users would be anxious to get access back to their Partner account, and will thus carefully balance caution with urgency as we proceed.
Thanks for your continued patience. We will share an update tomorrow.
We aim to complete the user swaps for all affected partners tomorrow.
We do understand that the remaining affected users would be anxious to get access back to their Partner account, and will thus carefully balance caution with urgency as we proceed.
Thanks for your continued patience. We will share an update tomorrow.
Update
We have restored the Partner account access for all unaffected authorised users.
We do understand that the remaining affected users would be anxious to get access back to their Partner account. Our aim is to complete the exercise at the earliest, but also observe abundant caution as this activity requires carefully undoing the swaps of Atlassian_ids.
In light of the above, while we will continue to actively work on restoring access for remaining affected users, for ensuring accuracy we will start rolling out these changes on Monday, 22 Mar 2021.
Thanks for your continued patience. We will share an update when the next set of changes are ready to be rolled out.
We do understand that the remaining affected users would be anxious to get access back to their Partner account. Our aim is to complete the exercise at the earliest, but also observe abundant caution as this activity requires carefully undoing the swaps of Atlassian_ids.
In light of the above, while we will continue to actively work on restoring access for remaining affected users, for ensuring accuracy we will start rolling out these changes on Monday, 22 Mar 2021.
Thanks for your continued patience. We will share an update when the next set of changes are ready to be rolled out.
Update
We will be restoring access to Marketplace partner functionality on Mar 19, 2021 ~10:30 UTC. This will allow partners to access their accounts again. In this process, we are removing all users (both authorised and unauthorised) modified between 12th Mar 2021 UTC and 15th Mar 2021 UTC from their respective Partner accounts.
After this process is complete:
- Unaffected authorized users will continue to appear in the correct Partner account.
- All affected authorized users will not be able to access their Partner account. They will be added back to their correct Partner accounts soon after.
- Any legitimate action done in the past by an affected authorized user in their Partner account will temporarily be attributed to an unauthorized user (i.e. app version release).
- All users will continue to be restricted from creating as well as deleting existing contacts.
- New Partner account creation will be temporarily disabled
Our target action steps for the next update are:
- Restoring correct mapping between Marketplace user_id and Atlassian account_id for affected users. This will restore correct attribution of actions previously done by their respective users.
- Adding back all affected authorized users to their correct Partner account.
The remaining steps after that will be:
- Opening up user modification functionality for all authorized users.
- Opening up new Partner account creation.
Please raise a ticket at AMKTHELP if your admin contact is affected. We appreciate your continued patience and support. We will keep you informed as we make progress.
After this process is complete:
- Unaffected authorized users will continue to appear in the correct Partner account.
- All affected authorized users will not be able to access their Partner account. They will be added back to their correct Partner accounts soon after.
- Any legitimate action done in the past by an affected authorized user in their Partner account will temporarily be attributed to an unauthorized user (i.e. app version release).
- All users will continue to be restricted from creating as well as deleting existing contacts.
- New Partner account creation will be temporarily disabled
Our target action steps for the next update are:
- Restoring correct mapping between Marketplace user_id and Atlassian account_id for affected users. This will restore correct attribution of actions previously done by their respective users.
- Adding back all affected authorized users to their correct Partner account.
The remaining steps after that will be:
- Opening up user modification functionality for all authorized users.
- Opening up new Partner account creation.
Please raise a ticket at AMKTHELP if your admin contact is affected. We appreciate your continued patience and support. We will keep you informed as we make progress.
Update
Our preliminary investigation of the incident has identified the root cause to be corruption within MongoDB triggered at the point of a failover event on 12th Mar 2021 20:20 UTC.
This modified the mapping between Marketplace user_id & Atlassian account_id. While Marketplace user_ids constituting the Partner contact groups remained intact, some Marketplace user_ids got linked to the wrong Atlassian account_id. This resulted in:
Certain Atlassian account_ids gaining access to incorrect Partner contact group
Certain Atlassian account_ids losing access to their correct Partner account and/or being added to an incorrect Partner account
We investigated the scope of potential unauthorized access by users who were inadvertently added to the Partner account.
Our next step was to restore functionality by:
Removing all impacted users (both authorized and unauthorized) from Partner contact groups, regardless of whether they were added or deleted by authorized users.
Unblocking Partner account access for all remaining unimpacted authorized users.
At 18th Mar 2021 14:35 UTC, we attempted to restore functionality by rolling out the above changes. Within 20 min of the roll-out, one additional user_id swap was reported to us. Out of abundance of precaution, we reverted the roll-out at 18th Mar 2021 15:00 UTC.
We are investigating the reported swap and will resume restoring functionality once we are confident the underlying root cause of the swap is resolved.
We will provide an update within the next 12 hours. Our priority is to restore functionality as soon as we can.
Thank you for your patience and continued support.
Note: We will share a detailed PIR after resolving the incident and completing the investigation.
This modified the mapping between Marketplace user_id & Atlassian account_id. While Marketplace user_ids constituting the Partner contact groups remained intact, some Marketplace user_ids got linked to the wrong Atlassian account_id. This resulted in:
Certain Atlassian account_ids gaining access to incorrect Partner contact group
Certain Atlassian account_ids losing access to their correct Partner account and/or being added to an incorrect Partner account
We investigated the scope of potential unauthorized access by users who were inadvertently added to the Partner account.
Our next step was to restore functionality by:
Removing all impacted users (both authorized and unauthorized) from Partner contact groups, regardless of whether they were added or deleted by authorized users.
Unblocking Partner account access for all remaining unimpacted authorized users.
At 18th Mar 2021 14:35 UTC, we attempted to restore functionality by rolling out the above changes. Within 20 min of the roll-out, one additional user_id swap was reported to us. Out of abundance of precaution, we reverted the roll-out at 18th Mar 2021 15:00 UTC.
We are investigating the reported swap and will resume restoring functionality once we are confident the underlying root cause of the swap is resolved.
We will provide an update within the next 12 hours. Our priority is to restore functionality as soon as we can.
Thank you for your patience and continued support.
Note: We will share a detailed PIR after resolving the incident and completing the investigation.
Update
As communicated in the previous update:
- No billing details were accessed or changed by unauthorized users in any impacted vendor partner account
- No reports and transaction details were accessed or changed by unauthorized users in any impacted vendor partner account
- No new app versions were created by unauthorized users in any vendor partner account
However, we are continuing to investigate and enumerate all unauthorized access to any Marketplace Partner account’s contact information and app details. In the process, we have already reached out directly to a subset of impacted Marketplace partners to communicate the impact to them and will be reaching out to others as we progress.
As no billing details were accessed or changed by unauthorized users in any impacted Partner partner account, our finance teams have started to process the payments for the month of February (due in March).
Up to now we have prioritized understanding the full scope of potential unauthorised access to partner data and communicating this to partners above restoring access to Marketplace Partner admin functionality.
We are now evaluating options to safely restore access to Marketplace Partner functionality, and hope to be able to do this tomorrow ( 18 Mar 2021 ) for the majority of functionality.
We will keep you informed as we progress.
Thank you for your patience and continued support as we work through this sensitive matter.
- No billing details were accessed or changed by unauthorized users in any impacted vendor partner account
- No reports and transaction details were accessed or changed by unauthorized users in any impacted vendor partner account
- No new app versions were created by unauthorized users in any vendor partner account
However, we are continuing to investigate and enumerate all unauthorized access to any Marketplace Partner account’s contact information and app details. In the process, we have already reached out directly to a subset of impacted Marketplace partners to communicate the impact to them and will be reaching out to others as we progress.
As no billing details were accessed or changed by unauthorized users in any impacted Partner partner account, our finance teams have started to process the payments for the month of February (due in March).
Up to now we have prioritized understanding the full scope of potential unauthorised access to partner data and communicating this to partners above restoring access to Marketplace Partner admin functionality.
We are now evaluating options to safely restore access to Marketplace Partner functionality, and hope to be able to do this tomorrow ( 18 Mar 2021 ) for the majority of functionality.
We will keep you informed as we progress.
Thank you for your patience and continued support as we work through this sensitive matter.
Identified
On March 14th 2021, the Atlassian team confirmed that unauthorized users were given access to Marketplace Partner accounts for marketplace.atlassian.com.
At that time, we froze access to all users of partner accounts and halted payment processing in order to properly investigate the scope of the incident. We have since confirmed this activity was happening because of a bug, and not because of malicious activity.
We have also confirmed:
No billing details were accessed or changed by unauthorized users in any impacted vendor account
No reports and transaction details were accessed or changed by unauthorized users in any impacted vendor account
No new app versions were created by unauthorized users in any vendor partner account
Our team is continuing to investigate potential unauthorized access to your vendor account’s contact information and app details.
We will reach out directly to each impacted partner when our investigation is complete. We will also provide further technical details on this incident in a public PIR.
Follow the Atlassian Developer Statuspage for further updates.
At that time, we froze access to all users of partner accounts and halted payment processing in order to properly investigate the scope of the incident. We have since confirmed this activity was happening because of a bug, and not because of malicious activity.
We have also confirmed:
No billing details were accessed or changed by unauthorized users in any impacted vendor account
No reports and transaction details were accessed or changed by unauthorized users in any impacted vendor account
No new app versions were created by unauthorized users in any vendor partner account
Our team is continuing to investigate potential unauthorized access to your vendor account’s contact information and app details.
We will reach out directly to each impacted partner when our investigation is complete. We will also provide further technical details on this incident in a public PIR.
Follow the Atlassian Developer Statuspage for further updates.
Update
The investigation is still underway, Marketplace Partner's admin features will continue to be disabled to prevent any potential abuse while we resolve this.
Traffic on the Marketplace itself continues to operate as normal and customers can continue to evaluate and purchase apps.
We will continue to provide updates on our investigations and fixes as they progress.
Traffic on the Marketplace itself continues to operate as normal and customers can continue to evaluate and purchase apps.
We will continue to provide updates on our investigations and fixes as they progress.
Update
We are still investigating the root cause, but based on our analysis we believe this to be an application-level bug that caused data corruption.
We are currently investigating how we can safely restore a backup and will keep you informed on further updates.
Traffic on the Marketplace itself continues to operate as normal and customers can continue to evaluate and purchase apps, but the block on all admin functionality on Marketplace for Vendors is still in place
We are currently investigating how we can safely restore a backup and will keep you informed on further updates.
Traffic on the Marketplace itself continues to operate as normal and customers can continue to evaluate and purchase apps, but the block on all admin functionality on Marketplace for Vendors is still in place
Update
We are still investigating the issue and the block on all admin functionality on Marketplace for Vendors is still in place, this includes access to vendor reports as well admin-only actions.
Traffic on the Marketplace itself is normal and customers can continue to evaluate and purchase apps.
Traffic on the Marketplace itself is normal and customers can continue to evaluate and purchase apps.
Update
We are continuing to investigate the root cause of this issue. In the meanwhile, we have disabled access to all vendor reports and vendor management actions for all vendor contacts to prevent any malicious activity on the site. Vendor payouts have been suspended until further notice.
Update
We are continuing to investigate this issue.
Investigating
Investigating the issue and blocking access to all vendor roles in manage vendor account and access to vendor reports.
This incident affected: Marketplace (Reporting APIs and dashboards, Vendor management).