Humorously I had this exact discussion with someone at work. And my point was that it’s not practical to test 1:1 against a production environment and that it arose because the use-case that caused it wasn’t in their test sample. Which this confirms. It is also a good reminder that it is a game made by people and people don’t trend to perfection and that is okay. Their response to all of this has been, imo, a good way to handle such an immediate recurrence.

Yep.

And it was still their first public data failure in 16 years of running games that consume and process persistent data.

The fact that both rollbacks were caused by the same bug and just sheer bad luck points that overall, they've been pretty good at what they do.

And like what happened to them, you can miss it because you got (un)lucky and hit a fresh instance in test while old instances are still out there.

Good writeup though, I dig the transparency.

The Bungie Community Team has posted a fascinating debriefing/retrospective from the Destiny Operations Center or Player Support Team regarding the recent game outages.

Bungie.net | Destiny 2 Outage and Rollback

I don't like the unfortunate circumstances but I LOVE reading these debriefing/retrospective/breakdowns so much! (I heard a talk recently from an IBM X-Force Team member that tickled my brain in such ways)
I Thank those who were involved with the actions and fix, as well as those who pieced everything together and put it into a readable document (which is not as easy as you might think it is)

This is exactly why I roll my eyes whenever some armchair developer suggests something is an "easy" fix. At a certain scale developers are literally wrestling with alternate realities existing at the same time...

Bungie.net | Destiny 2 Outage and Rollback

I don't like the unfortunate circumstances but I LOVE reading these debriefing/retrospective/breakdowns so much! (I heard a talk recently from an IBM X-Force Team member that tickled my brain in such ways)
I Thank those who were involved with the actions and fix, as well as those who pieced everything together and put it into a readable document (which is not as easy as you might think it is)

Destiny 2 servers are back online. Players may experience login issues as services return to normal.

Players who observe issues should report to the Help Forum:https://t.co/cOAIctERNC

— Bungie Help (@BungieHelp) February 12, 2020

Due to the rollback:

💠 Any progress or purchase made between 8:30-10:30 AM PST will need to be redone
💠 Silver spent during that time will be restored
💠 Platform store-purchased Silver bundles will not be affected by the rollback
💠 Fractaline will be returned to inventories https://t.co/Rq4SVfPNJX

— Bungie Help (@BungieHelp) February 12, 2020

Destiny Companion features have been re-enabled on Web, Mobile, and Third-Party Apps.

Players should report any issues with these features to the Help forum: https://t.co/dXNlQalio5

— Bungie Help (@BungieHelp) February 12, 2020

We are investigating the re-emergence of the issue causing missing currencies and materials after Hotfix 2.7.1.1 went live. Destiny 2 will remain offline, please stand by for further updates.

— Bungie Help (@BungieHelp) February 11, 2020

I’d just avoid D2 for a few hours after any hot fix at this point...

Although you may not even have a choice. According to Bungie, D2 will be offline until 10pm EST.