Avatar

Bungie done did it again... (Destiny)

by Korny @, Dalton, Ga. US. Earth, Sol System, Tuesday, February 11, 2020, 12:42 (18 days ago)
edited by Korny, Tuesday, February 11, 2020, 12:53

I’d just avoid D2 for a few hours after any hot fix at this point...

Although you may not even have a choice. According to Bungie, D2 will be offline until 10pm EST.

Avatar

Back up!

by INSANEdrive, ಥ_ಥ | f(ಠ‿↼)z | ᕕ( ᐛ )ᕗ| \[T]/, Tuesday, February 11, 2020, 16:46 (18 days ago) @ Korny

(See what I did there...eh... anyway...)



Avatar

A fascinating debriefing/retrospective from the DOC

by Pyromancy @, Tuesday, February 11, 2020, 21:19 (17 days ago) @ Korny
edited by Pyromancy, Tuesday, February 11, 2020, 21:46

The Bungie Community Team has posted a fascinating debriefing/retrospective from the Destiny Operations Center or Player Support Team regarding the recent game outages.

Bungie.net | Destiny 2 Outage and Rollback

I don't like the unfortunate circumstances but I LOVE reading these debriefing/retrospective/breakdowns so much! (I heard a talk recently from an IBM X-Force Team member that tickled my brain in such ways)
I Thank those who were involved with the actions and fix, as well as those who pieced everything together and put it into a readable document (which is not as easy as you might think it is)

Avatar

Such a good deep dive into the issue

by kidtsunami @, Atlanta, GA, Wednesday, February 12, 2020, 07:39 (17 days ago) @ Pyromancy

The Bungie Community Team has posted a fascinating debriefing/retrospective from the Destiny Operations Center or Player Support Team regarding the recent game outages.

Bungie.net | Destiny 2 Outage and Rollback

I don't like the unfortunate circumstances but I LOVE reading these debriefing/retrospective/breakdowns so much! (I heard a talk recently from an IBM X-Force Team member that tickled my brain in such ways)
I Thank those who were involved with the actions and fix, as well as those who pieced everything together and put it into a readable document (which is not as easy as you might think it is)

This is exactly why I roll my eyes whenever some armchair developer suggests something is an "easy" fix. At a certain scale developers are literally wrestling with alternate realities existing at the same time...

Avatar

Such a good deep dive into the issue

by Harmanimus @, Wednesday, February 12, 2020, 08:44 (17 days ago) @ kidtsunami

Humorously I had this exact discussion with someone at work. And my point was that it’s not practical to test 1:1 against a production environment and that it arose because the use-case that caused it wasn’t in their test sample. Which this confirms. It is also a good reminder that it is a game made by people and people don’t trend to perfection and that is okay. Their response to all of this has been, imo, a good way to handle such an immediate recurrence.

Such a good deep dive into the issue

by FyreWulff, Wednesday, February 12, 2020, 18:09 (16 days ago) @ Harmanimus

Humorously I had this exact discussion with someone at work. And my point was that it’s not practical to test 1:1 against a production environment and that it arose because the use-case that caused it wasn’t in their test sample. Which this confirms. It is also a good reminder that it is a game made by people and people don’t trend to perfection and that is okay. Their response to all of this has been, imo, a good way to handle such an immediate recurrence.

Yep.

And it was still their first public data failure in 16 years of running games that consume and process persistent data.

The fact that both rollbacks were caused by the same bug and just sheer bad luck points that overall, they've been pretty good at what they do.

Avatar

+1

by Robot Chickens, Wednesday, February 12, 2020, 12:51 (17 days ago) @ Pyromancy

- No text -

TL;DR - Spaghetti

by someotherguy, Hertfordshire, England, Wednesday, February 12, 2020, 13:44 (17 days ago) @ Pyromancy

Disabled timestamps on some quests, which lowered the stack limit on items to 0.

Good writeup though, I dig the transparency.

Avatar

Solid Write-Up!

by INSANEdrive, ಥ_ಥ | f(ಠ‿↼)z | ᕕ( ᐛ )ᕗ| \[T]/, Thursday, February 13, 2020, 14:12 (16 days ago) @ Pyromancy

I'm not sure I understood the whole thing mind you, but I enjoyed the peek behind the curtain all the same. :)

Cached / stale servers / data..

by FyreWulff, Wednesday, February 12, 2020, 16:00 (17 days ago) @ Korny

... yeah, can't say I haven't been bit by that a few times.

And like what happened to them, you can miss it because you got (un)lucky and hit a fresh instance in test while old instances are still out there.

Back to the forum index
RSS Feed of thread