Avatar

What Lag Switching Looks Like (Destiny)

by Beorn @, <End of Failed Timeline>, Friday, April 15, 2016, 16:40 (3238 days ago)

This is from Trials last weekend. Pretty amazingly blatant lag switch:

This guy tried it on us a couple of times, but I was front-and-center for this particular instance. Amusingly, they couldn’t even cheat their way to victory for a single round. Asshats.

Bungie really needs to put the hammer down on cheating in Trials. This is still a fairly common occurrence and it really sours the experience when it stops a Lighthouse run. Yes, we submitted reports on all three, but that is frequently "too little, too late," and it doesn’t seem to be making a dent in the number of cheaters out there.

Avatar

Good vid

by Kahzgul, Friday, April 15, 2016, 17:22 (3238 days ago) @ Beorn

The cheating ruins the game for me. Sucks to see that Bungie still hasn't found a way to prevent it or effectively deter it.

Avatar

Good vid

by Cody Miller @, Music of the Spheres - Never Forgot, Friday, April 15, 2016, 18:09 (3238 days ago) @ Kahzgul

The cheating ruins the game for me. Sucks to see that Bungie still hasn't found a way to prevent it or effectively deter it.

I don't understand why it doesn't just drop you if your connection becomes that bad.

Avatar

Does Achronos still peer deeply into your IP soul?

by dogcow @, Hiding from Bob, in the vent core., Friday, April 15, 2016, 18:20 (3238 days ago) @ Cody Miller

Or has he passed that on to someone else? I have a feeling that was just on bungie.net. Who over there handles this?

It'd be interesting to hear how Bungie came to their decisions on how to handle networking situations like this, from a technical standpoint.

Avatar

Cause some people have connections that bad.

by Funkmon @, Friday, April 15, 2016, 18:22 (3238 days ago) @ Cody Miller

- No text -

Avatar

And most times it's very quick

by ZackDark @, Not behind you. NO! Don't look., Friday, April 15, 2016, 18:24 (3238 days ago) @ Funkmon

The days it is downright awful for more than half an hour, I quit playing online games, but there are times when it goes awful for 3 seconds at a time.

Avatar

I also expect it's a bit of an arms race.

by dogcow @, Hiding from Bob, in the vent core., Friday, April 15, 2016, 18:40 (3238 days ago) @ Funkmon

If you fix one type of networking exploit then the jerk will find another way to exploit the tcp/ip stack that may be harder to detect.

But so far I've only heard of the famed "lag switch" where they interrupt the upstream flow of data while still receiving the downstream flow. What I don't understand is why they can't "referee" this with the console being able to detect a steady stream of data without their upstream data being acked. I wonder, is destiny using udp? If so it must be making it's own guaranteed delivery of key information. Anyway, I'd love to know more about the technical aspects of how they do their networking. It'd be fascinating.

Avatar

I also expect it's a bit of an arms race.

by Cody Miller @, Music of the Spheres - Never Forgot, Friday, April 15, 2016, 19:56 (3238 days ago) @ dogcow

Games pretty much all use udp. Multiplayer games probably could not function otherwise. Google the postmortem for Xwing va Tie Fighter for a great explanation why.

Avatar

I also expect it's a bit of an arms race.

by dogcow @, Hiding from Bob, in the vent core., Friday, April 15, 2016, 20:49 (3238 days ago) @ Cody Miller

Games pretty much all use udp. Multiplayer games probably could not function otherwise. Google the postmortem for Xwing va Tie Fighter for a great explanation why.

Yeah, I know about TCP and how spectacularly it can fail for speedy delivery in poor network conditions, I just forget sometimes. My degree in CS had an emphasis in Networking (should have went for the software engineering emphasis as that's what I've done for my entire career... anyway). Being in my wired world I see the HUGE improvements that the net's made in the past 18 years and forget that people still have really really crappy connections (wifi/LTE) and I think, "oh, maybe TCP doesn't fail so badly anymore". When I forget this I'm oh so very wrong, TCP still has those massive failures, especially when using OTA connections.

So, yeah, Destiny definitely uses UDP, but at the application layer they have to implement features similar to TCP and I was pondering how they could detect a break in just upstream communication. When detecting it they could flag subsequent packets sent/received as being "suspect" and thus trigger an improved reconciliation of conflicting game states (Damage Referee anyone?).

Anyway, I'd love to see an article done by Chris Butcher & crew about their networking stack and how they handle that sort of thing. Just so I could geek out over it. :)

Avatar

Well, it's tricky

by Kahzgul, Friday, April 15, 2016, 20:43 (3238 days ago) @ dogcow

In Destiny's case, my understanding is that they invented a new type of distributed "no-host" networking model where every console tells every other console where that console's player is and what that player is doing. When no data is received, the other consoles use predictive algorithms to continue animating the characters. This is why destiny looks so dang smooth, and also why your bullets don't hit where you were aiming all the time. It's... problematic, as far as I'm concerned.

Basically the player who lags his upstream connection *while moving* is predicted to continue moving in that direction by other consoles on the network. Then, when the upstream connection catches back up, the consoles say, oh, so you actually weren't there and the person appears to teleport. Keep in mind that they would still have taken damage from any attacks that were aimed at where they actually were during that time, because their downstream packets still resolve. It's the disconnect between displayed visuals and actual avatar position that causes the painful lag-related gameplay. And because of the distributed token-ring-esque network setup, a single laggy player can cause everyone in the game to periodically lose sync, even when they all have otherwise good connections, and it's not consistent because of how the buck is passed - sometimes only laggy mcghee is lagging, and sometimes (far less often) it's everyone, and that changes in the middle of matches.

In contrast, a player with an overall laggy connection behaves differently. They see everyone blinking around the map, and they themselves will blink around, too. In this case, where both upstream and downstream data packets are lagging, it will appear to the poor connection player that everyone else is lag switching, and has damage avatars separate from their visible avatars. To the non lagging players, the lagged guy will teleport a bit, but also move in fits and starts, taking no damage for a while, running into walls, firing every round in their mag at the same time, and then suddenly falling over dead. It's similar looking to lag switching from the outside but lacks the smooth damage dealing and evasive ability. It's also far more consistent.

On the lag switcher's screen, everything is smooth all the time.

On the lagger's screen, they are constantly blinking around the map, unable to shoot, and dying for no reason.

Traditional PvP FPS games have a host player (usually the guy with the fastest and most reliable connection) and that player's console is the referee for where everyone is. All packets move through his console, so if someone lags, his console makes them stand still *and continue to take damage normally* until they catch back up, and this applies to lag switchers as well. There are certainly still ways to cheat in those games, but lagging isn't one of them.

Avatar

Well, it's tricky

by dogcow @, Hiding from Bob, in the vent core., Friday, April 15, 2016, 21:00 (3238 days ago) @ Kahzgul

... And because of the distributed token-ring-esque network setup, a single laggy player can cause everyone in the game to periodically lose sync, even when they all have otherwise good connections, and it's not consistent because of how the buck is passed - sometimes only laggy mcghee is lagging, and sometimes (far less often) it's everyone, and that changes in the middle of matches.

I've heard mention of this "token-ringesque" model before. Where does this originate from? I don't recall any statements from Bungie regarding their network model, other than that wonderful presentation regarding how they split up hosting responsibilities among player & bungie servers. I don't recall any mention of passing a token between hosts (unless people mean when the host for an area is handed off to someone else due to the host leaving). What purpose would it serve, taking turns declaring authoritative game state? I can see how that would be advantageous in that nobody could exploit being host, but wonder if it could be solved in other (better?) ways too.

Avatar

Check this out

by slycrel ⌂, Friday, April 15, 2016, 21:21 (3238 days ago) @ dogcow

http://destiny.bungie.org/n/1500

It's a good read and will answer a bunch of your questions.

Avatar

Yeah, that's the presentation I meant.

by dogcow @, Hiding from Bob, in the vent core., Friday, April 15, 2016, 21:36 (3238 days ago) @ slycrel

- No text -

Avatar

Well, it's tricky

by Kahzgul, Friday, April 15, 2016, 22:30 (3238 days ago) @ dogcow

... And because of the distributed token-ring-esque network setup, a single laggy player can cause everyone in the game to periodically lose sync, even when they all have otherwise good connections, and it's not consistent because of how the buck is passed - sometimes only laggy mcghee is lagging, and sometimes (far less often) it's everyone, and that changes in the middle of matches.


I've heard mention of this "token-ringesque" model before. Where does this originate from? I don't recall any statements from Bungie regarding their network model, other than that wonderful presentation regarding how they split up hosting responsibilities among player & bungie servers. I don't recall any mention of passing a token between hosts (unless people mean when the host for an area is handed off to someone else due to the host leaving). What purpose would it serve, taking turns declaring authoritative game state? I can see how that would be advantageous in that nobody could exploit being host, but wonder if it could be solved in other (better?) ways too.

It's from that same presentation. They say (somewhere) that PvP works the same but without the server, meaning it's just players passing player data back and forth. There's not really a token, it's something totally new and Bungie hasn't told us what they call it. It's just similar to a token ring architecture in that it has a lot of the same flaws and advantages of one. To be fair, it's a really brilliant system and in a perfect network environment it is both more fair and faster than a traditional single host structure. The problem being that none of us are ever playing this game in a perfect network environment, and some people are intentionally degrading their network environment in order to exploit the architecture of this design.

So, to clear some of this up, in PvP there is not host and no token. Each player is sending their outbound packets to every other player. Each player is receiving inbound packets from every other player. Each player's system is using that data to simulate the actual state of the game, and is then showing that simulation to the people playing. Does that make sense? The true gamestate is distributed between the systems of all players, with each console only truly knowing where the player playing on that console is at any given moment, and guessing at where the other players are based on predictive algorithms and the packets they're sending that say "I'm here!" or whatever.

Everything beyond that is speculation, but it's based on my decade of QA and development work, specifically in MP PvP FPS games. It appears that if Someone at console B shoots a gun, Console B will generate a packet that says "bullet originated at point X and traveled to point Y with damage Z." That packet is sent to everyone, and if it generates an intersection with another player, say console A, then console A will send its own packet of "I took Z damage". This is the behavior I've observed.

Neither of these things would be issues except the following behaviors also apply:

- Bullets stop moving when they encounter an avatar of a character, even if that avatar is for a lagged player and is not a known position but a predictive one. This means lagging people cease to take any damage the moment they lag one pixel behind where the predictive algorithm thinks they are.

- Lagging players are allowed to fully catch up when their packets are finally delivered, meaning that they can effectively queue their actions and execute them all at once. This is how people empty a shotgun instantly and kill four members of your team by doing so. Furthermore, the packets appear to be timestamped, so instead of the actions happening at the time the lag ends and affecting the players' positions at that time, the actions affect where the player was when the packets were generated, so even though you didn't see the lagger shooting you (or didn't even see him because he started lagging around the corner from you), when his lag catches up, he shot you a whole bunch.

- Certain precious resources, such as Supers and Heavy Ammo are exceptionally rare, and are not refunded if launched into lagging player avatars, which can cost games.

Remember that when you're shooting at someone, you're actually shooting at your console's best guess as to where that person is. You are not truly shooting at where they truly are.

I'm not sure how the damage referee works, but since it was added it seems to at least kill some of the laggers some of the time once they catch up, but it doesn't prevent any of the damage they dealt while lagging. Some fix is better than no fix, though.

Avatar

Well, it's tricky

by narcogen ⌂ @, Andover, Massachusetts, Friday, April 15, 2016, 23:15 (3238 days ago) @ dogcow

... And because of the distributed token-ring-esque network setup, a single laggy player can cause everyone in the game to periodically lose sync, even when they all have otherwise good connections, and it's not consistent because of how the buck is passed - sometimes only laggy mcghee is lagging, and sometimes (far less often) it's everyone, and that changes in the middle of matches.


I've heard mention of this "token-ringesque" model before. Where does this originate from? I don't recall any statements from Bungie regarding their network model, other than that wonderful presentation regarding how they split up hosting responsibilities among player & bungie servers. I don't recall any mention of passing a token between hosts (unless people mean when the host for an area is handed off to someone else due to the host leaving). What purpose would it serve, taking turns declaring authoritative game state? I can see how that would be advantageous in that nobody could exploit being host, but wonder if it could be solved in other (better?) ways too.

I think he means token ring without the token, since as he described it, there was no token- all the boxes are updating all the other boxes all the time, not just the one with the token.

Avatar

Well, it's tricky

by uberfoop @, Seattle-ish, Sunday, April 17, 2016, 16:51 (3236 days ago) @ dogcow

I don't recall any statements from Bungie regarding their network model

Bungie's networking presentation explicitly states that they're using traditional hosts for managing core gameplay/physics in any given bubble, and they even mention a typical host migration interval (160 seconds).

That presentation was mostly about PvE, I guess PvP could use a different approach but that seems unlikely.

Avatar

Well, it's tricky

by ZackDark @, Not behind you. NO! Don't look., Sunday, April 17, 2016, 22:27 (3236 days ago) @ uberfoop

That presentation was mostly about PvE, I guess PvP could use a different approach but that seems unlikely.

Dunno. Considering their history with Halo, that's extremely likely. On top of that, considering the dynamics of PvE vs PvP, I find that even more likely.

Avatar

Good vid

by narcogen ⌂ @, Andover, Massachusetts, Friday, April 15, 2016, 23:12 (3238 days ago) @ Cody Miller

The cheating ruins the game for me. Sucks to see that Bungie still hasn't found a way to prevent it or effectively deter it.


I don't understand why it doesn't just drop you if your connection becomes that bad.

Because the only thing people like to complain about more than cheaters or bad matchmaking is Bungie's bad servers that are worse than everyone else's. There's already a plethora of "all my other games work fine, it's not me it must be you" posts with regard to lag in Destiny.

Avatar

You don't know about the space magic trap door on that box?

by Ragashingo ⌂, Official DBO Cryptarch, Friday, April 15, 2016, 19:02 (3238 days ago) @ Beorn

- No text -

Avatar

:thumbsup:

by Beorn @, <End of Failed Timeline>, Friday, April 15, 2016, 19:15 (3238 days ago) @ Ragashingo

- No text -

Back to the forum index
RSS Feed of thread