" qwave



You can already currently do these things in Path of Exile. You can 'speed hack' during the period of latency since your last packet based on movement speed. The server will validate it and not desync you. Every MMORPG works this way. The paper was describing basic position prediction against latency. Scrotie:You can already currently do these things in Path of Exile. You can 'speed hack' during the period of latency since your last packet based on movement speed. The server will validate it and not desync you. Every MMORPG works this way. The paper was describing basic position prediction against latency. No. You can't. All you can acomplish by this is getting out of sync with the server. Ther server does not "validate" your position from the client in any way, it accepts input that you issued a movement command at a certain time and knows how fast you can move. If you make yourself move faster on your client, that won't affect the server at all, and if you get too far apart, it will resync you when this is noticed.



The majority of Scrotie's criticism of this concept is correct. This is not remotely feasible to implement on the scale of something doing as many calculations on PoE due to bandwidth, and if it was, opens the door to incredibly easy hacks for simple things like the "stop sending data to fake disconnect when you die" so that the server never knows you died and thinks you disconnected just before then, and over time, more and more of the consequences of the seed would be mapped out, allowing more sophisticated hacks.



Furthermore, this entire thread is based on a false assumption: " qwave I. PoE is not an MMORPG, it's a fast-paced action RPG



The client currently acts as a 'dumb terminal' which is continually streamed the game state from the server. The client's only responsibility is to render the game state and send input to the server. The client performs very little actual calculations, and instead relies almost 100% on the server to validate the game state. This means that the server is likely running an ongoing simulation of the game instance and is continually notifying the client of changes to the state based. Even introductions of small amounts of latency can cause desyncs (forcing the game client to the current authoritative game state). The client currently acts as a 'dumb terminal' which is continually streamed the game state from the server. The client's only responsibility is to render the game state and send input to the server. The client performs very little actual calculations, and instead relies almost 100% on the server to validate the game state. This means that the server is likely running an ongoing simulation of the game instance and is continually notifying the client of changes to the state based. Even introductions of small amounts of latency can cause desyncs (forcing the game client to the current authoritative game state). Nothing here is correct with regards to the functionality of the game. The PoE client is not a dumb terminal.



If you want to know more about how PoE actually communicates between the client and server, I would recommend Chris's manifesto post (link here) as a good starting point. Link doesn't work anymore, sorry. No. You can't. All you can acomplish by this is getting out of sync with the server. Ther server does"validate" your position from the client in any way, it accepts input that you issued a movement command at a certain time and knows how fast you can move. If you make yourself move faster on your client, that won't affect the server at all, and if you get too far apart, it will resync you when this is noticed.The majority of Scrotie's criticism of this concept is correct. This is not remotely feasible to implement on the scale of something doing as many calculations on PoE due to bandwidth, and if it was, opens the door to incredibly easy hacks for simple things like the "stop sending data to fake disconnect when you die" so that the server never knows you died and thinks you disconnected just before then, and over time, more and more of the consequences of the seed would be mapped out, allowing more sophisticated hacks.Furthermore, this entire thread is based on a false assumption:Nothing here is correct with regards to the functionality of the game. The PoE client is not a dumb terminal.Link doesn't work anymore, sorry. Last edited by Mark_GGG on Nov 19, 2013, 1:45:26 AM Posted by

Mark_GGG

on Grinding Gear Games on Quote this Post

" qwave The client would still be sending the same amount of data. NThe client would need to send everything it already does, as well as a lot of extra data about monster rolls in combat/ai and so on that it currently doesn't. By definition, that's more.



" qwave " Mark_GGG and if it was, opens the door to incredibly easy hacks for simple things like the "stop sending data to fake disconnect when you die" so that the server never knows you died and thinks you disconnected just before then I have addressed this by saying that there would need to be a Town Portal / Log Out timer of 3-4 seconds in order to prevent this sort of activity. I have addressed this by saying that there would need to be a Town Portal / Log Out timer of 3-4 seconds in order to prevent this sort of activity. Chris has already posted in other threads why that's not desirable. We really don't want to do this.



" qwave The PoE client IS a dumb terminal. and its only role is to send input and update the game state received from the server. It does not perform any of the calculations locally Yes it does. When you perform a skill, that happens on the client immediately, and on the server as soon as it receives that input. The client doesn't handle damage, but it does do pathing, targeting, and so on, so that we can have immediate feedback on what you're doing (as discussed in chris's post), as opposed to giving an input and then waiting for the server to tell us we've started the skill.

I have been writing the skill code that executes and does calculations on the client and on the server for nearly three years now. If the client is a dumb terminal that does no calculations, then probably half the work I've done in that time never happened.



" qwave Mark, it's clear that you have not read my proposal, or you do not understand the manner in which Path of Exile's synchronization works at a technical level. This may further explain the root cause of these desync problems. I have read your proposal (but not the entire thread yet, so may have missed some later updates to it. However, I have been working on skills and fixing thei bugs with them that cause sync issues for some time now, and based on your comments here you have some misunderstandings about the way PoE works under-the-hood with regard to client-server communication. I am not an expert in the field, and my contributions are limited to stuff where skills operate differently on client and server because of bugs (for example, the check for an action being executed from too far away was happening a second time ont he client in some cases of charging rhoas - fixing that reduced the amount they got out of sync by quite a lot).

" qwave



Mark_GGG has clearly demonstrated that he does not have an understanding of how the game client works. I would like to know ONE thing that the game client is able to perform on it's own accord. Mack, by definition you've described a dumb terminal. It has no ability to perform these calculations on its own. The client DOES need authorization to take each step. This is why you desync when trying to walk through doors.Mark_GGG has clearly demonstrated that he does not have an understanding of how the game client works. I would like to know ONE thing that the game client is able to perform on it's own accord. Pathfinding and targeting all occur on the client, as well as simultaneously on the server. In the case where they disagree, the server is treated as authoritative because it can be trusted, but those things all still happen on the client, and mostly they agree with the server, and don't get corrected. NThe client would need to send everything it already does, as well as a lot of extra data about monster rolls in combat/ai and so on that it currently doesn't. By definition, that's more.Chris has already posted in other threads why that's not desirable. We really don't want to do this.Yes it does. When you perform a skill, that happens on the client immediately, and on the server as soon as it receives that input. The client doesn't handle damage, but it does do pathing, targeting, and so on, so that we can have immediate feedback on what you're doing (as discussed in chris's post), as opposed to giving an input and then waiting for the server to tell us we've started the skill.I have been writing the skill code that executes and does calculations on the client and on the server for nearly three years now. If the client is a dumb terminal that does no calculations, then probably half the work I've done in that time never happened.I have read your proposal (but not the entire thread yet, so may have missed some later updates to it. However, I have been working on skills and fixing thei bugs with them that cause sync issues for some time now, and based on your comments here you have some misunderstandings about the way PoE works under-the-hood with regard to client-server communication. I am not an expert in the field, and my contributions are limited to stuff where skills operate differently on client and server because of bugs (for example, the check for an action being executed from too far away was happening a second time ont he client in some cases of charging rhoas - fixing that reduced the amount they got out of sync by quite a lot).Pathfinding and targeting all occur on the client, as well as simultaneously on the server. In the case where they disagree, the server is treated as authoritative because it can be trusted, but those things all still happen on the client, and mostly they agree with the server, and don't get corrected. Last edited by Mark_GGG on Nov 19, 2013, 2:08:44 AM Posted by

Mark_GGG

on Grinding Gear Games on Quote this Post

" BoltThrower87 Yeah, not to mention Mark is a programmer. One of the main programmers if I'm not mistaken. I wouldn't go that far. I've evolved form skills to general implementation and in the process gained a fairly good understanding about lots of different game systems without specialising in any of them. The people who're focused on those things know them much better than me, but I've got a fairly wide (as opposed to deep :P) knowledge range at this point which makes me good to implement, say, arbitrary things for uniques. I wouldn't go that far. I've evolved form skills to general implementation and in the process gained a fairly good understanding about lots of different game systems without specialising in any of them. The people who're focused on those things know them much better than me, but I've got a fairly wide (as opposed to deep :P) knowledge range at this point which makes me good to implement, say, arbitrary things for uniques. Posted by

Mark_GGG

on Grinding Gear Games on Quote this Post

" Firecrest5 " Mark_GGG If you want to know more about how PoE actually communicates between the client and server, I would recommend Chris's manifesto post (link here ) as a good starting point.

Link doesn't work. Seems the manifesto posts got hidden when 1.0 was released. Link doesn't work. Seems the manifesto posts got hidden when 1.0 was released. Ah, I must be able to see them due to being on a staff account. I'll remove the link, since it's not useful to most viewers. Ah, I must be able to see them due to being on a staff account. I'll remove the link, since it's not useful to most viewers. Posted by

Mark_GGG

on Grinding Gear Games on Quote this Post

As the programmer responsible for maintaining the client/server synchronization of the random level generation, I can tell you that this sounds like a nightmare, for a few reasons.



1) The server must validate the snapshots in real-time, not once the client leaves the instance. This is because the client can potentially muck around in one instance for huge lengths of time. Hours, days, weeks even, if the realm stays up. Processing so much all at once would lag the instance to hell and back, and changing areas has to wait on this task. Also, you will still need the ability to resync players to a recent state, when desync inevitably occurs (see next point). Also also, you need to be able to save the player's data at a moment's notice to prevent item dupes with trading.



2) Floating-point calculations will still cause desync. Suppose the client needs to perform a mathematical calculation, such as 37 / 13. The client determines the result to be 2.84615384. But when the server performs the same calculation, with the same values, the exact same bit-patterns inputs, it might instead return 2.84615385. The numbers are slightly different. Why would it do this? There are several reasons: Windows client vs. Linux servers. Different CPU architectures. Different optimizations compiling the servers vs. client. Different system drivers. Whatever the cause, you now have a divergence point, where the simulation may or may not diverge, depending on how those values are used and rounded. In that example, rounding to an integer afterwards isn't a problem, but what if the two values were 7.499999 and 7.500001? One will round to 7 and the other will round to 8. Suddenly the tiny difference isn't so tiny, and so even legit players can end up desynced.



3) The client would effectively have the ability to see into the future. A hacked client could predict when hits, crits, evades, and dodges will occur, so a player could alternately attack minions/bosses, or even just pause a certain number of milliseconds, to ensure they are almost never hit by strong attacks and almost always crit against the boss. In both of these cases, the final simulation is perfectly legitimate, so the server cannot know the client is cheating, unless you want some statistical system that punishes players for being "too lucky", which will harm legit players who get an actual lucky streak.



If item creation and currency use is done from the server, then you have monsters only dropping items 2-3 seconds after they die, and waiting 2-3 seconds for every currency item to take effect, since you have to wait for the next snapshot to be verified (and on a turbulent connection, that could be a while). Using (say) 1000 Fusings to try for a 6-link would take forever (you couldn't spam shift+right-click).



4) How do you detect death? You can't rely on the client sending the "Oops I died" packet. Otherwise hacked clients would never die. The server must be able to continue the simulation without the client, and determine when a player dies. This means that the server must keep simulating "the future", and client commands occur in "the past", which then changes the simulated future (like what Valve's Source Engine does). Now, you suddenly need the ability to effectively "rewind time" as far as the simulation is concerned. This is not a trivial task, at all.



5) This entire system breaks down completely as soon as you have more than one player. Anytime one player moves or hurts a monster, everyone else is automatically desynced and must be jolted back to the last verified snapshot. This is because you cannot recover from small amounts of desync, since even tiny differences completely change the RNG and state hash.



The "rewinding time" mechanic would somewhat solve this, but you would STILL get desync (just like in Team Fortress 2 and other Source Engine games). The classic example is being headshotted through a wall, because even though you successfully took cover on your screen, you didn't on the sniper's.



Also, what happens when the level geometry is different for different players? For example, if one player has destroyed the Undying Blockage in the Sewers, then the corridor is clear for them, but another player who hasn't done that will have the corridor blocked. The simulations for projectiles and movement will be VERY different on each client, so which one does the server think is correct?



6) Maphack would be even easier, and better, because it could show all the monsters and chests and items on the map, too.



Just my 2c. Code warrior Posted by

Rhys

on Grinding Gear Games on Quote this Post

" qwave Does the game not use the IEEE754 standard for binary floating-point arithmetic? If Path of Exile is programmed in C++, you can set the floating-point behavior with the /fp compiler flag as long as none of the libraries are compiled with -ffast-math.

Unfortunately, changing the floating-point behaviour doesn't help.



" qwave The future would change each time a new roll occurs, which would make it virtually impossible to predict. Even a seasoned bot would not have the computational power to benefit from this.

The future is simple to predict - it's a deterministic simulation. That's the whole point. The only unknowns are the client actions which are you solving for anyway. At least for attacking a single monster, for a given RNG spin there are only two possible outcomes, one for attacking NOW and one for not. Sure, in general finding the optimal solution is difficult, but it's very easy to make significant optimizations by simply artificially delaying the player's attacks by a millisecond or two.



" qwave this solution does not apply to crafting.

But crafting can affect combat, and movement. You can use Whetstones/Armour Scraps on your equipped gear. You can use Baubles on your flasks. You can even unequip-reroll-requip gear. These can change your movement speed and combat stats, so it needs to be sync'd. Also, you might do the crafting using items dropped by chests or monsters.



" qwave " Now, you suddenly need the ability to effectively "rewind time" as far as the simulation is concerned.

This has been brought up several times. I indicated that the game would need a delay on town portal / logout of 2-3 seconds. As long as the snapshots are very frequent (a second or less), then you could not reliably prevent death. This has been brought up several times. I indicated that the game would need a delay on town portal / logout of 2-3 seconds. As long as the snapshots are very frequent (a second or less), then you could not reliably prevent death.

A "logout time" doesn't fix this. A legitimate player who has a near-death experience can still "die" on the server but survive on his client.



" qwave The server would still retain the map/items/monster spawning.

The client needs to know what happens when a fireball flies off-screen. Are there monsters out there in the fireball's path? In order to maintain synchronous simulation, the client must know about every monster on the map.



" ScrotieMcB

Also, UDP already. Pretty sure I brought up all points myself, with the exception of point 6.Also, UDP already.

I kinda skimmed this thread, I must admit. Also UDP isn't a magical cure-all. And would totally suck in conjuction with qwave's plan, because packet-loss would create massive desync.



" Eep I don't see any supporter packs under yer avatar. Just sayin'

Off topic, but... that doesn't mean anything. Supporters can hide their forum title if they wish.



" adghar I've often wondered why GGG doesn't just hire a few more codewarriors with lots of experience in the industry and/or with client/server synchronization and just hammer out some better code.

It's very hard to find experienced game programmers in NZ. And importing people from overseas is hard because we can't match the salaries of most US/EU companies. We kinda have to rely on finding people who wanted to move to NZ anyway... and even then, we aren't the only company here super-keen to hire these people. Unfortunately, changing the floating-point behaviour doesn't help. Possibly due to targeting x86 with an older compiler. The future is simple to predict - it's a deterministic simulation. That's the whole point. The only unknowns are the client actions which are you solving for anyway. At least for attacking a single monster, for a given RNG spin there are only two possible outcomes, one for attacking NOW and one for not. Sure, in general finding the optimal solution is difficult, but it's very easy to make significant optimizations by simply artificially delaying the player's attacks by a millisecond or two.But crafting can affect combat, and movement. You can use Whetstones/Armour Scraps on your equipped gear. You can use Baubles on your flasks. You can even unequip-reroll-requip gear. These can change your movement speed and combat stats, so it needs to be sync'd. Also, you might do the crafting using items dropped by chests or monsters.A "logout time" doesn't fix this. A legitimate player who has a near-death experience can still "die" on the server but survive on his client.The client needs to know what happens when a fireball flies off-screen. Are there monsters out there in the fireball's path? In order to maintain synchronous simulation, the client must know about every monster on the map.I kinda skimmed this thread, I must admit. Also UDP isn't a magical cure-all. And would totally suck in conjuction with qwave's plan, because packet-loss would create massive desync.Off topic, but... that doesn't mean anything. Supporters can hide their forum title if they wish.It's very hard to find experienced game programmers in NZ. And importing people from overseas is hard because we can't match the salaries of most US/EU companies. We kinda have to rely on finding people who wanted to move to NZ anyway... and even then, we aren't the only company here super-keen to hire these people. Code warrior Posted by

Rhys

on Grinding Gear Games on Quote this Post

" Yimi Why does desync seem so prevalent in Path of Exile while it seems to be much less of an issue in Diablo 3?

Technically off-topic, so I'll reply in spoiler tags...

Spoiler Diablo 3 has several tricks that alleviate desync (last I checked):

-You can't miss with melee attacks

-You can't move out of the way of melee attacks



These things allow the client to more reliably predict the server calculations, but make combat more shallow.



They also don't really have narrow corridors or doorways to the extent we do, which also helps a lot, though you lose the feeling of claustrophobia such level features create.



In other words, there are pro's and con's.



Diablo 3's combat feels smoother, I think, for three reasons:

1) Cut combat animations

Many combat animations (weapon swings, attacks, spells) don't naturally blend from the idle/run animations. They deliberately *cut* to a keyframe in a dramatic pose, which then flows into the rest of the attack. Path of Exile's combat animations transition smoothly from idle to attack to idle.



2) Skipping utility animations

Some animations only play if the player is standing still. For example, casting buffs or curses. If the player is running or attack at the time, the spell goes off with NO animation at all. It doesn't interrupt the existing combat. In contrast, in Path of Exile, you have to stop moving and wait while you Enduring Cry/Curse/etc.



3) Input queuing

Diablo 3 has a deceptively clever input queuing system. Suppose you are spamming right-click a lot to keep doing one action, then press a key to do another, then go back to spamming right-click. You can't just queue every key-press, otherwise you'd be stuck performing the right-click action a million times. But even if that solitary key-press was made at a "bad time" i.e. during another action (at a time when you couldn't immediately do anything new) you do want to queue up this new action once before going back the spam. In Path of Exile, this extra action frequently gets lost in the spam, making you feel like the skill didn't work for some reason.

Technically off-topic, so I'll reply in spoiler tags... Code warrior Posted by

Rhys

on Grinding Gear Games on Quote this Post

" qwave Well you actually have to follow the IEEE standard. I know that it's doable. That's the whole point of the standard.

I'm not sure how I, an application developer, can fail to "follow a standard" for compilers. Other than playing with compiler flags (which doesn't work), what else can I do?



" qwave Player actions take time to perform, so even if you predict the best action, by the time you perform the action the seed would have rerolled.

Yes, but the client knows exactly what the new seed will be, because it is a deterministic simulation with no user input in the meantime. Also, when I mentioned that a hacked client could wait a while before attacking to optimize the crits/evades etc. I'm not talking about waiting for a length of time, I'm talking about waiting for a number of RNG spins.



" qwave I have also recommended using a cryptographic hash to make it unfeasible for bots to calculate the permutations

I don't see how this would help. What exactly are you hashing? Do you mean encrypting the RNG state (which isn't hashing, btw)?? Even then, this is not a difficult challenge for hackers to work around.



" qwave Again, crafting checks would occur server-side and not be involved in this proposal.

In which case you get instant desync every time you fiddle with your gear and a monster happens to be nearby.



" qwave As long as snapshots are being sent quickly enough, a lapse in snapshot will indicate a disconnect and the server will continue the simulation to kill the character.

But the whole point is that the character isn't supposed to die. Even a 1ms lag can mean the difference between life and death. I think it will feel quite bad (especially for Hardcore players) to have the client as the authority for movement and combat but NOT death.



" qwave The server can use quad-trees to send portions of the map the player, it doesn't need to send the entire map.

But the server has to send map data to a client operating "in the future" as far as the simulation is concerned. The server doesn't know where the player is located on the client, and it has to send the map data in advance. So it still needs to send map data for regions the serverside player can't see. Regardless, it's a massive "buff" to maphack.



" qwave why can't this system be implemented?

-Because we'd have to overhaul the entire skill system (and, indeed, game engine)

-Because we'd have to overhaul our network architecture

-Because it doesn't work for multiplayer

-Because it's still very prone to desync I'm not sure how I, an application developer, can fail to "follow a standard" for compilers. Other than playing with compiler flags (which doesn't work), what else can I do?Yes, but the client knows exactly what the new seed will be, because it is a deterministic simulation with no user input in the meantime. Also, when I mentioned that a hacked client could wait a while before attacking to optimize the crits/evades etc. I'm not talking about waiting for a length of time, I'm talking about waiting for a number of RNG spins.I don't see how this would help. What exactly are you hashing? Do you mean encrypting the RNG state (which isn't hashing, btw)?? Even then, this is not a difficult challenge for hackers to work around.In which case you get instant desync every time you fiddle with your gear and a monster happens to be nearby.But the whole point is that the character isn't supposed to die. Even a 1ms lag can mean the difference between life and death. I think it will feel quite bad (especially for Hardcore players) to have the client as the authority for movement and combat but NOT death.But the server has to send map data to a client operating "in the future" as far as the simulation is concerned. The server doesn't know where the player is located on the client, and it has to send the map data in advance. So it still needs to send map data for regions the serverside player can't see. Regardless, it's a massive "buff" to maphack.-Because we'd have to overhaul the entire skill system (and, indeed, game engine)-Because we'd have to overhaul our network architecture-Because it doesn't work for multiplayer-Because it's still very prone to desync Code warrior Last edited by Rhys on Nov 20, 2013, 2:31:16 AM Posted by

Rhys

on Grinding Gear Games on Quote this Post

" qwave Most C++ compilers should be following the IEEE 754 standard. Here's a brief whitepaper on the topic.

Well, according to that white paper, it really does simply not work. The standard does not cover trigonometric functions, nor does it make any guarantees across different systems. Let me quote the relevant sections for you:

" 4 Issues not addressed by IEEE 754

Neither the IEEE 754 nor the C++ standard specify how transcendental functions like sin(), cos(), exp(), etc. have to be implemented Neither the IEEE 754 nor the C++ standard specify how transcendental functions like sin(), cos(), exp(), etc. have to be implemented " 6 Conclusions

To summarize,

...the same code may produce slightly different results on different systems; To summarize,...the same code may produce slightly different results on different systems;



" qwave you would generate each random number using a cryptographic hash against the seed. This would make it computationally expensive for bots to discover enough permutations to be efficient.

It's exactly as fast as a legit client. Any computational expense is equally applied to both hacked and legit calculations. I'm not exactly keen to artificially slow the game down.



" qwave I don't believe that you can craft with equipped gear, right? As long as gear must be unequipped to craft, then the server can perform the upgrade. Once new gear is equipped, no synchronization should be necessary because it's a known state of the character.

You can with quality-increasing items. But I was more thinking of unequipping/reequipping.



" qwave I think you misread my post. Death would only be simulated by the server in the event that there is a lapse in snapshots/input (a disconnect). You indicated that players could cheat death by ending their snapshots, I am saying they cannot cheat death because in that event the server would continue the simulation.

I wasn't disputing the fact you can't cheat death. I was pointing out that legit players can still die "to desync", even with minor lag/latency.



" qwave It's not prone to desync at all except in very high latency scenarios (1000+ ms).

I've already pointed out two different scenarios where desync will occur, even with low latency. Well, according to that white paper, it really does simply not work. The standard does not cover trigonometric functions, nor does it make any guarantees across different systems. Let me quote the relevant sections for you:It's exactly as fast as a legit client. Any computational expense is equally applied to both hacked and legit calculations. I'm not exactly keen to artificially slow the game down.You can with quality-increasing items. But I was more thinking of unequipping/reequipping.I wasn't disputing the fact you can't cheat death. I was pointing out that legit players can still die "to desync", even with minor lag/latency.I've already pointed out two different scenarios where desync will occur, even with low latency. Code warrior Last edited by Rhys on Nov 20, 2013, 4:01:50 AM Posted by

Rhys

on Grinding Gear Games on Quote this Post