Worrazen Senior Member

Join Date: Oct 2014 Location: Slovenia Posts: 1,482

DCS Multi-Threading Status Report February 2020



With quite some new people around since then, who aren't up to speed with the DCS performance stuff, I decided to make this overview of the DCS threads behavior, with some general observations and not trying to go into definitive detail, it should do the trick for most people to get a sense of the status of "Multi-Core Support" (there's no such thing :p)



Initially in this post, I got into a small A-10C mission, and because I've seen different behavior with different modules, maps, types of units on ground and/or air, radars, EWR, the types of actions you're performing (TGP usage, types of weapons), and so on, it is very important to always associate any such DCS performance stats with a specific scenario that was used when recording.



This case obviously did not have any missle AA units, scanners, radars, EWR, there was no ECM usage, no countermeasures, and I didn't even have GBU's to drop due to mission's default loadout choice.







The way you read/interpret this PerfMon graph is, .... well I'm not really sure as I used it more lightly, which is weird as It's me who should have figured out this before posting, but this is all niche stuff with scarse info on the web, the point is it's good enough to get you an idea how things may be under the hood.

It probably safe to say that 1 thread isn't taking, at some point over 80% of the whole CPU which is a Quad-Core, that wouldn't make any sense, so that wouldn't be correct interpretation.

One thread can only do 25% on a Quad-Core with HT disabled, so 100% of all cores divided by 4 is 25%. A lot of statistical data is lost when viewing total CPU usage of a process, that's why using Task Manager is not recommended, it's hardly reality at that point, most system administrators and other IT specialists within the industry know that Task Manager is garbage and it's solely made for the average user to get a rough idea of the system's health and status.

So interpreting this graph in a sense that when you look at a color, a thread, the scale on the left being how much this thread was taking in regards to the CPU core it was on. So Blue Thread was taking 80% of CPU Time on CPU2.

That is the correct idea and does apply in general how it works, but still take this specific graph with a grain of salt.



An example of a finding:



(There's so many threads that one got the same color, I forgot to change one of the reds to a different color)



So we got mostly Blue, Two Reds and Cyan, and spikes from a Yellow. So everything more than 4 is going to start filling into the same 4 CPUs we have in this case, but keep in mind that we're not maxing out the other 2-3 cores, there's still space for many threads with light load to fit into the 3 threads.



Now you can see where this case would get a potential slowdown, or a stutter, you can see in the middle roughly there's a spike of a Yellow graph, and that's a 5-th color, that would mean we got one thread here that has to fit in those 4 Cores and potentially be forced-in taking away headroom from the rest, we might be bottlenecking, well not necessairly ...



... Remember we still have headroom, that Yellow spike shouldn't affect the main thread, it's not big enoguh, it could fit in any of the cores that still have headroom, but what we can observe suspiciously is that there is a correlation with the main thread dropping down at the same time the Yellow thread is spiking, that's not what we would like to see if we want optimization, it could be a coincidence or a bug/issue/design in the DCS engine it self.

Having to pause some type of workload on the main thread while a different workload on a different thread is taking place is extinguishing the point of multi-threading.

Such cases, if true, would give us a false sense of optimization and multi-threading, if we were to just look at raw numbers, eye balling the data, looking at number of threads and their activities. So with all diagnostics and analysis it's very important to look at many different types of views and take things seriously, interpreting the diagnostic data correctly is very important and the last thing I would like to do here is to create some panic among the community by wrongly interpreting the data, or giving false sense of performance. What I can say for certain it's definitely better than what people think it is.



Back to our little example here, ... if the main thread has to stop for another thread to work then what's the point of that workload being on another thread right! But we still have to take all the factors and conditions into account, this graph is a replay of a 30 minute A-10C flying around doing stuff that you can see in the video, but also AIs doing it's own thing, it can get very complicated of who's responsible for what spike/event on the graph, you can separate with specific scenarios designed for that, but in general gameplay it's all molded together. I've did many such things in the past and boy the graphs get interesting when you destroy 500 units and there's a huge load of smoke, those are specific case examples which are complicated and I didn't want to post those on purpose because it's hard to explain all the tiny circumstances, without doing a live take video where you would see, unfortunately the other programs are extremely resource intensive and wouldn't be able to run both DCS and them at the same time, recording diagnostic data is already a resource hog by it self, but Windows Performance Toolkit isn't even designed for real-time operation anyway. So that's why it's key to not jump to conclusions, and the deeper you dig it actually presents more questions and more places to dig further, then it's obvious that everything's extremely relative to the scenario. People that report stuttering versus other people that don't may just be using different modules with different maps playing a different style of gameplay and using different weapons and systems.



So when looking at diagnostic data and graphs always keep a sharp lookout on the whole situation. Again, this scenario was not using any radars, any ships, and missle AA batteries, all the data is relative to this scenario in the video below.



There may be other benefits of putting things on a different thread but still being kinda tied to the main operation from various development point of views that I do not know about, but maybe that's a dud, I don't know, but generally something happening on another thread should not be stopping other things and getting in-the-way, that said this is still looking back at DCS as most of the core improvements haven't come, it's a look back at the last decade, so if anyone's reading this years down the line, there's no reason to be concerned, it's looking far better as it is, we all thought it's only running on 1 thread, well nope, so things are just going to get better from here on out.



There's no reason to concern overall, there's no cases where it's super bad, overall DCS is hard on the system yes, but I didn't see any hard bugs or indications of them, except the A10C_Protect.dll and FC3_Protect.dll seem to be kinda legacy things chugging along there taking a bit of CPU regularly, even tho if you're not flying A10C or any FC3 aircraft in a session, again those may just be names with other modules using it, while I'm speculating other newer modules seem to have their protection performance hit probably tied with DCS_protect.dll which isn't that big alone, let's just hope it's not stopping the main thread when it's taking CPU time.





A basic overview of the threads with approximated descriptions:









The basic case scenario video:









So what are the observations we can probably make:

There's one thread that seems to be taking more CPU time than any other one.

Audio seems to be on it's own threads

Storage Access (IO) seems to be on it's own threads

Copy Protections seem to be on their own threads

Some part of the graphics and driver stuff is separate

Some controls input may be separate as well

TacView plugin is on it's own thread too

Also, please note that the thread Start Addresses may not be 100% accurate/valid of what we're assuming the workload is, because it can be convoluted in code as things may be called from one place but work being done elsewhere, spread across differently below the calling piece of code. So don't take it as it's written in stone, althought it does make sense with the rest of the data for the most part, we know from the devs that IO is split from the main thread, we know from the devs that Audio is split from the main thread, at least that's what I'm aware of.

Just keep in mind that there could be other types of work on that thread apart from what it's Start Address may seem to tell us about, so an audio thread may not be exclusively doing just audio work, for example.





2020 And Beyond: (More opinion style!!!)



We'll see a major change in these stats when Vulkan API comes, unless some big new feature comes that takes up all that gained performance, but I doubt it, I believe all the driver overhead and draw calls run on the main thread and that's a big thing Vulkan based GFX engine will deal with.



The next big thing to alleviate off the Main ("simulation") thread is the AI if we are to believe the basic thread info, it has to be spun off the main thread ASAP, I don't see how AI logic per-unit would need to be on the main thread where physics calculations happen. There's proably loads of serial workloads that don't have anything to do with each other math-wise. While you fire a missile at an AI1 airplane and that missile calculations have to go on and it's taking a lot of CPU time on Thread1, what's keeping some distant AI3 from running it's separate AI logic on Thread2 for landing or whatever, right right!?!? So there's a lot of potential to split away these serial task into many threads. Each unit on the battlefield or rather squad could probably run not only in it's own thread, but it's own set of threads (by type) if you ask me.



Mission Editor logic and triggers, waypoints, etc, maybe don't have to run on the main thread either (I could be wrong a bit if there's a thing to do with events/detection, but conditions nah) So if the ME logic, a condition, checks for IF THIS UNIT IN ZONE A, it doesn't have to be on the main thread, sync with physics shouldn't be needed because it may fire what at most 100 ms later than it would otherwise, that's probably nothing in practice, and DCS matches with complex ME logic will probably have 1000x of factors so the chances of this discrepancy, but then again I'm talking worst case scenario. What I'm talking about is like, you entered zone A by a few meters, but because ME logic was late it did't register, but then you got hit by missile and then you happen to fail the mission. Chances of this being a problem? Then again, because the Main thread is bloated, moving it off might even make things better for this part. The thing I'm afraid is that this sync could be important for Multiplayer and you don't want each client have different away from what the main network is sending out, ... but what the heck am I saying, ME Logics should and probably is running exclusively on the dedicated server host, not the clients.



So we'll see how deep the devs are willing to go to split what is splittable, it's a big deal for engine programmers though and not something that will happen overnight, they should really hire some extra IMO because the longterm benefits are worth it especially when there's features coming that will need that extra performance headroom, but also new CPUs are going crazy on amount of cores.



Once this is all done, the Main thread may look less and less like the main one in terms of CPU Time, that's right, but you're still suppose to call it the Main one IMO, because in technical detail it is still the one holding the process together, the engine, being the parent and responsible and stuff So here we are, after quite a large gap between the last time I was doing diags with DCS, I kinda put that deep one on hold because I didn't want to jump to any conclusions, with a program a lot more detailed and sophisticated than the one in this post. I wanted to get in to the bottom of what's responsible for what, but I realized the task is quite a bit more complicated.With quite some new people around since then, who aren't up to speed with the DCS performance stuff, I decided to make this overview of the DCS threads behavior, with some general observations and not trying to go into definitive detail, it should do the trick for most people to get a sense of the status of "Multi-Core Support" (there's no such thing :p)Initially in this post, I got into a small A-10C mission, and because I've seen different behavior with different modules, maps, types of units on ground and/or air, radars, EWR, the types of actions you're performing (TGP usage, types of weapons), and so on, it is very important to always associate any such DCS performance stats with athat was used when recording.This case obviously did not have any missle AA units, scanners, radars, EWR, there was no ECM usage, no countermeasures, and I didn't even have GBU's to drop due to mission's default loadout choice.The way you read/interpret this PerfMon graph is, .... well I'm not really sure as I used it more lightly, which is weird as It's me who should have figured out this before posting, but this is all niche stuff with scarse info on the web, the point is it's good enough to get you an idea how things may be under the hood.It probably safe to say that 1 thread isn't taking, at some point over 80% of the whole CPU which is a Quad-Core, that wouldn't make any sense, so that wouldn't be correct interpretation.One thread can only do 25% on a Quad-Core with HT disabled, so 100% of all cores divided by 4 is 25%. A lot of statistical data is lost when viewing total CPU usage of a process, that's why using Task Manager is not recommended, it's hardly reality at that point, most system administrators and other IT specialists within the industry know that Task Manager is garbage and it's solely made for the average user to get a rough idea of the system's health and status.So interpreting this graph in a sense that when you look at a color, a thread, the scale on the left being how much this thread was taking in regards to the CPU core it was on. So Blue Thread was taking 80% of CPU Time on CPU2.That is the correct idea and does apply in general how it works, but still take this specific graph with a grain of salt.(There's so many threads that one got the same color, I forgot to change one of the reds to a different color)So we got mostly Blue, Two Reds and Cyan, and spikes from a Yellow. So everything more than 4 is going to start filling into the same 4 CPUs we have in this case, but keep in mind that we're not maxing out the other 2-3 cores, there's still space for many threads with light load to fit into the 3 threads.Now you can see where this case would get a potential slowdown, or a stutter, you can see in the middle roughly there's a spike of agraph, and that's a 5-th color, that would mean we got one thread here that has to fit in those 4 Cores and potentially be forced-in taking away headroom from the rest, we might be bottlenecking, well not necessairly ...... Remember we still have headroom, that Yellow spike shouldn't affect the main thread, it's not big enoguh, it could fit in any of the cores that still have headroom, but what we can observe suspiciously is that there is a correlation with the main thread dropping down at the same time the Yellow thread is spiking, that's not what we would like to see if we want optimization, it could be a coincidence or a bug/issue/design in the DCS engine it self.Having to pause some type of workload on the main thread while a different workload on a different thread is taking place is extinguishing the point of multi-threading.Such cases, if true, would give us a false sense of optimization and multi-threading, if we were to just look at raw numbers, eye balling the data, looking at number of threads and their activities. So with all diagnostics and analysis it's very important to look at many different types of views and take things seriously, interpreting the diagnostic data correctly is very important and the last thing I would like to do here is to create some panic among the community by wrongly interpreting the data, or giving false sense of performance. What I can say for certain it's definitely better than what people think it is.Back to our little example here, ... if the main thread has to stop for another thread to work then what's the point of that workload being on another thread right! But we still have to take all the factors and conditions into account, this graph is a replay of a 30 minute A-10C flying around doing stuff that you can see in the video, but also AIs doing it's own thing, it can get very complicated of who's responsible for what spike/event on the graph, you can separate with specific scenarios designed for that, but in general gameplay it's all molded together. I've did many such things in the past and boy the graphs get interesting when you destroy 500 units and there's a huge load of smoke, those are specific case examples which are complicated and I didn't want to post those on purpose because it's hard to explain all the tiny circumstances, without doing a live take video where you would see, unfortunately the other programs are extremely resource intensive and wouldn't be able to run both DCS and them at the same time, recording diagnostic data is already a resource hog by it self, but Windows Performance Toolkit isn't even designed for real-time operation anyway. So that's why it's key to not jump to conclusions, and the deeper you dig it actually presents more questions and more places to dig further, then it's obvious that everything's extremely relative to the scenario. People that report stuttering versus other people that don't may just be using different modules with different maps playing a different style of gameplay and using different weapons and systems.So when looking at diagnostic data and graphs always keep a sharp lookout on the whole situation. Again, this scenario was not using any radars, any ships, and missle AA batteries, all the data is relative to this scenario in the video below.There may be other benefits of putting things on a different thread but still being kinda tied to the main operation from various development point of views that I do not know about, but maybe that's a dud, I don't know, but generally something happening on another thread should not be stopping other things and getting in-the-way, that said this is still looking back at DCS as most of the core improvements haven't come, it's a look back at the last decade, so if anyone's reading this years down the line, there's no reason to be concerned, it's looking far better as it is, we all thought it's only running on 1 thread, well nope, so things are just going to get better from here on out.There's no reason to concern overall, there's no cases where it's super bad, overall DCS is hard on the system yes, but I didn't see any hard bugs or indications of them, except the A10C_Protect.dll and FC3_Protect.dll seem to be kinda legacy things chugging along there taking a bit of CPU regularly, even tho if you're not flying A10C or any FC3 aircraft in a session, again those may just be names with other modules using it, while I'm speculating other newer modules seem to have their protection performance hit probably tied with DCS_protect.dll which isn't that big alone, let's just hope it's not stopping the main thread when it's taking CPU time.A basic overview of the threads with approximated descriptions:The basic case scenario video:Also, please note that the thread Start Addresses may not be 100% accurate/valid of what we're assuming the workload is, because it can be convoluted in code as things may be called from one place but work being done elsewhere, spread across differently below the calling piece of code. So don't take it as it's written in stone, althought it does make sense with the rest of the data for the most part, we know from the devs that IO is split from the main thread, we know from the devs that Audio is split from the main thread, at least that's what I'm aware of.Just keep in mind that there could be other types of work on that thread apart from what it's Start Address may seem to tell us about, so an audio thread may not be exclusively doing just audio work, for example.(More opinion style!!!)We'll see a major change in these stats when Vulkan API comes, unless some big new feature comes that takes up all that gained performance, but I doubt it, I believe all the driver overhead and draw calls run on the main thread and that's a big thing Vulkan based GFX engine will deal with.The next big thing to alleviate off the Main ("simulation") thread is the AI if we are to believe the basic thread info, it has to be spun off the main thread ASAP, I don't see how AI logic per-unit would need to be on the main thread where physics calculations happen. There's proably loads of serial workloads that don't have anything to do with each other math-wise. While you fire a missile at an AI1 airplane and that missile calculations have to go on and it's taking a lot of CPU time on Thread1, what's keeping some distant AI3 from running it's separate AI logic on Thread2 for landing or whatever, right right!?!? So there's a lot of potential to split away these serial task into many threads. Each unit on the battlefield or rather squad could probably run not only in it's own thread, but it's own set of threads (by type) if you ask me.Mission Editor logic and triggers, waypoints, etc, maybe don't have to run on the main thread either (I could be wrong a bit if there's a thing to do with events/detection, but conditions nah) So if the ME logic, a condition, checks for IF THIS UNIT IN ZONE A, it doesn't have to be on the main thread, sync with physics shouldn't be needed because it may fire what at most 100 ms later than it would otherwise, that's probably nothing in practice, and DCS matches with complex ME logic will probably have 1000x of factors so the chances of this discrepancy, but then again I'm talking worst case scenario. What I'm talking about is like, you entered zone A by a few meters, but because ME logic was late it did't register, but then you got hit by missile and then you happen to fail the mission. Chances of this being a problem? Then again, because the Main thread is bloated, moving it off might even make things better for this part. The thing I'm afraid is that this sync could be important for Multiplayer and you don't want each client have different away from what the main network is sending out, ... but what the heck am I saying, ME Logics should and probably is running exclusively on the dedicated server host, not the clients.So we'll see how deep the devs are willing to go to split what is splittable, it's a big deal for engine programmers though and not something that will happen overnight, they should really hire some extra IMO because the longterm benefits are worth it especially when there's features coming that will need that extra performance headroom, but also new CPUs are going crazy on amount of cores.Once this is all done, the Main thread may look less and less like the main one in terms of CPU Time, that's right, but you're still suppose to call it the Main one IMO, because in technical detail it is still the one holding the process together, the engine, being the parent and responsible and stuff __________________

Temporairly off for PC Maintenance - DCS not installed - Specs will change

Specs: Win10 x64 1607/14393, 1440p@75"32 - CPU: Intel Core i7 3820 @ 4.0GHz - GPU: Radeon RX 480 8GB - RAM: 32 GB, DCS SSD Samsung 860 EVO 250GB, Saitek Cyborg X/FLY5 joystick.

Modules: A-10C, F/A-18C, Mig-21Bis, M-2000C, AJS-37, WW2AP, CA, FC3, SuperCarrier.

Terrains: NTTR, Normandy, Persian Gulf, Syria Last edited by Worrazen; 02-12-2020 at 11:34 AM . Reason: typos