skeldark Profile Joined April 2010 Germany 2223 Posts Last Edited: 2012-08-15 02:47:09 #1

Sc2 Ladder-Balance-Data

If you want to help to get more data:

Beside this little balance data, this program can do his main task: Show your real MMR!





This is the diffrence to the average MMR, of my Ladder-Data, per Race. Not more not Less.



You can not see on any statistic game-data, if the reason is game design or social aspects.

Not you, not me, not blizzard, not a single game designer!

So we have the choice of paying for a global sociology study to find it out (if you can call it this way in sociology ^^)

or just ASSUME it comes from game design like every game company does.





The data is biased towards EU/US and towards higher skill-rate.

+16/-16 MMR is the average for a single win/loose on ladder.

Result:

Difference to average MMR:



MMR Filter: 2000 MMR+ ( above master)

TIME Filter: 1 Jul 2012 00:00:00 GMT - 31 Jul 2012 23:59:59 GMT

T: -15.77

Z: -0.77

P: 12.23



MMR Filter: 2000 MMR+ ( above master)

TIME Filter: 1 Aug 2012 00:00:00 GMT - 11 Aug 2012 11:47:54 GMT

T: -15.24

Z: -7.24

P: 17.76



MMR Filter: No

TIME Filter: 1 Jul 2012 00:00:00 GMT - 31 Jul 2012 23:59:59 GMT

T: -45.24

Z: 28.76

P: 6.76



MMR Filter: No

TIME Filter: 1 Aug 2012 00:00:00 GMT - 11 Aug 2012 11:47:54 GMT

T: -46.82

Z: 23.18

P: 14.18









Old/First Results + Show Spoiler +

Source Main Data

+ Show Spoiler +

- The data is biased towards EU/US and towards higher skill-rate.



Gamescount: 125976

Sc2-Accounts: 45203



-worst to best player: 3200 MMR

-one average win/loose on Ladder: +16 / -16 MMR



TIME Filter: 13 Jun 2012 02:34:54 GMT - 15 Jul 2012 16:05:41 GMT

Games Left: 109028

MMR Filter: Above Master

Games Left: 19688





Average MMR per Race

+ Show Spoiler +

TIME Filter: 13 Jun 2012 02:34:54 GMT - 15 Jul 2012 16:05:41 GMT

Race account count: 15814

Data average MMR: 1539.46

Difference to average MMR per Race:

T-P: -62.14

T-Z: -117.03

P-Z: -54.89





TIME Filter: 13 Jun 2012 02:34:54 GMT - 15 Jul 2012 16:05:41 GMT

MMR Filter: Above Master

Race account count: 2840

Data average MMR: 2265.5

Difference to average MMR per Race:

T-P: -35.0

T-Z: -21.0

P-Z: 14.0









Win-ratio per Race over Game-Time

+ Show Spoiler +









Work: + Show Spoiler +



Preamble

+ Show Spoiler +

I thought long about if i should publish this data or not because i know the balance mentality on tl / reddit / sc2-forums.

I fight against balance whine on this forum since i joined it and even created an



However i had the data to calculate objective balance data and the balance threads will pop up so or so.

I thought long about if i should publish this data or not because i know the balance mentality on tl / reddit / sc2-forums.I fight against balance whine on this forum since i joined it and even created an script to ignore balance whiners on tl.However i had the data to calculate objective balance data and the balance threads will pop up so or so.



Work + Show Spoiler +

Some month ago, i created an



To find out more about mmr and functions behind it, i included an upload function, that uploads gamedata on our server.

"Not that" and me used this data to find out more about the MMR. We were able to solve the most secrets and calculate it very accurate.

One day decided to uploaded the race value too, without thinking much about it.

Than i realised that with race value and MMR and a lot of data, i can calculate the average MMR for an race.

This post is about such an calculation.



-I took all users and opponents and calculated their MMR.

-After that, i created a list of bnet accounts with their last MMR and race.

-Over this list i took the average of each race!





This steps are very easy to understand but everything else than easy to calculate.

It took us over 3 month and hundreds work hours to calculate accurate MMR.

Some month ago, i created an program to calculate the Hidden match making rating.To find out more about mmr and functions behind it, i included an upload function, that uploads gamedata on our server."Not that" and me used this data to find out more about the MMR. We were able to solve the most secrets and calculate it very accurate.One day decided to uploaded the race value too, without thinking much about it.Than i realised that with race value and MMR and a lot of data, i can calculate the average MMR for an race.This post is about such an calculation.-I took all users and opponents and calculated their MMR.-After that, i created a list of bnet accounts with their last MMR and race.-Over this list i took the average of each race!This steps are very easy to understand but everything else than easy to calculate.It took us over 3 month and hundreds work hours to calculate accurate MMR.



Proof of Concept

+ Show Spoiler +







1) Why average race mmr = data balance



We saw many ways to calculate balance in the past. Some can be indicators some are total useless.

Win/loose statistic of pro players was often used because they are easy accessible and can indicate balance problems.

However, this data dont take into account how strong the players are depending to each other.

The Skillfunction behind MMR is invented to do exactly this.



Arguments like "Race x players are stronger than race y players" are invalid because if this is the case we can call this already imbalanced.



2) Why mistakes in the MMR calculation don't affect the result or affect it



First: the accuracy of my mmr calculation is very good. But i can be wrong in some points or for some users.

However nothing in the calculation takes the race into account.



Theoretical it can be that my mmr calculation is race biased without even knowing the race.

However at the moment i dont see any indicator for that. But i will watch it closely



3) Why race populations dont change the result.

Because i take the average. This point is obvious but i better point it out.



4) Statistic independence

if you take the average of the race you must make sure that you don't have and depending factors in the data.

So what can be such a factor:



1) race/skill distribution of the user-group of my program is not representing battlenet user group



There is no reason why 1. should be true.

Also the data includes to 96% the opponents of my users and only to 4% of the users himself.

So the data have a random user base.

so we can exclude point 1.



2) Skill-range of my users is not skill-range of the battlenet



The users of my program have a way higher average skill than the bnet usergroup.

Also the opponent is allways in the range of the user.

So point 2 is true!

We have to remember that our result is not valid for the hole ladder, its only valid for our skillgroup.

Diamand, Master and Grandmaster are overrepresented in my usergroup.

This means this data show the balance on higher skillevels!



5) prove of small deviation and significance



lolcanoe make a nice analyses of the data. thanks for that!



On July 13 2012 07:41 lolcanoe wrote:

US DATA ONLY



Terran Average MMR, STD

1559.214909, 546.131097



Protoss Average MMR, STD

1620.764863, 509.5809733



Zerg Average MMR, STD

1672.129547, 495.3121321



TWO SAMPLE T-TEST RESULTS



T-Stat, T vs Z

T-Stat = -5.693

P = .0000001386



T-Stat, P vs Z

T-Stat : -2.872

P = 0.00472



T-Stat, T vs P

Tstat = -3.03

p = .00238



Histogram of T MMR for normality check:







Anderson-Darling Test for Normality (T only)



With a p slightly greater than .05, we cannot reject normality of the data. However, the weakness of this statistic indicates that normality should be scrutinized in the interpretation.



Assumptions

- MMR is an independent, fair indicator of skill.

- MMR is approximately normal.

- There is no sampling bias between races, however there is a sampling bias towards higher average skill.

- Cause-effect cannot be established by this test.



With over 99% confidence, we can reject the null hypothesis that the averages are equal in all 3 matchups. This is not surprising given the quantity of data, in addition to a maximum 7% difference between T and Z in average MMR.



The data for T appears approximately normal, but the study does not conclusively show that MMR is normal. Terran Average MMR, STD1559.214909, 546.131097Protoss Average MMR, STD1620.764863, 509.5809733Zerg Average MMR, STD1672.129547, 495.3121321T-Stat = -5.693P = .0000001386T-Stat : -2.872P = 0.00472Tstat = -3.03p = .00238Histogram of T MMR for normality check:Anderson-Darling Test for Normality (T only)With a p slightly greater than .05, we cannot reject normality of the data. However, the weakness of this statistic indicates that normality should be scrutinized in the interpretation.- MMR is an independent, fair indicator of skill.- MMR is approximately normal.- There is no sampling bias between races, however there is a sampling bias towards higher average skill.- Cause-effect cannot be established by this test.With over 99% confidence, we can reject the null hypothesis that the averages are equal in all 3 matchups. This is not surprising given the quantity of data, in addition to a maximum 7% difference between T and Z in average MMR.The data for T appears approximately normal, but the study does not conclusively show that MMR is normal.





6) Because some people have a problem understanding this:



-I calculate the unbalance of skill not the reasons for this unbalance!

-I calculate the average skill of an race not the general popularity of an race



7) Data:

Datafile



We saw many ways to calculate balance in the past. Some can be indicators some are total useless.Win/loose statistic of pro players was often used because they are easy accessible and can indicate balance problems.However, this data dont take into account how strong the players are depending to each other.The Skillfunction behind MMR is invented to do exactly this.Arguments like "Race x players are stronger than race y players" are invalid because if this is the case we can call this already imbalanced.First: the accuracy of my mmr calculation is very good. But i can be wrong in some points or for some users.However nothing in the calculation takes the race into account.Theoretical it can be that my mmr calculation is race biased without even knowing the race.However at the moment i dont see any indicator for that. But i will watch it closelyBecause i take the average. This point is obvious but i better point it out.if you take the average of the race you must make sure that you don't have and depending factors in the data.So what can be such a factor:1) race/skill distribution of the user-group of my program is not representing battlenet user groupThere is no reason why 1. should be true.Also the data includes to 96% the opponents of my users and only to 4% of the users himself.So the data have a random user base.so we can exclude point 1.2) Skill-range of my users is not skill-range of the battlenetThe users of my program have a way higher average skill than the bnet usergroup.Also the opponent is allways in the range of the user.So point 2 is true!We have to remember that our result is not valid for the hole ladder, its only valid for our skillgroup.Diamand, Master and Grandmaster are overrepresented in my usergroup.This means this data show the balance on higher skillevels!lolcanoe make a nice analyses of the data. thanks for that!6) Because some people have a problem understanding this:-I calculate the unbalance of skill not the reasons for this unbalance!-I calculate the average skill of an race not the general popularity of an race7) Data:



Concluding word

+ Show Spoiler +

Please keep in mind that the imbalance result is very small and there will be never perfect balance.

You only improve, in the long term, by ignoring the balance.

Your race can be underpowered today and overpowered tomorrow.



Also, the users make most of the balance, not the game designer.

But this is a different topic....







MMR distribution by races.

+ Show Spoiler + On July 11 2012 14:39 Not_That wrote:



Click for full version.





Amount of players:

2014 Zerg

1784 Protoss

1516 Terran





Same graph normalized, each bar representing the percentage of players of each race in the bin:







The server does matter as MMR is non comparable cross servers. I've decided to remove KR and SEA and keep EU and NA as they are closest to each other in terms of MMRs, and that's where most of our data comes from. Click for full version.Amount of players:2014 Zerg1784 Protoss1516 TerranSame graph normalized, each bar representing the percentage of players of each race in the bin:The server does matter as MMR is non comparable cross servers. I've decided to remove KR and SEA and keep EU and NA as they are closest to each other in terms of MMRs, and that's where most of our data comes from.





README before writing a long post why you think that is no scientific statistic prove.

+ Show Spoiler +

This is not an university paper about sc2 balance

I dont get money for this.

I dont personal care which race is op or not



I publish the data i collected with my own program that i wrote to back calculate mmr.

I found a very interesting anomalies in the race data.

So i published the result here.



If you want to do a more complex test or analyse with the data.

Feel free to do so!

datafile



If you read the text careful, i think will agree that this is not perfect but a way better method

than tldp win-ratios or random tournament results.



This is not an university paper about sc2 balanceI dont get money for this.I dont personal care which race is op or notI publish the data i collected with my own program that i wrote to back calculate mmr.I found a very interesting anomalies in the race data.So i published the result here.If you want to do a more complex test or analyse with the data.If you read the text careful, i think will agree that this is not perfect but a way better methodthan tldp win-ratios or random tournament results.

before writing a long post why you think that is no scientific statistic prove.



If you want to discuss the method and the significant of the data , first read my op and the analyses in it

+ the analyses and discussion of other people in this thread.

I did the best i can and willing to do to prove the significant of the data.

If you misunderstand the result this is not my problem.



GL & HF

Skeletor If you want to help to get more data: MMR-Stats Beside this little balance data, this program can do his main task:So we have the choice of paying for a global sociology study to find it out (if you can call it this way in sociology ^^)or just ASSUME it comes from game design like every game company does.The data is biased towards EU/US and towards higher skill-rate.+16/-16 MMR is the average for a single win/loose on ladder.Difference to average MMR:MMR Filter: 2000 MMR+ ( above master)TIME Filter: 1 Jul 2012 00:00:00 GMT - 31 Jul 2012 23:59:59 GMTT: -15.77Z: -0.77P: 12.23MMR Filter: 2000 MMR+ ( above master)TIME Filter: 1 Aug 2012 00:00:00 GMT - 11 Aug 2012 11:47:54 GMTT: -15.24Z: -7.24P: 17.76MMR Filter: NoTIME Filter: 1 Jul 2012 00:00:00 GMT - 31 Jul 2012 23:59:59 GMTT: -45.24Z: 28.76P: 6.76MMR Filter: NoTIME Filter: 1 Aug 2012 00:00:00 GMT - 11 Aug 2012 11:47:54 GMTT: -46.82Z: 23.18P: 14.18If you want to discuss the method and the significant of the data , first read my op and the analyses in it+ the analyses and discussion of other people in this thread.I did the best i can and willing to do to prove the significant of the data.If you misunderstand the result this is not my problem.GL & HFSkeletor Save gaming: kill esport