CrushDog5 Profile Joined March 2010 Canada 204 Posts Last Edited: 2013-12-06 18:38:59 #1



The data and scientific paper for the SkillCraft project are available online. In case you don’t know, the SkillCraft project is the largest scientific study of expertise ever conducted using StarCraft 2 players. Over three thousand players from bronze leaguers to professionals contributed replays to the project.



The published scientific paper is available to download



The dataset we used to create the paper is available



The dataset would be great to use in a project for a machine learning course, or for just poking around. The dataset consists of twenty variables for each of the 3395 games. The data are anonymized to protect participants’ privacy. There is documentation with the dataset at those links, and more details in the paper itself. If you have questions about either the paper, or the data, please post comments in this thread, and we’ll try to answer them quickly.



If you use the data for a project, or find anything interesting in there, we’d love to hear about it here, (or email us at cognitive-science-lab*AT*sfu.ca)



Our next project involves looking at how individual players develop over time. If you have more than 300 replays, please go to skillcraft.ca and contribute them to the study. Also, we now have ethics approval for participants younger than 16 to contribute. If you know anyone younger (perhaps your children, even) it would be awesome to get those replays, even if there are fewer than 300. There are interesting affects of age on performance and those data would be invaluable to understanding how age influences cognitive motor abilities. We have a paper under review now on some aspects of this. We'll post it when it's reviewed.



Thanks to all those players who have contributed replay files to the project, we couldn’t have done it without you!



The SkillCraft team (skillcraft.ca, Twitter) The data and scientific paper for the SkillCraft project are available online. In case you don’t know, the SkillCraft project is the largest scientific study of expertise ever conducted using StarCraft 2 players. Over three thousand players from bronze leaguers to professionals contributed replays to the project.The published scientific paper is available to download here. One of the reasons we chose the scientific journal PLoS ONE is that they are open access, and thus make the articles available without a paywall.The dataset we used to create the paper is available here. It's also available from the UCI machine learning repository here. The dataset would be great to use in a project for a machine learning course, or for just poking around. The dataset consists of twenty variables for each of the 3395 games. The data are anonymized to protect participants’ privacy. There is documentation with the dataset at those links, and more details in the paper itself. If you have questions about either the paper, or the data, please post comments in this thread, and we’ll try to answer them quickly.If you use the data for a project, or find anything interesting in there, we’d love to hear about it here, (or email us at cognitive-science-lab*AT*sfu.ca)Our next project involves looking at how individual players develop over time. If you have more than 300 replays, please go to skillcraft.ca and contribute them to the study. Also, we now have ethics approval for participants younger than 16 to contribute. If you know anyone younger (perhaps your children, even) it would be awesome to get those replays, even if there are fewer than 300. There are interesting affects of age on performance and those data would be invaluable to understanding how age influences cognitive motor abilities. We have a paper under review now on some aspects of this. We'll post it when it's reviewed.Thanks to all those players who have contributed replay files to the project, we couldn’t have done it without you!The SkillCraft team (skillcraft.ca, Facebook page SkillCraft.com - StarCraft + Science

csl249 Profile Joined April 2013 Canada 1 Post #2 Awesome!

jjthomps Profile Joined August 2011 Canada 9 Posts #3 Always very exciting

tili Profile Joined July 2012 United States 1316 Posts #4 This is really phenomenal!! I am actually very surprised at how important APM is. I thought workers produced would be more important, too...

Varanice Profile Blog Joined June 2011 United States 1513 Posts #5 Im honestly really confused at what this shows. All I really understood was that people at higher ranks tend to play with higher APM which I assumed was generally true. Someone want to explain to me what all this means in laymans terms? www.twitch.tv/varanice

WombaT Profile Blog Joined May 2010 Northern Ireland 14270 Posts #6 Sick, going to have a proper comb over this 'You'll always be the cuddly marsupial of my heart, despite the inherent flaws of your ancestry' - Squat

hansonslee Profile Blog Joined April 2011 United States 2013 Posts #7 Dang, I should spend some time reading this! Seed's # 1 fan!!! #ForVengeance

lailaiwd Profile Joined October 2012 United States 65 Posts #8 I just read the article in most of its length. I think it's quite academic but I suggest that to use primarily figure one to figure out what areas you should focus on as a player. For example, if you are a player in Platium, you should read the Plat-master column and then base your time management on the priorities explained. In Mvp I trust.

CrushDog5 Profile Joined March 2010 Canada 204 Posts #9 On December 07 2013 13:38 lailaiwd wrote:

I just read the article in most of its length. I think it's quite academic but I suggest that to use primarily figure one to figure out what areas you should focus on as a player. For example, if you are a player in Platium, you should read the Plat-master column and then base your time management on the priorities explained.



I think this is a very sensible approach, because it emphasizes the "normal" way skills in SC2 seem to develop. This is not to say that you couldn't emphasize other things earlier and have good results. We'd really need to do some experimental work to see how different training methods (emphasizing hotkeys earlier, for instance) might impact the development of the player as a whole.



Anyone out there got a good training map and what to see how it affects peoples game? I think this is a very sensible approach, because it emphasizes the "normal" way skills in SC2 seem to develop. This is not to say that you couldn't emphasize other things earlier and have good results. We'd really need to do some experimental work to see how different training methods (emphasizing hotkeys earlier, for instance) might impact the development of the player as a whole.Anyone out there got a good training map and what to see how it affects peoples game? SkillCraft.com - StarCraft + Science

CutTheEnemy Profile Joined November 2013 Canada 373 Posts Last Edited: 2014-01-18 08:35:02 #10 I wasn't able to find it: what is the average and std dev of people's apm per league? does it vary considerably based on race?



edit: eps file viewer. Very nasty to find a free program. Why not just bmp files that everyone can easily open? It's also hard to see actual numbers to calculate a mean and std dev. Can we help spread the word and create pressure to get Rob Pardo to replace Browder as head of Sc2? Pardo led the team for broodwar, frozen throne, and wow/BC. We need to make this a thing before LotV development starts. Think about it.

Lings_of_Wiberty Profile Joined September 2011 Germany 8 Posts #11 On January 18 2014 17:08 CutTheEnemy wrote:

I wasn't able to find it: what is the average and std dev of people's apm per league? does it vary considerably based on race?



edit: eps file viewer. Very nasty to find a free program. Why not just bmp files that everyone can easily open? It's also hard to see actual numbers to calculate a mean and std dev.



The authors of the study have uploaded the complete dataset, that is amazing (thanks thanks thanks!) and something you barely see in any publications. I personally think you should not blame them because you don't know how to read the data. I suggest using R, if you are interested I can provide you with the code to get started using their dataset.



I guess here is what you are looking for:







I love boxplots that's why I used them (the fat line is the median, the box includes 50% of people, from 25% quantile to 75% quantile but I also included mean and sd at the bottom).



Race was not include in the dataset, sorry. The authors of the study have uploaded the complete dataset, that is amazing (thanks thanks thanks!) and something you barely see in any publications. I personally think you should not blame them because you don't know how to read the data. I suggest using R, if you are interested I can provide you with the code to get started using their dataset.I guess here is what you are looking for:I love boxplots that's why I used them (the fat line is the median, the box includes 50% of people, from 25% quantile to 75% quantile but I also included mean and sd at the bottom).Race was not include in the dataset, sorry.

Loccstana Profile Blog Joined November 2012 United States 833 Posts #12 Perhaps with this data we come up with an absolute metric of skill that does not rely on MMR. [url]http://i.imgur.com/lw2yN.jpg[/url]

Lings_of_Wiberty Profile Joined September 2011 Germany 8 Posts #13 On January 19 2014 07:15 Loccstana wrote:

Perhaps with this data we come up with an absolute metric of skill that does not rely on MMR.



You could come up with a metric of MECHANICAL skill.



If you combine APM, Action Latency and Assign to Hotkeys you can predict 54% of the variance of what league someone is in (including ANY other measure in the study [including age] you never exceed ~58%...).

I think that's fucking good - especially considering none of them has anything to do with strategy.



But - assuming two players of similar mechanical skill - there will still be variation in strategy, game knowledge/understanding and adequate (or even creative) decision making - and none of the measures used in the study can really capture those... so... yes, based on data like this you can do a damn good job of predicting what leagues someone is in but you won't be able to replace MMR.

You could come up with a metric of MECHANICAL skill.If you combine APM, Action Latency and Assign to Hotkeys you can predict 54% of the variance of what league someone is in (including ANY other measure in the study [including age] you never exceed ~58%...).I think that's fucking good - especially considering none of them has anything to do with strategy.But - assuming two players of similar mechanical skill - there will still be variation in strategy, game knowledge/understanding and adequate (or even creative) decision making - and none of the measures used in the study can really capture those... so... yes, based on data like this you can do a damn good job of predicting what leagues someone is in but you won't be able to replace MMR.

Lings_of_Wiberty Profile Joined September 2011 Germany 8 Posts #14



As seen in my previous post, there is an interesting "APM" spike for Pros.







This spike in APM for pros seems to be based a lot on "Select by Hotkeys" (the correlation between "Select by Hotkeys" and "APM" increases from 0.56 in Bronze to 0.88 in Pros).







The pattern for Action Latency is impressively linear. Even though APM increases prediction accuracy at the highest level, Action Latency never loses its value.







Worker Production kind of sucks, knowing when to stop building workers could be a relevant skill as well I think the following graphs will help interested readers in understanding the content of the study on another level (assuming you have read it and have some kind of grasp on what the measures mean)... a level that is not dependent on a thorough understanding of the statistical methods used in the study but still allows to understand the basis for the conclusions drawn... and possibly even a quarter-decent informed opinion concerning some of the conclusions...As seen in my previous post, there is an interesting "APM" spike for Pros.This spike in APM for pros seems to be based a lot on "Select by Hotkeys" (the correlation between "Select by Hotkeys" and "APM" increases from 0.56 in Bronze to 0.88 in Pros).The pattern for Action Latency is impressively linear. Even though APM increases prediction accuracy at the highest level, Action Latency never loses its value.Worker Production kind of sucks, knowing when to stop building workers could be a relevant skill as well

CutTheEnemy Profile Joined November 2013 Canada 373 Posts #15 thanks for the diagrams Can we help spread the word and create pressure to get Rob Pardo to replace Browder as head of Sc2? Pardo led the team for broodwar, frozen throne, and wow/BC. We need to make this a thing before LotV development starts. Think about it.