Hi everyone!

April showers have blessed Central Texas this month and we’ve been hard at work on many fronts. We’ve seen record concurrency numbers with the 1.1.1 game update, and experienced multiplayer connectivity issues as a result. We mobilized all resources to focus on this issue, and you’ll see more detail on these activities from several teams in the detailed reports. We’ve got a lot to share, so please check out some detailed info from the teams in the Austin studio!

Persistent Universe Team

Art

The month of April saw many developments within the Art Department of CIG Austin. We hosted an Environment Mini-Summit where our friends from Foundry 42, BHVR, and Illfonic came to visit to discuss long term planning for Star Citizen Environments. There are a lot of cross-studio dependencies that require face-to-face time to discuss in more depth, and it is very valuable to interact with people in person that we work with on a daily basis across the world. All in all the Summit was a success, and our Global Environment Leads Ian Leyland and Cort Soest have helped establish a firm foundation upon which to build the best environments possible. Our focus specifically for the PU this year will be fleshing out the “First 5” planetary landing zones, wrapping up our first batch of shops that will be found on those landing zones, and R&D on creating modular space stations that will serve a variety of functions in the PU.

Our character team has been solely focused on supporting the FPS character revisions this month. There was a lot of work to do to help bring these characters up to speed with the rest of the amazing work Illfonic has been doing to make Star Marine look the best it can. We’ve added armor plating to help beef up the Medium Marine a bit more, tweaked materials and colors to help both the Marines and Outlaws look like they belong to the same unit, and Josh Coons even jumped over from the Ship Team for a while to help bring the Outlaw helmets up to the next level. These characters are looking fantastic, and we’re really excited for you guys to get to run around with them very SOON!

Other things the Art Team has been up to this month include concept work for Hangar weapon racks and Holo-Kiosks for the Astro Armada shop, concept work on shielding and radiators for space stations, explorations into Blue Collar clothing sets, flickering job board VFX for the TDD shop, and texturing work for the mobiGlas asset.

Our Animation Team has been supporting various aspects of the project, including prone weapon sets for FPS, and Grabby Hands animations for testing out picking up and putting down objects. Lead Animator Bryan Brewer has been testing and debugging the brand new skeleton we are going to be moving our characters over to in order to improve various aspects of the character’s proportions and range of motion. Our entire animation team is now transitioning over to helping retarget our whole library of animations (over 2,000!) to this new skeleton.

Design

There were many discussions and meetings that took place this month amongst our designers in order to break down what we are calling the “Game Loop” for our upcoming Social Module release. Where does a player start when he/she logs in? What do they experience as they participate in various activities around a landing zone? What happens when they interact with REDACTED??? All very important questions that ultimately helped shape the design for Shopping, Storage, NPC setup, and more. We’ll continue to break these down even further as we progress into next month.

Pete Mackay has been adding extensive developments to his newly dubbed “Ship Cruncher” tool (formerly Thruster Calculator). It’s a pretty robust tool that allows designers to balance discrete item parameters, create baseline ship items for each item type, properly size ship gear and visualize granular changes on a macro scale, and collect robust data sets for each ship. The Ship Cruncher also does several different physics calculations to determine how fast a ship can change direction, its top speed, its turn radius, etc. It’s an all-inclusive package that will significantly help with ship balancing in the future.

Evan Manning has been adding data to the Star Map to be used by our friends at Turbulent for the release of the Star Map on the Platform side. There is a lot of information to be documented and Evan has been doing a great job of ensuring every element in each system is in its proper place.

Work continues on fleshing out various occupations within the Persistent Universe. Tony Zurovec continues to whittle away at the details of the Pioneer occupation, and Rob Reininger and Nate Blaisdell have taken cracks at initial passes on the Mercenary/Escort and Bounty Hunter occupations, respectively. Before long we’ll have more detailed breakdowns of how each of these occupations will work in the PU.

Engineering

April was a busy month for our engineering team, as many of them were recruited to support the live patch that went out the first week of the month.

Our Server Team paused their work on the Multiplayer Hangars feature in order to work closely with DevOps and QA to improve and strengthen our Instance Manager, our Matchmaking Service and other services. As a result, we have had a dramatic decrease in reports of issues related to inability to load into multiplayer matches. Congrats to our hardworking team for this splendid accomplishment!

We have other, longer term systems in the works that continue to progress closer towards completion. These systems are being worked on out of Austin with strong cross-over with Manchester and Denver (by our friends at Wyrmbyte)…such as: Unique Global Entity ID system, Generic Instance Manager, Player Info Server, Presence Server, Universe Simulator, HUB Server, and Player Persistence. Much of this work is slated to have deliverables of first iterations in the month of May…the team is definitely making some solid progress!

The team has also been working with tools such as RAD Telemetry to help them profile our server metrics, in order to support the ongoing task of optimizing server performance. They have started profiling our live builds and analyzing the results with fine toothed combs to identify areas to focus on to implement improvements.

Work also continues on our various tools systems, including support for such items as the Sandbox Editor, Trackview and our Asset Validation Tool. And development continues to progress quickly on our various AI Editors…such as our Useable Editor and our Character Archetype Editor…which are needed by our design team to help bring our worlds to life.

And while juggling all of these tasks the team has also been busy working on such things as finalizing 64-bit networking support for our Large World system, finalizing work on the iPredictor (movement prediction) system, creating a series of prototypes for the Mining Occupation, and helping support other systems that are live or in the works, such as REC and Multi-Seat Ships.

Finally, behind the scenes our team has been hard at work re-optimizing our development stream workflow, which will help us streamline our development to ultimately be more efficient and stable. Everyone is looking forward to this new workflow, which has been rolling out step-by-step over the past several weeks.

Well, that’s it in a nutshell for the month on April, 2015! We’ll be back with more next month!

Live Operations

QA

For the month of April QA has been busy on multiple fronts. With the release of 1.1.1, our concurrency rose to a level we had not seen before. This exposed aspects of our back-end infrastructure that were preventing successful connections to multiplayer matches. QA worked closely with DevOps and our server engineers in a full investigation. After multiple hot-fixes to our servers, the issues were dramatically improved. Subsequently a call was made for everyone to help participate in a scheduled stress test. The turnout was such that we were able to surpass our previous concurrency record and were happy to see our back-end system could handle this increased load.

Melissa Estrada has been assisting our developers in the Frankfurt studio to track down and regress editor issues they’ve encountered when doing comparison testing of our version of the editor to pure CryEngine SDK. Melissa also headed up an initiative to log all errors found in the game logs which will be a huge help in ensuring we have a cleaner game for the release of Star Marine. The team assisted our Global Environmental Tech Lead Cort Soest to filter unused maps in the editor to reduce internal build space by more than 100GB helping to enhance work efficiency.

Our checklists have been updated to include more Star Marine and Squadron 42 features as well as deeper checks to the editor. With help from Jeffrey Pease, efforts have been made to improve the patch notes creation process which help to more easily incorporate multiple members of the QA team that can assist in the event that someone is absent.

Miles Lee has made significant progress in developing test automation. As a result, we are able to automate the capturing of performance data for each of our various builds. He is also working on automating our build sanity checks and assisting DevOps with the development of a headless client system that will enable us to automate stress tests on our back-end infrastructure.

The team was also continually testing Star Marine. FPS specialists Tyler Witkin and his UK counterpart Glenn Kneal ensured production was notified of all issues found that day and highlighted any severe outstanding issues that were blocking or significantly hindering testing.

Our QA team was also fortunate enough to visit Professor Paul Toprac and his aspiring game developers participating in the Capstone GAMMA program at the University of Texas. We had a lot of fun playtesting and providing feedback on their projects.

Next month we plan to incorporate Social Module in our regular testing. We will be working very closely with Jake Ross to ensure we begin testing each aspect of the Module as soon as it is available.

Game Support

The month of April was quite busy for everyone, including Game Support!

We’re continuing to learn and refine our processes of communicating and working with players, particularly on the Live Service Notification forums and on Reddit. We’ve seen nothing but appreciation from players for these efforts, which means we’re doing something right! Our goal of serving live players with information is carried out by giving you the most accurate and updated information on the health of the service, what kind of game experience you can expect as a player, and what we’re all doing to make it better (especially on patch days!).

Behind the scenes, we’ve made some changes that will ultimately result in you getting a better, faster response. We’re closely working with Customer Service, Community, and DevOps on a daily basis to make sure that you get the right timely support. We’ve got a good plan and we’re excited to start getting to where you would want us to be as a service provider.

On a more visible level, we’ve also begun the work of running and managing stress tests, and we’ll be involved in many more of these. The rollout of 1.1.1 and increased concurrency resulted in multiplayer instability, and we worked with multiple teams to identify and address the more common issues. To properly test these fixes, we coordinated with the player base in what turned out to be an incredibly successful test which set new concurrency records for Star Citizen. Not only was this helpful, but it was fun!

Lastly, we’ve spent some time internally on creating some plans on what we’re going to need to support future modules, namely the upcoming Star Marine module, which for us basically means supporting an entirely new game! We’ll be excited to roll these plans out in the coming weeks and months as we expand our services to better support the growing community and ever-expanding needs of the Star Citizen universe.

IT/Operations

The IT team has been hard at work improving internal systems, planning for future systems, growing storage, and fine tuning service performance. The term explosive growth is often used when talking about storage and in our case, we’re no different. As our dev teams continue to produce content and collaborate, we need to make the results of their work available to all teams within the company. As storage needs grow in LA for example, we must grow storage in all other studio locations to match so that data can be replicated locally to each team.

Growth of storage space is not the only aspect we’ve been focused on this month. Storage performance can also become an issue when lots of people need access to the same things at the same time. We’ve already surpassed the capability of standard replication methods and found it necessary to build our own in order to keep up with demand. I’m proud to say that the IT team has come up with some very creative and innovative strategies to make better use of existing systems and believe me, we push hardware to the limits here at Cloud Imperium Games.

For example, one of our replication systems pushes about 900 mb/s for about 12 hours each day pushing content to 5 replication targets simultaneously. It is important for this data to get to each studio as quickly as possible so the performance demand on the servers is pretty high. To solve this problem we’ve built a custom flash storage system in in order to provide the IOPS we need for this process. Each upgrade seems to expose new areas we can improve on. In this example we got our replication going so fast that it began to bog down servers on the receiving end so we’ve been upgrading those servers where necessary as well.

While the IT team has been continuing to provide support to individual developers around the world it seems like storage and replication tuning have been our main focus this month. Next month we’re looking forward to building even better systems to help the dev teams crank out content even faster.

Dev Ops

April was a month of firefighting and stability improvements! We launched patch 1.1.1 and worked to fix stability issues with our Instance Manager, Party, and Matchmaking services. After two weeks of deploying hotfixes from our Server Engineering team, stability increased. Due to some of the stability issues we have had over the last few patch releases, CIG has decided to change our approach to Operational Deployments. What this means to the players is that they should not expect Friday patch releases, or releases late at night. Instead we will be aiming for a more tempered roll out of patches and hotfixes to our live environment, taking full advantage of our PTU environment for config and game testing. We hope this will lead to more stable releases for our players.

The DevOps team has also been hard at work building the base infrastructure for the live environment, PTU, and internal test environments. Work on the new launcher and patcher continues, and we am sure you all will be as excited to use it as we are. We will be demo’ing it very soon with an upcoming PTU test and getting feedback and usability testing on it. Work on our automated dynamic infrastructure scaling system continues, around 40% of this system is already deployed live, and we expect the rest to begin operating over the next month. Internally, we continue to re-build our build server, change how we merge code and data between working branches, build better crash reporting, and work on automated testing.

There is still a lot of work to do to rework our internal data pipeline to get it to the point that both CIG developers and players do not have to patch and move around massive amounts of data each time there are minor changes. The DevOps team is working on a large roadmap to fix this with help from our engineering and tech art teams in Austin and Germany while being supported by our IT staff on the hardware build out. A full completion of this plan will take many months to roll out, but should drastically reduce the time it takes for a game build to be generated, the time for developers to test their changes, the size of patches, and the amount of data developers need to grab each morning. More on this plan in the months to come.