April showers have turned into May floods in Texas! We’ve been hit with severe weather and major flooding as you may have seen in the news. Thankfully our studio has been mostly spared from this destruction and we’ve been moving ahead as usual. We’ve been testing new builds of Star Marine and hard at work on the Persistent Universe and many technical activities. Here are some detailed reports from the head of each team.

Persistent Universe Team

Art

The PU Art team has had its hands in a lot of areas of the project this month, as usual. One of our main focuses has been moving the Nyx>Delamar>Levski landing zone into Final Art stage. We’re passed Greybox stage now and the environment is looking amazing. Art Director Mark Skelton, Lead Artist Patrick Thomas, and Global Environment Lead Ian Leyland have been working closely with BHVR to make this environment truly something to behold. VFX artist Lee Amarakoon and Lighting artist Marc Toscano have been busy adding that extra touch of fog, smoke, and lighting to breathe life into this environment. We can’t wait to see you guys participating in shady back-alley dealings, selling things on the black market, and traversing the winding asteroid tunnels that this environment has to offer.

Another major focus has been optimizing the Stanton>ArcCorp>Area18 landing zone for better performance in preparation for the Social Module release. The extremely high level of detail and fidelity that makes our environments so amazing are pretty taxing on performance, but with new tech in place from our Graphics team we are able to greatly improve our performance through various means.

Concept artist Ken Fairclough finished up a concept of a security turret prop that will eventually be found on landing zones like ArcCorp. These security turrets will be there to punish players who draw their weapons in areas that don’t allow them. They will also act as a deterrent for trolls and griefers who think it’s a great idea to open fire on random civilians.

Our Character Team has been wrapping up support on the FPS characters. The Marines and Outlaws are at a point where we have bid them adieu and now our attention has shifted to creating the characters that will be seen playing SATABall in Astro Arena. We’ve got some pretty slick new concepts we’ve been working from and should be wrapping up in the next couple of weeks. Our character team has also been doing some R&D on Hair and Swappable Clothing for the upcoming Social Module release.

The Austin Animation Team has been supporting various areas of the project as well. We’ve had our hands full retargeting old PU, FPS, and Arena Commander animations to our new skeleton. We’ve also been providing support for improving the G-Force animations and testing out the Grabby Hands system. Senior Animator David Peng has been focusing 100% of his attention on updating cockpit and gunner templates to match the character’s new proportions. We now have 7 cockpit types and 20 button combinations for our artists to choose from when creating cockpits.

Design

Much of Design’s time this month was spent on fleshing out some of our other occupations that players can eventually choose to adopt in the PU. Tony Zurovec spent much of his time working out the ins and outs of the Pioneer (formerly exploration) occupation and how it relates to the functionality of an upcoming ship. Rob Reininger completed a first pass of how the Mercenary/Escort occupation would work, while Nate Blaisdell and Evan Manning tackled the Bounty Hunter and Smuggler occupations, respectively. These designs are now sitting with Tony Z for review and once approved will be moved over to the queue with Andrew Nguyen for gameplay implementation.

Tony Z also finished his first draft of the new Universe Simulator (formerly the Economic Simulator) and forwarded it to the team over at Wyrmbyte. The new design incorporates much more than just the economy system that will be in the PU, now including things like the ability to create elementals and composites, the ability to create a planet within a solar system and assign it data, and the ability to create occupations for NPC’s and specify execution logic for each occupation.

Mark Skelton and Tony Z, along with Lead Writer David Haddock, met with BHVR to go over designs and layouts for upcoming planetside locations. Specifically this month our attention has been on Magnus>Borea>Odyssa and Helios>Tangaroa>Mariana. Both of these locations will offer several new exciting opportunities and diversity to our already impressive stable of landing zones. We’ll share some exciting look development with you guys as it comes online. Next month we will turn our attention to the other landing zones within the Stanton System: Crusader, Hurston, and MicroTech.

Engineering

The month of May was a very busy month for the Engineering Team here in Austin and for the Wyrmbyte Team. A lot of big ticket items were completed and ready for initial testing.

One big item that everyone has been waiting for is our new Generic Instance Manager, which promises to bring notable improvements to systems such as matchmaking and party. This was architected by Jason Ely, and he brought it home with strong support from Tom Sawyer. We will run initial testing on this new system, which is intended to be a part of the upcoming FPS release.

Meanwhile…over at Wyrmbyte…Scott Brown, Nathan Gray and Ryan Seabury have been working on our Universe Simulator and are about to deliver a proof-of-concept iteration with base functionality in. The next step is to continue to expand on this until we can create a working demo to share internally for further discussion on how to further build upon it. At the same time, Ian Guthrie is working on the first iteration of our Solar System Server, and will be wrapping up an initial iteration of that soon. The Wyrmbyte Team has also delivered a rewrite of our Player Info Server and our Presence Server in May and are continuing to work on their iPredictor ships and missiles movement system.

Our other engineers have been busy, too…Tom Davies has finished v1.0 of our Useable Editor for our NPC AI needs, and Jeff Uriarte is making solid progress on the Character Archetype Editor, which is another tool for our design team. Andrew Nguyen has been in R&D Land prepping for the initial prototyping of another occupation (tentatively called the “Discover” occupation), and he has just wrapped up an initial prototype for the Mining occupation.

Brian Mazza has been deep in crucial work needed by the team at large, such as implementing Google Brakepad for better crash reports, porting our Universe backend, and fixing server bugs that were blocking team progress and preventing build stability. Across the hall from Brian, James Wright has been working closely with QA on profiling and analyzing performance on our various builds, including a recent PTU build and our pending FPS build. This data is helping us determine problem areas that need attention in order to improve overall performance. And James has recruited the aid of Clive Johnson (who is close to wrapping up his work on implementing our Unique Global Entity ID system) and George Kidd from the UK Team to help investigate some of the critical performance issues uncovered in his profiling results.

On the tools side of things, Benjamin Bechtel has rolled out a new improved version of his Asset Validation Tool with an all new user interface. This tool is used daily by our artists. Benjamin is also awaiting a visit in June from UK tools engineer Ashly Canning to work together on our Dataforge Tool. Sean Tracy and Jeffery Zhu have rolled out the first phase of their stream restructuring, which will greatly help streamline our stream flows in getting our development streams all the way out to our live product. Lastly, these two gentlemen have spent a great deal of time and effort supporting Cort Soest on our much needed and super long task of cleaning up our Perforce data. The first two phases of this task have wrapped up in May.

The team is anticipating June and looking forward to offering their support on the upcoming FPS release, as well as continued work towards our upcoming Social Module.

Live Operations

QA

For the QA team, May began with testing and releasing 1.1.2 and subsequently 1.1.3.

With help from our Senior Engine Programmer James Wright, we conducted a specialized PTU test that utilized a separate server CPU thread to calculate physics. The results showed significant performance improvements during matches.

Changes have been made to how we utilize different streams and branches with Perforce, our versioning control software. This will help with a much more structured and smooth release and feature integration process. This has also resulted in an increased workload for QA to ensure these additional streams or branches of the game and editor are properly tested. However the team is doing a great job rising to the occasion to ensure this additional testing requirement is met.

Our FPS specialists have been very busy ensuring Star Marine is properly tested each day. The team was fortunate enough to be able to provide feedback directly to design director Todd Papy. Todd was very responsive to each point raised. After discussing these points, Todd tasked up and prioritized any agreed upon changes. As QA, it is very fulfilling to be able to be a part of the direction of development in such a way.

Meanwhile, Todd Raffray has been making sure the Social Module is properly tested. Todd has been doing a great job ensuring all features are documented into a comprehensive checklist and continually tested as they come online.

Congratulations to Miles Lee for transitioning into an official Associate Engineer role with our DevOps team! Miles has done a great job helping to create a framework for test automation. We can now capture performance data and automate the majority of our stability checks for each build as soon as they are available.

The team filmed their first episode of a special QA segment with Community where they showcase some of their favorite bugs.

The team was also fortunate enough to attend the University of Texas Denius-Sams Gaming Academy’s launch party of their first official release called “The Calm Before”. We had a lot of fun playing the game! Their team is very talented. Feel free to check out their free download!

It has been a busy month for us in QA. Next month we will be intently focus testing Alpha 1.2 in preparation for its imminent release.

Game Support

This month was quite busy for Game Support as we continued to build upon some of the work done in March and April.

The month of May also saw the publish of two Arena Commander updates (1.1.2 and 1.1.3). Update 1.1.2 greatly helped players with the dreaded Match Not Found, Kicked Back to Lobby, and Infinite Load Screen issues, and 1.1.3 assisted with the persistent “rubber banding” issues that saw players jump around the map. This is only the start of our optimizations, and we know you’ll be excited to see how we continue to use your information to help improve the game experience.

On that note, we also ran an incredibly successful playtest on PTU using 1.1.2. Players crammed into the PTU service to help us stress test and gather important server analytics, and along the way we had an absolutely EPIC 8v8 round of matches, which has set off a round of discussions and assignments internally on what we can do to start increasing the player cap.

In terms of continuing to build and scale our support operations for you, one of the bigger items this month was the creation of an internal workflow and knowledge base that greatly expands our ability to help you. By internally publishing a full matrix of possible problems, troubleshooting methods, and resolutions, we’ve been able to train more people in the company to help with technical issues. This means that we can help a greater number of people in a shorter amount of time, and provides a fantastic foundation for growing our team as we head into Star Marine, Social Module, Squadron 42, and ultimately the Persistent Universe.

We’ve also begun to work with Turbulent who is driving two big initiatives for Game Support: The Community Bug Council (a MUCH improved way for players to submit and see what bugs are active in the system) and a new revamped version of the Live Service Notification page, which will include a much-desired Server Status page.

You’ll see more about both of these in the coming weeks, but both of these are alignment with our larger goals of getting you more accurate information as quickly as possible about the state of the service.

IT/Operations

May has been a month of speed for IT. The IT group works hard to keep up with the expanding infrastructure needs of the development teams and performance of central storage systems at each studio continues to be one of our priorities. Last month we focused on internal build replication to studios from the build system in Austin. This month we’ve shifted our focus to the performance of the underlying hardware in the Austin build system itself. The goal of this project is to significantly reduce the time it takes to generate builds. Reducing build times speeds up development and internal testing but the entire build pipeline is large and complex so we have been approaching this from several angles.

First on the list has been the performance of the storage the build systems run on. The build system compiles code and assets for the game of course and most of this work really pushes the disks hard. I/O wait times can go through the roof on normal systems and since our build system is compiling many builds in parallel we saw a great deal of time being lost just waiting for disks to catch up. To solve this problem we built a custom server with all flash storage using special controllers to push our IOPS well beyond our current needs. This one improvement to our build system has shown impressive results with a 66% reduction in build times.

Next we plan to move more secondary systems over to fast storage which should result in additional reductions. Finally, we’ll incorporate parallel processing methods to bring more cpu cores in to the build process. This step plus some potential resource caching should bring our build times down from hours to minutes.

We’ve also been working closely with DevOps to improve our build automation systems which will add stability and usability to our current system. We’re hoping to roll out the rest of these changes over the next few weeks. For now, we’re very pleased with the early results and I think everyone is anxious to see how fast we can make these builds go now.

Dev Ops

We released two patches this month, Patch 1.1.2 and 1.1.3, and neither were on a Friday! Our team will continue to work with the company to attempt to corral these deployments to Tuesdays as industry standard, and then hopefully, move into a continuous deployment pipeline over the course of a few years which should eliminate the need for downtime (a lot of this will depend on our final database design).

We have been assisting in the deployment playtests of the FPS and using those opportunities to test our new launcher and patcher. So far the tests have been positive, but we have several more polish passes to go through before we move it to PTU for everyone to use. The Launcher development will be broken into three phases. The first, a PTU phase has already been mentioned. This takes the existing UI and combines it with our new patcher. The second phase will incorporate a complete re-write of the launcher into C++. Combined with the new patcher and the old UI, it will have stability improvements but will look and function much like the current launcher, and the upcoming PTU launcher. The third phase will be a complete new UI on top of the C++ core and patcher. These phases will take several months to roll out, and more information about each will be coming up soon.

Finally, a ton of work has been going on how data in the company is created, stored, and consumed. We are looking at how we use branches in Perforce, how data is copied to the build server, what data is excluded, how development leads look at that data. Initiatives on better Perforce tools, an Exclusion tool, and a new build server have been additionally occupying the teams time. The build server is an especially large project, that we are aiming to have the core phase done by July 14th. To that end, one of the engineers in the Frankfurt studio has flown out to Austin to work with us over the next three weeks so we can collaborate on implementation ideas.

Work on finalizing our database model and containerizing the servers has had to be put on hold while we finish our tools that handle all the data used by the studios. We will be returning to these topics when our tools currently in development are finished, and when our server team moves onto working on the Persistence Service that will be the main interface layer to the database. Hopefully, this work will be included in the June report!