Around the Verse: Connecting the Universe Written Thursday 4th of May 2017 at 07:13pm by CanadianSyrup, Desmarius and StormyWinters As per usual, anything said during the show is subject to change by CIG and may not always be accurate at the time of posting. Also any mistakes you see that I may have missed, please let me know so I can correct them. Enjoy the show!

TL;DR (Too Long; Didn't Read)

Studio Update Austin Design Moving forward with major features for 3.0 such as commodity trading which has been aided by the L.A. programmers to bring the shopping code rewrite, commodity kiosk and ship persistence to life.

Finished first batch of several usable request for ships.

Work on getting Miles Eckhart fully functional for 3.0 which brings a complex mission giver system that enables reputation gain and more.

Breaking down Levski in order to determine smuggling areas, quest giver locations and where content in general should be placed. Art Josh Coons is working on the damage modelling for the Cutlass black. The ship is nearly finished at this stage and only had LOD’s after to be completed. PU Animation Team Updating all usable animations to use the new usable system which saves on memory and create more unique animations quickly.

Austin also did a mocap shoot to gather animations for picking up crates at different heights and interacting with doors at various transitions. Ship Animation Team Wrapping up work on the Cutlass Black and creating zero g enter and exit animations for the Dragonfly.

Also working on the cockpit experience initiative by adding hit reactions, button presses, and overall cockpit layout. Engineering Team Bring more parts of Diffusion online and integrating it with various services currently being used such as friends, analytics, authentication, and presence.

Next stage is moving to the larger more complex services such as persistence cache, game server management/matchmaker, and persistence database.

The game server and client are also very close to being converted to Diffusion.

Optimizations using a new technique called “Router Biasing” to give more control over bandwidth management. QA Focus on 2.6.3 for the first part of the month.

Work is now focused the game dev branch and testing the PU and Squadron 42. PR Spent time with Turbulent to work on improvements for Spectrum and updating the Evocati list and PTU waves. Turbulent Release of Spectrum version 0.3.3

Nested threads were an addition to this release Planning to transform this into a view option so you can decide how to view a specific threads or all threads Looking into how unread status will work in nested threads and how to improve that as well as dual staff tracking

Now able to flag a post for moderation within a public forum as well as within orgs

Progress on mobile support for keyboard system

Working on 0.3.4 Looking to add some refinement on how tags are used in subforums so you can jump directly from a global community to a specific tag within a channel Second part of this is allowing you to bookmark a tag just like it was a channel itself

Adding more filters and working on the search subsystem This features set from search will power a new mini profile so you can jump directly to posts from a specific player directly from the mini profile

Working on virtual lists which will allow them to render what you see plus a buffer saving on performance and having everybody present in the lobby list They will also use virtual lists to fix the problem in chat lobbies where you can’t jump directly to a message then go back in time they’ll do this by changing the history mechanism for lobbies

Working on an overlay between the game and Spectrum so they can have an overlay that is over the game and your desktop client synchronized

Working on PM groups so you can have more of a party system for specific lobbies

Doing some infrastructure changes by moving the platform to a new set of hardware, this may result in some minor downtime Behind the Scenes Mike Jones has the longest title at CIG and is responsible for all hardware and infrastructure which is all custom designed

The introduction of a tri-build system full of VMs maintain stability though there is huge competition for resources and traffic; more than any other game that Mike Jones has worked upon, thus divide and conquer is the way to go

Three more hosts were added to isolate the tri-build and decrease resource contention in handling these divided up smaller pieces

Services are hosted externally for the flexibility and scalability without large capital investments, hence it's requiring the expansion of our build system

The process of adding an additional enclosure, racking the servers, loading and configuring the software is an interesting one and often requires fastidious cable management, but it's worth gaining room to grow

Full Transcript

Intro With Chris Roberts (CEO, Director of Star Citizen and Squadron 42), Sandi Gardiner (VP of Marketing). Timestamped Link.

Chris Roberts (CR): Hey, uh… we’re rolling, the camera’s rolling

Sandi Gardiner (SG): We’re rolling? Oh. Hello and welcome to Around the Verse, our weekly look at the development of Star Citizen. I’m Sandi Gardiner.

CR: And I’m Chris Roberts. So as you can see we have a nice little gift on our table here. So we had Jeremy visit us from New Zealand with his daughter Chloe and they left us this gift so we’ll open it on camera. Let’s see what happens here… Okay maybe I won’t open it that easily on camera, but impressively enough Chloe actually knew much more about Star Citizen than her father so that’s pretty cool. It boards well for the future of Star Citizen! So let’s have a look. Apparently I believe this is something that is very cool and New Zealandy.

SG: Oooh.

CR: Oh wow look at that. Okay so I’m going take it out.

SG: Exciting.

CR: There we go. Oh boy!

SG: Oooh, look at that.

CR: It is the “Kuro”. A symbol of life, new beginnings, growth, and harmony, support and loving protection.

SG: Exciting.

CR: Thank you very much, that is awesome Jeremy and Chloe.

SG: Who knew.

CR: Brilliant. Okay.

SG: Get that off the table then?

CR: I will get that off the table. Go back to reading your paper if you want.

SG: Nope I’m good.

CR: So we have some great news to share with you today. So later this month Star Citizen will be the featured story in 3D World Magazine. The magazine spoke with a number of our art directors about how they became game artists.



SG: Yes they did so if you’re interested in a career as a game artist, this will be a must read when it comes out on May 17th.

CR: Yeah so Josh our Character Art Director has spent some time updating various alien character designs. You’ve seen recently the Banu and even a sneak peak of the Vanduul this past week and what we’d like to do right now is share you the sort of updated Xi’An sculpt which will be shown in the 3D World Magazine, but giving you guys a sort of heads up. Here you go.

SG: The Xi’An has undergone some changes since we last saw them, pretty cool.

CR: Yes they certainly have. Now let’s go to Austin and Turbulent for their studio updates.

Studio Update With Jake Ross (Producer), Benoit Beausejour (CTO). Timestamped Link.

Jake Ross (JR): Hey guys, Jake Ross here, Producer here at Austin. There have been some exciting developments in the Austin studio this month and we’re excited to share them with you for this week’s Studio Update.

With 2.6.3 out the door we are finally moving forward with our major features for the big 3.0 patch. The Austin design team will be responsible for several things in the upcoming release. First and most notably is that we’ve made progress on adding commodity trading into the game. Since several things need to come together on the tech side for this first, the L.A. programmers have been furiously working on the shopping code rewrite, commodity kiosk and ship persistence so that commodity trading can be possible. Not much to show you guys visually on this just yet, but our next update should be in a great spot to give you a first look.

Second we have been finishing several of our first batch usable requests for our first round of shops. These are created so we can get the four requires disciplines: design, animation, tech art, and system design on the same page about each usables intended purpose and functionality. Once all the required assets are created, it will come back from design to do the final hookup.

Next we’re plotting out our first mission giver experience for 3.0. Getting Miles Eckhart into the game is our primary focus. There are necessary building blocks to make this happen. As you can see in this conversation tree there’s quite a bit that he will be able to do like give a constant stream of missions and enable players to earn reputation and higher tier options. As we’re still in our planning phase the sprint hasn’t quite started just yet, we’ll hopefulyl have more to show in our next update.

Finally we’re in the middle of breaking down the Levski landing zone. This write up details all the things you’ll be able to do in this location. Some of these include how players will smuggle into the city and which content will go into side tunnels. This placement of the mission givers and their content, the factions within Levski and the political aims of these factions. We don’t want to get into too much detail and spoil the end result for you, but we’ve got some awesome things planned for this landing zone and can’t wait for you to see it evolve into the vision we’ve currently got in the works.

On the Art side, Ship Artist, Josh Coons has been busy completing the damage model pass on the Cutlass Black. By far the most time consuming part during this phase is creating the intricate Trellis work on the parts that get blown off like the wings and the body. He worked closely with tech art to make sure the ship breaks apart and receives surface hull damage in the correct areas.

Once damage is completed it also marks the stage in the pipeline where the ship is pretty much ready to go. All that’s left is to make the LOD’s.

The PU animation team is currently updating all the exciting usable animation to use the new robust usable system. This will save us on our memory footprint and allow us to create a large number of unique animations more quickly. We took advantage of our desks getting rearranged to set up our mocap system. We needed to do a quick pickup shoot to capture a few things that has popped up since the last shoot.

We captured picking crates of different sizes from different heights and operating door controls from various other transition animations. We’re also working on picking up two handed objects as part of our looting system. With this functionality we will be able to interact with more objects ingame. This opens the possibility for more missions and activities. If you see a box on the ground and you want to pick it up, you’ll be able to walk up to it, pick it up, carry it back to your ship and put it into your cargo hold for transport back to your hangar.

The Austin ship animation team is wrapping up the reworked version of the drake Cutlass as well as creating new zero g enters and exit animations for the Drake Dragonfly. We’re also in the midst of improving our cockpit experience by adding hit reactions, button presses and updating our cockpit layouts to maximize the overall cockpit experience. More on this to come.

This month has been a busy one for the backend services engineering team as well. We’ve been focused on bringing Diffusion, our new backend services architecture online and ready for primetime. We’re converting some of the more trivial services such as friends, analytics, authentication and presence from legacy architecture to fully Diffusionize services running with our Ooze language. We are moving next into the larger more complex services like persistence cache, game server management and matchmaker, and persistence database. These services will be broken up into smaller micro services to meet our performance, scalability, and availability needs. The game server and client are very close to be Diffusionized as well which will close the communication gap between the backend and the frontend.

In addition we’re making various optimizations to the systems such as a technique we call “Router Biasing”. This allows us to apply advanced bandwidth control bandwidth techniques between services types in the Diffusion network.

Austin QA’s focus over the various early parts of April was testing to get 2.6.3 out to the players. With multiple PTU pushes leading 2.6.3 going live. The team spent a lot of time supporting the live build over the first week of April. Most of our new hires finished their basic training in the month and have all begun testing alongside their mentors fulltime on both Persistent Universe and Squadron 42.

After 2.6.3 went live, Austin QA shifted full time to game dev branch for testing. This has been both to continue work on stabilizing the branch and to begin more rigorous testing in preparation towards 3.0. Some of the items we’ve actively been testing are several new ships, in depth sweeps of some of procedural planet environments for bugs, testing MegaMap improvements, new implementations for player interactions, Item 2.0 conversions, movement system factors and new field of view controls to name a few.

The development teams have been lining up new documentation and data for QA to polish up our test cases to ensure we are ready to jump on new content as it comes online. One of our testers has begun working directly with the Austin animation team to clean up new mocap files for the development teams. Several other team members have also been engaged with in depth testing in new and updated engine tools alongside the Frankfurt QA team ensuring the new tools will function as designed so our designers can better implement and create content.

The player relations team spent a full week with Benoit and the Turbulent team working on Spectrum improvements. They’ve also added a couple new team members and are excited to continue to serve our growing community needs. A particular relevance to 3.0, the player relation team has started adding new Evocati and updating our PTU waves as we know everyone will be excited to test out the new content when it’s available.

That’s all I have for you this month guys. Thanks for watching and for your support, we’ll see you in the verse.

Benoit Beausejour (BB): Hey guys, here’s the April update from Turbulent and Platform team. So this month we’ve released Spectrum version 0.3.3 which contains a new type of discussion called nested threads which is basically when you reply to another reply it will go into a nested format. So you can see the discussion trail out from specific reply and the first level of these replies are sorted by default by votes, so most upvoted subtrees get upvoted to the top. This is very Reddit style type of discussion so when you start a new discussion on Spectrum 0.3.3 and up, you can decide which type of discussion you want to start.

Whether you want to start a chronological classic thread which goes in order or a nested thread. Now this new type of discussion we hope will allow more dynamic, faster threads and obviously the upvote feature is going to be useful for all the community teams for many of the features that they want to do like gathering questions. Now this project goes so fast that in a month we’ve been able to release 0.3.3 and get feedback on it so we know that not everybody likes nested threads so we plan on trying transform this from a discussion type to a view option. So that you can decide how you want to view a specific thread or all threads, so if you like chronological you’ll be able to view them as such and nested you’ll be able to choose that as such.

So we’re still looking into how we’re going to do that but we’re listening to your feedback and we’re thinking that this might be a more powerful feature, so we’re going to get it done. One other change that we’re looking for also for nested threads is how unread status works, currently it's a bit iffy so you can find your red marker we’re going to see how we can improve that as well as dual staff tracking in there so you can know from the top of the thread list which have had a staff response. This is going to be particularly useful in the Ask a Dev forums.

Also included in 0.3.3 is the ability to flag post for moderation so this also works within your orgs and obviously in the public community where you can flag a reply, a thread or a message for moderation and so public moderators in the case of the public Star Citizen community can come and intervene. In your private orgs then all your officers or people who have moderation permission will receive a notification and can go investigate. That is how we end all the flagging in Spectrum.

Also one more update in 0.3.3 is progress on mobile support for the keyboard system that we use and so it’s way useable than it used to be. It’s still not perfect but at least we’ve done massive progress on there and we’re hoping to get it fixed in the next release for all types of cases. So, look forward to that.

Currently the team is working on next release which is 0.3.4, the main feature we’re looking to add there is some more refinement on how we use tags as subforums and so the we’re going to surface these tags at the top level in the community index in the channel list. So that you can actually jump directly from a global community index to a specific tag within a channel and so the second part of that is that we’re going to allow you guys to bookmark a tag just like if it was a channel in itself. So this will I think give us the functionality of a subforum while keeping the tagging system in place. Which is a very nice way to just make this very dynamic so we’ll see that in 0.3.4.

Also the major undertaking we’re doing is adding way more filters and work on the search subsystem which will basically power ‘view my own posts’, ‘view somebody else’s post’ so search by author, search by role which is also allow you guys to search for dev posts. Then all of this feature set from search will power a new mini profile so you’ll be able to just jump directly to a list of posts from a specific player directly from the mini profile.

One other feature which is more a backend thing that we’re working on is virtual lists. You might have notice we have an issue rendering long presence lists in the chat lobbies so that’s why the backers group is usually collapsed by default in the general forums. So we’re working on a technical solution called virtual list which will allows us just to render what we see plus a buffer and so this will save on performance and allow us to have everybody present in the lobby list anyway. So we’re going to be able to use virtual list for that but also for other features like in the chat lobby, one of the most reported issues that we have is you can’t jump directly to a message and go then back in time, virtual list will allow us to do that because we’re going to change the history mechanism for lobbies to be able to do that. So, we’re working on this in 0.3.4.

In the meantime the research team from Spectrum is working on the overlay which is basically for Spectrum desktop and integration between the game and Spectrum. So this is basically taking our redux application store and moving it to an area where two processes can basically benefit from it and so this way we can have an overlay that is over the game and your desktop client synchronized without having to double the resources for it. So, that’s what the guys gave been working on.

Work has begun as well on PM groups so the ability to basically refactor the current one on one PM system to have more than one person in a group so you can have some kind of party system for specific lobby. So this is what we’re doing on the research side on top of what we’re already doing.

So, I’m just back from Austin with a few discussions with team at CIG over there where we presented and did a new design for some of the art that we’re doing for the site revamp. So that went really well and so we’re hoping, can’t show it just yet, but we’re hoping we’ll be able to showcase some of those changes to you guys before we actually get to do them. One thing that between now and our next monthly report that you’ll see is that we’re going to be doing some pretty major infrastructure changes by moving the platform to a new set of hardware, hopefully this will not cause any disruption but there should be a small downtime since we’re lift and shift everything. That’s going to be one of our major tasks upcoming when I’m done filing this video, which is pretty cool, otherwise we’ll see you guys in the ‘Verse.

Back to Studio With Chris Roberts (CEO, Director of Star Citizen and Squadron 42), Sandi Gardiner (VP of Marketing). Timestamped Link.

CR: So just looking at Eckhart's conversation tree really shows the complexity we’re putting into our mission givers. Building the logic system that drives these interactions is very complex and due to the size and demands of our universe and mission system, but I think when it all comes together it will be pretty cool.

SG: Pretty cool and when it comes to building that universe, there’s a department essential for keeping it all running, take a look.

Behind the Scenes With Mike Jones (IT Director). Timestamped Link.

Mike Jones (MJ): I'm Mike Jones, and I'm Director of Corporate & Publishing Technology for Cloud Imperium Games. And right now I think that's the longest title we have in the company. And it's based because of the merging of our Dev Ops and our IT departments. And so my responsibility for IT covers the hardware side of things and the infrastructure side of things, but the network and the security and the server deployments and all the automation and all the orchestration that goes into that is all built here with our Dev Ops team. So there's not a single piece of that, that is something we got from someone else. So the … you know there are a few services that we make use of that are common place, but for the most part our systems are all custom designed specifically for our needs. And the reason for that of course is this is a very unique game. We have very unique needs. And so one of my main focuses this month has been our builds system and the need for more build resources.

Our current build system is made up of six high density servers with a grand total of 240 Vcores and 2.3 TB of RAM. This is chalked full of build VMs, and we recently introduced the tri-build system which is just as many more build systems that are designed to do a pre-compile when there is a code check-in. This helps maintain the stability of our build system and the builds as they come out. As we continue to grow our system we're seeing more and more traffic coming through and more competition for resources. Part of what's happening is we're dealing with problems that we haven't had to deal with before. On previous game projects I've been on we thought we were doing something big, and it's nothing compared to what we are building here. And so the … just the amount of assets alone, the art, the pre-compressed data that's coming in is a tremendous amount. If I do a transfer of just one version of the build over a ten gig network it can take a half hour to 45 minutes. And this is why we have to break these jobs up. And it's the same with the rest of the game. We break things into pieces so that we can do more in parallel. That's how we get it going fast and that's the only way we can make this work.

Under our current deployment our primary build system consumes six hosts spread across a VMware clustered environment. This configuration allowed us to add tri-build to the existing environment, but as we grew we saw increasing resource contention. This pretty much leads to longer wait times on a tri-build job. By adding three more hosts we're able to completely isolate tri-build from the primary build system, and this is important particularly for the engineers who use tri-build constantly. A lot of people don't really realize why it takes so much more to build these builds inside the build system rather than the kind of compile that we do on our desktop computer, and in the traditional sense when you're compiling on your desktop you're doing one thing. You're bringing down source code and you're making that compilation of a client or a server whatever you happen to be working on, but in the build system we have many needs and we have to do all of it. So, in addition to building both the client and the server I'm building release and profile on multiple branches sometimes as many as three. So, this means that anytime a build come … a build request comes in we're spinning up a large number of virtual machines, and we've found that with the size of this game the build times can really spread out. So we break those jobs into many smaller pieces and this is why we can accordion the build process out into this much more server hardware.

The reason that we choose to host our services externally rather than internally is there are just a myriad of reasons, but it gives a tremendous scalability and flexibility. I can pay for just exactly what I need instead of having to make this large capital investment on a whole bunch of gear and then all the bandwidth so that people could access it, and if you think about the multi-region launch that we did recently we wouldn't be able to as easily do that if we had stuff located and physically stuck in one data center. This leads us to the need to expand our build system, so today we're expanding our current rack of six hosts by three more hosts. We'll be including a total of 72 more virtual cores and a whole lot of networking, and it's going to be really fun.

Since our current enclosure was at capacity we added a second enclosure and wired it in. Ultimately we plan to move all hosts to the first enclosure and all storage to the second enclosure for simplicity, but we have to take at that in phases. We then rack the servers from the bottom to the top in order to leave room for future expansion. Once we got them installed we realized our fiber cables weren't the right lengths, so we moved the servers up a bit to account for that temporarily. Once connected we installed and configured the software for use. We can do that pretty fast now, since that everything is virtualized on our own little cloud environment. From there it's a simple process to migrate the virtual machines over with VMotion. This means we can completely migrate a running system over the network with zero downtime. Each time we do one of these jobs we end up with a mess of cables. It happens all the time, and I feel like I'm constantly fixing it. In fact it's kind of a thing now and everyone likes to see how long it'll take before I find it and fix it, which usually isn't that long. I do this at pretty much at every studio I go to now. Once we had all the parts we were able to get the whole thing up and running in just under three hours. As a result we're now seeing much less contention in the build system and now we've got some room to grow again, and I'm sure we're going to need it.

Outro With Chris Roberts (CEO, Director of Star Citizen and Squadron 42), Sandi Gardiner (VP of Marketing). Timestamped Link.

CR: There you go, that’s pretty cool and as Mike said we’re developing a very unique game with very unique needs which tends to throw unforeseen obstacles at all our departments. While developing a game like this is exciting, it also takes time to overcome some of these obstacles as you guys know.

SG: And one thing that makes us unique is the variety of ships you can fly in the PU. So if you’re a subscriber you can fly the Drake Buccaneer as part of the ship of the month. Also subscribers get the Gladius Valiant as part of a flash sale so if you’re interested in learning more about this, check out the link in the description.

CR: Yup so that’s it for today’s episode. Please join us tomorrow for Happy Hour Friday at 8am Pacific. Jared Huckaby will interview Brian Chambers about everything that’s happening at the Frankfurt studio so that will be a pretty cool episode and after this week. Happy Hour will air bi-weekly so don’t miss tomorrow’s episode.

SG: But before we go I want to thank all the subscribers who make this show possible and also other shows like Happy Hour.

CR: yeah and we’re also grateful to all the backers for supporting the development of Star Citizen and Squadron 42, we wouldn’t be able to make this game without you. So thanks for watching and in the spirit of May the 4th, we thought it would be fun to show you a little glimpse of Mark Hamill in the cockpit of Squadron 42.

SG: Awesome. We’ll see you…

CR/SG: Around the Verse.

CanadianSyrup Director of Transcripts A polite Canadian who takes pride in making other peoples day brighter. He enjoys waffles with Maplesyrup, making delicious puns and striving for perfection in screaming at the T.V. during hockey games. Desmarius Transcriber When he's not pun-ishing his patients or moderating Twitch streams, he's at Relay pun-ishing their patience as a member of the transcription staff. Otherwise, he can be found under a rock somewhere in deep East Texas listening to the dulcet tones of a banjo and pondering the meaning of life. "If you can't do something smart, do something right." - Sheperd Book StormyWinters Director of Fiction Moonlighting as a writer in her spare time StormyWinters combines her passion for the written word and love of science fiction resulting in innumerable works of fiction. As the Director of Fiction, she works with a fantastic team of writers to bring you amazing stories that transport you to new places week after week.

Please enable JavaScript to view the comments powered by Disqus.