Around the Verse: Super Hornet & Multi-Region Servers Written Thursday 16th of February 2017 at 04:21pm by CanadianSyrup, Nehkara and The latest episode of ATV is here! Check out the next part of the ship pipeline. As per usual, anything said during the show is subject to change by CIG and may not always be accurate at the time of posting. Also any mistakes you see that I may have missed, please let me know so I can correct them. Enjoy the show!

TL;DR (Too Long; Didn't Read)

Misc Information Chris is back from the European studios

Monthly Report came out last week.

First ever Valentines Sale is on right now including a free fly event.

2.6.1 will feature the launch of Spectrum. Studio Update 200 staff at Foundry 42 UK

Shield generator, storage console, and mining module images

Damage states for the Dragonfly

Map of the Stanton system

Asteroid, nebula, satellite, and space station images Ship Shape: Hornet to Super Hornet The original Hornet paid homage to the Hornet from Wing Commander which was a light fighter you started out in.

At first the Hornet was to be a light fighter, but then after designs came back with it being heavily armed and beafy, Chris Roberts like it so much that it became the standard medium fighter for the UEE. The first Hornet came out in 2806 and replaced the Aegis Avenger.

The goal has always for a player to feel as if the ship and everything about is it real, down to the buttons on cockpit and Chris feels this is was a big draw for a lot of people to the project.

With the help of CGbot studios, Jim Martin, Ryan Churches, and Dave Haddock they created the demo shown at GDC and Kickstarter.

The first pass of the Hornet was in 2013 for the Hanger Module by Chris Smith. The next was in 2014 with updating it to PBR 2016 involved a complete upgrade to the Hornet and variants to bring them to the current standards The new Super Hornet will be in patch 2.6.1.

The old damage system was hand crafted and had states of 25, 50, 75, and 100. The new shader system dynamically applies the artists designs across any part of the ship as its destroyed, making it much easier to create damage models that look and feel realistic and varied. Behind the Scenes: Multi-Region Servers Game and platform run on public cloud, as has almost always been the case

Evaluated different public clouds to determine best option short term and long term

Huge overhead with running your own servers, cloud allows for lower cost and much greater flexibility

Cloud also allows you to have servers close to your players rather than centrally located

Three main types of virtual machines (hub, core, and game)

Current model allows for many hubs, many game servers, but singleton core services.

Core services currently being revamped and redesigned to become stateless and easy to scale to allow them to cope with any load they get.

Original build machine was a single computer which produced 1 build per day, now CIG uses an internal cloud as a build system that can scale based on need.

Ahmed and his team worked on multi-region servers through part of December, January, and early February

Happy that they have a feature done that can be seen and enjoyed by the player base directly

One database for all servers in the world, means you carry your game over when switching regions

Current servers in North Virginia (US), Frankfurt (Europe), and Sydney (Australia)

Players will be sent to closest load balancer, then closest hub server, then will be marked with a region ID and assigned to a game server. This can then be changed if the player wants.

They did the work in 2 months but the previous 18 months of work was crucial to getting it done so quickly.

Planning for failures and getting the network back up quickly

All networks are laid out and they are ready to expand to all 14 or 15 regions from the current 3.

Looking towards the long term and possibly having as many as 10,000 VMs reporting at the same time. Planning for that scale.

Automation systems are done and they can scale with high reliability.

Full Transcript

Intro With Chris Roberts (CEO, Director of Star Citizen and Squadron 42), Sandi Gardiner (VP of Marketing). Timestamped Link.

Sandi Gardiner (SG): Hello and welcome to Around the Verse, our weekly in depth look at the development of Star Citizen, I’m Sandi Gardiner and joining us back in the studio today is Game Director, Chris Roberts. Did you have a productive time overseas?

Chris Roberts (CR): Yes I did, but I’m glad to be back and the weather is a little nicer here, but it was a great trip. I spent some time with both the U.K. and Frankfurt teams and pushed ahead on some of the new systems coming online as we continue towards the release of Alpha 3.0 and Squadron 42. These systems are going to go a long way to support Star Citizens vast map size and huge array of player options. We’ll be showing off a few highlights when we check in with the studios during the up and coming weekly ATV updates in the next few months.

SG: Now you did mention the best part from last week's newsletter that you got to judge a bakeoff.

CR: Yeah [Laughs]. It was actually quite good. I had a big sugar rush, but it was the very first Foundry 42 cake bakeoff. The entries we got were spectacular, I’m not sure I got that much work done later in the afternoon, but well done to everyone there. I didn’t know we had that much baking talent in our U.K. office.

SG: There you go. The January monthly report was posted to the website last Friday and has a ton of information about all the progress that was made on Star Citizen and Squadron 42 heading into the release of Alpha 2.6.1 patch.

CR: Yup and thanks to all the Evocati, subscribers, and backers who have been testing the new Mega Map and multi-regional servers. The patch is expected to go live in the next few days so watch out for that.

SG: With the new patch out we wanted to give everyone an opportunity to get some new ships as well so we’re having our first ever Valentine's Day Sale right now.

SG/CR: Awe.

SG: Sweet.

CR: So on top of that we’re having a Valentine's freefly event through Sunday where you can check out one of our multi-crew ships, the Constellation which is sort of the grand daddy of the multi-crew ships in the game. So make sure to tell your love ones and visit the site for more details.

SG: Along with 2.6.1 we’re also launching our new chat and forum service Spectrum to the main site for everyone.

CR: Yeah this new service is bringing some impressive communication features which should make it even easier for hackers and devs alike to share ideas and interact. I’m really excited by Spectrum.

SG: Now it’s time for the weekly studio update so let's check in with the U.K. office to find out the latest developments from the Foundry 42 team.

Studio Update With Phil Meller (Lead Designer), Mici Oliver (QA Tester). Timestamped Link.

Phil Meller (PM): Hello and welcome back to Foundry 42 in the UK for our studio update. I am lead designer Phil Meller…

Mici Oliver (MO): and I’m QA tester Mici Oliver.

PM: Our team here in Wilmslow continues to grow and we now have over 200 staff here and while the team comprises nearly every aspect that goes into making Star Citizen and Squadron 42, here are some of the highlights of what the team has been working on lately.

MO: Our props team have a few components and ship parts to show us including a shield generator, a storage console, and a mining module.

[Shows off a Yorm medium-sized shield generator, looking finished or close to it; a storage console in whitebox; and a mining module rather fall along in the process.]

PM: So, our VFX team have been having fun blowing up ships again. Some people get all the fun.

[Shows off Dragonfly damage states.]

PM: And another example of the amount of detail that goes into our level environments is this in-fiction magazine cover from our UI team.

[Shows off Hitbox magazine cover featuring Star Marine.]

MO: As the game expands, and becomes more complicated, our design team uses some awesome internal tools such as this map of the Stanton system. We thought you might like it, so let’s take a look.

[Shows off Stanton system focusing in initially on Microtech before zooming out to see the entire system.]

PM: The artists are always adding detail and character to all areas such as asteroids, nebulas, modular satellites, and stations.

[Shows off intricately created asteroid, nebula, a variety of large satellites, and an artist creating a modular space station.]

MO: So there’s a look at some of the things that we’ve been working on.

PM: And now, back over to your hosts

Back to Studio With Chris Roberts (CEO, Director of Star Citizen and Squadron 42), Sandi Gardiner (VP of Marketing). Timestamped Link.

CR: Thanks for the update Phil and Mici! Fun to see that map sneak peek in there and some of the progress happening as we expand out the Stanton system for 3.0. There’s going to be so many things, and so many little areas, and so much content. It’s going to be pretty cool.

SG: And that magazine cover was a great example of the lengths our artists and narrative team go to, to expand out the immersiveness of the game.

CR: Yeah, especially with the new camera modes allowing players to zoom in on the environment and take in all of the little details. I’m always amazed by all of the great shots that I see online that people post of their play sessions.

SG: Coming up now, we have another edition of our ongoing series of Ship Shape.

CR: Yes, this week we’re going to take a comprehensive look at one of the ships that started it all - Anvil Aerospace’s Super Hornet. And, for 2.6.1, we have a new version of it which partly we’ll show in this little piece.

Ship Shape: Hornet to Super Hornet With Chris Roberts (CEO, Director of Star Citizen and Squadron 42), Chris Smith (Lead Vehicle Artist), Mark Abent (Gameplay Programmer), Ben Lesnick (Director of Community Engagement and Content Strategy), Kirk Tome (Lead Technical Designer), Dave Haddock (Lead Writer), Patrick Salerno (Associate Technical Artist). Timestamped Link.

Ben Lesnick (BL): If there was a single signature Star Citizen ship it would have to be the Anvil Hornet. The Hornets name sake was the F-36 Hornet from the original Wing Commander.

CR: For me when I started to do Star Citizen, obviously all my past games loom large and one of the inspirations for Star Citizen and Squadron 42 was my Wing Commander series as well as Privateer and Freelancer and so I felt like, “Well let's start with an iconic fighter”, and one of the most iconic fighters in Wing Commander which was the light fighter that you started in. The naming of the Hornet and quite a few other Wing Commander ships were always based on historical precedent because I took a lot from World War 2 or present day aerospace and military facts and so the Hornet was a light fighter in Wing Commander and so when I started on the Star Citizen I was like, “okay we need a ship, I need a ship to build, but I can fly around and I can sort of test out all the different things in terms of the flight model and how the look of it it’s going to be and how tactile, how visceral it’s going to feel.” So I thought, “Well let's come up with a fighter craft and call it the Hornet” as a nod to what I did in Wing Commander and a nod to historical ships whether it’s the F-18 Hornet today or back in World War 2 Days.

So that was essentially the inspiration. Originally I had given a brief to Rob McKinnon who is a Concept Artist that designed the Hornet and has worked on a whole bunch of stuff in Star Citizen. You see a lot of his designs ultimately in the game and I told them, “Well it should be a light fighter” and he would come back with these really cool designs, but he sort of gunned it up. He put this big ball turret on the top, and guns on the front with a cannon turret and it was like, super heavily armed, it really wasn’t a light fighter, but I liked the design so much that I was like, “Okay you know what, it doesn’t need to be a light fighter, this could be a medium or heavy fighter and we’ll call it the Hornet and it’ll be the base standard fighter for the UEE navy” which was the fiction that me and Dave Haddock who’d come up for the universe who you would fight on the side of in Squadron 42.

Dave Haddock (DH): The idea was that it was originally released in 2806. It was basically a replacement frontline kind of carrier dogfighter. The idea was the Avenger used to be their premiere dogfighter and then the Hornet came out and blew them out of the water and became the ubiquitous of the UEE navy.

CR: Once I had the design, I handed over to design with that, Ryan Churches design of the Bengal Carrier and also of the Vanduul Scythe which was done by Jim Martin. They were all handed CGbot when I was doing my early tech demo and prototype of what would eventually be carrying what I showed and the little movie we put together that hannes put this cinematic, five minute cinematic together and we headed to CGbot because I didn’t really have a team, it was a sort of loose affiliation of people helping me put this demo together and Sergio Rossi who used to work for me at Digital Anvil as an Art Director and Origins Art Director had a art company called CGbot which is mostly based in Monterrey, Mexico and had some people in Austin, Texas and he’s like. “Oh I’ve got some excess capacity and I’d love to help you out” and so we negotiated a very nice rate that he gave me because obviously this was very speculative and I said, “Well okay here’s the things i Need to build” and I gave him the Vanduul Scythe, I gave him the Hornet, I gave him the Bengal Carrier and they also did characters so they did the pilot character that you saw in the opening cinematic that you would have also seen in the cockpit of the Hornet and they also modelled an early version of the Vanduul.

They built those and then once I got those assets I, myself put them into the engine and was extending the engine to handle assets of that detail and fidelity with the articulation and animation that I wanted to see in the game because for me this was about proof of concept and visualising the feel, the tactical sense that I wanted the world of Star Citizen and the game itself to have and I think it’s one of the strengths we have is that our ships feel very real, they feel like you walk around them and you feel like you get into the seat and all the switches and all the bits work and they feel proper. It’s not just a cockpit that’s put in front of you, it’s not just an exterior view of your ship that the insides aren’t really figured out and so the Hornet was really the first one that did that and sort of showed what that could feel like and I think that was one of the big reasons why people responded to it.

Chris Smith (CS): The time first when I got the Hornet in my lap was when we decided to do like the first visual upgrade pass on it. It was already about two years at that time and then so the decision was made, we had to upgrade it a little bit. What I mainly did was take the landing gear components and stuff like that and uprezzed those. I didn’t do much on the mainship at that point, like the main hull, the wings, and all those things. The main features weren’t upgraded at that point, maybe the materials and the attachments and stuff like that.

Mark Abent (MA): Back then we didn’t have a lot of the technology that we have now. We just had some asset that an artist made and we just somehow had to make it work. So we had Chris Smith’s beautiful Hornet and then we had our hacked up versions over here. So we take his stuff, put it in, get it working how it want it to because we were still prototyping stuff so any one of us would be in that CGA mashing stuff together and to get what we needed it to be so we could go, “Alright Chris Smith, this is how we want it, take the beautiful Hornet and rearrange the hierarchy to how this should be”. So one day we come in and we have guns attached and then the next day we change the whole hierarchy and it would stop working. Then the following day we’ve got the parts breaking off and then we would have to redo parts of the item port system so we could get the hierarchy chain just there.

So then maybe next week we get the damage working and the hierarchy working so you could blow parts off, but now you can’t shoot. So it was kind of like that iterative process and then once… it took about a month to get everything there, it was a little bit buggy, but it was like, “Hey this is working, this is pretty cool”.

CR: Obviously after we had shown it at the 2012 sort demo, prototype, the Hangar Module was the first time that people actually were up close, get to see it. Now we worked pretty hard to get all the base ships like the Aurora, 300i and obviously the Hornet into the Hangar Module which we debuted at Gamescom back at 2013 which was a pretty cool moment and then not long after that we debuted a commercial for the Hornet which Hannes had put together which was still one our cooler commercials and it’s… A Hornet pilot being chased by various ships, an Aurora, an 300i, and then he comes back home and he revealed that he was bringing ice cream back for his Significant other.

It was really fun and then we sort of had this idea and I think this was one of the very early cases of coming up with idea of variants where we would have specialty versions of the ships. So there was a Tracker, or a Ghost and the Ghost was a stealth version and the Tracker was the AWACS version. Then the Super Hornet, that was the version of the F-18 Super Hornet that you have today where it’s a two seater, you’ve got a rico in the back. Maybe that person controls the gimballed turret or could control missile targeting and that’s certainly is going to become a much cooler role as Item system 2.0 comes out on the newer stuff, but that was the next update of the Hornet and Chris Smith sort of had taken the base Hornet and was working in these new variants that we had done designs and sketches on. CGbot helped out a little bit to add parts of it like the dome on the Tracker and stuff like that. So we updated it for 2013 and the Hanger Module and subsequent to that we started to roll out PBR: Physically Based Rendering which is a whole different, which is what the engine uses now and has been for a couple years, but it’s a way that the materials look much better, much more realistic, reaction to light. So metal really reacts like metal, Wood reacts like wood, concrete reacts like concrete.

So Chris had then took the Hornet and did a pass to bring up the materials to the PBR level and it stayed at that point until recently when he took a pass to take all our Hornets starting with the F7-C and then the Tracker and the Ghost and now with 2.6.1 it’ll be the Super Hornet to our new modeling with custom normals and poly mapping and UV2s and our special damage shader that we have.

CS: After that first upgrade pass it should pretty much remain unchanged aside from the PBR upgrade when we got that online and then it was decided that we do like another version of the ship, like the new 2018 BMW, it was supposed to be the next generation ship which was the F7-A and lucky for me I was tasked with doing that and that was great, it was fun since I had been involved with the Hornet for so long it made sense. I knew a lot about the ship already, I was very familiar with it.

So yeah I took on the F7-A and I sort of designed it up from the ground up and I kind of decided all of the things that bothered me on the original Hornet I sort of wanted to fix on this new one so that’s what I did.

So after the F7-A, the Mark Two was complete and I moved right onto upgrading the Mark One yet again, but this time it was a full upgrade. I wasn’t just upgrading pieces here and there, it was sort of rebuilding the whole ship with smart normals and the proper material PBR values and everything and so it could hold up with all the other ships that we’ve been coming out with for the last year and a half.

Since I had done the upgrade to the base Mark One Hornet, they all share parts so the change actually affects all the variants automatically which is great, it saves us time because it reuses the wing and the cockpit and everything like that. All that was needed was some material tweaks and colours and stuff like that.

After that, it was the Super Hornet and the Super Hornet was a little bit of a special case because A, it’s a two seater and B al ot of the parts aren’t directly shared with the original Hornet. There’s some tweaks and additions to the Super Hornet that the original base Hornet doesn’t have.

Kirk Tome (KT): We felt the Hornet line was due for an update and so we wanted to bring the Hornets up to the current standard of our latest batch of ships. We wanted to improve them both visually and technically. That did involve things like streaming the design shape. Taking a look at rebalancing the ships and changing their weapon loadout for example, their missile payload and also figuring out whether the current setup for the interior of the cockpit was going to be useful for our future multi-crew and item manipulation system that we’re applying in the near future.

We’re updating the damage tech on the ships so that they use our current UV2 damage so that they’ll blow up really nicely.

Patrick Salerno (PS): The old damage system used to have 25, 75, 50, 100 states. You would shoot the ship, after a certain amount of health, does a model swap, 100%, blows off. It was very cool damage, but it was very labour intense. We’d actually have to do a lot of hand modelling, each piece would end up looking very similar after awhile.

MA: If you picture my hand as the Hornet, a horribly drawn Hornet. Like each of these fingers would be parts and we should be able to shoot them and blow them off. In theory it sounds like an easy peasy thing, but we had it much more complicated than that. You could blow up this part of the thumb, followed by this part. So I could shoot here and it would take the whole thing away or shoot here and take the whole thing away.

PS: When the new shader came along, basically now we just blow off huge chunks. So wing, wing debris and when I’m taking a ship and I’m blowing it apart, I’m saying to myself, “which pieces detach from where and I’m setting up the proxies, I’m setting up the shields, I’m making sure things still animate and detach”. If there’s a turret or gun under the wing I have to make sure the helpers are attached to that piece of geo so it goes flying off with the debris and the destruction.

MA: It’s basically applying this streamlined texture around the whole ship and when you shoot it, you get these nice glow effects and decals that happen anywhere on the ship. With that we have this like underneath skelton carriage where if you shoot it so much, you’ll start seeing the bare bones skeleton of the ship.

PS: So now that I have pieces detaching, I have basic shader damage, I also need to go in and make sure that there are effects setup. Those basically represent a bunch of little cubes in that and they’re floating around and it kind of seems like C4 charges right , they explode and then you get the UV2 damage. So there’s burns and there’s wear on all the pieces that are exploded and melting and once those detach, it creates more randomized damage. So I can hand place squibs so even though say two wings detach on the same spot, there’s different charges in different spots and it looks more varied.

KT: The main impact is going to be visual. You will see that the lines are a lot sexier. You’ll see that when you die, explosions are more satisfying and it doesn’t feel as outdated as it used to.

CR: It’s really amazing to see all these back to back and there’s quite a difference in terms of detail and quality and the amazing thing was back in 2012 I thought, “Oh my god, this is amazing, I don’t think you could get something more detailed or cooler in a game back then and I sort of look at how we move along every year and how better we get and I’m always in awe at the ability and talent of our artists.

For us to build this world it feels that tactile with these ships that feel real, they don’t really feel like a fake or a digital ship, they feel like a real ship and that’s our goal is to lose yourself in the cockpit of one of these ships and you know, fly in space and live that fantasy.

Back to Studio With Chris Roberts (CEO, Director of Star Citizen and Squadron 42), Sandi Gardiner (VP of Marketing). Timestamped Link.

SG: That really takes me back to when we were putting together the original demo for the Kickstarter. Do you remember that?

CR: I do. Well, I don’t remember much sleep from it. Anyway, so much of Star Citizen’s development history is represented in the tech and artistry that can be found in the Super Hornet and the other Hornets. So, to see where we started and where we are now makes me really proud and even more excited where the project is going to go next.

SG: Speaking of what’s next, we’ve got a really interesting piece for this week’s deep dive featurette exploring some of the new server improvements that our engineers integrated into our network code for alpha 2.6.1.

CR: Yes, so have a look at how our new multi-regional server support and enhanced cloud architecture is allowing players to connect faster and easier no matter where they are in the world. This will be something we will be continuing to work on and roll out to other regions as it beds in and works well.

Behind the Scenes: Multi-Region Servers With Ahmed Shaker (DevOps Engineer), John Erskine (Studio Director), Mike Jones (IT Director). Timestamped Link.

John Erskine [JE]: Hey guys!

Ahmed Shaker [AS]: Hi John.

Mike Jones [MJ]: John.

JE: I’m here with Mike Jones and Ahmed. Today we’re going to talk about some of our server architecture and the cloud and some things that we’ve been working on from the operations and the Dev Ops side lately. Let’s start off first by talking about what we have today. Most people don’t know very much about what it takes to publish a game like Star Citizen. We’re in an alpha phase now and the game is available to anybody who is a backer. We have regular free fly events where even people who aren’t backers can check out the game. So, tell us a little bit about what is involved in actually running a game like Star Citizen today.

AS: Almost everything we use to serve the game or the platform to the players is run on public clouds and that has been almost always the case, as far as I know, and we have been trying different public clouds to see the perfect offering for what we need for the long term and also the short term. However, I believe IT under Mike Jones had been running our own internal clouds here as well. So, we still utilize the cloud technology whether it’s public or private on hardware owned by us.

MJ: The main thing Ahmed’s talking about is where do we put the servers and the server - or services - that we connect to when we play Star Citizen could be anywhere. It’s really just a computer on the internet.

So, we could host that here in our own servers - this is the way we used to do it in the old days. The problem with that model is that first you have to acquire a lot of servers and that’s a pretty big expense whereas with the cloud or hosted rental market we can only rent or buy what we need and then we can turn it off and stop paying for it if we’re not using it. With dedicated hardware we have a massive overhead in maintaining that equipment whereas if we’re using someone else’s, we don’t have to worry so much about that. If a fan goes out or a CPU or a power supply goes out in a virtual environment, these systems are designed to automatically roll over to the next available machine. So, this gives us a tremendous amount of flexibility at a much lower price point than if we had to acquire everything ourselves. One of the things that I think is really important to point out, and I hope that Ahmed can explain to you, is the additional flexibility we get in the network.

If we just hosted here, let’s say in Austin or at any of our studios, the players would be connecting to that one place. What we can do now by utilizing servers located throughout the world is that we could put servers anywhere we need them, wherever the players are, if that makes sense to do.

JE: Ahmed, do you want to tell us a little bit more about the services that actually comprise Star Citizen today.

AS: Absolutely. So, the game itself needs - any client that any player has needs another component. The DGS, the dedicated game server, we always call it DGS. However, you cannot have this connection between the player and the dedicated game server he wants to play with unless you have a lot of backend services taking care of a lot of business logic that the player needs. For example, the player needs to be able to declare his request for the game he wants to play. He wants to say that he wants to play with his friends, communicate with them, chat with them and these are the services that are being developed here in Austin by our backend services team - Jason Ely and Tom Sawyer.

Currently what we have, we have our layout - our VM mainly full in three main types. We have a hub server or a hub VM and a core VM and a game VM. The game VM is just a virtual machine that we acquire from our cloud provider, in our current case it’s EC2, and you would load it with DGS instances according to the amount of load that we decided on - how many instances would run on how many CPUs, how much memory, how much network, and they all get connected to the core services. The core box would have several services, most importantly we have GIM - the Generic Instance Manager - a lot of backers already are familiar with the name and know what it does, and we have the PCache, Persistence cache. Persistence cache sits before the Persistence database service that writes direct to the database, taking all of the traffic from all of the dedicated game servers and pushing it down to the database in more of a cubed organized manner rather than having all of the game servers hammering one single service.

The current model that we have, we can have as many hubs as we need, we can have as many game servers as we need, but we have only one singleton of every one of these services and that’s the current model. Because usually, due to my experience in startups you go out with proofing your concept, then having your MVP: Minimal Viable Product, make sure that’s what you need, and then you start moving from there.

So currently, the backend service team are working on revamping and redesigning all of these singletons into so many stateless, easy-to-scale, services that we can have whatever we want and cope with whatever load we get and that’s the main point that we are in the cloud. One of the main great things about being in the cloud is being able to scale whenever we want. Back in the days when you were trying to market the cloud, you always used to bring up examples from Black Friday. You have a shop that sells flour or sells toys, year-round you can be fine running on a single VM - on Black Friday you need 200. That’s when you go to the cloud, and we have the same concept because we need to be always able to provide resources and game servers to the players who play - no matter how many of them decided to play, no matter where they are.

JE: Do you want to tell us a little bit about how we’ve added functionality or added features over time and where that gets us today and I am particularly interested in understanding from your experience the level of complexity we have in Star Citizen today, already, compared to a lot of other games.

MJ: One way I like to describe that is a physical way that you can visualize. When we were developing the hangar module, of course we had fewer people, and everything we did as a company came together in one computer - a tall one but not a very big computer - and it sat under one guy’s desk and we got one build a day.

The needs of the publishing increased so rapidly that this one computer wasn’t going to be enough and when we took a look at this we realized, we would need a room full of computers to do this. So, this is what Ahmed talks about when we developed our own internal cloud. So, we’ve built a virtualization infrastructure where we can actually expand, much in the way the game server is run, we expand an internal build system that increases dynamically based on the needs.

JE: Tell me about what you’ve been working on lately. This is a major new feature that all of this infrastructure and architecture that we’ve talked about has gone through multiple generations and iterations and really set us up for something that you’ve been working on now with other engineers. Tell us about it.

AS: So, since December we started looking at - we released Star Marine and now we have a lot of players who want to play FPS and a lot of players have been asking us - where is the multiregion work? When we can we be able to play on servers closer to us? So, we all agreed this is the right moment. We have an FPS right now, so let’s have multi-regions. So, since December and including January and a little bit of February we have been working on the multi-regions servers to be able to have servers in as many regions as we want, without the player having to disconnect but he has to move from one region to another, because they all share a single database. So, the same player can play in the EU and then a few minutes later can switch and go back and play in the US and he’ll find the same information that he needs.

As any new feature that we come out with, as I was just explaining, we try to come out with a certain delivery between the PoC, a proof-of-concept, and an MVP, a minimum-viable product, and that’s what we have right now and I believe we have selection for the regions that we will start with and we have building or editing our automation tools, SaltStack, to be able to code with different regions, different naming, different availability zones which is a concept that happened inside AWS and how we can add resilience while we are adding availability at the same time so we always know if something went down over here or have a placement for it over there but also we are trying to get as close as we can to our player base. We have our current region in Virginia, North Virginia, US-East one and we will be adding Frankfurt and we will be adding Sydney. It’s been a lot of fun, every single minute that we spent on this project connecting these regions together, making sure that everything worked correctly. It was kind of different from any other changes that we did because we know that this will really enhance the way it’s experienced. We know that the players will feel that. There’s a lot of work that we get to do but we’re like, ‘We just did something really cool but the players won’t really get to notice it,’ but this one we really hope that you notice it, you like it and we can embrace it and expand it on it too.

JE: One thing I know about the multi-region work is that there’s really two levels to this experience from the players perspective of playing in the different region or being able to choose a region. One piece of it is where the game server is located. This is the DGS that you talked about, and the other piece which is invisible to the player is where the HUB server is located.

AS: Yes

JE: So from my understanding is that as we roll out to these different regions that players will always connect to the HUB server that closest to them.

AS: Absolutely.

JE: Which gets them into our network at that point instead of going over the open internet and then from there they can play on a dedicated game server that’s either located in that same region or that DGS may be located in another region.

AS: So the way this feature is going to overlap utilizing the technology of using a new extension of DNS that would allow the DNS server to know a little bit more about the location of the requester, we would be able to use a service like Cal53 to send players to the closest load balancer to them and the load balancer it would take them to the most suitable HUB server that you have. The moment you get in this HUB, you will get marked with a region ID, that’s the NID you walked in with and the matchmaker will respect that choice. You came from region ID that means Europe, then you’re going to get sent to a game server in Europe. You want to go play with your friends, make your selection then go play in the US. So Hopefully this will be rolled out really soon and players will enjoy it and we’ll expand on it and listen to all the feedback and try to make it much better.

MJ: Ahmed makes it sound so simple.

JE: I know right? Makes perfect sense.

AS: [Laughs]

MJ: We’ve been working on this for some time and it has been a fun project, but there is a tremendous amount of network engineering and network security that goes into this and also performance tuning that has to be considered for this to work.

JE: When you said a minute ago, “Well we worked on this in December and January”, but it’s also the case that if it wasn’t for the 18 months that came before it that you couldn’t do this in two months.

AS: Absolutely, absolutely. That’s where experience kicks in and that’s how you learn the hard lessons from before. You have to play everything ahead, you have to make that you have sure areas where you can expand and then get surprised that you’re going to leave that current region you work in and if you want to talk a little bit about technical, AWS the largest organisational body we have is called VPC: Virtual Product Cloud. The Virtual Product Cloud cannot go across regions so you cannot have one single VPC which is a cloud and by the cloud meaning it follows one single network CIDR like, it’s a single internal network that you can divide in between, but you cannot only make a VPC have the same IP address subnet or the same networking between two different regions. So you have to make sure that you make your own tunnels and you have to make sure that your VPCs were aligned correctly so that they’re not going to collide, you will get to know whose coming from what, who needs to what and no other traffic will not get in the way and also be able to recover because definitely while you being in the cloud is such a great thing, it’s an amazing thing, but always design for failure, everything will fail.

You have neighbours, you have a lot of very complicated stack beneath you to know what’s going on. As Mike was explaining, the disks might go bad or anything could happen so we have been working on that so that even if we lost our tunnels we can recover in a few minutes from a lurch on it and make sure that we are delivering that traffic needed from the distant game servers to the core service that runs the matchmaking or database and all of that with the best achieved latency and the most attainable bandwidth that we can attain between these different regions.

I think we start to kind of like, was a big stretch, we’re going to Sydney, we’re going from Virginia to Sydney and we’re going to Frankfurt. Frankfurt is actually central in Europe when you look at it. We could have one in Ireland which makes it a little bit easier for us, but we said, “Okay, let's do it’. So we’re going Sydney, Frankfurt, then Virginia for the first rollout and all the networks are laid out and we are ready to expand, I believe there are 14 to 15, we are ready to expand in every single region of AWS when we feel we need to.

JE: Well that’s perfect for my last question which is to say, where are you going from here? What do you see on the roadmap after this? What does this set you up for and what do we expect to see in the future.

AS: Yeah I mean when it comes to all the regions I’m not sure If can state all of them? I think I can. They have Ohio, Oregon, California, Ireland, London, Sydney, Mumbai, Sao Paulo Brazil, I believe Tokyo Japan, Seoul Korea. So yeah, skies the limit and they keep adding more regions, I believe they’re going to have a new region in France and one more in China soon. So as you mention, the way it works because we are here to build this project, we always try to lay our foundation and the support that we get from you and Mike that we can take our time to build our own foundations and techs so we can capitalise on it in the future and be able roll quick features, that what we have been doing.

Right now going to any region is not going to be of any limit to us, and once we’ve rolled out more regions, this year is going to be a little busy for us because we have a lot of… As much as we were talking about how the backend services started with having a proof concept and moved then to MVP and these are going to get revamped, we have a lot of infrastructure elements that we need to look at. We have a lot of areas that we monitor certain aspects, we need to group some of them, we need to diverse some of them. We need to look more longevity solutions, we can scale it out because the way you look at it, we need to be ready to have 10,000 VM’s reporting at the same time without crippling us and that’s the reward that we get from having the DevOps department early on. I hope this answers the question.

MJ: Every year we say the same thing, but this one really feels we’re making some big headway. The automation systems are done now, we feel like we can scale with a very high reliability. We have far fewer panic nights where we’re staying up all night trying to solve problems. So definitely the work we’ve put in has paid great dividends and so now with expansion into multi regions, it just gives us more that we can do.

JE: Cool we’ll I’m really proud of the work that you and your teams have accomplished and I'm really happy that we have a chance to share this in more detail with our fans and I look forward to all the great things that are yet to come.

MJ: Thank you John.

AS: Good times.

JE: Thanks guys

Outro With Chris Roberts (CEO, Director of Star Citizen and Squadron 42), Sandi Gardiner (VP of Marketing). Timestamped Link.

CR: Really happy we got take a deeper look at the team. They are often the unsung heroes of the project because what they do is critical to making Star Citizen work, but it’s not sexy as building a spaceship or making environments or a character and there’s been a ton of effort put into implementing multi-regional servers and some of that backend matchmaking so it’s great we got to highlight some of that work.

SG: It is, well that brings us to the end of this episode of ATV. As always we’d like to thank our subscribers for contributing to all of our behind the scenes content.

CR: Yup, thank you guys, and I’d also like to express our continued thanks to all of our backers out there. You do so much to support Star Citizen’s development. We’re trying to do something unique with this project and all of you are a large part of that.

SG: Yes you are. We’d also like to invite you all to join us tomorrow at 12 Pacific for the latest Star Citizen Happy Hour stream to watch some live game and discussion.

CR: Yes, so this week, Senior Writer Wiesbaum will be stopping by to answer some questions. It should be a great way to kick off your weekend.

SG: And do you know what else might make their weekend even better?

CanadianSyrup Director of Transcripts A polite Canadian who takes pride in making other peoples day brighter. He enjoys waffles with Maplesyrup, making delicious puns and striving for perfection in screaming at the T.V. during hockey games. Nehkara Founder Writer and inhabitant of the Star Citizen subreddit.

Please enable JavaScript to view the comments powered by Disqus.