September Update: PCC Optimizations

PCC and Optimizations Update

Today's update is to spotlight some of the team's recent work. The following information is to provide a developers side glimpse into what we've been up to most recently. We hope you enjoy and look forward to your feedback.

The location of Plymouth Control has a long history of edits since it was originally created. Our revamp brings the area to revision number four. Those who've been here for some time may have known it as either Plymouth Canyon, Plymouth Control Station, or most recently Plymouth Control Center. While the name hasn’t changed the visuals and functionality have over the years. This revision’s focus was to clean up the area, increase performance and do the prep work so we can finally finish it with the lore in mind.

You may be curious as to what the purpose of this pre-revamp work is right now instead of later during the world revamp. So we will jump right in.

Data Overload!

It came to our attention that the area Plymouth Control Center had several issues that could possibly hinder our upcoming patch test. This data is being replicated across multiple areas that are linked to PCC and can cause slowing frame rates, low fps in the area and larger than normal hitching. This hitching and eventual frame rate drop is caused by the amount of data that is being loaded and unloaded, as well as the order of it doing so.

Those issues were as follows:

Large number of individual assets

Large number of individual textures used in the Texture Banks

Large number of Sub-materials used in the Material Banks

Above caused an overly large amount of draws client side

Above caused an overly large amount of cache creating and cache removal time

Above caused some players to time out or crash to desktop

Above caused a type of memory leak over time, filling cpu and memory cache

Cache issues could cause some players to crash to do partially removed or added caching

Cleanup on Aisle Four

By addressing these issues we were able to further reduce the enormous amount of data that could potentially cause things to time out during our patch test. We are working at cleaning up the server and the game itself. The content for this area will not suffer from the reduced data. Instead we’ve reduced data while actually adding more visuals, and leaving additional space for future content to be placed during the world revamp.

Some of the ways this is done can be seen visually with the art assets themselves. By reducing the number of actual objects we can improve performance for the area and for the client. So in this case, instead of building an area using individual wall and floor tiles, we can build this in 3ds Max and export it in chunks or pieces and with a simple placement of one asset, we can then drop the rest in to fit into place and reduce the number of textures, assets and total draw counts. This is what we did with PCC, while adding a bit more to the world. By doing this, it allows us the room to add more clutter around the area (and world) that will help make it feel more alive and lived in.

As an old mentor of mine once said, "A poly saved is a poly earned." This was told to me over 20 years ago, and still applies to this day. It means we have more room to be able to add a bit more from art itself to features and functionality.

To give you a better idea of how much our changes have affected the area we’ve listed the before and after totals below.

Before: 48,251 Total HB Node counts for all asset types

HB Nodes:13,551

Heightmap Nodes: 12

Art assets total: 5,121 local

SpeedTree: 979 (includes surrounding areas)

Dynamic Detail: 10

Particles and emitters: 918

Textures total:1,627 local/10,121cached (all surrounding areas)

Waypoints: 926

Triggers: 24

Region Nodes:17

Asset Errors: 429

After: 6,709 total HB Node counts for all asset types

HB Nodes Total: 6,709

Heightmap Nodes: 9

Art Assets total: 133 local

SpeedTree: 100 estimated (139 surrounding areas) - add count once done

Dynamic Detail: 5

Particles and emitters: 121

Textures total: 29 local/798 cached (all surrounding areas)

Waypoints: 926

Triggers: 24

Region Nodes: 17

Asset Errors: 7 (Redefinition errors remaining from old art)



We plan to do a test the type 2, Normal Push, next week. Pending our findings from this test we may or may not need to proceed with additional pre-world revamp work on two additional areas. You can find below an informative summary of the different patch types. As always we encourage your feedback be it positive or otherwise.

Push Types

As we had explained in past updates, we have three types of pushes built into HeroEngine. We figured we would go into a bit more detail about these three types since some still had questions about the differences. If you still have questions, please feel free to ask them and we will do our best to answer them.

Last year we worked on making the version of HE that The Repopulation is on to work with our current Full Push system. There had been a lot of changes and updates to the HeroEngine systems over the last several years that The Repopulation was not included in since the Code Freeze in 2014 and Repopulation shutting down in 2015. Now we are currently working on the second type, Normal Push.

Here are the three types and what makes each one different.

Full push: This pushes ALL content and overwrites any old content. Can also add in full DB push as well .. this is generally done if a LOT of stuff was changed or you need to push a lot of major changes and do a full wipe.

- This tends to need the servers to be spun down to push to the server and out to all collocations.

- Server downtime can be between 2 hours to 12 hours depending on size/scale and any issues or conflicts found.

- Good for large scale Content Updates; expansions, rework, etc

Normal Push: This type is for larger but not massive updates. It runs a diff between old and new content, what needs to change, and where. It also runs debuggers to make sure there are no major conflicts between server, client and listener (dude server) however this tends to exclude any scripting gameplay conflicts)

- This generally takes a server restart.

- If there are no conflicts found restart tends to be between 15 and 45 minutes depending on the scale of the changes and updates.

- Good for quarterly or large scale updates

Live Push: This is to push smaller updates that might include art changes, some script changes, minor system changes, etc.

- This is generally done even when servers are up and running, and client will get the update when they log in next.

- No downtime generally other than perhaps a quick 5 minute respin.

- Good for monthly to bi-weekly updates that are smaller in nature.



Testing... and More Testing

With most of the work done in Plymouth Control Center and surrounding areas, the HeroEngine engineers will be doing another round of tests later this week once they get the go-ahead from the Repop dev team. The surrounding areas are done, including Ghetti Gorge. So what is left for the developers to do?

Smooth out the sculpting of the terrain for PCC and surrounding areas

Repaint to give a better blend between areas

Place trees and other flora

Test updates to the nodes (harvesting, triggers, etc)

Place working Doors

Though we still do not have an estimated time of when this will be going live, the Repop team is working hard on several aspects of the game and have not stopped. What started out to be a combat update is also turning into a cleanup phase with some new content to push and test for the Island then World revamp. We will continue to update you and once we have been able to do a successful test push without time out issues, you will all be the first to know!

We enjoy openly conversing with our community about ongoing development. Our internal testers from the community have been a tremendous help and we want to thank them for all the valuable input and finding issues like textures and helping with ideas on how playable the areas are. We are excited for when we can bring this update live to everyone.