The Bayou, April 2014 (Flickr)

Server Deployments – Week 30 Recap

On Tuesday July 22nd, the Main channel was updated with the infrastructure project deployed to the Magnum RC in week 29, and which adds support for the upcoming changes to the Skill Gaming policy, and includes the updates previously on LeTigre and BlueSteel- release notes

On Thursday July 24th, all three RC channel were updated with the infrastructure support for the upcoming changes to the Skill Gaming policy, and the updates previously deployed to LeTigre and BlueSteel – release notes.

SL Viewer

The Library Refresh viewer was updated to release candidate status with the release of version 3.7.13.292194 on July 23rd. This viewer contains an update to a large set of the libraries used by the viewer to provide security, stability and consistency improvements to this and future viewers.

Group Chat

The anticipated group chat test didn’t materialise at the Server Beta meeting on Thursday July 24th as a result of Simon Linden coming across a last-minute issue which needed to be resolved ahead of further tests. He and Oz did, however, explain some recent discoveries within the chat system.

“While the earlier update to group chat didn’t give us any significant performance boost, we got a lot more information out of the servers,” Simon said. “And what we found was a big part of the group chat system load is not the chat messages you care about, but the updates to who is in the session or not.”

Oz added, “Those updates happen whether or not you’re displaying who’s in the session, in every group you’re in.”

Simon continued, “You can actually see this in the viewer if you add a line of code to log something whenever an update comes in to tell you who’s in the group chat … you’ll be surprised how many you get. The load goes up as the group size goes up … with a larger group, people are joining and leaving more often, and there are more people to update.”

People joining / leaving a session are recorded by the chat server. “It has a list of who’s online and in the group chat at that moment,” Simon explained, “it’s adding and removing from that list, and [generating] the resulting updates, that are the problem, [causing them] to be sent to everyone else still active in the group as they do so.”

The growth curve of these updates is described as exponential, and there is a knock-on effect with them as well; as group chat sessions share server resources, it is possible that a large group chat session, with multiple users joining / leaving it and thus causing it to generate lots of updates can affect other group chat sessions hosted on the same group chat server, slowing them down as well.

While the chat servers are due for a hardware change, which is hoped will improve performance to a degree, simply adding more hardware to the chat service back-end isn’t seen as a solution, as it’s the exponential manner in which the updates grow which needs to be reduced and controlled. The testing Simon had hoped to run during the Server Beta meeting was to test some improvements he had been making to the queuing of the updates and in combining messages to hopefully reduced the load. However, in running over the code, he encountered a glitch that he needs to resolve before the testing can proceed.

Another issue with the group chat system is that when users log-in to a Second Life session, they automatically join all 42 of their groups, sending a “I’m here!” message to all 42 groups so that they can start receiving messages from active groups. This has obviously been exacerbated each time the limit on how many groups a person can join has been raised, so as Oz pointed-out during the meeting, “upping it again would make it even worse, so until this is dealt with, don’t even ask… 🙂 .”

HTTP Updates

As indicated by Monty Linden at the last TPV meeting, there are further server-side updates which should further assist with improvements to texture and mesh asset downloads. These are now nearing the point where they are likely to be surfacing (although quite when isn’t clear), prompting Oz to comment, “We’re setting up some experiments with server side changes that will complement the pipeline viewer, but are not strictly speaking dependent on it. When we’re confident that our test setup is ready, including how to measure the results, we’ll invite you folks to help us test.”