On November 1st, 2014, the Lab blogged about improvements seen from their side of things as a result of CDN support deployment. At the time the updates were being issued, the Lab was also asking for feedback from users as to how things were going for them.

As a result of this request for feedback, the Lab issued a further update on the improvements on Friday, November 7th,, and it is a tale of two halves.

The first part of the blog post re-states the core benefits that have been seen as a result of the CDN deployment for mesh and texture data, which is again split into two key areas: a considerable reduction in the load on some key systems on the simulator hosts, and a big performance improvement in texture and mesh data loading, resulting in users seeing faster rez times in new areas they’re visiting.

However, the experience of some users hasn’t been so good, as reported in the forum thread, and it could not be put down to matters of distance from the CDN nodes vs. the Lab’s simulators, or to people experiencing slower load times as a result of being the very first to enter a region which had not been cached at the local CDN node.

This feedback encouraged the Lab into further investigation and data-gathering of specific situations, allowing them to engage with CDN supplier Highwinds in order to try to determine possible reasons for the poorer experiences. The second part of the blog post notes the outcome of these efforts:

We believe that the problems are the result of a combination of the considerable additional load we added to the CDN, and a coincidental additional large load on the CDN from another source. Exacerbating matters, flaws in both our viewer code and the CDN caused recovery from these load spikes to be much slower than it should have been. We are working with our CDN provider to increase capacity and to configure the CDN so that Second Life data availability will not be as affected by outside load. We are also making changes to our code and in the CDN to make recovery quicker and more robust.

The blog post also points out some of the risks involved when trying to deploy large-scale changes to a complex and dynamic environment such as Second Life:

Making any change to a system at the scale of Second Life has some element of unavoidable risk; no matter how carefully we simulate and test in advance, once you deploy at scale in live systems there’s always something to be learned. This change has had some problems for a small percentage of users; unfortunately, for those users the problems were quite serious for at least part of the time.



The post concludes by thanking all those who contributed to helping the Lab understand the nature of the problems being experienced and in taking the time to help provide data on their particular circumstances which helped with further investigations, and with a note that it is hoped that the changes that are to be made as a result of this work will reduce such problems, allowing more people to enjoy the benefits offered through the use of the CDN for asset data delivery.