Last year, a group of awesome wiki editors and I forked the RuneScape and Old School RuneScape wikis from FANDOM/Wikia to an independent site. It was a monumental task that took a lot of time and effort, not least because of the insanely large amount of pages and revisions we have on the wikis (we had to import over 20 million revisions just for the RS Wiki!).

Here’s how we did it.

The logos of both the RuneScape Wiki and Old School RuneScape Wiki

Initial steps

When we first discussed forking the wikis back in late 2017, it was difficult to imagine how we were going to pull off such a move, though it was exciting to think about. The first hurdle was determining what kind of infrastructure we needed. The wikis use MediaWiki, an open-source wiki software developed in PHP. We couldn’t really diverge from this because we needed to maintain compatibility with the data from Wikia we would be porting, but it’s not like we actually wanted to either because MediaWiki is by far the most complete wiki software package available right now.

However, MediaWiki is quite old. It still gets very regular updates, but it relies on PHP, which can be slow, and there’s certainly some outdated code within the software that could do with some performance improvements. To compensate, we needed to make sure that we tuned the performance of MediaWiki, PHP, and other software we were using to ensure fast response times on the wikis and to decrease the chance of downtime.

To host the wikis, we needed servers that were fairly powerful and would be reliable. At first, we played around with the idea of using AWS EC2 instances, as it would’ve allowed us to scale up our infrastructure more easily, but the high costs associated with AWS made this a no-go at the time. We decided to settle on purchasing dedicated servers rather than a cloud-based solution. After setting them up with the basic software we’d need, such as a LAMP stack, we were ready to import the data over from the old Wikia sites.

It took several days to completely import Wikia’s data, mostly because some of the data was inconsistent, or missing. We plugged the gaps by scraping data in a number of different ways, such as by utilising the MediaWiki API on Wikia’s site. Wikia use a really old version of MediaWiki, so we had to put together our own tools to grab log entries and other data that are not included in their public dumps. Some of our archived files had been deleted by Wikia, so in some of those cases we even had to utilise archive.org to find them.

One of the important technical goals was to make it as easy as possible for people to migrate their accounts over to the forked site, so that nobody lost the record of their existing contributions. Importing users ended up being a relatively easy task. We filled our user database with “dummy users” based on the usernames assigned to the revisions we had imported, and thanks to some scripts and an extension by the owner of another forked site WikiDex, we were able to provide a method for those users to claim their old accounts. Users can simply login with their Wikia credentials, and we authenticate and move their account over.

The Great Merge

We had originally planned to launch the new wikis in January 2018, as a nice way to start the new year. Due to legal delays, our launch date had to be pushed back several times, which meant that we had diverged from the data that was on Wikia.

To make sure we were staying up to date with the content on the old wikis, we setup new databases that essentially acted as mirrors of the Wikia sites. The plan was simple: when we were ready to launch, we’d “merge” the new pages we’d made on the fork wikis with the last versions we pulled from Wikia in these new databases. To do this, we scraped Wikia’s recent changes feed and imported what we needed. This had to be done very frequently, because they do not keep a very long record of the recent changes on their wikis.