Matt: Visitors to guardian.co.uk between 9.30am and around noon on Thursday may have noticed that commenting was disabled. Early bird readers may even have seen the commenting form display a "Submit" button that was impossible to click.

Thankfully all is now fixed, and there's a new feature to boot, but this wasn't quite the way we planned to introduce the new functionality.

We release an update to our core software, R2, every two weeks at around 9.30am. Today we deployed release 125 to production. One of the changes for this release was an upgrade to our version of jQuery, the Javascript framework, from version 1.4.3 to 1.6.4. The newer version promised much faster performance, something we were keen to take advantage of.

One change to jQuery affected us in particular though: its .attr() function. jQuery modified the way this function behaved when used in this way:

$(element).attr("disabled", "");

In the past version of jQuery we used, this would change this kind of markup:

<input type="checkbox" disabled="disabled" />

Into this:

<input type="checkbox" />

After the upgrade, it instead changed the markup to this:

<input type="checkbox" disabled="" />

See the problem? Now, when we test to see if that element was disabled, it no longer returns false. In the case of Discussion, our commenting platform, we disable the Submit button when the page loads and enable it using jQuery once the user has started to type a comment. Our jQuery upgrade went out, and suddenly this code no longer behaved as we expected:

if ( $(element).attr("disabled") ) { // do stuff }

In jQuery 1.6, this method now returns "undefined", which Javascript gurus will know is not the same as false (unless you use === comparison). Long story short, we were no longer enabling our submit button, so comments couldn't be posted.

Now, of course, we knew about this bug a while before we pushed the jQuery upgrade out – we carefully checked for old-style uses of $(element).attr("disabled", ""); and rewrote them to use the more verbose (and correctly working) $(element).removeAttr("disabled");.

Unfortunately for us, we made the fix into our feature branch of Discussion which we were due to deploy before pushing the new jQuery to production. Due to timing constraints we eventually decided to push the new Discussion features after the jQuery upgrade. Oops. This then meant our scheduled release ("sometime next Monday morning") suddenly became a lot sooner ("right now").

Gideon: Every release we run a series of regression tests to make sure the key functionality on our site for editors and users still works, one of these is, of course, to post a comment. So how on earth did this not get spotted?

The answer as Matt says, is that we were testing a newer version of our Discussion platform (which sits outside our main CMS) on our test environments, than was running on Production. As we prepared to deploy the new commenting functionality we made the fatal mistake of upgrading the version of the Discussion on our Release environment, the final staging post which is supposed to be an exact mirror of what is to be deployed on Production.

So when performing final sanity checks the software appeared to be working, because it was running the newer version, which included the fix. Thus breaking one of the golden rules of testing (From the ISEB Software Testing syllabus):

The test environment should correspond to the final target or production environment as much as possible in order to minimize the risk of environment-specific failures not being found in testing.

If we'd followed this rule, this issue could have been spotted and averted. As it is, we learnt the hard way from this mistake!