For years now, I’ve been telling people to always take the results of tests done on one copy of a lens with a serious grain of salt. Not matter how carefully it was tested, it’s still just one copy. Others will be worse. Others will be better.

Because I’m so aware of copy variation, I (in theory at least) test at least 10 copies of a lens and average those results before I present them. I do that because it gives a much better idea of what the average copy will be like.

Sometimes, though, it seems the true purpose of my life is to serve as a warning to others. I just can’t pass up an opportunity, don’t listen to myself, give in to temptation, and test just one copy of a lens. I justify it to myself because sometimes one copy is all I’m going to get, at least for a good while.

So today I’m going to use myself as an example of why testing one copy of a lens is just a bad idea. While doing so I’ll eat some crow and also apologize to Sigma. Because a few weeks ago I published the MTF charts on one copy of the Sigma 24-35 f/2 DG HSM Art zoom that I was loaned for a day and published the results. After that Lensrentals got multiple copies in stock and I did my usual testing of 10 copies. It is very apparent that that first copy was the worst of the bunch. The other copies, each and every one, were better.

Sigma 24-35 f/2.0 MTF

I’m going to put our original results from one single copy we tested earlier on the left, the average of 10 other copies on the right, so that you can see how different the average results are compared to the original single test lens.

Roger Cicala, Olaf Optical Testing, 2015

Roger Cicala, Olaf Optical Testing, 2015

Roger Cicala, Olaf Optical Testing, 2015

Again, I want to emphasize that the MTF charts on the right are our official average results from testing 10 copies of the lens. I just put the original single test lens results on the left up so you can see the difference.

This gives an excellent example of why single-lens test results can be deceiving. If you notice, the single lens had excellent center resolution, a bit better than average. But off-axis (away from center) the test lens had a LOT of astigmatism, much worse than the average lens. In fact, it was worse than any of the 10 copies we ran for this test.

The average MTF curves for the Sigma zoom actually do (as Sigma claims) compare very well with prime lenses. Below are comparisons between the Sigma 24-35 f/2 and the Canon 24mm f/2.8 and 35mm f/2 IS lenses. The zoom is certainly right there with the two Canon primes.

Roger Cicala, Olaf Optical Testing, 2015

Roger Cicala, Olaf Optical Testing, 2015

Copy-to-Copy Variation

Now that we have our 10-copy testing done we can generate consistency numbers and variation graphs for the Sigma lens. These are really quite good, comparing very favorably to what we saw from Canon and Nikon 24-70 f/2.8 zooms.

Roger Cicala, Olaf Optical Testing, 2015

Roger Cicala, Olaf Optical Testing, 2015

Roger Cicala, Olaf Optical Testing, 2015

Conclusion

Looking at the results of the Sigma zoom now that we have tested a number of copies, it’s very obvious Sigma has done it again. They’ve broken new ground and made a zoom that is a full stop wider than other zooms, yet is optically on a par with similar aperture prime lenses. And they’ve maintained excellent copy-to-copy consistency while doing it. Superbly done, yet again, Sigma.

Roger Cicala, Brandon Dube and Aaron Closz

Lensrentals.com

September, 2015

Addendum: I know many of you want to know if I have an explanation as to why the first lens was the worst lens. I can’t say for certain, it was loaned to us for just one day, but I would guess that it may have gotten knocked around during use or shipping. The problem it had was subtle. Center resolution was still excellent and there were no clear signs of decentering (we checked carefully on some pretty expensive equipment as well as the usual test targets). It’s a good example of why it’s so difficult to decide, using just one copy, if a lens is as good as it can be or not. Even out of sorts, the Sigma was still excellent, about as good as we’d expect an excellent zoom to be. It just wasn’t as good as the other copies, which are better than we expect an excellent zoom to be.