Here’s a group of Canon 35mm f/1.4 lenses, for example, and there don’t appear to be any bad copies. It’s a nice, reasonably tight group of results. And knowing how other lenses look when they’re bad.
Unfortunately (for my sanity), all lenses aren’t primes with nice tight test patterns that make great examples. Below is a run of almost 100 copies of the Canon 24-70 f2.8 zoom lens tested at 70mm. Why so many? Because when I first tested two dozen copies they were a random smattering of results scattered around the chart. Data doesn’t make any sense? Get more data. Luckily we have a lot of lenses.
Interesting. And so is their summary (which would justify much of reviewers' neglect):
"For those of you into pixel peeping and “I demand a perfect copy” kind of stuff, I think this is the takeaway message:
Copy-to-copy variation is real, although barely detectable in actual photography. If you pixel peep you can find a difference that’s real, but not significant. Meaning you can see a small difference in test results, but couldn’t tell the difference in a print.
Bad lenses are usually massive outliers, easily detected at a glance (see the previous article for examples) or with the most rudimentary testing (like just taking some pictures) in the majority of cases.
There are some situations, like our Canon 24-70s, where a measured difference isn’t huge (at least compared to really bad lenses) but probably is significant enough to affect the sharpness of a print. While SQF isn’t the be-all, end-all measurement and has very real limitations, it can be a useful tool helping us to decide what is, and is not significant.
Finally, for those of you (and there are a couple of million of you) who own a Canon 24-70, please don’t go off the deep end because of this demonstration. Remember, these are rental lenses. They get used heavily an average of 90 days a year (and probably not as gently as you would use your own equipment). Plus they are shipped all over the country, an average of 20 round-trips a year. As I’ve always said, Lensrentals.com should be considered battle testing for photo equipment. Whatever can fail, will fail here. It doesn’t mean your copy is going to do this."
The only 24-70 I've used must have been one of the outliers. It was brand new and had worse CA at 2.8 than any other lens I'd ever used. It made the cheapie 18-55 look like a much better lens, which is sad. Ever since then I can't help myself from questioning Canon's quality control
Where tolerance mismatches between camera and lens really show up are in persistent auto-focus errors, particularly with fast lenses at long focal lengths. The most notorious offenders have been third-party zoom lenses and ultra-fast primes. However, Canon and Nikon lenses are not immune either. Fortunately, the problem is less common among mirrorless cameras like the GH2, which are always in Live Mode where auto-focus works directly on the image sensor.