I started something like this, but now I don´t have time to make comparison and no one follow... http://www.personal-view.com/talks/discussion/2812/only-settings-comparison-no-small-talk#Item_8

The best way I've found to see the details in video files is to display them at 400% magnification and scrub through the frames in a video editor. (Make sure the video frames are rendered at full resolution without any type of smoothing or filtering.) At this level of magnification, you can clearly see rectangular macroblocking artifacts and coarse quantization from inadequate bitrates. I use Adobe After Effects (which can display individual RGB channels) and Premiere Pro for this purpose, though I'm sure other NLE's can do this as well.

It seems some guys don't know the meaning of the word "objective"...

There are objective ways of analyzing encoder performance: looking at quantization parameters, comparing the unencoded input to the output, calculating the signal to noise ratio, estimating the signal to noise ratio.

Bitrate is still king. If this was not so, a lot of hi end codecs such as Pro Res wouldnt need for example 147Mbps for a normal transcode setting. On the GH2, a good employment of Q makes good use of the required bitrate alongside decent setup parameters. Lower bitrate recordings can look great - relying on predictive sampling of the reference frames but these subtle differences whichlook great on first acquisition to the human eye - may suffer on the edit and the grading. Many of the top testers go thru many hours (or days ) to achieve new settings. However, what looks good to one, looks not so good to others. Horses for courses.

I have settings which push nighttime recordings, daytime setups, closeups and wide shots with detail. Each have to be tempered for the GH2 as no single setting covers the majority well enough. All rounders? Canis, Sedna and Mysteron are very good, BKs GOLGOP stuff is good with its decent prediction, and there are others... but no single setting will satisfy everybody.

@mo7ies I was just reading some of your comments in the Sanity thread and I want to point out to you some things that may be helpful as you seek to get more information (and things that I am keeping mind as we plan stage 2 of the L.A. comparisons).

1) There is no entirely objective ranking system for image quality characteristics. All of them will be subjective to some extent because each viewer has differing aesthetics and priorities.

2) As regards "low noise", specifically, keep in mind that we are generally talking about different approaches to dealing with the same amount of noise, not actually ways of reducing the noise the sensor provides. This makes the question exceedingly subjective. Are we looking for settings that do so much smoothing and noise reduction in camera that we cannot see the noise that is there, or are we looking for the ones that give us the most detail and the most control of the noise in post? Do we want the noise to stay as smooth as possible and update infrequently, or do we want it to be very fine and update as frequently as possible? The settings that are often praised for their low-light performance are most often ones that give the finest rendering of noise and of details that are sometimes reduced by encoding quirks in other settings. Here is an example that illustrates that.

Please read the article referenced on the Vimeo page before asking any questions about the comparison.

Password Driftwood1

3) Because of "1)", no "objective" table can be compiled, only subjective ones. However, the images or videos with which to compare can be made available for people to make their own comparisons (objective data, though not always properly controlled) and subjective commentary or rankings could optionally be provided.

4) We need more comparisons and it would be helpful to have them consolidated. I am doing everything I can (within the time restrictions of my other work) to provide additional ones when I can, and to coordinate others in the area in doing so as well. I've posted informal guidelines for testing in the past and will be happy to codify them later on if (and only if) there is sufficient interest.

@LPowell I am not sure I follow. Are you referring to the range of settings tested or to the construction of the test?

@LPowell Ah, I see. Yes, that's already on track to be addressed with the inclusion of Sedna GOLGOP in an upcoming test. But just to be completely clear, I wasn't testing the applicability of the different matrices - my tests have (in the past) only been designed to evaluate specific pre-compiled settings.

So you're right on the money - I wasn't trying to test more than that.

Do you have notes in testing and comparing pre-compiled settings that use P and B frames? That test was really designed around the intra settings with the other settings included to compare against them, but some future tests may address other aspects. I plan to use fans to get (completely repeatable) motion in some shots, for instance.

Also, how do you feel about shutter speeds in motion tests like that? Should I keep them at 180 degrees to give a more realistic representation of normal usage? Or should I use narrower shutter angles to highlight sharpness differences while rendering motion?

Everybody is trying different ways to test things but there's no uniform test. Shooting resolution charts or color charts is good, but you need movement. So how do you test movement and color and resolution in such a way that ANYBODY can replicate it?

I've felt for a while now there should be a "Walmart" or "Amazon Test". I mean that there should be some items we could all agree on that anyone could easily buy at Walmart or order on Amazon and use them as a standardized test for between $20-30. A small personal fan and a motorized toy for movement. Some colorful items, watercolor set, markers, etc... for color and somethings fuzzy or textured for detail. Plus a box and some small lights so they could be inside the box with their own light source.

That way everybody could order the same items and arrange them similarly and we'd get some uniformity. Your thoughts?

@LPowell Heheh...I said $30 and that's $40....again...I am very cheap. >_< Even checking ebay the cheapest I see like that is around $25.

But to clarify...we need a self contained box with multiple objects inside it with their own light source. Something like this: http://www.amazon.com/Fulcrum-30010-301-Battery-Operated-Stick-On-Light/dp/B000R7PM36/ref=sr_1_1?s=home-garden&ie=UTF8&qid=1335751260&sr=1-1

A box, filled with lights and colorful objects with texture. Preferably under $30...at most $40. Stuff anybody could get.

EDIT: You could shoot the metronome, but not everybody would have the same wall behind it or same lighting. Put everything in the box, zoom in until it fills the screen or put the camera in the box as well and just record what's in it.

@LPowell They don't have to be directly aimed at the camera. They could be placed behind some of the objects or behind/above the camera if the camera is also in the box.

I do think one light should be visible so you could test the fringing or haloing(whatever it's called) of a light source at various ISOs or bitrates like the old Zacuto Shootout Lightbulb test or Blooms ultra low light/high iso match flame test.