WorkHorse

LPowell

the workhorse was developed because the original flow motion was stoping recording for me in 720p60 in the gh2 with the sandisk 32GB SDHC 45MBps card.

I suspect everyone who's tried to develop high bitrate 720p GH2 patches has discovered that it's just not as robust as the 1080p modes. I decided that it wasn't worth degrading FM2's 100Mbps SH bitrate for the few combinations of lens, settings, and subject matter where reliability could be marginal. Reason being - I just use the 60Mbps H mode instead of SH mode when I want guaranteed reliability.

One other thing. I've tested Sandisk 30MB/sec, 45MB/sec, and 95MB/sec SD cards extensively. I suspect there's something fishy with the 45MB/sec card, it really doesn't seem to work as reliably as the earlier 30MB/sec Sandisk. Bottom line, however, is hands down the 95MB/sec Sandisk is the card to get when you want to record at over 100Mbps.

apefos

If you want to be more conservative about 720p60/p50 safe settings here it is the setg with gop3 for all modes and gop6 for 720p60/p50 (seth is without matrices and tables for merge with another patches)

gop6 is great for 720p quality and gop3 for other modes is what improved fsh and hbr and was the default for 24p

apefos

The workhorse is not just a datarate changing compared to the original flowmotion, there are tweaking in other settings, it is a combination of settings which works together to make it stable.

I believe that the stop recording problem in 720p with the original flowmotion is not due to the card, I believe that it is something related to the camera hardware limits. i am not a panasonic engineer to know the limits but I did the workhorse to avoid these problems.

there are reports here in PV where people says that the camera cannot handle more than 45-50Mbps for 720p, I think these people perceived same things that I perceived testing patches for 720p and fsh / hbr. so this is the reason I did the workhorse using the flowmotion as a base because the flowmotion matrices and tables are amazing. the workhorse is a tweaking in the flowmotion to make it stable and reliable.

read this report from @Azo :

"I have tried just about every patch... All of the patches are reliable in 24p mode, the problems/reliability issues are with the other modes such as 30p/720p. Knock yourself out trying to see which one of those will record reliably in the 720/30p modes."

"All of the patches that record higher then about 45mbps in the 720/30p mode will fail on you. I am trying to save you the time and frustration of testing each patch and wondering why they are failing on you. You stated that you want to run the patches successfully like everyone else above 45mbps, this is exactly the problem that I am talking about. Based on all the tests that I have performed all of the patches that record above 45mbps in modes other then 24p will fail on you."

"So like I said knock yourself out, but you are wasting your time thinking that you will be able to record reliably with bitrates higher then 45mbps in the 720/30p modes."

apefos

GOP3 for all modes is SAFE!

sete.ini improves the HBR and FSH to make the dark low contrast areas better quality!

sete.ini gop3 CONTINOUS recording death chart= APPROVED! for all modes

720p60 gop3: 10 minutes recording = APPROVED!

1080i60 FSH gop3: 9 minutes continous recording = APPROVED!

1080p30 HBR gop3: 9 minutes continous recording = APPROVED!

1080p24 24H gop3: 5 minutes continous recording = APPROVED!

1080p24 24L gop3: 10 minutes continous recording = APPROVED!

sete.ini patch is the best workhorse!

Spanning was not tested.

All tests above was started and ended recording by the camera operator, the camera operator stoped the recording before the file reaches 4GB.

Azo

@apefos

Hey man thanks for going to town on this setting. Please upload this latest setting so I can try it out. What I am looking for is complete stability and if LPowell approves your methods I am sure it is a lot better then my modified version. Thanks ...

apefos

@Azo

the last patch sete.ini gop3 for all modes was uploaded here in this topic some posts ago, stable.

(I uploaded it again in the last post below)

apefos

workhorse sete.ini gop3 for all modes patch, SDC4 death chart, 10 seconds recording, streamparser screen shoots frame mode and time mode, from 24H, 24L, HBR, FSH and SH modes:

apefos

37 original mts files with street shoots, people, flowers, buildings, fountains, using workhorse sete.ini gop3 for all modes, showing performance of 24H, 24L, HBR, FSH, SH modes, isos 800, 1600, 3200, 5000:

https://www.dropbox.com/sh/9uya2iibb6vxkxt/AAAN5pqDBlnrcaURK7QeAuwpa?dl=0

apefos

This is the last version of the workhorse, the sete.ini version, tested, stable, with gop3 for all modes,

it improves the hbr and fsh quality and keeps the already achieved quality for 24p and 720p

this version was used to shoot the 37 mts files in the link showed in previous post

used to shoot the death charts to show the streamparser screenshoots in previous post

used to do the 4GB approved continous recording tests as reported in previous post

download it, load to your gh2 and enjoy: (tested with Sandisk 45MBps 32GB SDHC)

apefos

About memory cards:

All the developing and tests was done with the Sandisk 32GB SDHC 45MBps U1 Class 10 Extreme.

My math calculations points that the 24H mode will only work with this card or higher speed cards.

If you avoid using the 24H, all other modes are expected to work with 30MBps or higher write speed cards, but this is just math calculations, these cards was not tested.

The Variable Mode shares the same settings from 24H and 24L, so before using the Variable Mode set the camera to 24H or 24L depeding on your card.

apefos

I left the camera recording alone in 720p60 SH mode and the camera recorded two 4GB files with 12 minutes and 11 seconds each file, the camera ended recording after generating the second file. no message of error, it just ended the recording. both files can playback in computer with sound. the camera was over the table pointed to some objects and room walls. this was with the sete gop3 patch.

apefos

second attempt to left the camera recording alone, this time in 720p60 H mode. the camera recorded for more than one hour and ended the recording when the card got full. the card was not empty when I started the recording, there was some files in it including the two 4GB files from the first attempt.

in this second attempt the camera generated five files, four files are 4GB and one file is 3.36GB

four files are around 13m40s and last file is 11m35s

the 4GB files are around 4.290.750.000 bytes in size, not exactly the same but very close numbers.

all files can playback in computer with sound.

So my conclusion is that the lower bitrate modes are better to left the camera recording alone because it generates more files and can record until the card gets full. i only tested 720p60 SH and H for this, other modes needs tests. there are chances of the other modes work because datarate are similar.

duartix

AFAICR, Encoder Setting 1 1080i/p = 2 will disable B-Frames for 1080 modes.

It does add another problem though... The quantizer will not be respected anymore. Choose your poison. :(

apefos

thanks for the tip, but I will not touch it anymore. I am very satisfied with the results of the sete gop3 for all modes.

the image quality is excelent in all modes, and it is rock stable, it never stops.

and it is spanning in 720p60 H mode with gop3 at 42Mbps... this is pretty good

apefos

The 24H does not span, the camera end recording and save the file when it reaches 4GB. The file can playback in computer with sound, no problem.

LPowell

One other point I should mention. All 1080p Flow Motion video modes were designed explicitly for a 3-frame GOP and will not work as intended with a longer GOP. (However, 720p modes were designed for a 6-frame GOP and may be extended to a longer GOP.)

The reason behind this is complex and specific to Panasonic's software implementation of the AVCHD encoder in the GH2. Regardless of the GOP length, the first GOP of each recording will contain an extra P-frame, the second frame which follows the initial I-frame of the recording. The encoder uses statistics from this initial P-frame to gauge the complexity of the scene and estimate the bit allocations it will need to make to fit subsequent frames within its available bitrate. In all but a few shooting circumstances, this works as intended to produce high quality IPB encodings.

During testing of 100Mbps 1080p patches, I discovered several situations where this encoding mechanism could fail and cause the encoder to crash. If you start a recording with the camera pointed at a featureless scene like a blank sky, there are few details for the encoder to gauge and it predicts low bitrate requirements for the video. If you then immediately pan into a complex scene, the encoder may be overloaded with details and fail to encode the scene properly. In the stock firmware, this may cause it to revert to Fallback Mode, producing a poor quality recording, but at high bitrates it can cause a crash.

Since I was using a 3-frame GOP, that initial P-frame was the only one in the recording, the rest were all I and B-frames. That allowed me to use the encoder's P-frame 1080p quantization tables solely for the purpose of tailoring the response of this initial P-frame. To protect the encoder from these types of crashes, I greatly boosted its P-frame sensitivity to fine details, insuring that it would allocate enough bitrate to handle highly detailed scenery without crashing. This is one of the techniques that make 1080p GOP-3 recordings work reliably at 100Mbps in Flow Motion v2.

If you lengthen the GOP beyond 3, FM2 will start producing P-frames after the initial I-frame in each GOP, and will use these exaggerated quantization tables to encode content. This may produce noticeable discrepancies between P-frames and I-frames, and will most likely waste bitrate on spurious details. The camera will still work without crashing, but FM2 will not work as intended.

Other issues will arise if you disable the custom quantization or deblocking tables completely. The side effects may not be apparent immediately, because it required extensive testing to flush out the subtler issues of high bitrate recording of panning and fast moving objects. FM2 juggles a wide range of constraints and optimizations to achieve consistent 100Mbps performance across the GH2's perplexing variety of video modes. It is very tightly integrated and not at all modular - all settings interact in myriad ways with each other, and many things were tuned by careful observation rather than formulaic theory. Consequently, it is not easy to evaluate the results of patch modifications, because they may well affect FM2's performance in particular shooting conditions that you may not have anticipated.

One example of the pitfalls can be seen in the Stream Parser chart in sete_gop3_sh_tm.jpg above. After an initial burst of 67Mbps frames, the encoder throttles down into a 52Mbps stream of frames. This is suboptimal behavior related to encoder bit allocation strategies and can be very difficult to diagnose and resolve. Once you find a solution, it is then necessary to repeat all reliability tests to make sure your bitrate allocation fix has not inadvertently undermined the encoder's stability in any of a number of specific shooting conditions.

driftwood

It takes time apefos.

Its easy to establish how much weighting should be allocated to P frames by setting GOP Related to 0 on the P frames and measuring with Streameye the results of the automated bitrate calculations based on the matrix, bitrate allocated to i frames and b frames together with your suggested Initial Quantisation to provide a guide as to how much P frame bitrate will be consumed in the current GOP/matrix design.

We can mess with what we have available in ptools to provide a a couple of different rate controls.

Inside the three Quantisation settings the 1080 and 720 Quantisation are misnamed in ptools (I believe should be quant min and quant max) as is the '... Bottom Setting=' section which is actually the cpb picture buffer allocation. Some of my settings experiment with a designed / measured matrix to knock out artefacts and then let the power of the max/min quantiser adjust the QP on a frame-by-frame basis. Some of my previous settings use the aforementioned will attempt to find the frame best quantisation available based on the previous frame. Or you set all to a fixed QP across the board measured to all your settings and desired bitrate (as in flowmo at 17, 17, 17 or Nebula T8).

My Intra settings with Encoder Setting 1 = 1 do not employ the extra initial p frame allocation (note no P frame is found within the stream). With anything greater than all-intra such as Encoder Setting 1 = 2 (uses only I and P frames) or Encoder Setting 1 = 3 (Uses I, P, B frame GOP structure encoding) the 'extra' P frame will surface. As to its true meaning there are several explanations and I am still awaiting an official explanation from Panasonic as to why its employed though lpowell's explanation is perfectly plausable.

apefos

@duartix @LPowell @driftwood thanks for useful information.

a first comment I have is about the boost in datarate in the first second of the video file. yes i saw it in streamparser, the begining of the video has a higher datarate, but after the first second the codec stabilizes itself inside the correct datarate, so it seems there is no need to worry about it because the codec is handling it well, it is stable this way. only the first second has this boost, and then the video file turns into a very well balanced datarate over time.

the _tm screen shoots are better than the _fm screen shoots to perceive this because the _tm screen shoots are zoomed out showing the graphs for all the video duration and it is possible to see that the boost is only in the first second, the _fm screen shoots are zoomed in to show better the frames so it enphasizes the boost area in the first second of the graphs.

the second comment is about the relationship between the workhorse and the flowmotion. as LPowell said in his post the flowmotion was developed to work better in gop3 and to use the only one P frame in the video file to make the quality/stability work better. so I believe that the workhorse is done correctly because it got mature, high quality and rock stable when it turned into gop3 for all modes, and it also have the only one P frame in the begining of the video files, so the workhorse is respecting the flowmotion design, maybe due to this the tweakings in the workhorse are working so good for quality / stability.

about the I, P and B frames manipulation, to turn the B frames into P frames, I am evaluating if it deserves the efforts because the sete.ini gop3 workhorse with I and B frames is already great quality for all modes and it is rock stable, and respecting the flowmotion design.

so my feelings are showing to me that this idea of change the B frames into P frames will not worth the risk of damaging the stability and the need of start tweaking stability again... the information you bring to me shows that there is a deep sea to dive into... and the amount of improvement will be so small to worth the effort.

the last workhorse sete.ini version solved the quality for the dark low contrast areas for the HBR and FSH modes and the cinema 720p60 and 1080p24 modes are excelent quality, so my advice for people who is following this topic is: use the sete.ini because it is already OK!

many thanks!

apefos

TORTURE TEST:

sete.ini is perfect for who needs continous recording for 12 minutes or less, rock stable and great quality in all modes! it is the perfect stable version of the FlowMotion 2.02 for short time recording with highest possible stable datarate and gop3.

sete.ini Codec Torture Stability Test:

Camera recording the fullhd computer monitor with the death chart movie playback, iso 1600, 1/60 shutter, on tripod, when reach 4GB the camera end recording alone without the need of press the button, and save the file to the memory card, in some modes it span.

SH 720p60 - Time Recorded: 12m10s File Size: 3.99 GB (4,287,830,592 bytes) Total datarate: 46938kbps - APPROVED! (ended recording)

FSH 1080i60 - Time Recorded: 11m07s File Size: 3.99 GB (4,293,224,448 bytes) Total datarate: 51480kbps - APPROVED! (started another file)

HBR 1080p30 - Time Recorded: 11m08s File Size: 3.99 GB (4,291,971,072 bytes) Total datarate: 51389kbps - APPROVED! (started another file)

24L 1080p24 - Time Recorded: 12m01s File Size: 3.99 GB (4,292,229,120 bytes) Total datarate: 47577kbps - APPROVED! (ended recording)

24H 1080p24 - Time Recorded: 6m03s File Size: 3.99 GB (4,294,967,295 bytes) Total datarate: 94586kbps - APPROVED! (ended recording)

about spanning: when the image is more steady, and not too much complex in detail, there are more chances of spanning, the SH recorded 2 files pointed to some static objects and the H recorded 5 files as reported in previous post. in the death chart movie it is very difficult to span, but the continous 4GB recording in the first file works ok on it in all modes.

for shoots smaller than 12 minutes, the sete.ini gop3 is just perfect, it is rock stable and have great image quality. it has the highest possible stable datarate and it is based on flowmotion 2.02

the card used for the tests is the Sandisk 32GB SDHC Class 10 U1 Extreme 45MBps, formated before the tests.

kit lens 14-42mm f10 image stabilization enabled.

this is the death charts movie used for tests, I downloaded the fullhd version:

apefos

This is the seti.ini which changes the quantization to 0 (zero) for 1080 modes.

seti.ini is also APPROVED in the Torture Test.

I do not know if the seti.ini improves the image quality in the dark low contrast areas compared to the sete.ini , Please do your own tests.

sete.ini is here also, which is already tested in the dark low contrast areas, working ok.

There is nothing more to try...

Please, close the topic.

apefos

I am tired about testing patches, but I could not leave the job without doing the last test:

the difference from quantizer 14 to quantizer 0 (zero) in the dark low contrast areas.

for SH 720p60 there is no difference to my eyes, so I keep it 0 (zero)

for HBR 1080p30 there is significant difference, for my surprise, the quantizer 14 is much better and the results are perfect texture, similar to the results of the 720p. the quantizer 0 (zero) for 1080 shows posterization in the dark areas, texture mud, so it is no good.

just avoid using the seti.ini and keep using the sete.ini because the sete.ini has the correct quantizer settings.

you may think this way: if q14 is better than q0 for 1080p and q14/q0 does not make difference for 720p, so it would be good idea to keep q14 for all modes...

if you think this way, that a quantizer 14 will be better to all modes, so here it is the setj.ini

now my job is complete.

apefos

to satisfy my curiosity I did a try in quantizer 42. the shadows are completely mud, the image turned into a flat color.

so, as said in PTool pop up help, the lower the quantizer the better. but I think the quantizer must respect the limit of what the datarate can handle, maybe the quantizer 0 (zero) for 1080 was more than the datarate could handle so some areas of the image got posterized to save datarate for other areas. the quantizer 14 did a good calibration on this.

the quantizer 0 (zero) works for 720p SH but not for 1080p HBR.

the quantizer 14 for all modes is pretty good to my eyes.

so you can use the sete.ini or the setj.ini because I did not see significant difference in quantizer 14 or 0 for 720p

apefos

If you are a pixel peeper, and if you download and see the original mts files shoot with the sete.ini gop3 for all modes and quantizer 14 for 1080 and 0 for 720 you will perceive:

comparing the 24H at 96000Kbps which is perfect image quaity to the the 24L at 48000Kbps, you will perceive that the 24L is the lower limit to the minimum datarate for excelent image quality.

in the 24L file, in the girl with pink shirt, the texture in the middle of the pink shirt is in the limit of quality, before starting to show the compression behavior. in the fountains shoot, in the dark grass in lower right corner of the frame, the 24L holds the texture ok with good quality noise, but also in the limit of great quality, starting to become gross. in these two scenes the 720p60 and 1080p30 are starting to show the compression.

considering this, and doing some math calculations, the ideal minimum datarate for 720p60 would be 53333Kbps and the ideal minimum datarate for 1080p30 would be 60000Kbps.

the problem is: the gh2 hardware cannot handle this, it will stop recording... the 48000Kbps results for 720p60 and around 52000Kbps for 1080p30 are already amazing.

in theory, considering the numbers, the gh2 with the sete.ini or setj.ini is in a similar quality level of the sony XAVCS 50000Kbps codec found in the A7S and the A5100.

the uploaded mts files shows that all the hack patches with lower datarate compared to the workhorse 24L, SH, HBR and FSH will not be the best thing for cinema production, big screen projection, considering you will playback the intermediate codec files or deliver in DCP. for bluray output the workhorse can be better but the difference can be minimized by the 40000Kbps maximum video datarate of the BR disc.

apefos

I have other tasks to do, due to this I was desperate to finish this job, sorry... but I could not stop... and now I have good news for 720p:

If you need stable HBR, stop reading from now, use the sete.ini or setj.ini uploaded in previous posts.

These new settings keeps the 24L and 24H from sete.ini and setj.ini and have new settings for 720p60.

The 720p SH and H are now only I and P frames, there are no B frames anymore. In streamparser the P frames are a little bigger than B frames and this makes difference in low light textures in high iso. The image is improved, the flowmotion matrices and tables can handle P frames for 720p! Thanks @duartix @driftwood @LPowell for informations which made this possible for me to do.

The datarate from the sete.ini SH is now moved to the H slot, so the highest stable options are 24H, 24L, H and FH.

The HBR is unstable, not for use, because there is no way to have two datarates for HBR in the patch.

The SH settings datarate was increased and is showing 70Mbps in streamparser for the death chart and showed around 64Mbps for a steady low light scene. this datarate combined with the I and P frames only and quantization 0 (zero) removed all the mud from the 720p. My GH2 is a new camera now for slow motion!

But there is a price to pay: it stops recording in the death charts after one minute recording. In the steady scenes, not so much complex in detail, it records more time. And the quality of noise and textures in high iso low light is AMAZING, can intercut with 1080p24 in timeline and no problem!!!

no worries, because you are safe for stability, for rock stable 720p60 just use the H mode because it is the old SH datarate from the sete.ini

this is the patch I am keeping in my GH2, perfect for indie cinema.

apefos

GREAT NEWS, really great, pretty amazing, see:

The new SH 720p60 settings with only I and P frames and 70Mbps datarate is reliable.

Using a manual focus lens with adapter it passed the death charts movie torture test in iso 6400:

SH 720p60 time recorded: 8m12s file size: 3.99 GB (4,287,830,016 bytes) Total datarate: 69674kbps APPROVED! - (ended recording, no span)

Using the kit lens 14-42mm it recorded the torture test for 5m32s and the stop recording message shows up in the camera LCD.

So you may ask: why this is great news? keep reading...

the 5m32s with the kit lens was with the image stabilization enabled, but the camera was on tripod, so it was enabled but inactive, quiet...

I also did tests in the green grass with lots of camera movements and shutter 1/4000 iso 6400, and when I disable the image stabilization the camera did not stop.

So I did the the torture test again with the camera handheld and image stabilization off, the camera recorded 5m38s and then stop recording, similar time.

the reasons for the great news are:

when the camera stops recording in the SH mode, there is no freeze, no need to remove the battery, the message shows up, go away, and the camera return to the stand by mode, and you can start recording again.

and the best news is:even when the stop recording happens the camera saves the file, the file is usable, it shows the thumbnail in the computer and playback with sound!!!

I think this proves that the patch settings are very well done.

For me, for my use, this is the perfect patch, because I have perfect stable 1080p24 in both 24L and 24H and I have perfect stable 720p60 in H and when I want to go to the limits I have the SH for the best possible 720p60 without big worries. All modes are excelent quality.

tests done with the Sandisk 32GB SDHC 45Mbps U1 Class 10 card

Howdy, Stranger!

Categories

Tags in Topic

Top Posters