I think @stonebat has it close to right. I'm just guessing according to theoretical criteria - you'll have to test them. In some cases the AQ may need to be lowered a little.
For GOP 6 I would estimate that you could go to AQ3 at any bitrate. With GOP1 I think it's AQ 1-2 no matter the bitrate. This is because AQ is automatically scaled according to bitrate.
When you lower frame limit it will end up raising the effective QP value to accommodate the lower bandwidth allocated to I frames and do other strange things. It's sort of like clipping - not really a good thing. Better to leave frame limit a little higher and lower AQ until it works. According to my calculations (based in part on your feedback), the ideal AQ value for GOP3 at 100Mb would be between 1 and 2. It seems that 2 works pretty well, 3 is clearly too high according to my latest math. With GOP1, you probably want to use AQ1.
@driftwood Unfortunately stream parser several times showed the same thing. Maybe this card? Maybe this is my model of camera? When I changed FBs and Fl to 1.3 .... Hmmm, I think I'm doing something wrong. Anyone can help me ?
"The h setting is 110Mbps, GOP3, AQ2, X1.5 on FBs and FL. The j setting is 110Mbps, GOP3, AQ3, X1.39 on FBs and FL. Both worked. The attached diagram is from the j setting. My gut feeling is that... the B-frame seems little too big. Or is that a typical behavior from AQ3 setting?"
I think it is typical of AQ3 and 4... and to my mind it's a good thing, those big b-frames, especially with non moving shots. Going to test your latest right now.