Grass Valley Forums Facebook   Twitter   YouTube  

Go Back   Grass Valley Forums > Editors > Editing with EDIUS

Reply
 
Thread Tools Display Modes
Old 02-19-2019, 02:03 AM   #1
dpalomaki
Senior Member
 
Join Date: Oct 2007
Location: USA
Posts: 1,874
Default Video noise vs. compression and quality

Ran a couple simple tests using E9 to look at the impact of noise on video compression. I used NTSC SD format for simplicity, and because it is the format of the noisiest video many of us deal with in the USA (Video8, Hi8, VHS, and S-VHS home video).

I used a 5 second clip of three pople with good light (low noise), and the Ignite Noise filter. I use Grass Valley Lossless as the codec.

A 5 sec clip of 3 people resulted in 42 MB file.
Adding the Ignite Noise filter set to 10 resulted in a 59 MB file.
Adding the Ignite Noise filter set to 20 resulted in a 68 MB file.

As a point of reference, a 5 second DV clip runs about 18 MB.
and 5 sec of DVD compression would run around 6 MB.

It is clear that the popular compressed lossy formats that are limited to a set bandwidth will throw away a lot of data, both noise and information. It argues that initial capture from the analog format is best done to a lossless format, and that any noise reduction and restoration/color correction and other editing be completed before saving to a lossy distribution format.
dpalomaki is offline   Reply With Quote
Old 02-20-2019, 03:02 AM   #2
BernH
Senior Member
 
BernH's Avatar
 
Join Date: Sep 2014
Location: St. John's, NL, Canada
Posts: 1,441
Default

Quote:
Originally Posted by dpalomaki View Post
Ran a couple simple tests using E9 to look at the impact of noise on video compression. I used NTSC SD format for simplicity, and because it is the format of the noisiest video many of us deal with in the USA (Video8, Hi8, VHS, and S-VHS home video).

I used a 5 second clip of three pople with good light (low noise), and the Ignite Noise filter. I use Grass Valley Lossless as the codec.

A 5 sec clip of 3 people resulted in 42 MB file.
Adding the Ignite Noise filter set to 10 resulted in a 59 MB file.
Adding the Ignite Noise filter set to 20 resulted in a 68 MB file.

As a point of reference, a 5 second DV clip runs about 18 MB.
and 5 sec of DVD compression would run around 6 MB.

It is clear that the popular compressed lossy formats that are limited to a set bandwidth will throw away a lot of data, both noise and information. It argues that initial capture from the analog format is best done to a lossless format, and that any noise reduction and restoration/color correction and other editing be completed before saving to a lossy distribution format.
Hey Don, I have never bothered to run a series of measurable file size tests with regard to differing video noise amounts, but your results make sense.

For those that may be scratching their heads, wondering what this means, simple video compression essentially works in one of or a combination of 3 ways:

1) Don't encode what the eye doesn't see or doesn't see well. This is the theory behind the whole 444, 422, 420, and 411 colour sampling. Our eyes are much more sensitive to luminance (the first number 4) than they are to colour (the other two numbers). 422 in a lot if areas (ie. broadcast) is deemed as having enough colour information for the image to look good and still be able to be manipulated, so the decision was made a long time ago to not bother trying to transmit/encode the extra colour present in the 444 image. This saves transmission bandwidth, processing power, file size requirements, etc.

This is what would be encoded as an uncompressed video, and in an indirect way, this is the first step in video compression (colour sub sample reduction).
Although it isn't really compression in and of itself, it aids in the actual compression steps, by reducing the amount of data the compression has to deal with.

2) Look for and simplify repeating patterns or colours. For example, if I have 50 pixels in a row that all have an RGB value if R217 G136 B75, I don't have to write the R217 G136 B75 individually 50 times. I can encode it as an expression that says 50 X R217 G136 B75, which takes up a lot less room. This would be the simplest form of lossless compression, similar to the compression scheme used in a zip file.

3) Look for pixels that are similar in value and encode the bunch as the average value. For example, if I have a series of pixels that are R217 G136 B75, R215 G136 B80, and R219 G136 B70, they could possibly all be encoded as R217 G136 B75, since it is the average value, assuming that the other values fall within the threshold of the evaluation algorithm. This series of pixels can form lines or blocks, depending on the codec. This would probably be the simplest form of a lossy compression, similar in concept to jpeg, motion jpeg or mpeg I-Frame compression.

More advanced compression also relies on other steps like DCT, RLE, temporal difference encoding, using larger sample areas/macro blocks, etc., but in all cases, video noise will reduce the efficiency of the compression, resulting in larger files, by throwing off the accuracy/uniformity of the chroma sub sampling, breaking repeating patterns into very small segments, or causing too much deviation in adjacent pixels, resulting in a signal/image that looks bad due to the non-uniform areas that are continually changing from frame to frame, and therefore have problems compressing both spatially and temporally.

Video noise is indeed the worst enemy of any video codec, regardless of the type of codec it is. Better colour sampling will not make the compression more efficient, but it will make the footage easier to manipulate accurately with effects such as a noise reduction plugin and then be able to compress further without major visible degradation, simply because there is more detail/information in what the plugin is trying to work with. It is the cleaner and/or more detailed source that allows the noise reduction to work better, that in turn will make the heavier compression codec work more efficiently without causing errors that are seen as compression artifacts.

That said, don't try to eliminate all traces of noise or grain, unless you want everything to look like it is made from plastic. ;-)
__________________
Edius WG 9.50.5351, various 3rd party plugins, VisTitle 2.8, Win 7 Ultimate SP1, i7-4790K @ 4GHz with HD4600 GPU embedded, MSI Z97 Gaming 7 Motherboard, 32GB Kingston HyperX RAM, nVidia GTX680 4GB GPU, Matrox MX02 Mini MAX, Corsair 750W PSU, Corsair H110i GT Water Cooler, Corsair C70 case, 4TB Internal RAID 0/stripe (2x2TB Seagate SATAIII HDD's, Win7 Software RAID 0/stripe), 1TB Crucial MX500 SSD, Pioneer BDR-207D, Dual 1920x1080 monitors (one on GTX680 and one on Intel HD4600).

Last edited by BernH; 02-20-2019 at 03:35 PM.
BernH is online now   Reply With Quote
Old 02-20-2019, 01:09 PM   #3
dpalomaki
Senior Member
 
Join Date: Oct 2007
Location: USA
Posts: 1,874
Default

The other factor that argues for lossless file formats when restoration or extensive filtering is in order is the reduction of cumulative processing artifacts that will arise due to compression/decompression cycles and round-off/truncation errors.

The old PCX graphics file was an early form of lossless compression that used a form of the the repeated value (run-length encoding) shorthand mentioned in BernH's above post.

Quote:
That said, don't try to eliminate all traces of noise or grain, unless you want everything to look like it is made from plastic.
Which is no doubt why every beauty with perfect complexion has added a beauty mark to her cheek. ;-)
dpalomaki is offline   Reply With Quote
Old 02-20-2019, 02:16 PM   #4
Bassman
Senior Member
 
Join Date: May 2007
Location: Texas
Posts: 2,392
Default

Nice thread. The longer I do this stuff the more I appreciate the HQX format. Hard drives are less expensive and the benefits are solid. Dave had a post a while back regarding HQX and how nice it is.

Compressed formats can be a pain after a while but it is all about time vs file size in the end. I have a few concerts in house now that needed Neat Video applied to the entire camera files. While nice it takes forever! Once you get it to HQX it edits like butter but then you have these monstrosities sitting on your edit drives. :)
__________________
Asus Prime X299-A - Intel i9 7900x all cores @4.3GHz 2 cores @4.6GHz - 32GB RAM - NVidia GTX670 - Edius 8.53.2808WG - BM Intensity 4k - Boris RED - Vitascene 2 - Windows 10
Bassman is offline   Reply With Quote
Old 02-20-2019, 02:44 PM   #5
BernH
Senior Member
 
BernH's Avatar
 
Join Date: Sep 2014
Location: St. John's, NL, Canada
Posts: 1,441
Default

Quote:
Originally Posted by dpalomaki View Post
The other factor that argues for lossless file formats when restoration or extensive filtering is in order is the reduction of cumulative processing artifacts that will arise due to compression/decompression cycles and round-off/truncation errors.
This is indeed a factor to consider in sequential compression/decompression cycles, but is less of a concern, although not completely eliminated, if all processing is done in one cycle. Regardless of that fact though, the lighter the compression in the source, often the faster and more accurately that cycle can be processed, simply because there is less/simpler decompression math to be done, and therefore less chance of an error in the decompression half of the cycle that can get translated into the compression half of the cycle.

As always, the "Holy Grail" for quality is working with and ultimately producing uncompressed video, but this can mean huge file sizes and the need for extremely fast storage in order to keep up with the bitrates of the huge files. If the ability of uncompressed is not a reality, then the next best thing is a high quality, lightly compressed, I-Frame, mezzanine codec such as our beloved HQX, DNxHD/HR, ProRes, etc. This is the very reason why codecs like those were created in the first place.

These are some of the reasons I always try to work with one of those as a source, but preferably HQX as a source and as a master codec, in addition the the fact that, since it is the native codec for Edius, it is by far the easiest source to edit, as Tim just alluded to.
__________________
Edius WG 9.50.5351, various 3rd party plugins, VisTitle 2.8, Win 7 Ultimate SP1, i7-4790K @ 4GHz with HD4600 GPU embedded, MSI Z97 Gaming 7 Motherboard, 32GB Kingston HyperX RAM, nVidia GTX680 4GB GPU, Matrox MX02 Mini MAX, Corsair 750W PSU, Corsair H110i GT Water Cooler, Corsair C70 case, 4TB Internal RAID 0/stripe (2x2TB Seagate SATAIII HDD's, Win7 Software RAID 0/stripe), 1TB Crucial MX500 SSD, Pioneer BDR-207D, Dual 1920x1080 monitors (one on GTX680 and one on Intel HD4600).
BernH is online now   Reply With Quote
Old 02-21-2019, 07:46 AM   #6
Haddock
Senior Member
 
Join Date: May 2007
Location: Belgium
Posts: 1,145
Default

Quote:
Originally Posted by BernH View Post
This is indeed a factor to consider in sequential compression/decompression cycles, but is less of a concern, although not completely eliminated, if all processing is done in one cycle. Regardless of that fact though, the lighter the compression in the source, often the faster and more accurately that cycle can be processed, simply because there is less/simpler decompression math to be done, and therefore less chance of an error in the decompression half of the cycle that can get translated into the compression half of the cycle.

As always, the "Holy Grail" for quality is working with and ultimately producing uncompressed video, but this can mean huge file sizes and the need for extremely fast storage in order to keep up with the bitrates of the huge files. If the ability of uncompressed is not a reality, then the next best thing is a high quality, lightly compressed, I-Frame, mezzanine codec such as our beloved HQX, DNxHD/HR, ProRes, etc. This is the very reason why codecs like those were created in the first place.

These are some of the reasons I always try to work with one of those as a source, but preferably HQX as a source and as a master codec, in addition the the fact that, since it is the native codec for Edius, it is by far the easiest source to edit, as Tim just alluded to.
Very interesting subjects and advices...
__________________
Yvon durieux alias "Haddock" Belgium GMT + 1

Sorry for my poor english, I am french native speaking

Main System: Azus Z87 Pro, 4770K@3.5Ghz, 16gb ram, Nvidia GeForce GT 630, Windows 7 Pro 64, Samsung 840 pro, Edius 8.53.2808 WG and 9.40.4896 + NXexpress or HDspark, 2T separate video SSD.
Haddock is offline   Reply With Quote
Reply
 
Go Back   Grass Valley Forums > Editors > Editing with EDIUS
 

Bookmarks

Thread Tools
Display Modes

Similar Threads
Thread Thread Starter Forum Replies Last Post
Video noise reduction Barry Wilkes Editing with EDIUS 9 10-06-2018 01:52 AM
Robuskey video noise in ProRes Edius 7.4 dlmonak Editing with EDIUS 20 11-13-2015 10:18 AM
Segment encoding Mpeg video noise mattie Editing with EDIUS 1 09-22-2008 10:29 PM
Edius 4.0 V Edius Broadcast quality and compression? rallyman2007 Editing with EDIUS 2 08-06-2008 01:37 PM

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are Off
[IMG] code is On
HTML code is Off

Forum Jump


All times are GMT +1. The time now is 01:00 PM.


Copyright 2014 Belden Inc. All rights reserved.