I regularly use the VMAF high quality metric to research encoding parameters and examine codecs. In my expertise, VMAF scores are likely to match my very own subjective high quality observations and the truth that Netflix makes use of it provides the metric nice credibility. Initially, there was just one VMAF model, however Netflix has added each a 4K mannequin and a telephone mannequin, growing the metrics’ utility.
For probably the most half, I compute VMAF scores with both the Moscow State College Video High quality Measurement Device (VQMT) or the Hybrik cloud encoder which has a really strong analytical engine. Nevertheless, to collaborate with a colleague who had neither of those instruments, we used one other software which is on the market free from Netflix.
I name it VMAF-Grasp as a result of that’s the identify of the ZIP file you obtain from Github right here, although the Home windows executable that I used and downloaded from right here was referred to as vmafossexec.exe. It’s a command line utility that’s quick and practical, although missing a number of the creature comforts of instruments that you simply truly pay for. Extra on this under.
Because the identify suggests, you’ll be able to compute VMAF, but in addition the PSNR, SSIM, and MS SSIM metrics. On this fast assessment, I’ll make it easier to get the Home windows executable put in, present you how one can use all variations of this system, after which talk about its strengths and weaknesses.
Getting Up and Operating
All customers ought to obtain the code from Github right here as proven in Determine 1 after which extract the contents to a folder in your pc. Most Home windows customers will need to achieve this on their C: drives; Linux and Mac customers ought to decide the analogous drive on their methods.
Determine 1. Obtain VMAF Grasp from Github.
VMAF-Grasp was initially written for Linux and the Mac with the Home windows executable provided by some open-source contributors. You’ll need to compile your personal model for Linux and Mac; in case you don’t understand how I’m afraid it’s past my talent set. Determine it out and be a part of us within the following part.
Click on right here to obtain the Home windows executable. It will take you to the web page proven in Determine 2. Click on vmaf-win-x64.zip on the decrease left to start out the obtain.
Determine 2. Downloading the Home windows executable.
The zip will extract right into a folder entitled x64; it’s in all probability not important that it’s situated within the vmaf-master folder, however it’s going to make it straightforward to seek out should you drag it in there. Right here’s what the setup seems to be like on my C: drive.
Determine three. All of the information of their locations.
You possibly can see the x64 folder on the underside with the executable situated within the Launch subfolder. Word additionally the mannequin folder which you’ll additionally seek advice from within the command string.
VMAF-Grasp Command Syntax
Like all well-mannered command-line instruments, in case you sort -h after the executable identify you’ll see the assistance information. I present this in Determine four which reveals the required command string.
Determine four. Operating vmafossexec.exe.
Right here’s what this seems like outdoors of the Command window.
Utilization: vmafossexec.exe fmt width peak ref_path dis_path model_path [–log log_path] [–log-fmt log_fmt] [–disable-clip] [–disable-avx] [–psnr] [–ssim] [–ms-ssim] [–phone-model]
Right here’s an evidence of the switches that I used.
vmafossexec.exe – this calls this system.
fmt – this identifies the enter format of the 2 video information; have to be one among these — yuv420p, yuv422p, yuv444p, yuv420p10le, yuv422p10le, yuv444p10le
width peak – you bought these.
ref_path – path to the reference file and reference file.
dis_path – path to the “distorted” or compressed video file and file identify.
model_path – path to the mannequin and mannequin. As I write this, this have to be both the default mannequin (vmaf_v0.6.1.pkl) or the 4K mannequin (vmaf_4k_v0.6.1.pkl). You add the telephone mannequin by way of the –phone-model change proven under. Technically, the telephone mannequin is a customized model of the default mannequin, so select the default mannequin to make use of the telephone mannequin.
–log log_path – log file identify and path. If no path specified, the log file is saved within the folder with the distorted file.
–log-fmt – format for log file (have to be both XML or JSON). In case you don’t specify, this system shops out a CSV file in XML format.
–disable-clip – have requested Netflix; will add as soon as they reply.
–disable-avx – have requested Netflix; will add as soon as they reply.
–psnr – run the PSNR metric.
–ssim – run the SSIM metric.
–ms-ssim – run the SSIM metric.
–phone-model – run the telephone mannequin.
Prepping the Information
Observe that vmafossexec.exe solely works with YUV inputs so that you’ll need to convert your reference and distorted information to YUV format. FFmpeg is the device of selection and you may modify this command string in your conversion.
ffmpeg -y -i distorted.mp4 -pix_fmt yuv420p -vsync zero distorted.yuv
ffmpeg -y -i reference.mp4 -pix_fmt yuv420p -vsync zero reference.yuv
Clearly, it is best to substitute the precise extension for MP4 if crucial, and if FFmpeg doesn’t decode the file discover a totally different decoder to take action. Notice the pixel format used because you’ll enter that into your VMAF-Grasp command string.
Right here’s the command string I used throughout my current venture. Notice that each distorted and reference information are in the identical folder which additionally accommodates the batch file so there are not any paths to both file. Right here I’m calling the default mannequin, computing VMAF, PSNR, SSIM, and MS SSIM, and storing the leads to XML format in distorted.csv.
C:vmaf-masterx64Releasevmafossexec.exe yuv420p 1920 1080 reference.yuv distorted.yuv C:vmaf-mastermodelvmaf_v0.6.1.pkl –psnr –ssim –ms-ssim –log distorted.csv
Right here’s what it seems to be like within the Command window. When you paste the command line in you’ll see the message “Begin calculating VMAF rating…”
Determine 5. Computing, ready, and output.
Then, as soon as the computations are full, you’ll see the scores within the command window. This could take some time and there’s no progress meter. For instance, a six-second 1080p check file took about 80 seconds to compute on a fairly quick pocket book. Do the maths and because of this a 1-minute check file would take over 13 minutes.
Scanning the CSV File
In most main tasks, the subsequent step is to open the CSV file and replica and paste the outcomes right into a spreadsheet. I don’t like opening the CSV file in Excel as a result of it’s cumbersome, however in case you open the XML-formatted CSV in Notepad, you get the gobblygook you see in Determine 6.
Determine 6. The CSV in Notepad.
My software of selection for opening and grabbing the info is Notepad ++, a free software you’ll be able to obtain right here (Determine 7). As you possibly can see, the software codecs the XML construction and shade codes the info, making it straightforward to seek out.
Determine 7. I favor Notepad++ for VMAF-Grasp output.
Deploying the Telephone Mannequin
In the event you can comply with the command line within the second batch run in Determine eight, you’ll see that it accommodates the –phone-model change. The leads to the command window don’t specify that it’s the telephone mannequin, although clearly, you possibly can seek advice from the command string. Nevertheless, there doesn’t look like a telephone mannequin designator within the CSV outcomes file, although I’ll have missed that. So ensure you designate phone-model computations within the file identify, in any other case, once you return to your knowledge after a few weeks, you might have hassle checking out which computations have been accomplished with the telephone mannequin and which with out.
Determine eight. Computing scores with the telephone mannequin.
Word that the one rating that the telephone mannequin designator modified was VMAF; the others are the identical, indicating that the telephone mannequin computation is VMAF-only. Notice additionally that the rating is greater, reflecting that decrease high quality movies look higher on small screens. I discover the implications of this within the weblog publish, The VMAF Telephone Mannequin and Saving on Streaming to Cellular Viewers.
Going Past the Fundamentals
This describes VMAF-Grasp’s most elementary performance. For these trying to automate VMAF-Grasp, take a look at the VMAF Python Library which discusses batch operation and different outputs, together with JSON. There’s additionally a reference to an software referred to as ffmpeg2vmaf that permits you to keep away from the conversion to YUV step in your reference and distorted information. This web page additionally paperwork the best way to create and validate a dataset and practice a brand new mannequin, and there’s a BD-Price calculator right here. It appears as if VMAF-Grasp could be very tightly built-in into the Netflix encoding stack (I’m guessing right here) and Netflix is supplying you with a lot of what you could do the identical.
What doesn’t VMAF-Grasp do? Most significantly, as close to as I can inform, there’s no approach to synchronize the reference and distorted movies if they’re out of sync. It doesn’t occur that always, however typically an encoder will add or drop a body and if there’s no approach to get the frames in sync, you’ll be able to’t precisely measure high quality. You could possibly work round these points however it usually turns into an enormous time sink. Some instruments synchronize information routinely, like Elecard VideoQuest, and the Moscow State device enables you to sync manually in each GUI and batch operation.
The opposite main deficit is the lack to visualise the output as you see in Determine 9 from the MSU device, which compares VP9 and HEVC output at three Mbps. The general scores are literally shut, with HEVC edging VP9 94.83 to 94.05, an irrelevant distinction. Nevertheless, you see the downward inexperienced spikes within the determine which characterize brief however probably vital reductions in high quality. The VQMT Outcome Plot enables you to spot these drawback areas, and by clicking Present Body on the underside proper, you possibly can truly examine the distorted frames with the reference.
Determine 9. The Moscow State College VQMT End result Plot.
Hybrik Media Analyzer additionally has its charms (Determine 10). Not solely do you get a single-file graph, however you additionally get the usual deviation of the metric, a worthwhile measure of high quality variability.
Determine 10. The Hybrik Media Analyzer provides you graph and commonplace deviation.
To be truthful, body sync points seldom happen in a mature encoding workflow, and as soon as in manufacturing, you care extra about numbers than precise visible comparisons. So whereas these deficits may hinder VMAF-Grasp’s use in an experimental capability, neither impacts its utility inside an encoding workflow, which is the place Netflix has doubtless deployed it. If all you’re in search of is the power to blast out VMAF scores with out spending a grand or two, or are on the lookout for a strategy to tightly combine VMAF scoring into your encoding stack, VMAF-Grasp seems to be best.