EEVblog #726 - Dual Xeon Video Editing Machine Build

[barnacle2k]

Like many in this thread already suggested:


Change your video encoding process, export uncompressed or lightly compressed (mpeg2) from your editing software and then shove the result into handbrake.
Yes that intermediate file will be absolutely humongous but you can delete it immediately after the handbrake pass is through.
That file should go onto an ssd because of the high datarate (100mbps+).

One thing on the pc build:
The GPU power cable looked not fully plugged into the power supply (they take a LOT of force to get right).

[elgonzo]

@barnacle2k, thank you for making sure that also people who broke their glasses or misplaced their contact lenses are able to read your post...

[true]

Why use one SSD for OS and the other for video? This is an offline render rig. Stripe the SSDs and use along with uncompressed / light compressed export from Vegas.

If the export is still slow, blame Vegas.

(But, from the video, there are other PEBKAC-type issues...)

[Armxnian]

Some quick tips:

1. Connect the 2nd cpu 8pin... it doesn't actually draw more power unless it is needed. The motherboards for 6 core processors all have an 8 pin 12v header, so why would you think two 6 cores don't require 2 connectors?

2. ECC ram is useless for video encoding/rendering, and is slower than non-ecc ram as it has to detect potential data corruption. 1333mhz is also bottom of the range for ddr3.

3. x264 can use as many processors/cores/threads as the OS supports. The Sony application itself isn't what needs to support x amount of cores, it's the codec itself. You will find some are single threaded, others can use as much as you have. All professional codecs should support 24 threads.

I never really understood your choice for 60/50fps Dave. Yeah people say they like it, but that's compared to 25fps. Why not invest in a 4k camera? Youtube 4k isn't the best right now but it still looks better than 1080p, and will only improve. I don't see the point of 50/60fps for electronic videos, where you're staring at something that isn't moving.

[muffenme]

The real reason is that Hyper-Threading may be Killing your Parallel Performance.  http://www.pugetsystems.com/blog/2014/07/02/Hyper-Threading-may-be-Killing-your-Parallel-Performance-578/ Dr Donald Kinghorn wrote about the problem with Hyper-Threading.

[EEVblog]

So, why did Dave not use all the DIMMs he already has?

Because the manual told me not to do that.

[true]

1. Connect the 2nd cpu 8pin... it doesn't actually draw more power unless it is needed. The motherboards for 6 core processors all have an 8 pin 12v header, so why would you think two 6 cores don't require 2 connectors?

Why would you think it does? It works. Dave isn't a computer guy, he's an electronics guy. I am sure the CPU and memory controller won't be burdened with his CPU choice and one connection is fine.

2. ECC ram is useless for video encoding/rendering, and is slower than non-ecc ram as it has to detect potential data corruption. 1333mhz is also bottom of the range for ddr3.

Not sure you understand how memory works. For his use case, any RAM performance difference between different types or speeds of modules for his RAM channel count is almost non-existent. Likewise, 1333MHz vs 1600MHz vs faster won't make much of a difference for his use case. As stated before, using more RAM channels may improve performance.

3. x264 can use as many processors/cores/threads as the OS supports. The Sony application itself isn't what needs to support x amount of cores, it's the codec itself. You will find some are single threaded, others can use as much as you have. All professional codecs should support 24 threads.

Codec support is likely limited by Vegas. I haven't used Vegas in some time - if there is a way to use x264 encoder through Vegas export that might be easiest for Dave but really he should be exporting to an uncompressed or lightly compressed output to fast storage and transcoding that. OR see Bud's post below regarding using a frameserver and tools Dave is already familiar with.

I never really understood your choice for 60/50fps Dave. Yeah people say they like it, but that's compared to 25fps. Why not invest in a 4k camera? ...

...uh

[rs20]

I never really understood your choice for 60/50fps Dave. Yeah people say they like it, but that's compared to 25fps.

Um, yeah.... what frame rate are you suggesting?

Why not invest in a 4k camera? Youtube 4k isn't the best right now but it still looks better than 1080p, and will only improve. I don't see the point of 50/60fps for electronic videos, where you're staring at something that isn't moving.

Most people's monitors are barely in excess of Full HD right now, and in any case 4k would put huge demands on your optics (both the camera and the end-viewer) and offer no advantage to anyone sitting more than 30cm from the screen. In summary, 4k makes no sense, 50 fps is beautiful. Also, Dave moves quite a lot.

[Bud]

Lots of great advices here, so i feel obligated to give one, too:

Dave should alltogether skip rendering step in Vegas and stream straight to Handbrake via a frameserver.

http://www.vegasvideo.de/vegas-2-handbrake-en.html

Edit: sorry mean Encoding step

[true]

So, why did Dave not use all the DIMMs he already has?

Because the manual told me not to do that.

The manual also told you to use the same size and type throughout the system "for best performance" but you aren't doing that either. But it did _not_ say you could not do this.

Use motherboard manuals for advice, not stringent guidelines. Same for any modern PC hardware. What is supported and what works fine are different things. In fact, for your configuration the manual section 2-4 explicitly identifies a 2 CPU + 6 DIMM loadout. I understand your DIMMs are different brands and sizes but this mixing, even though not ideal, should still work. All of your memory is registered ECC. The recommendation from Supermicro in the manual is sound (4 DIMM in CPU1, 2 DIMM in CPU2) and may make more sense than the one stated in this forum (3 DIMM in each) but your software is the limitation so both are worth trying out. There is no risk of frying things by trying.

To reiterate, 3-channel mode should work. 4ch in CPU1 and 2ch in CPU2 is recommended by SuperMicro. You can mix modules. If it doesn't work or shows no improvement, you can remove the RAM.

[rs20]

Lots of great advices here, so i feel obligated to give one, too:

Dave should alltogether skip rendering step in Vegas and stream straight to Handbrake via a frameserver.

http://www.vegasvideo.de/vegas-2-handbrake-en.html

Edit: sorry mean Encoding step

Your message made me wonder... why is everyone assuming that the codecs are to blame*? The evidence in Dave's video shows the CPU utilization dropping when the video transitions from requiring 60 --> 50 translation to just compressing 50 fps. How does the codec even know what the input file frame rate is, let alone have its performance be affected by it? Observed behaviour seems consistent with the rendering step being limited to having, say, 8 frames in flight at a time, leaving the codec starved of input.

                    +-----------+                  +-------+         
  50/60 fps frames  |           |   50fps frames   |       |         
+-----------------> | Rendering | +--------------> | Codec | +------>
                    |           |                  |       |         
                    +-----------+                  +-------+         

A frameserver may be a neat way to test this theory, because I assume that you can tell frameserver to just dump the frames to /dev/null, if you do this and the "rendering" still takes ages and has CPU utilisation all over the place, then we've found our culprit: the video editing software isn't rendering (as opposed to encoding/compressing) enough frames in parallel.

* Disclaimer: I haven't really been reading these threads, so it's probably false that everyone is assuming this.

[george graves]

Yes.  Frameservers are another option.  I've done that in the past.  But it's kinda of a PITA.  More professional software will allow you to send a project to the encoding software - either directly, or via a quicktime reference file (as I already showed in this thread).  Export and then encode for youtube is a waist of time.

[Armxnian]

1. Connect the 2nd cpu 8pin... it doesn't actually draw more power unless it is needed. The motherboards for 6 core processors all have an 8 pin 12v header, so why would you think two 6 cores don't require 2 connectors?

Why would you think it does? It works. Dave isn't a computer guy, he's an electronics guy. I am sure the CPU and memory controller won't be burdened with his CPU choice and one connection is fine.

2. ECC ram is useless for video encoding/rendering, and is slower than non-ecc ram as it has to detect potential data corruption. 1333mhz is also bottom of the range for ddr3.

Not sure you understand how memory works. For his use case, any RAM performance difference between different types or speeds of modules for his RAM channel count is almost non-existent. Likewise, 1333MHz vs 1600MHz vs faster won't make much of a difference for his use case. As stated before, using more RAM channels may improve performance.

3. x264 can use as many processors/cores/threads as the OS supports. The Sony application itself isn't what needs to support x amount of cores, it's the codec itself. You will find some are single threaded, others can use as much as you have. All professional codecs should support 24 threads.

Codec support is likely limited by Vegas. I haven't used Vegas in some time - if there is a way to use x264 encoder through Vegas export that might be easiest for Dave but really he should be exporting to an uncompressed or lightly compressed output to fast storage and transcoding that. OR see Bud's post below regarding using a frameserver and tools Dave is already familiar with.

I never really understood your choice for 60/50fps Dave. Yeah people say they like it, but that's compared to 25fps. Why not invest in a 4k camera? ...

...uh

His two processors draw 80w each, I thought it was more towards 140w for each, so never mind on that point.

He seemed happy for a 20% gain. Non ecc and faster memory can improve performance, not dramatically, but it can.

As for uncompressed, you need to test it. x264 might be faster since it does not have to decode the stream but it has more data to process. You can also try just using a lower bitrate straight out of Sony and not encoding a 2nd time. I doubt the quality of Sony's codec at low bitrates though.

I never really understood your choice for 60/50fps Dave. Yeah people say they like it, but that's compared to 25fps.

Um, yeah.... what frame rate are you suggesting?

Why not invest in a 4k camera? Youtube 4k isn't the best right now but it still looks better than 1080p, and will only improve. I don't see the point of 50/60fps for electronic videos, where you're staring at something that isn't moving.

Most people's monitors are barely in excess of Full HD right now, and in any case 4k would put huge demands on your optics (both the camera and the end-viewer) and offer no advantage to anyone sitting more than 30cm from the screen. In summary, 4k makes no sense, 50 fps is beautiful. Also, Dave moves quite a lot.

I'm suggesting 25/30fps at 4k. You don't need a 4k monitor to get increased quality. 4k downsampled to 1080p is going to look better than 1080p.

30cm? Where did you pull that figure out of? Did you forget to factor in screen size, eye sight, and overall subjective bias? I wear glasses and sit 2ft. away from my monitor and can notice the difference of 4k vs 1080p on a 1080p monitor. The difference is more on native 4k.

In summary, 50fps makes no sense. You are staring at a board of test instrument for 95% of the video. The only thing moving is Dave's hand when he points at things. Huge demands on your optics? You don't have to watch it at 4k, 1080 will be available. 50fps is a waste of bandwidth if you really want to get picky about network strain. 4k monitors are getting relatively cheap and more are adapting, no point in getting comfy with high frame rate with eevblog type content.

[EEVblog]

1. Connect the 2nd cpu 8pin... it doesn't actually draw more power unless it is needed. The motherboards for 6 core processors all have an 8 pin 12v header, so why would you think two 6 cores don't require 2 connectors?

I don't have a 2nd cable.

[EEVblog]

Or maybe its time to look for another editing software and re-learn your workflow. Almost any editing software do the job you need, since you only need the basics, just trim and join clips looking at waveforms, and text overlay.

What makes you think I haven't.
In fact I have, many times, and I have not always used Sony.
I've tried every package that everyone has suggested, including the high end professional ones everyone swears by, and they ALL have issues in some way that have caused me not to take them up beyond the sometimes one trial video
I have downloaded the latest Cyberlink power director to try (again, have tried it years ago, it was not suitable) as it now promises lighting speed and several have testified to that.

[EEVblog]

In summary, 50fps makes no sense.

Ok, I'll just switch to the 25/30fps 4K cameras I have.
Oh, that's right, I don't have 4K cameras 

[Armxnian]

In summary, 50fps makes no sense.

Ok, I'll just switch to the 25/30fps 4K cameras I have.
Oh, that's right, I don't have 4K cameras 

I said think about it for a future investment. No one "needs" 4k, you provide high resolution images of teardowns. At the same time, no one "needs" 50fps when looking at a non moving object on a non moving camera. It's not like you magically miss out on information on a whiteboard if you watch it in 25fps. I'm simply stating the fact that for optimal quality, higher resolution > higher frame rate for your style and content. Those who say there is no noticeable difference in image quality in 1080 vs 4k are the same who said there is no difference in 720 vs 1080 and 480 vs 720 a few years ago. If there is no difference, an entire industry wouldn't make the jump as the technology becomes available.

You've mentioned often that more content and higher quality content = more success. You went from 4x real time to 1/3rd real time. If it actually effects the amount of content you put out it's not worth it. 4k could be worth it.

[Dr. Frank]

Dave,

I really enjoyed your video, just sitting in front of my self assembled, downgraded 20W  µATX / E350 PC..which barely manages to display this nice video in HD.

That Xeon machine is a really wonderful big beast.. and I also can understand your disappointment.. encountered that also quite often..

Good luck in finding the tin-opener for appropriate performance!

Frank

[Stonent]

Another thing to keep in mind: LGA2011 CPUs have a quad-channel memory controller so unless you put four similar DIMMs on each CPU, you are only enabling half of the RAM bandwidth each CPU is capable of. This could be a massive bottleneck when all 24 threads are enabled.

Would also probably be best that each CPU has an identical memory configuration to the other.  I think the Dell Precision Workstations with Xeon cpus will warn you if your memory layout is not optimal at start-up.

[Psi]

I was very surprised the OS actually booted when moved from a generic single CPU machine to a radically different server grade motherboard with dual cpu and totally different bios.

Usually the windows kernel/reg settings are fixed at install time for your chipset (AMD vs Intel) and your multiprocessor architecture.

There's so many things that could have caused a BSOD doing that i am amazed it actually worked.
I've done it before myself but i've only had luck when the computers were similar.

[EEVblog]

I said think about it for a future investment. No one "needs" 4k, you provide high resolution images of teardowns. At the same time, no one "needs" 50fps when looking at a non moving object on a non moving camera.

Sure, but:
a) Countless people have commented on the excellent quality of the 50fps footage. Who are you or anyone else to tell them that they don't need it? trust me, if hardly anyone saw the difference or didn't mention it then I wouldn't bother doing it. But they do, and I've been inundated with people saying they love it, so I've kept doing it.

b) I can do 50fps now with the gear I have invested in, which is why I am doing it. It doesn't cost me anything except render time. 4K on the other hand is a massive investment in all new cameras (multiple), all new workflows (so was 50fps, but not in new camera workflow, just rendering), almost certainly a much higher bitrate than I'm running now (greatly increases my expense and effort for raw footage backup), perhaps new 4K monitors to match, etc.

It's not like you magically miss out on information on a whiteboard if you watch it in 25fps. I'm simply stating the fact that for optimal quality, higher resolution > higher frame rate for your style and content.

I do not disagree at all.

Those who say there is no noticeable difference in image quality in 1080 vs 4k are the same who said there is no difference in 720 vs 1080 and 480 vs 720 a few years ago. If there is no difference, an entire industry wouldn't make the jump as the technology becomes available.

I have been considering moving to 4K for while now, but was disappointed in the camera releases at the CES in Jan, so did not make the switch.

If it actually effects the amount of content you put out it's not worth it.

It doesn't really, it's just rendering time. Screws up getting videos to market quickly, e.g. Shooting Mailbag on Monday and releasing it on a Monday, but apart from that, no.
BTW, the great thing about Sony Moviestudio/Vegas is that I can (and sometimes do) run more than one instance at once. e.g. I can be editing another video while the previous one is rendering. I think I once tweeted a photo of me with 3 instances running at once for something I was doing.

[EEVblog]

UPDATE:
1) 3 DIMM's does work. Not any faster

2) Hyperthreading is confirmed slower with Handbrake

3) Direct uncompressed output using the Sony YUV Video For Windows codec actually works (it had issues on my old machine)
1:46 for the 1min test video which is pretty good and the fastest yet. That was to the SSD. File size was 12.163GB for the 1 min, so that means a 1 hour video would need 730GB.
Such an output would need about 114MB/s write speed, so that makes a 7200 rpm drive a WD Black suitable. 1GB+ of SSD would of course be very expensive and of no improvement.
So this seems like the best solution for now.
I might try some of the other VFW codecs again, but I've been advised by top people close to the metal that the Sony CODEC is pretty darn good at this.

4) x264vfw still does not work. Major problems with output file audio/video sync. But FYI, render time was 2:27 for the 1min video. For those not aware, x264vfw is essentially directly streamaing the output from Sony to handbrake (which uses the x264 codec as well)

[bktemp]

3) Direct uncompressed output using the Sony YUV Video For Windows codec actually works (it had issues on my old machine)
1:46 for the 1min test video which is pretty good and the fastest yet. That was to the SSD. File size was 12.163GB for the 1 min, so that means a 1 hour video would need 730GB.
Such an output would need about 114MB/s write speed, so that makes a 7200 rpm drive a WD Black suitable. 1GB+ of SSD would of course be very expensive and of no improvement.
So this seems like the best solution for now.
I might try some of the other VFW codecs again, but I've been advised by top people close to the metal that the Sony CODEC is pretty darn good at this.

You could try the Lagarith Lossless Video Codec. It probably compresses a bit better and is quite fast.
http://lags.leetcode.net/codec.html

[rs20]

3) Direct uncompressed output using the Sony YUV Video For Windows codec actually works (it had issues on my old machine)...

Purely out of curiosity, how did CPU usage trend using this codec? Are the CPUs still starved when going from 50fps to 50fps?

[EEVblog]

Purely out of curiosity, how did CPU usage trend using this codec? Are the CPUs still starved when going from 50fps to 50fps?

Yes, they are, very little use, so it basically seems to become primarily an I/O issue when doing it this way. Although I'm sure the CPU and memory width helps also.
Full pelt CPU at 60fps to 50fps.
I just realised the source video material was coming from a slow 5400 hard drive, so not sure if that impacts that or whether or not the clips were already in memory or however it works.
Also, in this case the drives should be put on the two SATA3 ports. Currently on the SATA2 ports I think.