A lot of people have set up their systems to utilize compressor’s Qmaster feature on their own system in order to use all available processor cores when encoding videos.
But what if you have multiple machines with access to file-level shared storage, like XSAN. All client computers can have access and write access to the SAN simultaneously, so how can we utilize this fact to process distributed encoding tasks across multiple computers?
The typical setup goes something like this; Setting up the local machine as a “QuickCluster with Services” and setting the cluster storage to something local or leaving the default.
When I was first setting up a system for distributed rendering via Qadministrator into a single client managed cluster, there was a lot of regular maintenance and command line resets for all the client machines. Often, the cluster would choke or freeze or clients would drop off unexpectedly. Sometimes I would have to stop what I was doing to rebuild the cluster from scratch altogether. But why not utilize one of the features built into Qmaster and Compressor to create a more flexible cluster environment?
Instead of forcing all the computers to adhere to the rules of a single managed cluster set up each machine on the SAN as an independent unmanaged QuickCluster with the “Include unmanaged services from other computers” checkbox checked. This allows for clients to manage their own contribution to the distributed encoding process. You can still utilize command line to turn on client services at night for example with just:
qmasterprefs -cluster on
There are a couple of other setup things you should do, like setting your cluster storage to somewhere on the SAN. Make sure this directory has full Read/Write permissions or better yet, ACL’s. Also, you will have to make sure you also have Read/Write permissions to whatever folder your source is located if you decide your target destination is source. Ideally your source material and destination should always be on the SAN in order to take full advantage of the cluster. If you want your client machines to administer their own contribution, an easy way for them to monitor and quickly change their status is by checking “Show Apple Qmaster service status in menu bar”. Finally, open Compressor > Preferences on each client and be sure to set to never copy files to/from cluster. This is done over the network interface you have selected and can be slow if you are transferring a massive quicktime to everyone. Besides, everyone already has access in a SAN environment, right?
Now when you submit a batch from Compressor, you will have to option of which cluster to send the batch to. Select your own cluster so you can monitor if needed, and check the box next to the cluster selection pulldown to Include unmanaged services. Submit your batch and you should be good to go. Now say edit03 has to drop off the cluster services to do some multicam editing. Simply stop services in the Qmaster System Preferences pane and in most cases, the segments assigned to edit03 will either be finished or reallocated to available services. Worst case is that current target won’t correctly reallocate and therefor won’t finish compressing completely and you have to do that one again. In a large environment, this probably isn’t an issue because there are so many computers involved that compressions are extremely fast and don’t overly strain a single machine anyway. In smaller environments, you can simply communicate your need to do large compressions and give everyone the option to drop off or keep services active.
I hope this helps some people get the most out of their equipment. I am a big fan of distributed processing. It just takes a bit of forethought and a little setup, but then you should be pretty happy with the results. If you want more command line resources for a distributed processing setup, refer to the Distributed Processing Setup Guide.