Many ISyE research problems involve running many separate iterations of the same code with either random or predetermined inputs. Condor excels at these types of problems.

When running on a single computer, the most common solution is set up loops and let just one or a few processes iterate through the inputs (sometimes generating random inputs along the way).

However, it is much more convenient and faster to let Condor handle the looping. If you can feed Condor multiple small jobs - anywhere from a minute to an hour in expected run time - it can spread them out onto all available resources and finish them very quickly.

Example 1: C program

We need to process 1000 different sets of input constraints using a compiled executable named "program0", written in C with the first argument being the input file.

First in our working directory we do 'mkdir in; mkdir out'. Put the data in 1000 separate files in the input directory, with names containing the integers 0 through 999. Here we will use 0.in through 999.in.

Then create the Condor submit file, 'program0.cmd':

universe    = vanilla
executable  = program0
arguments   = in/$(Process).in
output      = out/$(Process).out
error       = out/$(Process).err
queue 1000
Submit this script, and you should have 1000 jobs queued up that will iterate through your data and create 1000 output files.

Example 2 : Matlab

First, we need for our Matlab program to have an input variable for the process number, taken from Condor's $(Process) macro. Then we use that variable within the Matlab program to determine the name of the file to read.

Then the Condor submit file looks like this:

universe = vanilla
executable  = /opt/matlab/latest/bin/matlab
arguments   = -singleCompThread -r clear -r procnum=$(Process)
input       = program0.m
output      = out/$(Process).out
error       = out/$(Process).err
queue 1000

Please note: Make sure to NOT setup notifications on jobs like these. In the above example you will get 1000 emails in your inbox even though you only submitted once.

Example 3 : gurobi

Submitting Multiple jobs Using Queue Statements

Then the Condor submit file looks like this:

universe = vanilla
executable  =  lp.py
arguments   =  $(file)
Log = out/$(Cluster).log
output      = out/$(Process).out
error       = out/$(Process).err
request_memory  =  1024
request_cpu  =  1
queue file from testlp.txt

Example 4 : gurobi

Submitting Multiple jobs Using InitialDir

Then the Condor submit file looks like this:

universe = vanilla
initialdir = job$(Process)
executable  =  lp.py
arguments   =  file$(Process).in file$(Process).out
transfer_input_files = input /file$(Process).in
log = log /job$(Process).log
output = out/$(Process).out
error = err /job$(Process).err
request_memory  =  1024
request_cpu  =  1
queue 3

Random inputs:

The above examples assume that the user has a predetermined set of input data to use, but what about generating the data at random in the program?

That can be done, but there is a major catch: the random seed needs to be the same to ensure valid randomness.

This could be accomplished by importing a constant random seed within the program. But the far easier method is to use a single program to generate the random input data you need, then set it up as in the examples above.

Search