1. Create a submit file.
Sample Script
# first-job.cmd # Universe should always be 'vanilla' unless you know explicitly what you're doing universe = vanilla # this sets the environment variables of the login environment to the compute node environment # generally, you will want to leave this set to true getenv = true # this is the command or program you are running. # common executables and their locations are: # R: /usr/bin/Rscript # matlab: /opt/matlab/latest/bin/matlab # python 2: /opt/python/latest/bin/python # python 3: /opt/python3/latest/bin/python # many other applications can be found in /opt # anaconada 4: /home/kross48/anaconda3 executable = $ENV(HOME)/gurobi/lp.py # this is a space delimited list of arguments sent to the program listed above. the path to and name of your script go here for example, along with any other arguments # if you need to run multiple copies of your job you can pass $(Process) as an argument which will increment by one, starting at zero, for every process spawned # for any files that you are using, you should pass the entire path. for example, for a file 'script.py' in your home directory use: #arguments = test$(ProcID).lp should_transfer_files = yes when_to_transfer_output = on_exit # this is the name and location of the Condor logs for your job. This will show details of what Condor itself is doing in relation to your job, but not the output of your job (for that, see the next entries) log = $ENV(HOME)/gurobi/$(Cluster).log # this is the name and location of the actual output from your job. # output will contain whatever would've been sent to the screen (aka STDOUT) # error will contain any errors generated by your application (aka STDERR) # the variable $(cluster) will generate one number per run of "condor_submit" # the variable $(process) will generate a unique number for every job generated within this script, starting at zero output = $ENV(HOME)/gurobi/$(Cluster).$(ProcId).out error = $ENV(HOME)/gurobi/$(cluster).$(ProcId).error # these options will notify the email listed below if the job errors out or when it completes # comment or delete these lines if you do not wish to get an email # you will get an email for each instance of your job, so if you spawn 50 jobs you will get 50 emails. # use notification = none if you do not want to get any email about the job notification = error notification = complete # If you need a specific amount of memory, Change this number to the minimum amount of RAM you need for your job. This number is in megabytes. request_memory = 1024 # If you need more than one CPU, Uncomment and change this number to the number of cores you need. The more CPUs you request the longer the job will take to queue. request_cpus = 1 # the queue command is REQUIRED # if you need to run this more than once, enter the number of copies after the queue command # for every copy spawned, the variable $(process) will increment by one, starting at zero. # if you only wish this to run once, leave the queue command with no number. # for example "queue 50" will spawn 50 copies of this job queue 3
2. The following is a sample gurobi python script.
#!/opt/gurobi/latest/linux64/bin/python3.7 #copyright 2020, Gurobi Optimization, LLC # This example reads an LP model from a file and solves it. # If the model is infeasible or unbounded, the example turns off # presolve and solves the model again. If the model is infeasible, # the example computes an Irreducible Inconsistent Subsystem (IIS), # and writes it to a file import sys import gurobipy as gp from gurobipy import GRB if len(sys.argv)
3. Now, submit your job to the queue using condor_submit.
[kross48@isye-hps0401 gurobi]$ condor_submit first-job.cmd Submitting job(s)... 3 job(s) submitted to cluster 116013.
4. To check on the status of your jobs, run the following command.
[kross48@isye-hps0401 gurobi]$ condor_q
(If you want to see everyone's jobs, use condor_q -all.)
The output of condor_q should look like this:
[kross48@isye-hps0401 gurobi]$ condor_q -- Schedd: isye-hps0401.isye.gatech.edu :
You can run the condor_q command periodically to see the progress of your jobs. By default, condor_q shows jobs grouped into batches by batch name (if provided), or executable name. To show all of your jobs on individual lines, add the -nobatch option. For more details on this option, and other options to condor_q, see our condor_q guide.
5. When your jobs complete after a few minutes, they'll leave the queue. If you do a listing of your home directory with the command ls -l, you should see something like:
And, if you look at one of the output files, you should see something like this:
[kross48@isye-hps0401 gurobi]$ ls -l total 140 -rw-r--r--. 1 kross48 gtperson 0 Apr 16 17:13 116013.0.error -rw-r--r--. 1 kross48 gtperson 1413 Apr 16 17:13 116013.0.out -rw-r--r--. 1 kross48 gtperson 0 Apr 16 17:13 116013.1.error -rw-r--r--. 1 kross48 gtperson 1079 Apr 16 17:13 116013.1.out -rw-r--r--. 1 kross48 gtperson 0 Apr 16 17:13 116013.2.error -rw-r--r--. 1 kross48 gtperson 1079 Apr 16 17:13 116013.2.out -rw-r--r--. 1 kross48 gtperson 3365 Apr 16 17:13 116013.log
6. Useful information is provided in the user log and the output files.
HTCondor creates a transaction log of everything that happens to your jobs. Looking at the log file is very useful for debugging problems that may arise.
[kross48@isye-hps0401 gurobi]$ cat 116013.log
000 (116013.000.000) 04/16 17:13:24 Job submitted from host:
...
000 (116013.001.000) 04/16 17:13:24 Job submitted from host:
...
000 (116013.002.000) 04/16 17:13:24 Job submitted from host:
...
001 (116013.000.000) 04/16 17:13:41 Job executing on host:
...
001 (116013.002.000) 04/16 17:13:41 Job executing on host:
...
001 (116013.001.000) 04/16 17:13:41 Job executing on host:
...
006 (116013.001.000) 04/16 17:13:42 Image size of job updated: 69392
0 - MemoryUsage of job (MB)
0 - ResidentSetSize of job (KB)
...
006 (116013.002.000) 04/16 17:13:42 Image size of job updated: 2
0 - MemoryUsage of job (MB)
0 - ResidentSetSize of job (KB)
...
006 (116013.000.000) 04/16 17:13:42 Image size of job updated: 69392
0 - MemoryUsage of job (MB)
0 - ResidentSetSize of job (KB)
...
005 (116013.001.000) 04/16 17:13:42 Job terminated.
(1) Normal termination (return value 0)
Usr 0 00:00:00, Sys 0 00:00:00 - Run Remote Usage
Usr 0 00:00:00, Sys 0 00:00:00 - Run Local Usage
Usr 0 00:00:00, Sys 0 00:00:00 - Total Remote Usage
Usr 0 00:00:00, Sys 0 00:00:00 - Total Local Usage
0 - Run Bytes Sent By Job
0 - Run Bytes Received By Job
0 - Total Bytes Sent By Job
0 - Total Bytes Received By Job
Partitionable Resources : Usage Request Allocated
Cpus : 1 1
Disk (KB) : 2 2 946345
Gpus : 0
Memory (MB) : 0 1048 1152
...
005 (116013.002.000) 04/16 17:13:42 Job terminated.
(1) Normal termination (return value 0)
Usr 0 00:00:00, Sys 0 00:00:00 - Run Remote Usage
Usr 0 00:00:00, Sys 0 00:00:00 - Run Local Usage
Usr 0 00:00:00, Sys 0 00:00:00 - Total Remote Usage
Usr 0 00:00:00, Sys 0 00:00:00 - Total Local Usage
0 - Run Bytes Sent By Job
0 - Run Bytes Received By Job
0 - Total Bytes Sent By Job
0 - Total Bytes Received By Job
Partitionable Resources : Usage Request Allocated
Cpus : 1 1
Disk (KB) : 2 2 946345
Gpus : 0
Memory (MB) : 0 1048 1152
...
005 (116013.000.000) 04/16 17:13:42 Job terminated.
(1) Normal termination (return value 0)
Usr 0 00:00:00, Sys 0 00:00:00 - Run Remote Usage
Usr 0 00:00:00, Sys 0 00:00:00 - Run Local Usage
Usr 0 00:00:00, Sys 0 00:00:00 - Total Remote Usage
Usr 0 00:00:00, Sys 0 00:00:00 - Total Local Usage
0 - Run Bytes Sent By Job
0 - Run Bytes Received By Job
0 - Total Bytes Sent By Job
0 - Total Bytes Received By Job
Partitionable Resources : Usage Request Allocated
Cpus : 1 1
Disk (KB) : 2 2 946345
Gpus : 0
Memory (MB) : 0 1048 1152
...
Congratulations. You've run your first jobs in the HTCondor!