代码之家 › 专栏 › 技术社区 › Charlie Parker

使当前文件和扩展名与condor匹配输出和错误文件(使PBS和Slurm具有相同的输出文件)

condor pbs slurm hpc

0

Charlie Parker · 技术社区 · 4 年前

如何将condor命名为我的文件,如下所示:

元学习实验提交.py.e451863元学习实验提交.py.o444375

$(FILENAME).e$(CLUSTER)
$(FILENAME).e$(CLUSTER)

我试过了,但似乎不起作用。

e、所以当我这样做的时候,它符合PBS的默认行为 qsub ?

我还想知道如何使slurm也匹配它,这将是完美的(所以所有三个集群都在工作)。

我试过:

#SBATCH --error="(filename).%j.%N.err"

但那没用

赏金

我想自动设置文件名而不需要硬编码。

相关:

https://www.quora.com/unanswered/How-do-I-make-a-current-file-the-extension-with-condor-match-output-error-files-to-have-PBS-and-Slurm-have-the-same-output-files

对于上下文,这里是我当前的提交文件:

####################
#
# Experiments script
# Simple HTCondor submit description file
#
#
# chmod a+x test_condor.py
# chmod a+x experiments_meta_model_optimization.py
# chmod a+x meta_learning_experiments_submission.py
# chmod a+x download_miniImagenet.py
# chmod a+x ~/meta-learning-lstm-pytorch/main.py
# chmod a+x /home/miranda9/automl-meta-learning/automl-proj/meta_learning/datasets/rand_fc_nn_vec_mu_ls_gen.py
# chmod a+x /home/miranda9/automl-meta-learning/automl-proj/experiments/meta_learning/supervised_experiments_submission.py
# chmod a+x /home/miranda9/automl-meta-learning/results_plots/is_rapid_learning_real.py
# condor_submit -i
# condor_submit job.sub
#
####################

Path = /home/miranda9/automl-meta-learning/
Path = /home/miranda9/ML4Coq/

# Executable = /home/miranda9/automl-meta-learning/automl-proj/experiments/meta_learning/supervised_experiments_submission.py
# Executable = /home/miranda9/automl-meta-learning/automl-proj/experiments/meta_learning/meta_learning_experiments_submission.py
# Executable = /home/miranda9/meta-learning-lstm-pytorch/main.py
# Executable = /home/miranda9/automl-meta-learning/automl-proj/meta_learning/datasets/rand_fc_nn_vec_mu_ls_gen.py
# Executable = /home/miranda9/automl-meta-learning/results_plots/is_rapid_learning_real.py

## Output Files
Log          = experiment_output_job.$(CLUSTER).log.out
Output       = experiment_output_job.$(CLUSTER).out.out
Error        = experiment_output_job.$(CLUSTER).err.out

Output = %(FILENAME).o$(CLUSTER)

# Use this to make sure 1 gpu is available. The key words are case insensitive.
# REquest_gpus = 1
# requirements = (CUDADeviceName != "Tesla K40m")
# requirements = (CUDADeviceName == "Quadro RTX 6000")

# requirements = ((CUDADeviceName = "Tesla K40m")) && (TARGET.Arch == "X86_64") && (TARGET.OpSys == "LINUX") && (TARGET.Disk >= RequestDisk) && (TARGET.Memory >= RequestMemory) && (TARGET.Cpus >= RequestCpus) && (TARGET.gpus >= Requestgpus) && ((TARGET.FileSystemDomain == MY.FileSystemDomain) || (TARGET.HasFileTransfer))
# requirements = (CUDADeviceName == "Tesla K40m")
# requirements = (CUDADeviceName == "GeForce GTX TITAN X")

# Note: to use multiple CPUs instead of the default (one CPU), use request_cpus as well
# Request_cpus = 4
Request_cpus = 16

# E-mail option
Notify_user = me@gmail.com
Notification = always

Environment = MY_CONDOR_JOB_ID= $(CLUSTER)

# "Queue" means add the setup until this line to the queue (needs to be at the end of script).
Queue

或者,如果我可以使用我的可执行脚本作为提交脚本,上面的参数也可以

#!/homes/miranda9/.conda/envs/automl-meta-learning/bin/python
#PBS -V
#PBS -M mee@gmail.com
#PBS -m abe
#PBS -lselect=1:ncpus=112

import sys
import os

for p in sys.path:
    print(p)

print(os.environ)

0 回复 | 直到 4 年前

1

0

Greg 4 年前

HTCondor默认不设置文件名,但您可以自己设置,例如。

FILENAME = meta_learning_experiments_submission.py
output = $(FILENAME).o.$(CLUSTER)
error  = $(FILENAME).e.$(CLUSTER)