代码之家 › 专栏 › 技术社区 › lampShadesDrifter

bash脚本的多线程信号量(子进程)

bash

lampShadesDrifter · 技术社区 · 6 年前

对于类似信号量的结构,有没有任何方法/二进制?例如,在文件目录中循环运行固定数量的(后台)子进程(此处使用“子进程”而不是“线程”,因为使用附加的 &

我的实际用例是尝试使用一个名为 bcp 在CentOS 7上,将一组(可变大小)TSV文件写入远程MSSQL Server DB,并观察到运行太多线程时程序似乎有问题。有点像

for filename in $DATAFILES/$TARGET_GLOB; do

    if [ ! -f $filename ]; then
        echo -e "\nFile $filename not found!\nExiting..."
        exit 255
    else
        echo -e "\nImporting $filename data to $DB/$TABLE"
    fi

    echo -e "\nStarting BCP export threads for $filename"
    /opt/mssql-tools/bin/bcp "$TABLE" in "$filename" \
        $TO_SERVER_ODBCDSN \
        -U $USER -P $PASSWORD \
        -d $DB \
        $RECOMMEDED_IMPORT_MODE \
        -t "\t" \
        -e ${filename}.bcperror.log &
done
# collect all subprocesses at the end
wait

这将启动一个新的子进程 一下子 以不受限制的方式,似乎使每个子进程崩溃。我想看看在循环中添加一个类似于信号量的结构来锁定将要启动的子进程的数量是否会有所帮助。例如(这里使用一些非bash类的伪代码)

sem = Semaphore(locks=5)
for filename in $DATAFILES/$TARGET_GLOB; do

    if [ ! -f $filename ]; then
        echo -e "\nFile $filename not found!\nExiting..."
        exit 255
    else
        echo -e "\nImporting $filename data to $DB/$TABLE"
    fi

    sem.lock()
    <same code from original loop>
    sem.unlock()

done
# collect all subprocesses at the end
wait

3 回复 | 直到 6 年前

muru Rider44 6 年前

这不是严格等价的,但您可以使用 xargs 要同时启动给定数量的进程:

-P max-procs, --max-procs=max-procs
      Run  up  to max-procs processes at a time; the default is 1.  If
      max-procs is 0, xargs will run as many processes as possible  at
      a  time.   Use the -n option or the -L option with -P; otherwise
      chances are that only one exec will be  done.   While  xargs  is
      running,  you  can send its process a SIGUSR1 signal to increase
      the number of commands to run simultaneously, or  a  SIGUSR2  to
      decrease  the  number.   You  cannot decrease it below 1.  xargs
      never terminates its commands; when asked to decrease, it merely
      waits  for  more  than  one existing command to terminate before
      starting another.

比如:

printf "%s\n" $DATAFILES/$TARGET_GLOB |
  xargs -d '\n' -I {} --max-procs=5 bash -c '
    filename=$1
    if [ ! -f $filename ]; then
        echo -e "\nFile $filename not found!\nExiting..."
        exit 255
    else
        echo -e "\nImporting $filename data to $DB/$TABLE"
    fi

    echo -e "\nStarting BCP export threads for $filename"
    /opt/mssql-tools/bin/bcp "$TABLE" in "$filename" \
        $TO_SERVER_ODBCDSN \
        -U $USER -P $PASSWORD \
        -d $DB \
        $RECOMMEDED_IMPORT_MODE \
        -t "\t" \
        -e ${filename}.bcperror.log
  ' _ {}

TABLE TO_SERVER_ODBCDSN , USER , PASSWORD DB 和 RECOMMEDED_IMPORT_MODE 变量,以便在 参数代换 bash -c 在一个单独的脚本中,并将变量放入该脚本中。

lampShadesDrifter 6 年前

https://stackoverflow.com/a/2546509/8236733 ))与

bcpexport() {
    filename=$1
    TO_SERVER_ODBCDSN=$2
    DB=$3 
    TABLE=$4 
    USER=$5
    PASSWORD=$6
    RECOMMEDED_IMPORT_MODE=$7
    DELIMITER=$8 # DO NOT use format like "'\t'", nested quotes seem to cause hard-to-catch error
    <same code from original loop>
}
export -f bcpexport
parallel -j 10 bcpexport \
    ::: $DATAFILES/$TARGET_GLOB \
    ::: "$TO_SERVER_ODBCDSN" \
    ::: $DB \
    ::: $TABLE \
    ::: $USER \
    ::: $PASSWORD \
    ::: $RECOMMEDED_IMPORT_MODE \
    ::: $DELIMITER

一次最多运行10个线程,其中 $DATAFILES/$TARGET_GLOB 是一个全局字符串,用于返回所需目录中的所有文件(例如,“$storagedir/tsv/*.tsv”),我们要遍历它(并将剩余的固定参数与该glob返回的每个元素相加,作为剩余的并行输入(如图所示) $TO_SERVER_ODBCDSN 变量实际上是“ -D -S <some ODBC DSN> ,因此需要添加引号以作为单参数传递)。所以如果 $DATAFILES/$TARGET\全局 glob返回文件A,B,C,…,我们最终运行命令

bcpexport A "$TO_SERVER_ODBCDSN" $DB ...
bcpexport B "$TO_SERVER_ODBCDSN" $DB ...
bcpexport C "$TO_SERVER_ODBCDSN" $DB ...
...

parallel 是

gnupallel确保命令的输出与按顺序运行命令时得到的输出相同。

Sputnik 6 年前

使用(&A);

#!/bin/bash
xmms2 play &
sleep 5
xmms2 next &
sleep 1
xmms2 stop