Shell多线程与控制线程数实现

概述

在工作当中执行某个脚本时常常因为循规蹈矩的进度导致缓慢，需要通过多线程来进行解决速度慢的问题。

生成测试文件，模拟多个任务

[root@VM_0_9_centos ~]# for num in {1..10};do echo $num >> test.txt;done

单进程

[root@VM_0_9_centos ~]# cat test.sh
#!/bin/bash

File_Name=${1:?we need a parameter!}
if [ ! -f "${File_Name}" ];then
    echo "Can't find the file ${File_Name} !"
    exit 1
fi
while read ;do
    {
        echo "$REPLY"
        sleep 1
    }
done < ${File_Name}
echo done

运行:

[root@VM_0_9_centos ~]# time ./test.sh test.txt
1
2
3
4
5
6
7
8
9
10
done

real    0m10.014s
user    0m0.006s
sys     0m0.008s

多进程

&实现多进程，wait实现父进程等待子进程完成后退出

[root@VM_0_9_centos ~]# cat test.sh
#!/bin/bash

File_Name=${1:?we need a parameter!}
if [ ! -f "${File_Name}" ];then
    echo "Can't find the file ${File_Name} !"
    exit 1
fi
while read ;do
    {
        echo "$REPLY"
        sleep 1
    } &
done < ${File_Name}
wait
echo done

运行，数字全部一起输出，耗时1s：

[root@VM_0_9_centos ~]# time ./test.sh test.txt
1
2
7
3
8
4
9
5
10
6
done

real    0m1.010s
user    0m0.012s
sys     0m0.006s

通过管道控制进程数

[root@VM_0_9_centos ~]# cat test.sh
#!/bin/bash

File_Name=${1:?we need a parameter!}
if [ ! -f "${File_Name}" ];then
    echo "Can't find the file ${File_Name} !"
    exit 1
fi

#线程数
thread_num=4
# create pipe
[ ! -p tmp ] && mkfifo tmp
exec 9<>tmp

# fill in the pipe
for ((i=0;i<thread_num;i++)); do
    echo 1>&9
done

while read ;do
    # remove one line per thread
    (read -u 9) # In order to avoid the "read" command in while loop, we use the parentheses here
    {
        echo $REPLY
        sleep 1
        # thread run finish, put line back
        echo 1>&9
    } &
done < ${File_Name}
wait
echo done

运行如下，数字四个一组输出，共三组，耗时3s：

[root@VM_0_9_centos ~]# time ./test.sh test.txt
1
2
3
4
5
6
7
8
9
10
done

real    0m3.017s
user    0m0.012s
sys     0m0.011s

另辟蹊径，通过 xargs 制进程数

[root@VM_0_9_centos ~]# cat test.sh
#!/bin/bash

whereami=`cd $(dirname $0); pwd`
file_name=${1:?we need a parameter!}
if [ ! -f "${file_name}" ];then
    echo "Can't find the file ${file_name} !"
    exit 1
fi

#线程数
thread_num=4

cat ${file_name} |xargs -n 1 -I {} -P ${thread_num} sh -c "$whereami/print.sh {}"
echo done

[root@VM_0_9_centos ~]# cat print.sh 
#!/bin/bash

echo $* 
sleep 1

运行如下，数字四个一组输出，共三组，耗时3s：

[root@VM_0_9_centos ~]# time ./test.sh test.txt
1
2
3
4
5
6
7
8
9
10
done

real    0m3.022s
user    0m0.025s
sys     0m0.023s

END