Simple Bash concurrency

By using wait and &, Bash scripts can be made concurrent.

Passing wait a process id will cause it to wait until that process has terminated. More interestingly, when no process id is given, wait will wait until all child processes of the invoking shell terminate.

To see how this pairs with & to achieve concurrency, let's explore a few simple scripts.

The sequential case

Say you have some time-consuming process that needs to run many times. Here's a script that emulates that:

#!/bin/bash
# sequence.sh

function expensive_process() {
  sleep 1
}

for i in {1..10}
do 
  expensive_process
done

In the script, we loop ten times over a function that calls sleep for one second. Given it runs in sequence, we can expect the script to take about ten seconds to finish.

Let's check the execution time by using the time command.

time ./sequence.sh

On my machine, this time outputs just over ten seconds for the total:

./sequence.sh  0.01s user 0.02s system 0% cpu 10.075 total

Background processes

To push execution of a command to the background, we can use ampersand.

#!/bin/bash
# long.sh


function expensive_process() {
  sleep 100
}

expensive_process &

If you run the previous script, find its process id, then run pgrep, you should see the child process running the sleep command.

For example, for me, running:

pgrep -lP 50509

Outputted:

50510 sleep

Waiting for child processes to finish

Since we can spawn child processes with ampersand and wait for child processes with wait, we can improve the performance of our original script by making the expensive function calls concurrent.

#!/bin/bash
# concurrent.sh

function expensive_process() {
  sleep 1
}

for i in {1..10}
do 
  expensive_process &
done

wait

Measuring the run time of this script with time outputs:

./concurrent.sh  0.01s user 0.03s system 4% cpu 1.018 total

A huge improvement! From 10 seconds down to 1 second.

Conclusion

This example is, of course, trivial and assumes that the each call to expensive process is independent. More clever methods for bash concurrency exist, including using xargs or GNU parrallel. Experiment with each option to find the one that fits your use case.