The GNU project is proud to announce a new version of the command-line utility Parallel which is now nearly 10 years old. The new version supports grouping jobs and there is quite a few bug fixes.

Never heard of GNU Parallel? It is a tool for running multiple shell jobs (commands) in parallel. Consider this really meaningless example:

Shell command: parallel echo 'Job for:' ::: *.png

This simple command, when ran in a folder with some png images, will produce something like:

Job for: Guix-installing-software-fs8.png Job for: guix.png Job for: KDE_Katie.png Job for: kdenlive-wjsn.png Job for: lives-2.10.2-fs8.png

You may or may not wonder what's the point of using parallel when you can do the exact same thing, echo a filename, by running

Shell command: for f in *.png;do echo 'Job for: '$f;done

and answer is simple: You can run 1000 echo jobs in an instant so it does not matter if it's single-threaded or multi-threaded but it does matter if you want to actually do something with a command like convert[1] and that something is single-threaded. Running a mostly single-threaded program in sequence or parallel can make a big difference if you have a 4 or 6 or 8-core system. There's is a very real and clear advantage in executing 8 or 10 single-threaded jobs instead of just one if you have 8 cores. This is what GNU parallel can do for you.

Shell command: parallel pngquant ::: *.png

would obviously be faster than using for f in *.png;do on a larger collection of images since pngquant[2] takes up to half a second per image.

For more advanced use-cases the same syntax as xargs[3] applies.

The parallel manual page is rather long. You should want to read it, but if you don't want to there's a short video introduction series on YouTube. BTW, if you do look at the manual page and think "This is as long as a book" you'd be wrong. There is a paper-back book and it is 112 pages long.





Versions of Parallel are available in all distributions repositories. Just install it and you will get the new version when you update your system in the not too distant future.

You can read more about GNU Parallel at http://www.gnu.org/s/parallel/

footnotes



last edited 2019-05-23



