Streaming data is a common data processing problem. But luckily, the Bash xargs command, a versatile tool, can help process and manipulate streaming data more efficiently.
In this tutorial, you will learn how to use the Bash xargs command through practical examples.
Stay tuned and take control of command behaviors with the Bash xargs command!
Prerequisite
This tutorial houses hands-on demonstrations, and you will need a Linux system with the Bash shell installed to follow along. This tutorial uses Ubuntu 20.04, but any recent Linux distribution should work.
Finding and Removing Files with the Bash xargs Command
The Bash xargs command is a Linux utility used to build and execute commands from standard input. This command reads input from stdin, parses into arguments, and executes the specified command one or more times with the parsed arguments.
The general syntax for the xargs command is as below:
xargs <options> <command>
The most common use case for the xargs command is to find and remove files based on specific criteria. For example, you can use the xargs command to find all files older than a specified number of days and delete them.
Run the below command to find all files older than 30 days (-mtime +30) and pipe the output to the Bash xargs command. The xargs command then executes the rm command to forcefully (-f) delete each file without prompting for confirmation.
The -print0 option below prints the filenames to stdout, separated by a null character, since the xargs command expects its input to be null-delimited rather than newline-delimited.
find -mtime +30 -print0 | xargs -0 rm -fThis command does not provide output, but you will verify the deletion is successful in the following step.
💡 The Bash
xargscommand is often used with pipes to process data that would otherwise be too long for a single command.
Now, run the command below to list (ls) all contents of the working directory, sorted by time (t) in reverse order (r).
ls -ltrAll the files older than 30 days have been deleted, as shown below. Only new files remain in the current directory.
Finding and Archiving Files
Another everyday use case for the xargs command is to find all files based on a particular pattern and execute a command on each. Suppose you wish to find all text files and compress them to an archive (ZIP file), which lets you save some disk space. If so, the Bash xargs command is up to the task.
Run the below command to find all text files (*.txt) and pipe the output to the xargs command, which executes the tar command to archive the files to *archive-txt.tar.gz*.
The -0 option below tells xargs to treat its input as null-delimited instead of the default newline-delimited, so filenames with spaces in them are handled correctly.
find -name "*.txt" -print0 | xargs -0 tar -czvf archive-txt.tar.gz
Now, run the following command to list all files containing the name archive-txt.tar.gz.
ls -la archive-txt.tar.gzBelow, you can verify the archive (archive-txt.tar.gz) exists and that the compression is a success.
Running Multiple Commands with Bash xargs
Perhaps a file needs to be processed, and you wish to parallelize the processing by running multiple commands simultaneously.
For example, instead of running one echo command for each filename to insert to the text file, as shown below, let the xargs do the trick. The xargs command supports running multiple commands on its input.
echo commands, one for each file, to processSuppose you have to create directories for multiple files. If so, you just need a list of filenames and run an xargs command.
1. Run the below command to create a text file named bash-xargs-demo.txt containing a list of filenames, one per line.
# Creates a bash-xargs-demo.txt containing a list of filenames, one per line.
cat > bash-xargs-demo.txt << EOF
One
Two
Three
EOF
# Viewing the bash-xargs-demo.txt content
cat bash-xargs-demo.txt
2. Next, run the command below to read the filenames from the text file and create directories for each (sh -c 'echo %; mkdir %').
The -I % option tells xargs to replace % with the input it reads from stdin, which is *bash-xargs-demo.txt*.
cat bash-xargs-demo.txt | xargs -I % sh -c 'echo %; mkdir %'
💡 Perhaps you want to see what happens in the background as you run the command. If so, append the
-toption to thexargscommand to get a verbose output, as shown below.cat bash-xargs-demo.txt | xargs -t -I % sh -c 'echo %; mkdir %'
xargs command3. Lastly, run the command below to list (ls) all contents of the working directory in a long list (-la).
ls -laYou can verify that the directories were created successfully by running the ls command below.
In the output below, you can see three new directories (One, Two, and Three), one for each input line in the text file.
Enabling Prompting Before Execution
By default, the Bash xargs command executes specified commands for all input without asking for confirmation. This behavior can be dangerous if you are not careful, such as when deleting files with the rm command.
For example, the rm command below deletes all files in the current directory recursively without a confirmation prompt. You could delete many important files if you accidentally ran this command in the wrong directory.
rm -rf *💡 Perhaps you have mistakenly deleted important files. Luckily, Linux lets you recover deleted files.
To avoid untoward actions, append the -p option to enable prompting before execution.
Run the below command to find all files (*) in the working directory and delete (rm) them recursively (-rf) with a confirmation prompt (-p).
find * | xargs -p rm -rf
Notice below that even though you used the force (-f) option in the rm command, you will still get a prompt since the -p option is appended to the xargs command. The prompt below lets you verify that the command will do what you expect before confirming.
Type Y and press Enter to confirm the action. Or type N and press Enter if you made a mistake to cancel the command, and no harm will be done.
Reading Files Using the -a Option
So far, you have been reading input from stdin. But did you know that xargs can also read input from files? Yes! This feature can be useful if you have a large amount of data that needs to be processed and you do not want to keep the data all in your memory.
Run the below command to read and print input (-a) from the foo.txt file to stdout.
💡 Note that for
xargsto read the input from files, you must append the-aoption before any other options.
xargs -a foo.txt
Filtering Search Result with xargs and grep
The grep command is handy when searching texts in files. But one drawback is that the process may take a long time if you search through many files. The good news is that xargs can parallelize the grep command, speeding up the search significantly.
Run the following command to find all text files (*) in the working directory, filter, and print (grep) any files that contain the Examples string.
find . -name "*.txt" | xargs grep 'Examples'As you can see below, two files contain the Examples string.
Conclusion
Streaming input from stdin is a powerful feature of the Bash xargs command, but that is not all. In this tutorial, you have learned how to read input from files using the -a option. In addition, you touched on using xargs with other commands, such as grep, to speed up your workflows.
At this point, you should be comfortable with using the Bash xargs command. Experiment with different ways of using xargs to find the method that works best for you!
Now, why not try running programs in parallel using xargs?
