Functional Programming with the Unix Command Line

Jeff Aigner — Fri, 26 Jun 2009 06:29:30 +0000

I was pondering the unix command line the other day while reading sed & awk, and I came to the following revelation: variables aside, the unix command line is a lot like functional programming. Using the command line, you can build a functional program by redirecting I/O between various programs, or functions. Many of the programs one uses to build one-liners are essentially pure functions. An example of a pure function on the unix command line might be the programs uniq or sort. Various other functions such as ps and wget are not so pure, as their output relies on I/O with the operating sytem or server.

All theory aside, here are a few good one-liners.

Uploading Files

tar cjvf - yourDirectory | ssh uname@host "cat > yourDirectory.tar.bz2"

This command will create an archive of “yourDirectory” and upload it to your server via SSH, all in one chained command. Quite handy for uploading local content for sharing or backup.

Edit: Apparently this command transfers large files extremely slow. Oops.

Checking FreeBSD UPDATING File Automatically

portversion | awk '/ updates.txt

Ok. So this command pertains to the FreeBSD ports system. It is not recommended that you upgrade all your ports at once. There is often information contained in the file /usr/ports/UPDATING about special instructions you may have to follow while upgrading a package. Often times you may have to recompile other packages, or add a line to a configuration file, etc. The string above will check all of the ports tht are out of date, and compare them to the UPDATING file to see if the package is contained. If it is, it writes the name of that port to updates.txt. You can then use this file to know which ports have special instructions. You should probably write an alias for this one.

Converting Line Endings

The following two commands should help with reformatting files. Often times I find myself getting web designs from someone who uses windows. Most editors on windows will save files with DOS line endings. When you open one of these files in unix (in certain editors), the line endings won’t appear properly. The following commands will circumvent this problem, in either situation.

Convert from unix (\n) to DOS (\r\n)

awk '{sub(/$/, "\r")};1' unix_endings.txt > dos_endings.txt

Convert from DOS (\r\n) to unix (\n)

awk '{sub(/\r$/, "")};1' dos_endings.txt > unix_endings.txt

And yes, I realize that I use too many commas when I write: I’m working on it.

Replace Text Recursively with sed

Jeff Aigner — Wed, 17 Jun 2009 20:53:31 +0000

Here is a quick little trick I discovered today when I needed to replace text in multiple files recursively:

grep -rl -e  * | xargs sed -i .bak 's/searchterm/newterm/i'

There might be a problem with files that have spaces in the names (because xargs will take a space as the start of a new argument). To solve this problem, you should be able to do something like this:

grep -rl --null -e  * |

xargs -0 sed -i .bak 's/searchterm/newterm/i'

If you are using GNU grep, you could also use -Z instead of –null. I happen to be using the BSD version of grep, so -Z isn’t an alias for –null.