Pipes and Redirection

In the first lesson I said the shell lets you combine small programs in ways nobody anticipated. This is that lesson.

Everything so far has been individual commands. What makes the command line genuinely powerful — more powerful than any GUI, for a large class of problems — is that its programs are designed to be connected.

Three streams

Every process starts with three channels already open:

• stdin (0) — where input comes from. Usually your keyboard.
• stdout (1) — where normal output goes. Usually your screen.
• stderr (2) — where error messages go. Also usually your screen.

The screen is just the default. Every one of these can be pointed somewhere else, and that's all redirection is.

The split between stdout and stderr is deliberate and useful: it means you can capture a program's results while still seeing its complaints, because they travel on separate channels even though they land in the same place by default.

Redirecting to files

ls > files.txt          # stdout into a file, replacing its contents
ls >> files.txt         # stdout appended to the end instead

The difference between > and >> is worth burning in: > truncates the file first. Pointing > at something that already has contents destroys them, instantly, with no warning. It's the second-most common way people lose work at a terminal, after rm.

Errors need their own redirect, because they're on channel 2:

./script.sh 2> errors.txt         # errors to a file, output still on screen
./script.sh > out.txt 2>&1        # both into one file
./script.sh &> everything.txt     # shorthand for the same thing (bash)

That 2>&1 reads as "send channel 2 to wherever channel 1 is currently going". It looks cryptic and it's worth recognising, because it appears in nearly every cron job and CI script you'll ever read.

And input can come from a file instead of the keyboard:

sort < names.txt

The pipe

> sends output to a file. The pipe, |, sends it to another program's input:

ps aux | grep node

ps aux writes a long list to stdout. Instead of your screen, that list becomes grep's stdin. grep reads it, keeps only lines matching "node", and writes those to its stdout — which is your screen.

Neither program knows the other exists. ps doesn't know it's being filtered; grep doesn't know where the text came from. That independence is the whole design, and it's why pipes compose without limit:

cat access.log | grep " 500 " | wc -l

Read left to right: take the log, keep only lines containing a 500 status, count them. Three simple tools answering a question none of them was written for.

That second one finds the three IP addresses hitting your server hardest: take each line's first field, sort them, count each unique value, sort those counts numerically in reverse, show the top three. Six tools, one line, no script, no program written.

The pieces worth knowing

A small vocabulary covers most real pipelines:

• grep pattern — keep matching lines (-v inverts it, -i ignores case, -c counts)
• wc -l — count lines
• sort — sort lines (-n numerically, -r reversed)
• uniq -c — collapse adjacent duplicates and count them (needs sorted input)
• head / tail — first or last N lines
• cut -d, -f2 — pull out a field by delimiter
• awk '{print $1}' — pull out a column by position

Where this goes next

You now have the model the rest of Linux is built on — a filesystem tree, permissions, processes, and streams you can wire together.

The natural next step is to stop typing these one at a time and start saving them: shell scripts, variables, loops, and conditionals. After that, containers — because a Docker container is, at bottom, a process with its own view of the filesystem, and everything in this course applies directly.

That's what the next courses cover.