sed

sed is the ancient UNIX “stream editor”, mastery of which is every UNIX newbie’s test of true geekness. I find it vastly preferable to the Perl behemoth for parsing simple files in embedded systems - or brain-dead semi-proprietary UNIX boxes that have nothing but the most basic userland components.

Handy One-Liners for sed

Originally compiled by Eric Pement (version 5.4, Apr. 26, 2004)

File Spacing

# double space a file
sed G

# double space a file which already has blank lines in it
sed '/^$/d;G'

# triple space a file
sed 'G;G'

# undo double-spacing (assumes even-numbered lines are always blank)
sed 'n;d'

# insert a blank line above every line which matches "regex"
sed '/regex/{x;p;x;}'

# insert a blank line below every line which matches "regex"
sed '/regex/G'

# insert a blank line above and below every line which matches "regex"
sed '/regex/{x;p;x;G;}'

Numbering

# number each line of a file (simple left alignment)
sed = filename | sed 'N;s/\n/\t/'

# number each line of a file (number on left, right-aligned)
sed = filename | sed 'N; s/^/     /; s/ *\(.\{6,\}\)\n/\1  /'

# number each line of file, but only print numbers if line is not blank
sed '/./=' filename | sed '/./N; s/\n/ /'

# count lines (emulates "wc -l")
sed -n '$='

Text Conversion and Substitution

# convert DOS newlines (CR/LF) to Unix format
sed 's/.$//'               # assumes that all lines end with CR/LF
sed 's/^M$//'              # in bash/tcsh, press Ctrl-V then Ctrl-M
sed 's/\x0D$//'            # gsed 3.02.80

# convert Unix newlines (LF) to DOS format
sed 's/$/\r/'              # gsed 3.02.80

# delete leading whitespace (spaces, tabs) from front of each line
sed 's/^[ \t]*//'

# delete trailing whitespace (spaces, tabs) from end of each line
sed 's/[ \t]*$//'

# delete BOTH leading and trailing whitespace from each line
sed 's/^[ \t]*//;s/[ \t]*$//'

# substitute "foo" with "bar" on each line
sed 's/foo/bar/'           # replaces only 1st instance in a line
sed 's/foo/bar/4'          # replaces only 4th instance in a line
sed 's/foo/bar/g'          # replaces ALL instances in a line

# substitute "foo" with "bar" ONLY for lines which contain "baz"
sed '/baz/s/foo/bar/g'

# change "scarlet" or "ruby" or "puce" to "red"
sed 's/scarlet/red/g;s/ruby/red/g;s/puce/red/g'   # most seds
gsed 's/scarlet\|ruby\|puce/red/g'                # GNU sed only

# reverse order of lines (emulates "tac")
sed '1!G;h;$!d'

# join pairs of lines side-by-side (like "paste")
sed '$!N;s/\n/ /'

Selective Printing

# print first 10 lines of file (emulates "head")
sed 10q

# print last 10 lines of a file (emulates "tail")
sed -e :a -e '$q;N;11,$D;ba'

# print only lines which match regular expression (emulates "grep")
sed -n '/regexp/p'

# print only lines which do NOT match regexp (emulates "grep -v")
sed -n '/regexp/!p'

# print section of file from regular expression to end of file
sed -n '/regexp/,$p'

# print section of file based on line numbers (lines 8-12, inclusive)
sed -n '8,12p'

# print line number 52
sed -n '52p'

# print section of file between two regular expressions (inclusive)
sed -n '/Iowa/,/Montana/p'

Selective Deletion

# delete duplicate, consecutive lines (emulates "uniq")
sed '$!N; /^\(.*\)\n\1$/!P; D'

# delete the first 10 lines of a file
sed '1,10d'

# delete the last line of a file
sed '$d'

# delete ALL blank lines from a file
sed '/^$/d'

# delete all leading blank lines at top of file
sed '/./,$!d'

# delete all trailing blank lines at end of file
sed -e :a -e '/^\n*$/{$d;N;ba' -e '}'

Special Applications

# get email message header
sed '/^$/q'

# get email message body
sed '1,/^$/d'

# add a leading angle bracket and space to each line (quote a message)
sed 's/^/> /'

# remove most HTML tags (accommodates multiple-line tags)
sed -e :a -e 's/<[^>]*>//g;/</N;//ba'

Notes

Typical use: Sed takes one or more editing commands and applies all of them, in sequence, to each line of input. After all the commands have been applied to the first input line, that line is output and a second input line is taken for processing, and the cycle repeats.

This page is referenced in:

  • iWorkNov 23rd 2007