The program is a stream of rules
awk reads records, tests each against a pattern, and runs the corresponding action. The default record separator is a newline; the default field separator treats runs of whitespace specially. $0 is the whole record, $1 the first field, NF the field count, and NR the cumulative record number. BEGIN runs before input; END runs after it.
A missing pattern means every record, and a missing action means print the record. The two are easy to confuse: awk "/error/" prints matching records, while awk "{count++}" increments once per record and prints nothing until you explicitly do so. Quote the entire program so the shell does not expand dollar signs.
awk -F: '$3 >= 1000 { print $1, $3 }' /etc/passwd
awk 'BEGIN { n=0 } /ERROR/ { n++ } END { print n }' service.logWhitespace splitting is a special case
With the default FS=" ", awk trims leading and trailing blanks for field splitting and treats adjacent whitespace as one separator. FS=":" treats a single colon as a separator, so empty fields survive. This distinction matters for /etc/passwd, delimited exports, and log formats with optional columns.
Changing FS after a record is read may not resplit it the way you expect. Set it before input with -F or in BEGIN. Assigning to a field rebuilds $0 using OFS; that makes field replacement a compact transformation, but it can normalize spacing unintentionally.
printf ' a b \n' | awk '{ print NF, "[" $1 "]", "[" $2 "]" }'
printf 'a::c\n' | awk -F: '{ print NF, "[" $2 "]" }'
printf 'red:blue\n' | awk -F: 'BEGIN { OFS="," } { $2="green"; print $0 }'Records need not be lines
Setting RS="" in standard awk groups paragraphs separated by blank lines. A two-line record is often easier to reason about than two independent grep matches. GNU awk extends RS to regular expressions and provides RT, the text that matched the separator; that is powerful for irregular logs but is not portable awk.
If records may be huge, remember that awk stores a record while processing it. An input file lacking expected separators can become one enormous record. Bound the input and test the actual implementation before putting a paragraph or regex separator in a log pipeline.
# Portable paragraph records: print blocks mentioning a failed check.
awk 'BEGIN { RS=""; ORS="\n---\n" } /FAILED/ { print }' report.txt
# GNU awk only: split on blank lines with optional spaces.
gawk 'BEGIN { RS="\n[[:space:]]*\n" } { print NR, RT }' report.txtUse arrays as a streaming group-by
Associative array keys are strings; an array can accumulate counts or totals without sorting first. For small cardinality data this is a shell-friendly group-by. For millions of distinct keys, memory grows with the number of keys, so aggregate in a database or prefilter when the cardinality is unbounded.
Array traversal order is not a portable sort order. Pipe the result into sort when presentation order matters. In a pipeline, be deliberate about which component owns numeric sorting and which owns the data transformation.
awk '{ count[$1]++ } END { for (key in count) print count[key], key }' status.log | sort -nr
awk -F, 'NR>1 { bytes[$2]+=$4 } END { for (host in bytes) printf "%12.0f %s\n", bytes[host], host }' usage.csv | sort -nrThe range pattern is stateful
A pattern of the form start,end activates at a matching start record and stays active through a matching end record, inclusive. If the end never appears, it keeps selecting to end of input. If start and end match the same record, that one record can form a complete range. This makes it useful for bounded sections in human-readable reports, but surprising in repeated logs.
For machine-readable data with nested records, explicit state variables are clearer. A range pattern is not a parser and cannot reliably match arbitrary nesting.
awk '/^BEGIN CHECK$/,/^END CHECK$/' audit.txt
awk '/^BEGIN CHECK$/ { inside=1; next } /^END CHECK$/ { inside=0 } inside' audit.txtJoining two files with FNR and NR
FNR resets to one for each input file; NR continues across files. That gives a compact two-pass lookup: read keys from the first file into an array, then test the second. A common idiom is NR==FNR, but it fails if the first file is empty because the condition remains true in the second file. FILENAME or an explicit marker can make intent clearer.
Treat both filenames and field separators as data boundaries. The example assumes one key per line and a colon-delimited second file, not arbitrary CSV with quoted delimiters.
awk 'FILENAME==ARGV[1] { wanted[$1]=1; next } ($1 in wanted) { print }' wanted-users.txt FS=: /etc/passwd
# Easier to read as two steps when the field formats differ:
awk 'NR==FNR { wanted[$1]=1; next } ($1 in wanted)' wanted-users.txt other-users.txtNumbers, formatting, and exit behavior
awk converts between strings and numbers according to context. Numeric-looking strings may compare numerically, while text compares as strings. For repeatable reports, use explicit arithmetic and printf formats. The default print adds OFS between arguments and ORS at the end; printf does not add a newline for you.
An explicit exit status is useful in health checks. END still runs when exit is executed from a record action. Set a status variable and exit from END if you need final aggregation; avoid accidentally overriding an earlier failure in END.
awk -F: '$1=="service" { found=1 } END { if (!found) exit 1 }' /etc/passwd
awk '{ total+=$2; n++ } END { if (n) printf "%.2f\n", total/n; else exit 1 }' samples.txtSubstitution has a second parser
sub(regex,replacement,target) changes the first occurrence; gsub changes all occurrences. If target is omitted, awk edits $0. In replacement text, & stands for the entire match, and backslashes may be processed more than once. That is why assembling replacement expressions from arbitrary input requires care even when the regex itself is fixed.
For simple field transformations, assign to a field instead of searching the whole line. For a report that strips a suffix from one column, sub on $2 leaves the rest of the record data model intact, though rebuilding $0 later may alter separators.
printf 'api-ms api-ms\n' | awk '{ gsub(/-ms/, "", $1); print $1, $2 }'
printf 'a a\n' | awk '{ n=gsub(/a/, "x"); print n, $0 }'getline changes what “current” means
getline can read another record inside an action. It is tempting for header-plus-body logs, but it advances the input stream and affects counters and control flow. A plain getline updates $0, NF, NR, and FNR; getline into a variable has different effects. Its return value distinguishes success, end of file, and error.
If you use it, write a state diagram or a small fixture with missing final lines. For many record formats, an explicit pending-header variable is easier to audit than a nested read. Never assume the next line exists just because the current line says BEGIN.
awk '/^HEADER:/ { header=$0; if ((getline body)>0) print header, body; else print "missing body" > "/dev/stderr" }' report.txtLog analysis without losing the original question
A quick count of status codes is easy, but a production log may contain spaces, quoting, escapes, and optional fields. A naïve FS value that looks right on three sample lines can silently attribute data to the wrong service on the fourth. Test against representative records and reject malformed ones by checking NF or a known prefix.
When accuracy matters, let a parser for the real format produce clean tabular data, then use awk as the aggregation stage. That division keeps awk powerful without pretending it is a JSON or quoted-CSV parser.
awk -F '\t' 'NF!=4 { bad++; next } { requests[$2]++; bytes[$2]+=$4 } END { for (s in requests) print s, requests[s], bytes[s]; if (bad) print bad " malformed records" > "/dev/stderr" }' clean.tsvSecurity: awk is a programming language
GNU awk and many other implementations can read files, redirect output, and invoke programs with system() or a command pipeline. An unrestricted sudo grant to awk, or a service that accepts caller-supplied awk source, grants far more than a harmless text filter. Treat the program as code, not as a safe query string.
Use a fixed reviewed awk script and pass untrusted values as data with -v, while still validating file paths and output destinations. -v protects against shell expansion of the value; it does not turn an attacker-controlled awk program into safe code. Avoid running a rich interpreter with elevated privileges to parse untrusted logs.
# Fixed program, value passed as data.
awk -v sought="$username" -F: '$1==sought { print $1, $3 }' /etc/passwd
# Audit privileged rules before allowing an interpreter.
sudo -lA vulnerability belongs to an implementation
CVE-2023-4156 is a reported heap out-of-bounds read in gawk builtin.c that could crash the program and potentially expose information. It is not a statement about every awk implementation. Debian tracks fixed gawk package revisions; a distribution may backport a fix without changing the upstream version string in the way you expect.
Check which awk implementation your system selects and its package update status. GNU awk has extensions that mawk, BWK awk, and BusyBox awk may not share. Keep portable scripts to the common language and explicitly label GNU-only examples.
command -v awk
awk -W version 2>&1 | head -n 2 # implementation-dependent
dpkg-query -W gawk mawk 2>/dev/null || rpm -q gawk mawkField exercise: a report worth keeping
Given a bounded sample of space-delimited service events with the service name in field one and a numeric latency in field two, compute count and average latency per service. Then add a threshold that exits nonzero when any average exceeds it. This combines field extraction, state, formatting, and a deliberate status code.
For real logs, establish the format before trusting a field number. A quoted CSV file or JSON stream is not made into a correct parser merely by choosing a clever FS regex.
printf 'api 12\nweb 8\napi 18\n' | awk '{sum[$1]+=$2; n[$1]++} END {for (s in n) printf "%s count=%d avg=%.1f\n",s,n[s],sum[s]/n[s]}'