The handoff is a format, not just a pipe
A report says Alice ran two jobs for 32 seconds. The pipeline exits successfully. Then you discover that awk rejected half the input—and sort concealed its exit status. Combining familiar tools is easy; making their agreement visible takes more work. This article builds on the sed guide and awk guide.
A pipe transfers bytes, not a schema. Decide what counts as a record, how fields are separated, whether empty fields are valid and how malformed input is handled. sed is a good fit for a small local transformation; awk is a good fit for field validation, aggregation and decisions across records. Neither should guess a format that the producer has not promised.
For this article, the main examples use three tab-separated fields: user, elapsed seconds and result. User names contain no tabs or newlines, elapsed seconds are nonnegative integers, and result is OK or FAIL. These restrictions are deliberate. Real scheduler exports may include headers, units and additional fields; define and test their contract before adapting a pipeline.
printf 'alice\t12\tOK\nbob\t8\tFAIL\nalice\t20\tOK\n' > jobs.tsv
cat jobs.tsvNormalize only what you are willing to discard
CRLF input is a common reason to put sed before awk: strip a trailing carriage return without rewriting the rest of the record. The shell variable below holds the literal byte, so the sed script does not rely on a GNU-only regex escape. Do not globally delete every carriage return unless embedded ones are explicitly forbidden.
Whitespace cleanup is not always harmless. Collapsing spaces can corrupt a free-text field; removing comments can delete literal hash characters inside quoted values. Normalize the envelope you understand, then let awk validate the schema. Keeping the raw file lets you reproduce an error and distinguish a producer defect from a transformation defect.
cr=$(printf '\r')
sed "s/${cr}$//" jobs.tsv | awk -F '\t' '{ print $1, $2, $3 }'Validate before you aggregate
This program rejects malformed records and exits nonzero rather than silently converting arbitrary text into zero. awk numeric conversion is helpful for well-formed data and dangerous when it conceals a broken input contract. A header, an empty duration or a unit suffix must be handled deliberately.
The END block still runs after exit. The bad flag therefore suppresses partial results. Output order from an associative array is unspecified; use sort if consumers need deterministic output. Here the tab character separates fields, and count, total and failure arrays share the same user key. A sample with no records legitimately produces no summary.
This example uses ordinary awk numbers, typically floating point. Integer-shaped input does not imply arbitrary-precision arithmetic. Keep elapsed times and their totals within the exact range of your implementation; use an appropriate exact-number tool for very large accounting totals.
# Bash: preserve a failed stage’s status.
set -o pipefail
cr=$(printf '\r')
sed "s/${cr}$//" jobs.tsv | awk -F '\t' '
BEGIN { OFS="\t" }
NF != 3 || $1 == "" || $2 !~ /^[0-9]+$/ || $3 !~ /^(OK|FAIL)$/ {
bad=1; exit 2
}
{ count[$1]++; total[$1]+=$2; failures[$1]+=($3=="FAIL") }
END {
if (bad) exit 2
for (user in count) print user,count[user],total[user],failures[user]
}' | LC_ALL=C sort
# alice 2 32 0; bob 1 8 1 (tab-separated)Order changes the question you answer
Filtering failures before aggregation gives totals for failed records. Aggregating everything first gives totals for all records. That sounds obvious, but many production dashboards drift because a cleanup stage also silently becomes a selection stage. Keep normalization, validation and selection visibly distinct.
A substitution can alter a later regex match, and sed addresses operate on the current pattern space. If you lowercase or redact status labels before awk sees them, the awk predicate must match that new representation. In a review, write down an example record at each boundary. When two people disagree on the expected output, the intermediate representation often reveals why.
awk -F '\t' '
NF != 3 || $1 == "" || $2 !~ /^[0-9]+$/ || $3 !~ /^(OK|FAIL)$/ { bad=1; exit 2 }
$3 == "FAIL" { n[$1]++; seconds[$1]+=$2 }
END {
if(bad) exit 2
for(u in n) print u,n[u],seconds[u]
}' jobs.tsvExpected summary: bob 1 8. A malformed row or empty username makes this program exit 2 and suppress its summary.
Use sed to carve a trusted region
For a plain-text report with explicit standalone BEGIN JOBS and END JOBS markers, sed can select the region and remove the markers. awk can then interpret its records. This example assumes at most one non-nested region and tabs between fields. If the start marker has no end marker, sed continues to end of input; the wrapper must validate completeness if a partial report is unacceptable.
This is not a general-purpose parser for markup or nested blocks. If a header or marker can appear inside a quoted field, the line-based extraction contract is already wrong. Use a format with unambiguous boundaries or parse the source format before passing records to the pipeline.
sed -n '/^BEGIN JOBS$/,/^END JOBS$/{
/^BEGIN JOBS$/d
/^END JOBS$/d
p
}' report.txt | awk -F '\t' '$3=="FAIL" { print $1,$2 }'Sometimes awk should run first
awk can compute a summary, then sed can format a presentation-specific label. That is a reasonable final stage for a human-readable report, provided downstream programs do not need to parse the decorated version. Keep a machine-readable result separate from a display intended for a terminal or email.
The example computes one total and prefixes the resulting line. Do not use sed to make arithmetic appear to work through digit substitutions. Similarly, if you need to remove duplicate users after computing groups, awk already has the keys. Another process should serve a distinct purpose, not merely make the pipeline look more Unix-like.
awk -F '\t' '$2~/^[0-9]+$/ { seconds+=$2 } END { print seconds+0 }' jobs.tsv |
sed 's/^/Total elapsed seconds: /'Failure status must survive the pipeline
In a traditional shell pipeline, the final command usually determines the exit status. A successful sort can therefore conceal an awk validation failure. Bash set -o pipefail makes failure in an earlier stage affect the pipeline status. It is a Bash feature, not something to assume in every /bin/sh implementation.
Error status does not retract bytes already written. For important reports, write to a fresh temporary file and publish the result only after the pipeline succeeds. A downstream side effect, such as sending a message or overwriting a configuration, must happen after validation. For privileged output, use a directory whose ownership and permissions you control.
# Bash; publish no partial report on failure.
set -o pipefail
tmp=$(mktemp) || exit 1
trap 'rm -f "$tmp"' EXIT
if sed -f normalize.sed jobs.tsv | awk -f validate-and-summarize.awk | LC_ALL=C sort > "$tmp"; then
cat "$tmp"
else
printf '%s\n' 'Input rejected; no report published.' >&2
exit 1
fiKeep values out of program source
A shell value inserted into a sed script can become regex syntax, replacement syntax or an additional command. A value inserted into an awk program can become awk source. Shell quoting alone does not fix the second interpretation step. Keep scripts fixed and pass data through a separate channel.
Try this harmless local injection demonstration: shell quoting keeps the argument together, but the interpolated slash closes the replacement and the e flag asks GNU sed to execute it. No elevated privileges are needed. This is a demonstration of program injection, not a report of an observed intrusion.
For a literal username filter, the environment lets awk receive the value without treating it as program source or interpreting -v assignment escape sequences. Compare with == rather than a regex operator when a literal match is intended. This is not a license to trust every downstream use: handing the resulting data to system(), a shell or another interpreter reintroduces an execution boundary.
# Deliberately unsafe construction: local demonstration, no sudo.
replacement='printf injected/e;#'
printf 'NAME\n' | sed "s/NAME/$replacement/"
# Prints injected. Keep source fixed instead:
wanted='alice'
WANTED_USER="$wanted" awk -F '\t' '$1 == ENVIRON["WANTED_USER"] { print }' jobs.tsvAbuse and vulnerabilities cross the handoff
GNU sed execution commands and awk system() or command pipes are powerful documented features. Arbitrary scripts running with elevated privilege can read, write or execute far beyond the intended transformation. The risk is in program authority; a text-processing tool is not automatically safe because its output usually looks like text.
Our stdout-only pipeline avoids the in-place/symlink combination involved in the sed race vulnerability. The awk guide discusses GNU awk CVE-2023-4156, a heap out-of-bounds read that may cause a crash or expose information, rather than a blanket command-execution claim about all awk implementations. Connecting patched tools still leaves a pipeline vulnerable if it interpolates untrusted values into source.
Know when to collapse the pipeline
awk can perform substitutions itself with sub() and gsub(). If sed only performs a simple cleanup that awk can clearly express, one awk program may be easier to test and report errors from. Conversely, a stable reviewed sed script shared by several consumers can justify a separate normalization stage.
For huge inputs, think about the memory behavior of every stage: ordinary sed transformations can use bounded state, awk aggregation grows with unique keys, and sort may spill to disk. Multi-line sed accumulation can hold the entire input. A pipe does not guarantee constant memory. Test with the expected number of keys and longest record, not merely total file size.
Use fixtures for blank input, CRLF, a malformed duration, an empty user, duplicate users and a missing region terminator. Assert both output and status. The point of combining these tools is not to win a contest for the shortest command: it is to make the data contract and each transformation easy for the next administrator to understand.
# Collapse CRLF normalization and a literal field selection into awk.
awk -F '\t' '{ sub(/\r$/, "", $0) } $3=="FAIL" { print $1,$2 }' jobs.tsv