# Bash Scripting for DevOps

## 1. A Safe Script Skeleton

```bash
#!/usr/bin/env bash
set -Eeuo pipefail          # exit on error (-e), on unset vars (-u), on failure anywhere in a pipe; ERR trap inherited (-E)
IFS=$'\n\t'                 # safer word splitting

readonly SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
log()  { printf '%s [%s] %s\n' "$(date -u +%FT%TZ)" "${2:-INFO}" "$1" >&2; }
die()  { log "$1" ERROR; exit "${2:-1}"; }

tmpdir="$(mktemp -d)"
cleanup() { rm -rf "$tmpdir"; }
trap cleanup EXIT
trap 'die "failed at line $LINENO: $BASH_COMMAND"' ERR

main() {
  [[ $# -ge 1 ]] || die "usage: $0 <environment>" 2
  local env="$1"
  log "deploying to $env"
}
main "$@"
```

`set -e` has surprising exceptions (commands in `if` conditions, `&&`/`||` chains, and pipelines without `pipefail`), so check critical commands explicitly. Run **ShellCheck** on every script — it catches most of the pitfalls below.

## 2. Variables and Quoting

```bash
name="pet pal"
echo "$name"                 # ALWAYS quote expansions: "$var", "$@", "$(cmd)"
files=("a b.txt" c.txt)      # arrays keep elements intact
for f in "${files[@]}"; do echo "$f"; done
echo "${#files[@]}"          # length
: "${ENVIRONMENT:?must be set}"       # fail if unset/empty
port="${PORT:-8080}"                  # default
echo "${path%/*}" "${path##*/}"       # dirname / basename via parameter expansion
echo "${name^^}" "${file%.log}.gz"    # uppercase (bash 4+), replace suffix
readonly VERSION=1.4.2
local count=0                         # inside functions
```

Unquoted `$var` undergoes **word splitting** and **globbing** — the #1 source of bash bugs (`rm -rf $dir/*` with an empty `$dir`...).

## 3. Tests and Control Flow

```bash
if [[ -f "$file" && -r "$file" ]]; then ...; fi    # [[ ]]: no splitting, supports && || =~
[[ "$branch" == release/* ]]                         # glob match
[[ "$version" =~ ^v([0-9]+)\.([0-9]+)$ ]] && major="${BASH_REMATCH[1]}"
(( retries < 5 ))                                    # arithmetic
case "$1" in
  start|run) start ;;
  stop)      stop ;;
  *)         die "unknown command: $1" ;;
esac
while IFS= read -r line; do ...; done < "$file"      # read lines safely
for i in {1..5}; do ...; done
```

File tests: `-e` exists, `-f` file, `-d` dir, `-s` non-empty, `-x` executable, `-nt` newer than. String: `-z` empty, `-n` non-empty.

## 4. Functions, Exit Codes, Input/Output

```bash
retry() {                   # retry <attempts> <cmd...>
  local attempts="$1"; shift
  local n=1 delay=1
  until "$@"; do
    (( n >= attempts )) && return 1
    log "attempt $n failed; retrying in ${delay}s" WARN
    sleep "$delay"; (( n++, delay *= 2 ))
  done
}
retry 5 curl -fsS https://api.petpal.io/health
```

- Functions return status codes (`return 0..255`) and output via stdout; capture with `$(...)`.
- Redirections: `>` `>>` `2>` `2>&1` `&>` `< file`, here-docs `<<'EOF'` (quoted = no expansion), here-strings `<<< "$var"`, process substitution `diff <(sort a) <(sort b)`.
- `$?` last status; `${PIPESTATUS[@]}` statuses of a pipeline.

## 5. Text Processing Toolkit

```bash
grep -E 'ERROR|WARN' app.log | grep -v healthcheck
awk '$9 >= 500 {c++} END {print c+0}' access.log            # count 5xx (field 9)
awk '{print $1}' access.log | sort | uniq -c | sort -rn | head -5   # top IPs
cut -d, -f2,5 data.csv · sed -n '10,20p' file · sed -i.bak 's/old/new/g' file
jq -r '.items[] | select(.status=="failed") | .id' report.json
xargs -P 4 -n 1 ./process.sh < ids.txt                       # parallel
sort -t, -k3,3n · tr -s ' ' · column -t · tee
```

Know when to stop: if a script grows past ~150 lines, needs data structures or real error handling, rewrite it in Python/Go/TypeScript.

## 6. Scripts for Operations

- **Arguments**: `getopts` for short flags (`-e prod -n`), or a `while [[ $# -gt 0 ]]; case $1 in --env) ...` loop for long ones; always print usage.
- **Idempotency**: running twice should be safe (`mkdir -p`, check-before-create, `ln -sfn`).
- **Locking**: `exec 9>/var/lock/deploy.lock; flock -n 9 || die "already running"`.
- **Dry-run** flag that prints commands instead of running them.
- **Logging** to stderr with timestamps; exit codes that mean something.
- **Secrets** from the environment or a secret manager — never as command-line arguments (visible in `ps`).

## Under the Hood: How Bash Processes a Command Line

For each command, bash performs a fixed sequence of **expansions**: (1) brace expansion `{a,b}`; (2) tilde expansion `~`; (3) parameter/variable expansion `$var`, command substitution `$(...)`, arithmetic `$((...))`, process substitution — left to right; (4) **word splitting** of the *unquoted* results on `IFS`; (5) **pathname expansion** (globbing) of unquoted words containing `*?[`; (6) quote removal. Then redirections are set up, and the command is run: builtins and functions in the current shell; external commands via `fork()` + `exec()` after searching `PATH` (cached in a hash table). Pipelines run each part in its own subshell process connected by pipes — which is why `cmd | while read x; do count=$((count+1)); done` doesn't change `count` in the parent shell, and why a pipeline's status is the last command's unless `pipefail` is set. Quoting stops steps 4 and 5, which is precisely why `"$var"` is safe and `$var` is not.

![Bash expansion order: brace, tilde, then parameter, command and arithmetic substitution, then word splitting and globbing applied only to unquoted results, then quote removal, redirections, and finally a builtin runs in-shell or an external command is forked and exec'd; pipelines run in subshells](diagrams/01-expansion-order.svg)

## Common Pitfalls

1. **Unquoted variables** — splitting and globbing surprises.
2. **No `set -euo pipefail`** — scripts continue after failures.
3. **Parsing `ls` output** — use globs or `find -print0`.
4. **`cd` without checking** — `cd "$dir" || exit`.
5. **Variables set inside a pipeline's `while` loop** lost in the parent.
6. **Secrets on the command line**.
7. **Bash 4+ features** (`declare -A`, `${x^^}`, `mapfile`) on macOS's bash 3.2.

## Quick Reference

```
set -Eeuo pipefail · trap cleanup EXIT · trap ... ERR · mktemp -d · shellcheck
"$var" "${arr[@]}" ${VAR:-default} ${VAR:?msg} ${p%/*} ${p##*/}
[[ -f x && $a == b* ]] · [[ $s =~ re ]] BASH_REMATCH · (( n < 5 )) · case · while IFS= read -r
functions return codes, output via stdout · getopts · flock · idempotent · dry-run
grep -E · awk · sed · cut · sort | uniq -c | sort -rn · jq · xargs -P