Linux Interview Guide

Strong DevOps engineers do not just troubleshoot faster. They eliminate uncertainty faster.

This guide covers 112 Linux and Shell Scripting interview questions, answered the way a calm, senior Cloud/DevOps engineer would actually think in an interview — clarifying before concluding, gathering evidence, weighing trade-offs, and preventing recurrence.

Linux boot process (BIOS/UEFI → bootloader → kernel → systemd)
Linux filesystem hierarchy (/, /etc, /var, /opt, /tmp, /proc, /sys)
File permissions (read, write, execute, chmod, chown, chgrp)
Special permissions (SUID, SGID, sticky bit, umask)
File types (regular, directory, symlink, hard link, device, socket, pipe)
Users and groups (/etc/passwd, /etc/shadow, useradd, usermod)
Process management (ps, top, htop, kill, nice, renice, nohup)
Process states (running, sleeping, stopped, zombie, orphan)
Package management (apt, yum/dnf, rpm, dpkg, snap)
systemd (services, units, journalctl, systemctl)
Cron jobs (scheduling, crontab syntax, /etc/cron.d, at)
Disk management (df, du, lsblk, fdisk, mount, fstab)
Filesystem types (ext4, xfs, btrfs, tmpfs, NFS)
I/O redirection (stdin, stdout, stderr, >, >>, 2>&1, pipes)
Text processing (cat, head, tail, grep, wc, sort, uniq, cut)
Find and locate (find by name, size, time, type, -exec)
Archiving and compression (tar, gzip, bzip2, xz, zip)
SSH (key-based auth, ssh-keygen, ssh-agent, config, SCP, rsync)
Environment variables (export, PATH, .bashrc, .bash_profile)
vi/vim basics (modes, navigation, editing, saving)
Linux networking (ip, ifconfig, ping, traceroute, netstat, ss)
DNS tools (nslookup, dig, /etc/resolv.conf, /etc/hosts)
Firewall basics (iptables, ufw, firewalld)
Log files (/var/log/syslog, auth.log, dmesg, journalctl)
Screen and tmux (persistent terminal sessions)
Special files (/dev/null, /dev/zero, /dev/urandom, /dev/stdin)
Signals (SIGTERM, SIGKILL, SIGHUP, SIGUSR1, SIGINT)
Man pages, help, whatis, apropos (navigating documentation)
lsof, fuser, inotifywait (file monitoring and debugging)
tee, watch, diff, comm (everyday utility commands)

Linux Fundamentals

Q1. Linux boot process — BIOS/UEFI → bootloader → kernel → systemd

Scenario: A server reboots after a kernel update and never comes back. You're on the KVM console watching it hang. You need to know exactly which stage failed to know where to look.

Engineer-style Answer: Before guessing, I locate the failure by stage. Firmware (BIOS/UEFI) runs POST and hands off to the bootloader — if I see no GRUB menu, the problem is disk/boot-order/EFI partition. GRUB then loads the kernel and initramfs; a hang here usually means a bad initramfs or missing driver for root storage. The kernel mounts the real root and hands PID 1 to systemd, which brings up targets and services. If I get to a systemd emergency shell, the kernel booted fine and the issue is a failing mount or service. I'd check the last change first (the kernel update), boot the previous kernel from GRUB to confirm, read journalctl -b -1 for the failed boot, and check /etc/fstab for a bad UUID. Prevention: keep the previous kernel installed, test in staging, and never remove the known-good kernel.

Key Interview Signal: Locates failure by boot stage instead of randomly rebooting; treats "what changed" (the kernel update) as the first suspect.

Q2. Linux filesystem hierarchy — /, /etc, /var, /opt, /tmp, /proc, /sys

Scenario: An app writes ephemeral cache to / and fills the root filesystem, taking the box down. A teammate asks where things "should" live.

Engineer-style Answer: I think of the hierarchy by mutability and purpose. /etc is static config, /var is variable data that grows (logs, spool, caches), /tmp is scratch that may be wiped on reboot, /opt is add-on/third-party software, and /proc and /sys are virtual — kernel and device state, not real disk. The practical lesson here is that anything that grows (logs, uploads, DB data) belongs on /var or a dedicated mount, never on /. My fix is to move the cache to a sized, separate volume and set alerts on the root filesystem. Understanding the hierarchy is really about isolating growth so one runaway directory can't take down the OS.

Key Interview Signal: Frames the hierarchy around operational risk (growth isolation), not memorized definitions.

Q3. File permissions — read, write, execute, chmod, chown, chgrp

Scenario: A deploy user can't read a config file another team dropped in. You need to fix access without opening it up to the world.

Engineer-style Answer: I read permissions as three triplets — owner/group/other — and prefer fixing access through group membership rather than loosening other. First ls -l to see current owner, group, and mode. If the deploy user is meant to have access, I put it in the right group with chgrp/usermod -aG and give the group read (chmod g+r), rather than chmod 644 blindly or, worse, 777. On directories, execute means "can traverse," which people forget. I avoid the assumption that "permission denied" is always the file itself — it can be a parent directory missing x, or an ACL, or SELinux. Least privilege is the rule: grant the narrowest access that solves it.

Key Interview Signal: Reaches for group-based least privilege, not 777; knows directory x and parent-traversal gotchas.

Q4. Special permissions — SUID, SGID, sticky bit, umask

Scenario: A security scan flags a custom SUID-root binary in /usr/local/bin. You have to explain the risk and decide whether to keep it.

Engineer-style Answer: SUID means the binary runs as the file owner (often root) regardless of who launches it — powerful and dangerous, because a bug becomes a privilege-escalation path. SGID on a directory makes new files inherit the group, which is genuinely useful for shared team folders. The sticky bit on /tmp stops users deleting each other's files. umask sets default permissions for new files. For the flagged binary I'd ask: does it truly need root, or can I replace SUID with a capability (like cap_net_bind_service) or sudo with a tight rule? I'd audit with find / -perm -4000 to inventory all SUID binaries. Prevention: no custom SUID unless justified, prefer capabilities, and monitor for new SUID files.

Key Interview Signal: Treats SUID as an attack surface and proposes capabilities/sudo as safer alternatives.

Q5. File types — regular, directory, symlink, hard link, device, socket, pipe

Scenario: A backup job silently skips some data. You suspect symlinks pointing outside the backup root.

Engineer-style Answer: ls -l's first character tells me the type: - regular, d directory, l symlink, c/b char/block device, s socket, p named pipe. The key distinction here is symlink vs hard link: a symlink is a pointer to a path (can dangle, can cross filesystems), a hard link is another name for the same inode (same filesystem, survives the original being deleted). For the backup, I'd check whether the tool follows symlinks (tar -h, rsync -L) and whether targets are outside the root. I avoid assuming missing data means corruption — it's often a symlink the backup didn't follow, or a bind mount not traversed. Fix is to make link-following explicit and verify with a restore test.

Key Interview Signal: Distinguishes symlink vs hard link precisely and connects it to a real backup failure mode.

Q6. Users and groups — /etc/passwd, /etc/shadow, useradd, usermod

Scenario: A contractor left; you must revoke access cleanly without breaking services that run as service accounts.

Engineer-style Answer: /etc/passwd holds account metadata (UID, GID, home, shell); /etc/shadow holds password hashes and aging. My first move is to distinguish human accounts from service accounts — I don't want to delete a service account that a daemon depends on. For a departing human, I lock rather than delete first (usermod -L, set shell to /sbin/nologin, expire the account), revoke SSH keys in authorized_keys, and kill live sessions. Deleting immediately risks orphaned files and broken cron. I'd audit group membership (groups, /etc/group) for lingering sudo access. Prevention: central identity (LDAP/SSO) so offboarding is one action, and no shared accounts.

Key Interview Signal: Locks before deleting, separates human vs service accounts, thinks about centralized identity for offboarding.

Q7. Process management — ps, top, htop, kill, nice, renice, nohup

Scenario: A batch job is starving an API on the same host. You need to tame it without killing it mid-write.

Engineer-style Answer: I first see what's running with ps aux --sort=-%cpu or top/htop. Rather than immediately killing the batch job (which could corrupt a partial write), I lower its priority with renice and, if it's I/O-heavy, ionice. kill sends SIGTERM (graceful) by default; I only escalate to SIGKILL (-9) if it ignores TERM, because -9 gives no chance to clean up. nohup/disown matters when I want a process to survive logout. The real fix is not co-locating a noisy batch job with a latency-sensitive API — I'd move it to its own node or a cgroup with capped resources.

Key Interview Signal: Prefers renice/ionice over kill -9; knows SIGTERM-before-SIGKILL discipline and the real fix (isolation).

Q8. Process states — running, sleeping, stopped, zombie, orphan

Scenario: Monitoring shows a growing number of processes, and ps reveals many in Z state. The app team says "it's fine, they're using no CPU."

Engineer-style Answer: A zombie (Z) is a finished process whose parent hasn't reaped its exit status — it uses no CPU but holds a PID slot, so enough of them exhaust the PID table. The bug is in the parent not calling wait(). An orphan is different — its parent died, so init/systemd adopts and reaps it. So I don't kill zombies (you can't — they're already dead); I find and fix or restart the parent (ps -o ppid= -p <zombie>). I avoid the assumption that "no CPU" means harmless. Prevention: fix the parent's child-reaping logic, or in containers use a proper init (--init/tini) that reaps orphans.

Key Interview Signal: Knows you fix the parent, not the zombie, and cites PID exhaustion and container init reaping.

Q9. Package management — apt, yum/dnf, rpm, dpkg, snap

Scenario: A dnf update on one node pulled a newer library and broke the app, while other nodes are fine. You need consistent, reversible package state across the fleet.

Engineer-style Answer: apt/dnf are the high-level tools that resolve dependencies; dpkg/rpm are the low-level ones that install a single package without resolving deps. The root issue here is unpinned, drifting versions. Short term, I identify what changed (dnf history), and roll back with dnf history undo. Long term, I pin versions, use a controlled internal mirror/repo so all nodes see the same package set, and roll updates through staging first. I avoid --nodeps or force-installing, which just defers breakage. Prevention: immutable/base images or config management so package state is declarative and identical everywhere.

Key Interview Signal: Diagnoses version drift, uses transaction history to roll back, and pushes for pinned/declarative package state.

Q10. systemd — services, units, journalctl, systemctl

Scenario: A service keeps dying and "restarting forever." You need to know why and stop the crash loop from hiding the real error.

Engineer-style Answer: systemd manages services as units; systemctl status shows current state and the last few log lines, and journalctl -u <svc> -b gives full logs for this boot. A crash loop with Restart=always can mask the root cause, so I read the journal from the first failure, not the latest. I check the unit file for a wrong ExecStart, missing WorkingDirectory, or an env var only present in an interactive shell. I confirm dependencies (After=/Requires=) — often it starts before the DB or network is ready. Fix is usually a corrected unit plus Restart=on-failure with StartLimitBurst so it fails visibly instead of looping silently.

Key Interview Signal: Reads logs from first failure, checks unit dependencies and ordering, and hardens restart policy to surface errors.

Q11. Cron jobs — scheduling, crontab syntax, /etc/cron.d, at

Scenario: A nightly report "sometimes" doesn't run, and no one noticed for a week. You must make it reliable and observable.

Engineer-style Answer: Cron's five fields are minute, hour, day-of-month, month, day-of-week. The classic failures are: it runs with a minimal environment and bare PATH; overlapping runs collide; and failures are silent because no one reads mailed output. I'd confirm the schedule, check /var/log/cron or journald for whether it even fired, and wrap the job so it logs to a file, locks with flock to prevent overlap, and emits a success/failure signal to monitoring (a dead-man's-switch alert if it doesn't run). For anything critical I'd move it to a systemd timer, which gives better logging and dependency handling. at is for one-off future jobs.

Key Interview Signal: Knows cron's silent-failure and PATH traps, adds locking and dead-man's-switch alerting, and considers systemd timers.

Q12. Disk management — df, du, lsblk, fdisk, mount, fstab

Scenario: You need to add a new 500GB volume to a running database server for data growth, with no downtime.

Engineer-style Answer: lsblk shows block devices and mounts, df -h shows filesystem usage, du shows what's consuming space within a path. For the new volume I attach it, confirm the device with lsblk, partition if needed, make a filesystem, mount it, then add it to /etc/fstab — by UUID, not /dev/sdX, because device names can reorder on reboot. I test the fstab entry with mount -a before trusting it, since a bad fstab can prevent the next boot. I'd mount it at the DB data path (or migrate data with the service stopped/using replication). Prevention: use LVM so I can grow the volume later without re-doing partitions.

Key Interview Signal: Mounts by UUID, validates fstab before reboot, and reaches for LVM for future growth.

Q13. Filesystem types — ext4, xfs, btrfs, tmpfs, NFS

Scenario: You're choosing a filesystem for a high-throughput database volume and someone suggests "just use the default."

Engineer-style Answer: I pick by workload. ext4 is the safe, well-understood general default. xfs handles large files and high parallel I/O well — a common choice for databases and big data. btrfs gives snapshots and checksums but adds complexity. tmpfs is RAM-backed — great for scratch/cache, gone on reboot. NFS is network-shared and brings latency and locking concerns, so I'd avoid it for a primary DB. For this DB volume I'd lean xfs, size the volume with headroom, and confirm mount options like noatime to cut needless writes. The point is matching the filesystem to access pattern and durability needs, not defaulting blindly.

Key Interview Signal: Chooses filesystem by workload characteristics and knows tmpfs/NFS trade-offs.

Q14. I/O redirection — stdin, stdout, stderr, >, >>, 2>&1, pipes

Scenario: A script "logs nothing" on failure because errors go to the terminal but your logfile only captures stdout.

Engineer-style Answer: Three streams: stdin (0), stdout (1), stderr (2). > truncates, >> appends. The bug here is capturing only stdout — errors go to stderr. cmd > file 2>&1 sends both to the file, but order matters: 2>&1 > file does not work as people expect because redirection is evaluated left to right. Pipes connect stdout of one command to stdin of the next; to pipe stderr too you redirect it first. For scripts I send structured logs to a file and let errors flow to both the log and monitoring. Prevention: standardize a logging function so every script captures both streams consistently.

Key Interview Signal: Gets the ordering subtlety of 2>&1 and standardizes stream capture in scripts.

Q15. Text processing — cat, head, tail, grep, wc, sort, uniq, cut

Scenario: During an incident you need the top offending IPs from an access log fast, without loading a 10GB file into an editor.

Engineer-style Answer: These compose into a pipeline. tail -f to watch live, grep to filter, cut/awk to pull a field, sort | uniq -c | sort -rn to rank frequencies, wc -l to count. For top IPs it's roughly awk '{print $1}' access.log | sort | uniq -c | sort -rn | head. I avoid cat file | grep (useless-use-of-cat) and reach for streaming tools that never load the whole file into memory. uniq only dedupes adjacent lines, so it must follow sort. The habit that matters in an incident is composing small tools into a fast, memory-safe pipeline rather than opening the file.

Key Interview Signal: Builds a streaming pipeline, remembers uniq needs sort first, avoids loading huge files.

Q16. Find and locate — find by name, size, time, type, -exec

Scenario: You must delete log files older than 30 days across many directories to reclaim space — safely, without nuking anything active.

Engineer-style Answer: find searches live by predicate — name, -size, -mtime, -type — while locate uses a prebuilt (possibly stale) database, so for a cleanup I use find. My discipline is to run it read-only first: find /var/log -type f -name '*.log' -mtime +30 -print and eyeball the list before adding -delete or -exec rm {} +. I avoid -exec rm on unreviewed matches and never combine broad paths with delete casually. -mtime +30 means older than 30 days — off-by-one confusion here can delete the wrong set. Prevention: proper logrotate config so manual cleanups aren't needed.

Key Interview Signal: Dry-runs with -print before deleting, knows mtime semantics, and treats logrotate as the real fix.

Q17. Archiving and compression — tar, gzip, bzip2, xz, zip

Scenario: You need to ship a 40GB directory to another region nightly; the transfer window is tight and CPU is shared.

Engineer-style Answer: tar bundles files; the compressor is a separate choice with a speed/ratio trade-off: gzip is fast and light, xz compresses best but is CPU-heavy and slow, bzip2 is in between. For a nightly transfer on shared CPU I'd weigh whether the bottleneck is bandwidth or CPU — if bandwidth is scarce, better compression pays off; if CPU is contended, gzip or parallel pigz is safer. Often the smarter answer isn't tar at all but rsync, which only sends deltas and resumes on failure. Prevention: incremental strategy (rsync or tar snapshots) so I'm not re-shipping unchanged data every night.

Key Interview Signal: Reasons about the compression trade-off by bottleneck and proposes rsync/incremental over full re-transfer.

Q18. SSH — key-based auth, ssh-keygen, ssh-agent, config, SCP, rsync

Scenario: The team is still using passwords and shared keys. You're asked to move to secure, auditable SSH access.

Engineer-style Answer: Key-based auth replaces passwords with a keypair — private key stays with the user, public key goes in authorized_keys. ssh-agent holds the decrypted key so you're not retyping the passphrase; ~/.ssh/config stores per-host settings (user, jump host, key). I'd disable password auth and root login in sshd_config, require per-user keys (no shared keys, so actions are attributable), and use a bastion/jump host. scp and rsync ride over SSH for transfers, rsync being preferable for deltas and resume. Prevention: short-lived certificates or an SSH CA so keys rotate automatically and offboarding is instant.

Key Interview Signal: Pushes per-user keys for auditability, disables password/root login, and mentions SSH CA for rotation.

Q19. Environment variables — export, PATH, .bashrc, .bash_profile

Scenario: A script that works in your terminal fails under a service because a command "isn't found," even though it's installed.

Engineer-style Answer: export makes a variable available to child processes; PATH decides where commands are looked up. The subtle part is which file runs when: .bash_profile runs for login shells, .bashrc for interactive non-login shells — and services and cron get neither, so an interactive-only PATH or env var won't exist there. My fix is to not rely on the login environment: use absolute paths or set the needed env explicitly in the unit file / script. I avoid the assumption that "installed" means "on PATH for this context." Prevention: define required environment in the systemd unit (Environment=) so it's explicit and reproducible.

Key Interview Signal: Knows login vs non-login vs service environments differ and sets env explicitly rather than depending on shell rc files.

Scenario: You're on a minimal recovery console with only vi available and must edit /etc/fstab to fix a boot problem.

Engineer-style Answer: vi is modal — normal mode for navigation/commands, insert mode (i) for typing, and command-line mode (:) for save/quit. In a recovery situation the essentials are: i to edit, Esc to leave insert, :wq to save and quit, :q! to bail without saving if I made a mistake, and u to undo. On a read-only root I'd remount read-write first (mount -o remount,rw /). The reason vi matters is that it's on virtually every system, including rescue images where nothing else is installed. I'd make the smallest change, save, and verify before rebooting — a broken fstab edit can leave the box unbootable again.

Key Interview Signal: Comfortable in a rescue context, remounts rw, makes minimal verified edits with an escape hatch (:q!).

Q21. Linux networking — ip, ifconfig, ping, traceroute, netstat, ss

Scenario: An app can't reach a downstream service and the team blames "the network." You need to localize the fault fast.

Engineer-style Answer: I work up the stack instead of accepting "the network is down." ip a / ip r confirm the interface has an address and a route. ping tests L3 reachability (but ICMP may be blocked, so absence isn't proof). traceroute/mtr shows where packets die. ss -tulpn (modern replacement for netstat) shows what's listening locally and whether the app is even bound to the right port/interface. Often "network" is actually a wrong bind address, a missing route, a firewall/security-group rule, or DNS — not the wire. I test from the same host to isolate local vs remote, then communicate a specific hypothesis ("app can resolve but connection is refused at the security group") rather than a vague "network issue."

Key Interview Signal: Localizes layer by layer, uses ss over netstat, and refuses the vague "it's the network" framing.

Q22. DNS tools — nslookup, dig, /etc/resolv.conf, /etc/hosts

Scenario: Some requests resolve to a stale IP after a DNS record change. You must find whether it's caching, the resolver, or a stray hosts entry.

Engineer-style Answer: dig is my precise tool — it shows the answer, the TTL, and which server answered, and I can query a specific resolver with @. I first check /etc/hosts, because a static entry silently overrides DNS and is a classic "why is this one box wrong" cause. Then I check /etc/resolv.conf for which resolver is used, and compare dig @<resolver> vs dig @8.8.8.8 to spot stale caches. TTL explains how long stale answers linger. nslookup works but dig gives more detail. Prevention: shorter TTLs before planned changes, and no ad-hoc /etc/hosts overrides in production.

Key Interview Signal: Checks /etc/hosts first, uses dig @server to compare resolvers, and reasons about TTL/caching.

Q23. Firewall basics — iptables, ufw, firewalld

Scenario: After tightening the firewall, a service becomes unreachable and you're unsure if it's the firewall or the app.

Engineer-style Answer: A firewall filters by rules evaluated in order with a default policy. Before blaming it, I confirm the app is actually listening (ss -tulpn) and then check rules (iptables -L -n -v or firewall-cmd --list-all). The most common self-inflicted outage is a default-DROP policy without an allow rule, or rule ordering where a broad DROP precedes the specific ALLOW. ufw and firewalld are friendlier frontends to netfilter. Critical habit: when editing remote firewall rules I use a safety net — a scheduled iptables flush or a rollback timer — so I don't lock myself out. Prevention: manage rules as code and test in staging.

Key Interview Signal: Confirms the app listens first, checks rule order/default policy, and guards against locking itself out remotely.

Q24. Log files — /var/log/syslog, auth.log, dmesg, journalctl

Scenario: A server had "a problem overnight" and you have no monitoring detail — just the logs on the box.

Engineer-style Answer: I go to logs by category: journalctl (or /var/log/syslog/messages) for general system events, auth.log/secure for logins and sudo, dmesg/journalctl -k for kernel and hardware (disk errors, OOM kills, link resets). I anchor to the incident window with journalctl --since/--until rather than scrolling blindly. I look for the first anomaly, not just the loudest error, because the first event is usually the cause and the rest are cascades. I correlate across files by timestamp. Prevention: ship logs to central storage so I'm not dependent on a box that may be the very thing that failed.

Key Interview Signal: Anchors to the time window, hunts the first anomaly, and pushes for centralized logging.

Q25. Screen and tmux — persistent terminal sessions

Scenario: You're running a 3-hour migration over SSH and your laptop Wi-Fi is flaky. A disconnect mid-run would be disastrous.

Engineer-style Answer: The risk is that a long foreground command dies when the SSH session drops (SIGHUP). tmux (or screen) keeps the session alive on the server — I start tmux, run the migration inside, and if I disconnect I reattach with tmux attach and it's still going. This is far safer than nohup for interactive, multi-window work, and it lets a teammate attach to the same session to help. For truly unattended jobs I'd prefer a systemd unit or a job runner, but for a hands-on migration tmux is the right tool. Prevention: never run long operations in a bare SSH foreground.

Key Interview Signal: Understands SIGHUP-on-disconnect and uses tmux for resilience and collaboration on long ops.

Q26. Special files — /dev/null, /dev/zero, /dev/urandom, /dev/stdin

Scenario: A script generates gigabytes of debug output you don't need, and separately you need to create a test file of a specific size.

Engineer-style Answer: These are virtual devices. /dev/null is the bit bucket — redirect unwanted output there (2>/dev/null) to silence noise. /dev/zero emits endless zeros — handy to create a fixed-size file with dd. /dev/urandom gives cryptographically-usable random bytes for keys, tokens, or randomized test data. /dev/stdin lets you treat piped input as a file path. I'm careful discarding to /dev/null — dropping stderr can hide real errors, so I only silence output I've confirmed is noise. For sizing test files I'd use /dev/zero for speed or /dev/urandom when incompressible data is needed.

Key Interview Signal: Uses the right device per need and warns against silently discarding stderr.

Q27. Signals — SIGTERM, SIGKILL, SIGHUP, SIGUSR1, SIGINT

Scenario: A service leaves corrupt state when restarted. You suspect it's being hard-killed before it can flush.

Engineer-style Answer: SIGTERM (15) is a polite "please shut down" the app can trap to flush and close cleanly; SIGKILL (9) is unblockable and gives no chance to clean up — so hard-killing is exactly how you get corrupt state. SIGINT is Ctrl-C. SIGHUP often means "reload config" for daemons or "terminal closed" for shells. SIGUSR1/2 are app-defined (e.g., rotate logs). My fix is to ensure the shutdown path sends TERM, waits (systemd's TimeoutStopSec), and only escalates to KILL if it hangs — and to make sure the app actually handles TERM to flush. I'd avoid any script or orchestrator that jumps straight to -9.

Key Interview Signal: Ties corruption to premature SIGKILL and prescribes graceful TERM-then-KILL with a timeout.

Q28. Man pages, help, whatis, apropos — navigating documentation

Scenario: You're on an air-gapped production box with no internet and need the exact flag for a command you rarely use.

Engineer-style Answer: Offline, the docs are already on the box. man cmd is authoritative and section-numbered (section 1 commands, 5 file formats, 8 admin). apropos keyword (or man -k) searches man page descriptions when I know what I want but not the command name. whatis gives the one-line summary. For shell builtins help is the source, since they have no man page. cmd --help is the quick reference. The senior habit is trusting the installed man pages over half-remembered flags or random web snippets — especially in production where a wrong flag is costly. No internet is never an excuse to guess.

Key Interview Signal: Uses on-box docs (apropos/man sections/help) confidently instead of guessing flags.

Q29. lsof, fuser, inotifywait — file monitoring and debugging

Scenario: You can't unmount a volume — "device is busy" — and you need to know exactly what's holding it.

Engineer-style Answer: "Device busy" means a process has an open handle or a working directory on that mount. lsof +D /mountpoint or fuser -vm /mountpoint lists exactly which PIDs and users are holding it. I identify the offender, decide whether it's safe to stop (a stuck process vs a critical service), and coordinate before killing anything. inotifywait is different — it watches for file events in real time, useful to confirm whether a process is actually reading/writing a file I expect. I avoid the lazy umount -l (lazy unmount) as a first move because it can hide the real dependency and cause surprises. Find the holder, resolve it cleanly.

Key Interview Signal: Finds the exact holder with lsof/fuser before force-unmounting, and resolves rather than lazy-unmounting blindly.

Q30. tee, watch, diff, comm — everyday utility commands

Scenario: You're running a risky command and want its output both on screen and saved, and later you need to compare two config files across servers.

Engineer-style Answer: tee splits a stream to both stdout and a file — perfect for capturing an audit trail of a risky command while still watching it live (cmd | tee run.log). watch re-runs a command periodically so I can eyeball a changing value (like watch -n1 'ss -s') during an incident. diff compares two files line by line to spot config drift between servers; comm compares two sorted lists for common/unique lines. For config drift I'd diff the two files and, better, manage them with config management so drift can't silently accumulate. These small tools save time and create evidence during incidents.

Key Interview Signal: Uses tee for an audit trail, watch for live monitoring, and treats diff drift as a config-management problem.

Shell Scripting

Q31. Shell scripting fundamentals — shebang, variables, quoting, exit codes

Scenario: A shared script behaves differently on two machines and sometimes mangles filenames with spaces.

Engineer-style Answer: The shebang (#!/usr/bin/env bash) pins the interpreter — relying on the default shell is exactly how a script works on one box and breaks on another that links sh to dash. Unquoted variables are the spaces-in-filenames bug: I always quote "$var" and use "$@" for arguments. Exit codes matter — 0 is success, non-zero is failure, and I check $? or rely on set -e. My defaults for any production script are set -euo pipefail, quoted expansions, and shellcheck in CI. I avoid assuming the environment; I make dependencies and the interpreter explicit. That turns "works on my machine" into "works everywhere."

Key Interview Signal: Pins the interpreter, quotes expansions, checks exit codes, and standardizes on set -euo pipefail + shellcheck.

Q32. Conditionals — if/elif/else, test, [[ ]], comparisons

Scenario: A script's condition silently misbehaves on empty input, taking a destructive branch it shouldn't.

Engineer-style Answer: I prefer [[ ]] over [ ] in bash because it's safer with unset/empty variables and doesn't word-split. The classic bug is [ $x = y ] blowing up when $x is empty — quoting ([[ "$x" == y ]]) fixes it. Numeric vs string comparison is a common trap: -eq/-lt for numbers, ==/< for strings; mixing them gives wrong results. Before a destructive branch I explicitly guard against empty/unset ([[ -z "$x" ]] and set -u), because "assume input is present" is how scripts delete the wrong thing. I test conditions with representative bad input, not just the happy path.

Key Interview Signal: Uses [[ ]], guards empty/unset before destructive actions, and knows numeric vs string operators.

Q33. Loops — for, while, until, break, continue

Scenario: A for loop over filenames breaks whenever a file has spaces, and a while read loop mysteriously skips the last line.

Engineer-style Answer: Looping over $(ls) or unquoted command output word-splits on spaces — I loop over a glob (for f in *.log) or use find -print0 | while read -r -d '' for arbitrary names. For reading files, while IFS= read -r line is the safe idiom, and the "skips last line" symptom is usually a file with no trailing newline — I handle it with the || [[ -n "$line" ]] pattern. break/continue control flow; I avoid piping into while when I need to mutate outer variables, because the pipe creates a subshell and the changes vanish. Correct loop idioms prevent a whole class of silent data bugs.

Key Interview Signal: Knows the safe read idiom, -print0 for odd names, and the subshell-variable pitfall of piping into while.

Q34. Functions — declaration, arguments, return, local vars, getopts

Scenario: A growing script has variables leaking between functions and causing intermittent, hard-to-reproduce bugs.

Engineer-style Answer: In bash, variables are global by default, so the leak is expected unless I declare local inside functions — that's my first fix. Functions take positional args ($1, $@), and return only passes a small exit status (0–255), so to return data I echo it and capture with command substitution, or use a nameref. For flag parsing I use getopts rather than hand-rolled $1/shift chains, because it handles combined flags and errors cleanly. I keep functions small, single-purpose, and testable. Preventing the intermittent bug is really about discipline: local everything, quote expansions, and run shellcheck, which flags exactly these leaks.

Key Interview Signal: Uses local to stop leaks, knows return is status-only, and uses getopts for real argument parsing.

Q35. Arrays and associative arrays in Bash

Scenario: A script builds a command from a list of arguments (some with spaces) and it keeps splitting incorrectly.

Engineer-style Answer: The right tool is a real array, expanded as "${args[@]}" — that preserves each element as a distinct word even with spaces, which a flat space-separated string cannot. Indexed arrays hold ordered lists; associative arrays (declare -A map) are key/value, great for lookups like host→role. The common bug is "${args[*]}" (joins into one string) vs "${args[@]}" (separate words) — I want @ for command building. I also remember associative arrays need bash 4+, so on old/macOS bash they may not exist. Using arrays instead of string concatenation is the clean fix for the splitting problem.

Key Interview Signal: Reaches for "${arr[@]}" to build commands safely and knows @ vs * and the bash-4 requirement.

Q36. String manipulation — substring, replacement, length, splitting

Scenario: You need to sanitize and reformat thousands of filenames or config values inside a script, and shelling out to sed for each one is slow.

Engineer-style Answer: Bash parameter expansion does most of this in-process, no subprocess needed: ${var:offset:len} for substrings, ${var//old/new} for replace-all, ${#var} for length, ${var%%.*}/${var#*/} for trimming suffixes/prefixes. Doing it natively avoids spawning sed/cut thousands of times, which is the actual performance fix. For splitting I set IFS and read into an array rather than fragile cutting. I keep transformations explicit and quote results. When logic gets genuinely complex (multi-field parsing, regex-heavy), that's my signal it's outgrown bash and Python would be clearer and more maintainable.

Key Interview Signal: Uses native parameter expansion for speed and knows when complexity signals a move to Python.

Q37. jq for JSON processing — parsing API output and configs

Scenario: A deploy script greps AWS/Kubernetes JSON output and breaks whenever the API adds a field or changes formatting.

Engineer-style Answer: Parsing JSON with grep/awk is fragile because JSON isn't line-oriented — key order and whitespace can change. jq parses the structure properly: jq -r '.items[].metadata.name' extracts exactly what I want regardless of formatting. I use -r for raw strings so downstream commands aren't confused by quotes, and I handle missing keys with // empty or ? so one absent field doesn't crash the pipeline. For robustness I validate that the JSON is well-formed before acting on it. Switching from grep to jq removes a whole class of "worked until the API changed" failures. Prevention: never text-scrape structured data.

Key Interview Signal: Rejects grepping JSON, uses jq -r with null-safety, and states the principle: parse structured data structurally.

Q38. cURL deep dive — headers, auth, debugging API calls

Scenario: A service-to-service API call "sometimes" returns 401 in production but works from your laptop. You must debug it precisely.

Engineer-style Answer: I make the request reproducible and observable. curl -v (or -i) shows request/response headers so I can confirm the exact Authorization header, content type, and status. The laptop-vs-prod difference is usually auth: a token that's expired, scoped differently, or picked up from a different environment. I check the response body and WWW-Authenticate header for the real reason, use -w to capture timing and HTTP code for scripting, and --fail/--fail-with-body so scripts actually detect HTTP errors instead of treating a 401 body as success. I avoid assuming "works locally" means the request is identical — I diff the actual headers. Prevention: centralized secret management and health checks on token validity.

Key Interview Signal: Makes the call observable with -v/-w, uses --fail so scripts catch HTTP errors, and diffs environments rather than assuming.

Q39. Command substitution, process substitution, subshells

Scenario: A script sets a counter inside a while loop fed by a pipe, and the counter is always zero afterward.

Engineer-style Answer: That's the classic subshell trap: the right side of a pipe runs in a subshell, so variable changes there don't survive into the parent. The fix is process substitution — while read ...; do ...; done < <(command) — which runs the loop in the current shell, preserving the counter. Command substitution $(...) captures output but also runs in a subshell and strips trailing newlines. Backticks are the legacy form; I use $(...) because it nests cleanly. Knowing where a subshell boundary sits is what separates "why is my variable empty" confusion from a correct script. Prevention: prefer < <(...) over piping into loops when state must persist.

Key Interview Signal: Diagnoses the pipe-subshell variable loss and fixes it with process substitution.

Q40. grep, sed, awk — the text processing power trio

Scenario: During an incident you need to extract and summarize fields from millions of log lines quickly, without a heavy tool.

Engineer-style Answer: I pick by job: grep to find/filter lines, sed to edit/substitute in a stream, awk for field-based extraction and aggregation. For "sum request bytes by status code" or "count by field," awk is the right tool in one pass — awk '{sums[$9]+=$10} END{for(k in sums) print k,sums[k]}' — no need to load the file. I stream rather than slurp, so memory stays flat even on huge files. I reach for grep first (fastest), sed for transforms, awk when there's logic or math per line. If the parsing gets truly complex (nested structure, JSON), that's my cue to switch to jq or Python instead of an unreadable awk one-liner.

Key Interview Signal: Chooses the right tool per task, aggregates in a single streaming awk pass, and knows when to graduate to Python/jq.

Q41. Regular expressions — basic vs extended, classes, quantifiers

Scenario: A log-parsing regex works in grep -E but fails in plain grep, and a "simple" pattern matches far more than intended.

Engineer-style Answer: The tool-to-tool difference is BRE vs ERE: in basic regex +, ?, |, () are literal unless backslash-escaped, while grep -E/awk treat them as operators. So a pattern that "works in -E" naturally fails in plain grep — I make the mode explicit rather than guess. Over-matching is usually greedy quantifiers (.*) swallowing too much; I tighten with character classes ([0-9]), anchors (^/$), and word boundaries. I test regexes against known good and bad samples before trusting them in a pipeline that filters production data. Prevention: for anything non-trivial, prefer a real parser (jq for JSON, structured fields in awk) over an ever-growing regex.

Key Interview Signal: Knows BRE vs ERE, tames greedy .* with anchors/classes, and tests against negative cases.

Q42. xargs and parallel execution — processing argument lists

Scenario: You must run an operation across 10,000 items; a naive loop takes hours and a naive xargs blows up on filenames with spaces.

Engineer-style Answer: xargs batches items into command invocations, and with -P it parallelizes — turning an hours-long serial loop into minutes. The safety rule is find ... -print0 | xargs -0 (or -d '\n') so spaces and odd characters don't split arguments. I control fan-out with -P sized to CPU/IO so I don't overwhelm the box or a downstream API, and -n to batch. For rate-limited targets I cap concurrency deliberately. I avoid unbounded parallelism, which can self-DoS a dependency. If the per-item logic is complex, GNU parallel or a small script gives better error handling. The trade-off is speed vs load, and I tune it consciously.

Key Interview Signal: Uses -0 for safety and bounded -P for controlled parallelism, weighing speed against downstream load.

Q43. ACLs — setfacl, getfacl, when basic permissions aren't enough

Scenario: Two teams need different access to the same directory, and the standard owner/group/other model can't express it cleanly.

Engineer-style Answer: Classic Unix permissions give exactly one owner and one group, so "team A read-write, team B read-only, everyone else nothing" doesn't fit. POSIX ACLs solve this: setfacl -m g:teamA:rwx,g:teamB:rx dir grants per-group rights beyond the base model, and getfacl shows them. I set default ACLs on the directory (-d) so new files inherit the policy instead of drifting. The trade-offs: ACLs are powerful but less visible than ls -l (a + hints at them), so they can surprise the next engineer, and not every filesystem/backup tool preserves them. I document them and prefer group design first, reaching for ACLs only when the base model genuinely can't express the requirement.

Key Interview Signal: Knows the limit of the base model, uses default/inherited ACLs, and flags the reduced visibility as a maintenance cost.

Q44. trap and signal handling — cleanup on exit/error

Scenario: A deploy script that grabs a lock and creates temp files sometimes dies mid-run and leaves stale locks that block the next run.

Engineer-style Answer: Stale state after a crash means the script isn't cleaning up on abnormal exit. trap 'cleanup' EXIT runs cleanup on any exit path — success, error, or signal — so the lock and temp files always get released. I also trap INT/TERM so a Ctrl-C or a systemd stop still cleans up. Inside cleanup I remove the lockfile and temp dir idempotently (safe to run twice). This is the reliable pattern for locks, temp resources, and partial deploys. Combined with set -euo pipefail, it means the script fails safely instead of leaving landmines. Prevention: every resource acquisition pairs with a trap-based release.

Key Interview Signal: Uses trap ... EXIT for guaranteed idempotent cleanup, covering signals, not just the happy path.

Q45. Error handling — set -euo pipefail, error trapping, exit codes

Scenario: A backup script reported "success" every night for weeks, but the backups were empty because a step failed silently.

Engineer-style Answer: "Reported success while failing" is the exact failure set -euo pipefail prevents: -e exits on any command failure, -u errors on unset variables, and pipefail makes a pipeline fail if any stage fails — without it, false | tee log returns success. I add an ERR trap to log where it died, and I verify the result, not just the exit code — a backup script should confirm the archive exists and is non-empty, then test a restore. I'm careful that set -e has gaps (it won't trigger inside some conditionals), so I don't treat it as a substitute for checking critical steps explicitly. Prevention: validate outcomes and alert on absence of success, not just presence of errors.

Key Interview Signal: Explains why pipefail matters, validates the actual result, and knows set -e's blind spots.

Q46. Here documents, here strings, template generation

Scenario: A provisioning script generates config files, and variable interpolation is either not happening or happening when it shouldn't.

Engineer-style Answer: A heredoc (<<EOF) feeds a block to a command; the key control is the delimiter. Unquoted <<EOF expands $vars and backticks — good for templating with values — while quoted <<'EOF' is literal, which I use when the file itself contains $ I must not expand (like another script or a systemd unit). That single quoting choice is usually the whole bug. A here string (<<<) feeds one line to stdin. For real templating with logic I'd prefer a proper tool (envsubst, Jinja via Python) over ballooning heredocs, but for a few files heredocs are clean. Prevention: pick quoted vs unquoted deliberately and test the rendered output.

Key Interview Signal: Controls interpolation via quoted vs unquoted delimiter and knows when to graduate to a templating tool.

Q47. File descriptors, ulimits — too many open files, production tuning

Scenario: A busy service starts throwing "Too many open files" under peak load and begins dropping connections.

Engineer-style Answer: Every socket and file is an FD, and the process hit its nofile limit. I confirm by checking the limit (cat /proc/<pid>/limits) and current usage (ls /proc/<pid>/fd | wc -l, or lsof -p). But before just raising the limit, I check whether it's a genuine load ceiling or an FD leak — a service that never closes connections will exhaust any limit. If it's real load, I raise nofile via systemd LimitNOFILE= (the durable place, not just ulimit in a shell) and the system-wide fs.file-max. If it's a leak, raising the limit only delays the crash, so I fix the code. Prevention: monitor FD usage as a leading indicator and set sane limits in the unit file.

Key Interview Signal: Distinguishes real load from an FD leak before raising limits, and sets LimitNOFILE durably.

Q48. fstab and mount options — noexec, nosuid, nodev, defaults

Scenario: Security review requires that /tmp and upload directories can't be used to execute malware or escalate privileges.

Engineer-style Answer: Mount options are a cheap, strong control. noexec blocks running binaries from that mount, nosuid neutralizes SUID escalation, nodev blocks device-file tricks — the standard hardening trio for /tmp, /var/tmp, and world-writable upload areas. I add them in /etc/fstab and remount to apply, then verify with mount | grep tmp. I test that legitimate apps don't rely on executing from /tmp first, because some installers do, so I coordinate before flipping noexec. I always validate fstab with mount -a before rebooting, since a bad entry can block boot. Prevention: bake these options into the base image so every new host is hardened by default.

Key Interview Signal: Applies noexec/nosuid/nodev to writable mounts, tests for app impact, and validates fstab before reboot.

Q49. File locking and concurrent execution — flock

Scenario: A cron job occasionally overlaps with a still-running previous instance and they corrupt shared output.

Engineer-style Answer: Overlapping runs mean the job has no mutual exclusion. flock gives a lockfile-based lock so only one instance runs: flock -n /var/lock/job.lock -c 'the_command' — the -n makes the second instance exit immediately instead of piling up. This is more reliable than a hand-rolled "does a PID file exist" check, which has race conditions and goes stale after crashes (flock releases automatically when the process dies). I decide the policy consciously: skip if locked, or wait — for cron I usually skip and log. Prevention: wrap every periodic job that touches shared state in flock, and alert if it's been skipped repeatedly (a sign runs are taking too long).

Key Interview Signal: Uses flock over racy PID files, chooses skip-vs-wait deliberately, and alerts on repeated skips.

Q50. Process management from scripts — background, wait, job control

Scenario: A script launches several long tasks in the background but exits before they finish, or doesn't notice when one of them fails.

Engineer-style Answer: Backgrounding with & returns immediately, so without wait the script exits and may orphan or lose track of the children. I capture each PID, wait on them, and check each exit status so one failed task isn't silently ignored (wait "$pid"; status=$?). For bounded parallelism I cap how many run at once rather than launching hundreds. I trap to ensure children are cleaned up if the parent is killed, avoiding orphans. If the coordination gets complex — retries, dependencies, partial failure handling — that's the signal to use a real job runner or Python rather than hand-managing PIDs. The core discipline is: never fire-and-forget work whose success I care about.

Key Interview Signal: Waits on and checks each background job's status, bounds concurrency, and cleans up children on exit.

Q51. Bash debugging — set -x, bash -x, PS4, shellcheck

Scenario: A production script fails intermittently and the logs don't show which command or variable value caused it.

Engineer-style Answer: I make execution visible. set -x (or running bash -x script) traces every command with expanded variables, so I see the actual values at the failure point rather than guessing. I enrich the trace with a custom PS4='+${BASH_SOURCE}:${LINENO}: ' so each line shows file and line number. For intermittent issues I enable tracing to a separate log so I don't spam stdout. Before it ever ships, shellcheck catches the common bugs statically — quoting, unset vars, subshell traps. So my order is: shellcheck in CI to prevent classes of bugs, and set -x/PS4 to diagnose the ones that slip through. Prevention: shellcheck as a required gate.

Key Interview Signal: Combines static analysis (shellcheck as a gate) with runtime tracing (set -x + informative PS4).

Q52. Backup strategies — rsync, dd, incremental tar, snapshots

Scenario: You must design backups for a growing dataset with tight backup windows and a real requirement to restore quickly after data loss.

Engineer-style Answer: I start from requirements: RPO (how much data can I lose), RTO (how fast must I restore), data size, and change rate. rsync with --link-dest gives space-efficient incremental snapshots and only moves deltas — good for the tight window. dd is a raw block copy, useful for full-disk images but heavy and offline. Incremental tar works but restores are slower. Filesystem/LVM/cloud snapshots are fast and low-impact for large volumes. The rule I never skip: a backup you haven't restored isn't a backup — I test restores regularly. I also keep offsite/immutable copies (3-2-1) so ransomware or a bad actor can't delete them. Prevention: automate, monitor for backup success and restore validation, and alert on absence.

Key Interview Signal: Designs from RPO/RTO, uses incremental/snapshots for scale, and insists on tested restores + 3-2-1 immutability.

Advanced Linux & Systems

Q53. Systemd targets (run levels) — multi-user, rescue, emergency

Scenario: A box boots to a broken graphical/networked state and you need a minimal environment to fix it without full services starting.

Engineer-style Answer: systemd targets replace old runlevels: multi-user.target is the normal server state (networking, all services, no GUI), graphical.target adds a desktop, rescue.target is single-user with basic mounts for repair, and emergency.target is the barest shell with root mounted read-only — for when even rescue won't come up. To fix a broken boot I add systemd.unit=rescue.target (or emergency) to the kernel line in GRUB, repair the offending service or fstab, then boot normally. I check the default with systemctl get-default. The skill is choosing the least environment that still lets me fix the problem, so a bad service can't interfere with the repair.

Key Interview Signal: Maps targets to recovery needs and boots into the minimal one that still allows repair.

Q54. Network bonding/teaming — HA networking

Scenario: A single NIC or top-of-rack switch failure takes a critical server offline. You're asked to remove that single point of failure.

Engineer-style Answer: Bonding aggregates multiple NICs into one logical interface for redundancy and/or throughput. For availability I'd use active-backup (mode 1) — traffic flows over one NIC and fails over if it drops — which tolerates a NIC or cable failure without app impact. For throughput with switch support there's 802.3ad/LACP (mode 4), but that needs the switch configured too, and to survive a switch failure the two NICs must land on two different switches. I verify failover by physically/logically pulling a link and watching it hold. Trade-offs: LACP needs switch cooperation and matching config; active-backup is simpler and more forgiving. Prevention: pair NIC redundancy with redundant switches and monitor bond member state.

Key Interview Signal: Chooses bonding mode by goal (HA vs throughput), notes switch requirements, and actually tests failover.

Q55. Bash vs sh vs zsh vs dash — shell compatibility

Scenario: A script with #!/bin/sh uses bash-only features and breaks on a Debian/Ubuntu host where /bin/sh is dash.

Engineer-style Answer: sh is a POSIX spec, not one program — on Debian/Ubuntu /bin/sh links to dash, which is fast and minimal but lacks bash extras like [[ ]], arrays, and ${var//}. So a script using those with a #!/bin/sh shebang is misdeclared and breaks. The fix is honest labeling: if I use bash features, the shebang must be #!/usr/bin/env bash; if I need broad portability, I write strict POSIX sh and test with dash/shellcheck -s sh. zsh is interactive-focused and not something I'd assume on servers. The principle is matching the shebang to the features I actually use, and testing on the target shell rather than assuming bash everywhere.

Key Interview Signal: Knows /bin/sh→dash on Debian, aligns shebang with features used, and tests portability explicitly.

Q56. LVM — physical volumes, volume groups, logical volumes

Scenario: A database volume is filling up and you need to grow it online, and later move it to faster disks without downtime.

Engineer-style Answer: LVM abstracts physical disks (PVs) into a pool (VG) that you carve into resizable logical volumes (LVs) — which is exactly what makes online growth possible. To extend: add/enlarge the disk, pvresize/vgextend to grow the pool, lvextend -r to grow the LV and its filesystem in one step (xfs grows online). To migrate to faster disks I add the new PVs and pvmove data live, then remove the old ones — no downtime. I always snapshot before risky operations and verify free space in the VG first. Trade-off: a thin layer of complexity and a small performance/consideration overhead, well worth the flexibility. Prevention: provision volumes on LVM from day one so growth and migration are routine, not emergencies.

Key Interview Signal: Grows LVs online with lvextend -r, migrates live with pvmove, and provisions on LVM proactively.

Q57. RAID levels — 0, 1, 5, 6, 10 — when to use which

Scenario: You're speccing storage for a write-heavy database and separately for a large media archive, and must justify the RAID choice for each.

Engineer-style Answer: I match RAID to the workload's need for performance vs capacity vs fault tolerance. RAID 0 is stripe-only — fast but zero redundancy, so never for data I care about. RAID 1 mirrors — simple, survives one disk, halves usable capacity. RAID 5 (single parity) and RAID 6 (double parity) give capacity efficiency but carry a write penalty and painful, risky rebuilds on big disks — RAID 6 tolerates two failures, which matters as rebuild times grow. RAID 10 (mirror+stripe) gives strong performance and fault tolerance at the cost of capacity — my choice for a write-heavy DB. For a media archive where capacity dominates and writes are rare, RAID 6 is reasonable. Crucially, RAID is availability, not backup — it doesn't protect against deletion or corruption, so I still back up.

Key Interview Signal: Chooses RAID by workload (10 for write-heavy DB, 6 for capacity), knows the write penalty, and stresses RAID ≠ backup.

Q58. Swap space — sizing, swapon/swapoff, swappiness tuning

Scenario: A latency-sensitive service occasionally stalls, and you find the box is swapping despite having free RAM.

Engineer-style Answer: Swap is disk used as overflow memory — a safety valve, but disk is orders of magnitude slower than RAM, so active swapping causes exactly the stalls I'm seeing. vm.swappiness controls how eagerly the kernel swaps; on a latency-sensitive server I lower it (e.g., 1–10) so it only swaps under real pressure, not to reclaim cache. I confirm actual pressure with vmstat/free before tuning — swap used isn't the same as active swap in/out. I don't disable swap entirely on most systems, because a little swap plus a controlled OOM policy is safer than sudden OOM kills. The real fix is usually right-sizing memory or fixing a leak, not just tuning swappiness. Prevention: alert on swap-in/out rate, not just swap usage.

Key Interview Signal: Distinguishes swap-used from active swapping, lowers swappiness for latency, and treats memory sizing as the real fix.

Q59. Linux kernel — modules, sysctl, /proc tuning, kernel parameters

Scenario: A high-connection server drops connections under load, and you suspect kernel network limits rather than the application.

Engineer-style Answer: Kernel behavior is tunable at runtime via sysctl (backed by /proc/sys), and modules add features like drivers or netfilter. For connection drops I'd examine parameters like net.core.somaxconn (accept queue), net.ipv4.tcp_max_syn_backlog, and conntrack table size — a full conntrack table silently drops new connections. I change values live with sysctl -w to test, watch the effect, then persist in /etc/sysctl.d/ so it survives reboot. I tune conservatively and one variable at a time so I can attribute the improvement, and I document why. I avoid copy-pasting "10 sysctls that make Linux fast" blogs blindly — wrong tuning can hurt. Prevention: capture the tuned values as code in the host baseline.

Key Interview Signal: Targets specific network sysctls (somaxconn/conntrack), tests live then persists, and avoids cargo-cult tuning.

Q60. Linux capabilities — fine-grained privileges replacing root

Scenario: A service needs to bind to port 80 but security policy forbids running it as root.

Engineer-style Answer: Root is all-or-nothing; capabilities split root's power into discrete pieces so a process gets just what it needs. Binding to a low port needs only CAP_NET_BIND_SERVICE, which I grant to the binary (setcap 'cap_net_bind_service=+ep') or, better, via the systemd unit (AmbientCapabilities=CAP_NET_BIND_SERVICE) while the service runs as an unprivileged user. That removes the "run as root" requirement entirely and shrinks the blast radius if the service is compromised. I audit which caps a process holds (getpcaps) and grant the minimum. Trade-off: capabilities are subtle and easy to over-grant, so I keep them explicit and reviewed. Prevention: default services to non-root with a minimal capability set rather than reaching for root.

Key Interview Signal: Replaces root with a single targeted capability via systemd, shrinking blast radius and applying least privilege.

Q61. Namespaces and cgroups — the foundation of containers

Scenario: An interviewer asks what a container actually is under the hood, and how one container can starve the host.

Engineer-style Answer: A container isn't a VM — it's a normal Linux process with two kernel features applied. Namespaces give it an isolated view (its own PID, network, mount, UTS, user namespaces) so it thinks it's alone; cgroups limit and account its resources (CPU, memory, I/O). "One container starves the host" happens when it has no cgroup memory/CPU limits, so it consumes everything — the fix is setting limits (requests/limits in Kubernetes, --memory/--cpus in Docker). Understanding this demystifies containers: it's isolation (namespaces) plus resource control (cgroups) plus a layered filesystem, all on the shared host kernel. That shared kernel is also why container escape is a real security concern, unlike VMs.

Key Interview Signal: Explains containers as namespaces (isolation) + cgroups (limits) on a shared kernel, and ties resource starvation to missing cgroup limits.

Q62. SELinux and AppArmor — mandatory access control

Scenario: An app gets "permission denied" writing to a directory it clearly owns with correct Unix permissions, and disabling SELinux "fixes" it.

Engineer-style Answer: MAC adds a policy layer on top of standard permissions — even root is constrained by it. So a process can own a file yet be denied because its SELinux domain isn't allowed to write that file's type. The wrong move is setenforce 0 permanently; that's turning off a security control to mask a labeling issue. The right move is to read the denial in the audit log (ausearch/audit2why), see it's a context mismatch, and fix the label (restorecon, or a proper policy/boolean) so the app works with SELinux enforcing. AppArmor is the path-based equivalent on Ubuntu/SUSE. Prevention: understand the denial, adjust policy correctly, and keep enforcement on — disabling MAC to fix a bug is trading a real security boundary for convenience.

Key Interview Signal: Reads audit logs and fixes labels/policy instead of disabling SELinux; treats MAC as a boundary worth keeping.

Q63. Linux networking deep dive — routing, iptables chains, NAT, bridging

Scenario: Containers on a host can't reach the internet, and you must trace whether it's routing, NAT, or a firewall chain.

Engineer-style Answer: I follow the packet's path. Routing (ip route) decides the next hop — I confirm there's a default route and the container's traffic is directed out the right interface. A bridge connects container veths to the host network; I check the bridge and that forwarding is enabled (net.ipv4.ip_forward). NAT (the nat table's POSTROUTING MASQUERADE) rewrites the container's private source IP to the host's — missing that rule is a classic "outbound fails" cause. iptables/nftables chains (raw→mangle→nat→filter) can drop it, so I inspect FORWARD. I test at each layer (ping the gateway, then external IP, then a hostname to isolate DNS). Prevention: manage these rules declaratively (the container runtime usually does) and don't hand-edit iptables under a runtime that rewrites it.

Key Interview Signal: Traces routing → bridge/forwarding → NAT/MASQUERADE → filter chain systematically, isolating each layer.

Q64. TCP/IP stack tuning — socket buffers, conntrack, TIME_WAIT

Scenario: A high-throughput proxy shows connection failures and lots of sockets in TIME_WAIT; ops wants to "just increase limits."

Engineer-style Answer: I diagnose before tuning. High TIME_WAIT is normal on the side that closes connections — it's a 2×MSL safety wait, and it's rarely the actual problem. Real failures usually come from ephemeral port exhaustion, a full conntrack table (drops new connections silently — check nf_conntrack_count vs max), or undersized accept queues (somaxconn). The right fixes depend on cause: enable connection reuse/keep-alive so you open fewer connections (attacking the root), raise conntrack max if it's genuinely full, tune socket buffers for bandwidth-delay product on long fat links. I avoid tcp_tw_recycle (removed/harmful) and blanket buffer inflation. Prevention: measure which limit is actually hit, tune that one, and reduce connection churn architecturally.

Key Interview Signal: Knows TIME_WAIT is usually a symptom not a cause, checks conntrack/ephemeral ports, and prefers connection reuse over blind limit-raising.

Q65. Storage performance — I/O schedulers, fio, disk latency

Scenario: A database shows rising query latency and you suspect the disk, but need evidence before asking for faster storage.

Engineer-style Answer: I gather evidence rather than assume. iostat -x shows per-device utilization, await (latency), and queue depth; high await with high %util points at the disk being the bottleneck. I confirm with fio using a workload that matches the DB's pattern (random vs sequential, block size) to measure real IOPS/latency, not vendor marketing numbers. The I/O scheduler matters: on SSD/NVMe none/mq-deadline usually beats older elevator schedulers meant for spinning disks. I check whether it's actually disk or something masquerading as it (page-cache pressure, a noisy neighbor on shared storage). Only after evidence do I recommend faster storage, a scheduler change, or query/index fixes. Prevention: baseline storage performance and alert on latency, not just utilization.

Key Interview Signal: Uses iostat await + fio to prove the bottleneck and matches scheduler to device before recommending spend.

Q66. Memory management — OOM killer, page cache, hugepages

Scenario: A process is killed at 3am; the app team insists "the server has plenty of free memory."

Engineer-style Answer: The "plenty free" confusion usually comes from misreading free — Linux uses spare RAM as page cache, which is reclaimable and not a shortage. The OOM killer fires only under true memory pressure, and it picks a victim by an oom_score, which may not be the actual culprit. I confirm the kill in dmesg/journal (it logs the score and RSS of processes), identify which process really grew, and check for a leak vs legitimate load. Fixes: cap the offending service with cgroup/systemd MemoryMax, protect critical processes via oom_score_adj, and right-size the box. Hugepages help specific workloads (databases) by reducing TLB pressure but must be tuned deliberately. Prevention: memory limits per service and alerting on real available memory (MemAvailable), not "free."

Key Interview Signal: Corrects the free-vs-available/page-cache misconception, reads OOM logs, and caps memory per service.

Q67. Linux hardening — CIS benchmarks, automated scanning, lynis

Scenario: A new fleet must pass a security audit, and manually hardening each host is error-prone and unrepeatable.

Engineer-style Answer: I standardize against a recognized baseline rather than inventing my own — CIS Benchmarks give concrete, auditable controls (disable unused services, SSH hardening, filesystem mount options, auditd, password policy). I scan with tools like lynis or an OpenSCAP CIS profile to get a measured score and a gap list, so hardening is evidence-driven, not vibes. Crucially I apply it as code (Ansible/config management or a golden image) so every host is identical and re-scannable, and drift gets corrected automatically. I balance security against operability — some CIS items break real workflows, so I document exceptions with justification rather than silently skipping. Prevention: continuous scanning in CI/CD and on running hosts, so a regression is caught immediately, not at the next audit.

Key Interview Signal: Anchors to CIS, measures with scanners, applies hardening as code, and manages exceptions explicitly.

Q68. /proc and /sys — runtime kernel inspection and tuning

Scenario: You need to inspect a live process's limits, open files, and environment during an incident without restarting anything.

Engineer-style Answer: /proc and /sys are virtual filesystems exposing live kernel and process state. For a running process, /proc/<pid>/ is a goldmine: limits (its actual ulimits), fd/ (open file descriptors — great for finding leaks or held-open deleted files), status (memory, threads), environ (its environment, which explains "works in my shell but not here" bugs), and cmdline. /sys exposes device and kernel-subsystem state and some tunables. I read these to gather evidence non-intrusively — no restart, no attaching a debugger. I'm careful writing to /proc/sys live (it's a real tuning change); I test then persist via sysctl. The value is diagnosing a live problem from ground truth instead of guessing.

Key Interview Signal: Uses /proc/<pid>/ for live, non-intrusive evidence (fd, limits, environ) during incidents.

Q69. Disk encryption — LUKS setup and key management

Scenario: Compliance requires encryption at rest for a database volume, and the servers reboot unattended.

Engineer-style Answer: LUKS provides full-volume encryption at rest — if a disk is stolen or improperly decommissioned, the data is unreadable. The hard part isn't the crypto, it's key management: an unattended reboot can't prompt for a passphrase, so I don't hardcode the key on the same disk. Instead I use a network key server / TPM binding (clevis + Tang, or a KMS) so the volume unlocks automatically only in the trusted environment. I keep a securely stored recovery key (escrow) so a lost key doesn't mean lost data, and I rotate keys via LUKS key slots without re-encrypting. Trade-offs: a small CPU cost (negligible with AES-NI) and added boot dependency on the key service. Prevention: test the unattended unlock and the recovery path before going to production.

Key Interview Signal: Treats key management (not the cipher) as the real problem — automated unlock via TPM/KMS plus tested escrow and rotation.

Q70. Systemd advanced — custom units, dependencies, resource limits

Scenario: A custom daemon must start after the database and network are ready, restart on crash, and never consume more than a set memory budget.

Engineer-style Answer: I write a unit that encodes all of this declaratively. Ordering and readiness: After=network-online.target postgresql.service plus Wants=/Requires= for the dependency strength — and I use Type=notify or a real readiness check rather than assuming "started" means "ready." Resilience: Restart=on-failure with RestartSec and StartLimitBurst so it recovers but doesn't hot-loop invisibly. Resource control: MemoryMax=, CPUQuota=, TasksMax=, which put the service in a cgroup and cap it — so a leak can't take the host. I run it as a dedicated non-root user with minimal AmbientCapabilities and hardening (ProtectSystem, NoNewPrivileges). Prevention: the unit is the single source of truth, version-controlled, so behavior is reproducible across the fleet.

Key Interview Signal: Encodes ordering/readiness, restart policy, cgroup resource caps, and least-privilege hardening in one declarative unit.

Q71. Infrastructure automation with shell scripts

Scenario: You inherit a pile of ad-hoc provisioning scripts that are non-idempotent — re-running them corrupts state.

Engineer-style Answer: The core defect is non-idempotency: a script should be safe to run repeatedly and converge to the same state. I refactor so each step checks-then-acts (create user if not present, append to a file only if the line is missing), uses set -euo pipefail, logs clearly, and cleans up with trap. But the bigger judgment call is scope: shell is fine for small, single-host glue, but for real fleet provisioning I'd move to a purpose-built tool (Ansible/Terraform) that gives idempotency, state, and drift detection for free — hand-rolling that in bash is a maintenance trap. So I stabilize the scripts to stop the bleeding, then migrate the durable parts to declarative tooling. Prevention: idempotency as a design rule, code review, and shellcheck in CI.

Key Interview Signal: Makes scripts idempotent, and knows when to graduate from bash to declarative config-management tooling.

Q72. Log aggregation and rotation — logrotate, rsyslog, journald

Scenario: A server's disk fills because an app log grew to hundreds of GB, and rotation "was configured" but clearly didn't work.

Engineer-style Answer: First I reclaim space safely — but I check whether the app holds the file open, because deleting an open logfile doesn't free space until the process reopens or is signaled (a common trap). Then I fix rotation: logrotate config with size/time limits, compression, retention count, and crucially a postrotate signal (or copytruncate) so the app writes to the new file — the usual reason rotation "ran but disk still filled" is the app kept writing to the old inode. journald should have SystemMaxUse caps. Long term I ship logs off-box to central aggregation so the local disk isn't the log store and I keep searchable history. Prevention: alert on disk usage and on log growth rate, and validate rotation actually reopens the file.

Key Interview Signal: Understands open-file-not-freeing-space and the reopen/copytruncate rotation gotcha, and pushes for centralized logging.

Q73. Security hardening — SSH hardening, fail2ban, audit framework

Scenario: Auth logs show constant SSH brute-force attempts, and leadership wants assurance the servers are locked down.

Engineer-style Answer: Layered defense. First, remove the attack surface: disable password auth and root login, use keys only, and ideally put SSH behind a bastion or VPN so it isn't internet-exposed at all. fail2ban watches auth.log and temporarily bans IPs after repeated failures — useful, but it's mitigation, not the primary control; key-only auth already defeats brute force. I add the Linux audit framework (auditd) to record privileged actions and file access for forensics and compliance, and I ship those logs off-box so an attacker can't erase them. I verify with a scan (lynis/CIS). The mindset is defense in depth plus attribution: make intrusion hard, and make everything auditable. Prevention: manage this config as code so no host drifts back to password auth.

Key Interview Signal: Prioritizes key-only/no-root and network exposure reduction over fail2ban, adds auditd with off-box logs, and manages as code.

Q74. Certificate management — openssl, CSR, self-signed, Let's Encrypt

Scenario: A production TLS cert expires over a weekend and takes down the service; leadership wants this to never happen again.

Engineer-style Answer: An expired cert is an avoidable, self-inflicted outage, so the real deliverable is automated renewal, not just a new cert. Short term I generate a CSR with openssl, get it signed (or issue via Let's Encrypt/ACME), install it, and reload the service — verifying the chain and expiry (openssl x509 -noout -dates, openssl s_client). Long term I automate issuance/renewal with ACME (certbot / cert-manager in Kubernetes) so certs rotate well before expiry, and I add monitoring that alerts weeks ahead on approaching expiry — belt and suspenders. Self-signed certs are only for internal/testing, never public. I keep private keys protected and rotate them. Prevention: automation + expiry alerting means "cert expired" stops being an incident category.

Key Interview Signal: Treats expiry as an automation failure — ACME auto-renewal plus proactive expiry alerting, with chain verification.

Q75. chroot — jail environments and their role before containers

Scenario: An interviewer asks how isolation worked before containers and why chroot isn't considered real security isolation.

Engineer-style Answer: chroot changes a process's apparent root directory so it sees only a subtree — historically used for build environments, recovery (chrooting into a broken install from a live USB), and simple service jails. But it only isolates the filesystem view; it shares the host's PIDs, network, and users, and a root process can escape a chroot, so it's not a security boundary on its own. Containers extended the idea with namespaces (PID/net/mount/user isolation) and cgroups (resource limits), which is why they're meaningfully stronger. I still use chroot for legitimate cases like OS recovery, but I'd never rely on it to contain untrusted code. The takeaway: chroot is the conceptual ancestor of containers, not an equivalent.

Key Interview Signal: Explains chroot isolates only the filesystem, is escapable by root, and positions namespaces/cgroups as the real evolution.

Scenario: An application intermittently hangs, and you trace it to an NFS mount whose server briefly became unreachable.

Engineer-style Answer: NFS (Unix/Linux) and SMB/CIFS (Windows-interop) share files over the network, which introduces failure modes local disks don't have — latency, and hangs when the server disappears. A hard-mounted NFS share will block processes indefinitely on server loss (which is why the app hung); soft/intr or timeout options let it fail instead, at the risk of I/O errors, so it's a durability-vs-availability trade-off I choose deliberately. I verify with showmount, check locking (NLM/lockd) for apps that need it, and watch for stale file handles. For anything latency-sensitive or a primary datastore I'd avoid networked filesystems. Prevention: monitor mount health, choose mount options for graceful failure, and design apps to tolerate the shared storage being temporarily unavailable.

Key Interview Signal: Identifies the hard-mount hang, weighs hard vs soft mount trade-offs, and avoids network FS for latency-critical data.

Q77. Linux in cloud — cloud-init, user-data, metadata service

Scenario: New EC2/VM instances must self-configure on first boot (hostname, packages, keys, app bootstrap) with zero manual steps.

Engineer-style Answer: cloud-init is the standard first-boot provisioner across clouds — it reads user-data (a script or cloud-config YAML I pass at launch) to set hostname, install packages, write files, and inject SSH keys, so instances come up configured without me touching them. The instance metadata service (IMDS) exposes instance-specific data and, importantly, temporary IAM credentials — I fetch config or secrets from it rather than baking them into the image. Two things I'm careful about: keep user-data idempotent and minimal (heavy logic belongs in a baked image or config management), and secure IMDS (enforce IMDSv2) because SSRF against the metadata endpoint is a real credential-theft vector. Prevention: golden images + thin cloud-init, and treat metadata credentials as sensitive.

Key Interview Signal: Uses cloud-init/user-data for hands-off bootstrap, fetches creds via IMDS, and calls out IMDSv2/SSRF security.

Q78. Container runtime from scratch — unshare and nsenter

Scenario: In an interview you're asked to demonstrate that a container is "just namespaces" by building a mini one by hand, and to debug inside a running container's namespace.

Engineer-style Answer: I'd show that unshare creates new namespaces for a process — e.g. unshare --pid --net --mount --fork bash gives a shell with its own PID/network/mount view, which is the essence of a container without any runtime. Add a cgroup for limits and a chrooted rootfs and you've basically hand-built one. nsenter is the debugging counterpart: it enters an existing process's namespaces, so I can drop into a running container's network namespace from the host to run tcpdump/ss even when the container image lacks those tools. This demystifies containers and is genuinely useful in production — debugging a distroless container's networking from the host via nsenter is a real technique. The signal is understanding containers deeply enough to build and inspect them at the primitive level.

Key Interview Signal: Builds isolation with unshare and debugs live containers with nsenter, proving containers are kernel primitives.

Q79. strace and ltrace — system call tracing

Scenario: A process hangs or fails with a vague error and the logs say nothing useful. You need to see what it's actually doing.

Engineer-style Answer: When logs are silent, strace shows the truth — every syscall the process makes. strace -p <pid> on a hung process reveals what it's stuck on: often blocked on a read/connect (waiting on a slow dependency), spinning on EAGAIN, or failing open/stat with ENOENT/EACCES (a missing file or permission problem the app swallowed). strace -f follows children; -e trace=network or -e trace=file narrows the noise. ltrace does the same for library calls. I use it as a last-resort diagnostic because it has real performance overhead and can perturb timing, so I attach briefly and detach. It frequently turns "mysterious hang" into "it's blocked connecting to a host that's firewalled" in seconds. Prevention: better app-level logging so I don't need strace as often.

Key Interview Signal: Reaches for strace to find the exact blocking syscall/failed open when logs fail, while respecting its overhead.

Q80. Bash vs Python for DevOps — when to switch languages

Scenario: A "quick" bash script has grown to 500 lines with nested logic, arrays of structured data, and JSON parsing, and it's becoming unmaintainable.

Engineer-style Answer: I choose by the shape of the problem. Bash is unbeatable for gluing commands, pipelines, and short automation — starting a process, moving files, chaining CLI tools. But once I have complex data structures, real error handling, JSON/API work, unit tests, or anything a teammate must maintain, that's the signal to switch to Python: it has real data types, exceptions, libraries, and testability. The 500-line bash script with nested logic is already past that line — it's harder to read and riskier than the Python equivalent. I don't rewrite reflexively, but I stop adding complexity in bash. Rule of thumb: if I'm fighting the language (arrays, quoting, parsing) more than solving the problem, it's outgrown bash. Prevention: recognize the threshold early rather than after 500 lines.

Key Interview Signal: Has a clear heuristic — bash for glue, Python once data structures/error handling/testing dominate — and applies it before the script rots.

Incident & Troubleshooting Scenarios

Q81. Server disk 100% full — find and fix without losing data

Scenario: Alerts fire: root filesystem at 100%, the app is now failing writes. You must recover fast without deleting something critical.

Engineer-style Answer: I stay calm and locate before deleting. First df -h to confirm which filesystem is full (root vs a data mount). Then find the big consumers: du -x -h --max-depth=1 / | sort -rh | head walking down into the largest directory (-x stays on one filesystem). Usual suspects: /var/log, a runaway app log, old package caches, orphaned temp files. Before deleting anything I confirm it's safe and reversible — I truncate or rotate logs rather than removing files an app holds open (deleting an open file won't free space; I signal the app or use > file to truncate). I reclaim just enough to restore service, communicate status, then do root cause. Prevention: separate volume for /var, disk alerts at 80%, and working logrotate so it never reaches 100% again.

Key Interview Signal: Locates the consumer with du before deleting, truncates open logs safely, and prevents recurrence with alerts + separate volumes.

Q82. 100% disk but du shows free space — deleted files held open

Scenario: df says the disk is full, but du on everything adds up to far less. Someone already "cleaned up" by deleting a large log.

Engineer-style Answer: This is the classic deleted-but-open-file gap: a process still holds a deleted file open, so the space isn't freed and du can't see it (no directory entry) while df still counts it. The "cleanup" deletion is exactly what caused it. I find the culprit with lsof | grep deleted (or lsof +L1), which lists processes holding deleted files and their sizes. The fix is to make the process release the handle: signal it to reopen its log, restart it gracefully, or truncate via its /proc/<pid>/fd/<n> descriptor if I can't restart. I avoid killing blindly. Prevention: never rm a live logfile — rotate it (logrotate with copytruncate or a reopen signal) so this stops happening.

Key Interview Signal: Immediately recognizes the deleted-open-file signature, uses lsof +L1, and fixes it by releasing the handle, not more deleting.

Q83. Cannot SSH into server — systematic debugging

Scenario: A production host won't accept SSH. On-call needs it back and everyone's guessing.

Engineer-style Answer: I debug outward-in instead of guessing. Is the host even up — does it ping, does the cloud console say running? If it pings but SSH refuses, is port 22 open (nc -vz host 22) — that separates network/firewall/security-group from the SSH daemon. ssh -v shows exactly where it fails: connection refused (sshd down or firewall), timeout (network/SG), or auth failure (keys/permissions). Common causes: security group/firewall change, sshd crashed, disk full so sshd can't write, wrong key, or /home permissions too open. If SSH is truly dead I use the cloud serial/console or a recovery instance. I communicate which layer I've ruled out as I go. Prevention: a break-glass access path (SSM/console), config-managed sshd, and disk alerts so a full disk doesn't lock everyone out.

Key Interview Signal: Isolates network vs daemon vs auth with ping/nc/ssh -v, uses out-of-band console, and keeps a break-glass path.

Q84. Running out of inodes but disk not full

Scenario: Writes fail with "No space left on device," yet df -h shows plenty of free space. The team is baffled.

Engineer-style Answer: Free bytes but failing writes points at inode exhaustion, not space. Every file consumes an inode, and a filesystem has a fixed count set at creation — millions of tiny files (session files, mail queue, cache fragments, PID-named temp files) exhaust inodes long before bytes. I confirm with df -i (the giveaway), then hunt the directory with the file count: for d in /*; do echo "$d: $(find $d -xdev | wc -l)"; done and drill down. The fix is cleaning up or archiving the many small files and stopping whatever generates them. I can't easily add inodes to ext4 after the fact (xfs allocates dynamically), so a rebuild or a different mount may be needed long term. Prevention: monitor df -i alongside df -h, and fix the app pattern creating file spam.

Key Interview Signal: Reaches for df -i immediately, finds the file-count offender, and knows inodes are fixed at creation on ext4.

Q85. Process consuming 100% CPU — identify and resolve

Scenario: A server is pegged; latency is up. You need to find the offender and decide whether to kill, throttle, or investigate.

Engineer-style Answer: I identify precisely before acting. top/htop (press 1 for per-core) shows which process and whether it's one thread or many; pidstat and ps -eLf drill into threads. Then I ask why it's hot: strace -p/perf top shows if it's stuck in a syscall spin loop, GC, or genuine compute. That distinction drives the response — a real workload I'd renice or move; a runaway bug I'd capture a stack/thread dump (evidence for the app team) before restarting so we can root-cause. I avoid an immediate kill -9, which destroys the evidence and may corrupt state. I communicate impact and what I'm doing. Prevention: CPU limits (cgroups), autoscaling, and profiling the hot path so it doesn't recur.

Key Interview Signal: Pinpoints the process/thread, captures evidence before killing, and distinguishes real load from a runaway bug.

Q86. OOM killer killed production process — diagnose and prevent

Scenario: A critical service was terminated overnight; monitoring shows it vanished with no app-level error.

Engineer-style Answer: A silent disappearance points at the OOM killer. I confirm in dmesg/journalctl -k for the "Out of memory: Killed process" line, which logs the victim, its RSS, and oom_score. Then I ask whether it was a leak (steady growth) or legitimate demand exceeding RAM — memory graphs over time answer that. If it's a leak, raising memory only delays the next kill, so I get a heap profile to the app team. If it's real demand, I right-size the host or cap other services. I protect the critical process with oom_score_adj (make it less likely to be picked) and isolate memory hogs with cgroup MemoryMax so they die instead of the important service. Prevention: per-service memory limits, alert on MemAvailable trending down, and fix the leak at the source.

Key Interview Signal: Confirms via kernel log, separates leak vs demand, protects critical procs with oom_score_adj, and caps others via cgroups.

Q87. Zombie processes accumulating — find parent and fix

Scenario: Process count climbs steadily; many are <defunct>. Left alone, the box will hit the PID limit.

Engineer-style Answer: Zombies are already-dead children the parent hasn't reaped, so I don't try to kill the zombies (they're dead) — I fix the parent. I find it with ps -o ppid= -p <zombie_pid>, then look at why that parent isn't calling wait() — usually a bug in signal handling (ignoring/mishandling SIGCHLD) or a supervisor that spawns but never reaps. Short term, signaling or restarting the parent lets init adopt and reap the orphans. Long term the parent code must reap its children, or in containers run a proper init (tini/--init) that reaps. I avoid ignoring it because enough zombies exhaust the PID table and block new processes entirely. Prevention: correct child-reaping in the app and a real init as PID 1 in containers.

Key Interview Signal: Targets the non-reaping parent (not the zombies), understands SIGCHLD/wait, and prescribes a container init.

Q88. Cron job running but producing no output — debug

Scenario: A scheduled job "runs" (it's in the cron log) but the expected file/report never appears, and there's no error anywhere.

Engineer-style Answer: Cron ran the job, so the problem is inside its execution environment. The #1 cause is the minimal cron environment: no interactive PATH, different HOME, no sourced profile — so a command found in my shell isn't found under cron, or a relative path resolves elsewhere. I make failures visible first: redirect both streams to a log (>> /tmp/job.log 2>&1) and re-check, because cron's output otherwise goes to local mail nobody reads. Then I compare environments: run the exact command with env -i to mimic cron's bare env and watch it break the same way. Fixes: use absolute paths, set needed env explicitly in the script, and cd to a known directory. Prevention: every cron job logs its output and emits a success signal to monitoring, so silent no-ops are caught.

Key Interview Signal: Knows cron's stripped environment is the usual cause, captures both streams, and reproduces with env -i.

Q89. Script works manually but fails in cron — environment issues

Scenario: A backup script runs perfectly when you execute it, but fails every night under cron. Same script, same server.

Engineer-style Answer: "Works by hand, fails in cron" is almost always environment differences, and I debug it methodically rather than blaming the script. Cron runs with a stripped PATH, a different HOME, no .bashrc, possibly a different shell (SHELL/dash vs bash), and no TTY — so anything relying on the login environment or interactive features breaks. My approach: capture cron output to a log, then reproduce the cron environment locally with env -i /bin/sh -c '...' to trigger the same failure. Typical fixes: absolute paths to every binary, explicit export PATH and any required vars at the top of the script, correct shebang, and no assumptions about the working directory. Prevention: write scripts to be environment-independent from the start (absolute paths, explicit env), and test them under cron's conditions before trusting them.

Key Interview Signal: Attributes it to cron's environment, reproduces with env -i, and makes scripts environment-independent.

Q90. Permission denied despite correct permissions — SELinux

Scenario: A web server returns 403/permission-denied writing a file it owns with mode 644, and standard permission checks all look fine.

Engineer-style Answer: When Unix permissions clearly allow it but access is still denied, I suspect a layer above them: SELinux/AppArmor, ACLs, or a parent directory missing execute. I check getenforce and, decisively, the audit log — ausearch -m avc -ts recent or audit2why will show an SELinux denial with the mismatched context. The wrong fix is disabling SELinux; the right one is correcting the file's context (restorecon, or set the proper type, or flip the relevant boolean like httpd_can_network_connect). I also rule out an unexpected ACL (getfacl) and directory traversal permissions. Confirming the actual denial source turns a confusing 403 into a one-line fix. Prevention: keep SELinux enforcing with correct contexts baked into deployment, so files land with the right labels automatically.

Key Interview Signal: Looks past mode bits to SELinux/ACLs, reads the AVC denial, and fixes the context instead of disabling enforcement.

Q91. DNS resolution suddenly stopped on a server

Scenario: An app on one host starts failing with name-resolution errors; other hosts are fine. Nothing was "changed" according to the team.

Engineer-style Answer: I isolate the layer. First, is it DNS or connectivity? dig @<resolver> example.com vs dig @8.8.8.8 tells me whether the configured resolver is the problem or DNS broadly. I check /etc/resolv.conf — often overwritten by DHCP/NetworkManager/cloud-init back to a bad or unreachable resolver — and /etc/nsswitch.conf ordering. A stray /etc/hosts entry can also hijack one name. Because it's one host, I compare it against a healthy peer to spot the difference fast. "Nothing changed" rarely holds — I check recent DHCP renewals, package updates, or a resolver outage. Once I find it (say resolv.conf points at a dead resolver), I fix it and make it stick (immutable resolv.conf or correct NetworkManager config). Prevention: manage resolver config declaratively and monitor DNS resolution as a health check.

Key Interview Signal: Separates resolver-vs-connectivity with dig @, checks resolv.conf/nsswitch/hosts, and diffs against a healthy peer.

Q92. Server time drifting — NTP causing auth failures

Scenario: Users get intermittent auth failures (Kerberos/TLS/JWT), and you notice the server clock is off by minutes.

Engineer-style Answer: Clock skew breaks time-sensitive security: Kerberos, TLS cert validity, TOTP, and JWT exp/nbf all reject requests when the clock drifts beyond tolerance, so the auth failures are a symptom, not the disease. I confirm with timedatectl / chronyc tracking — is NTP synced, what's the offset and drift? Common causes: the NTP/chrony service is dead, blocked from reaching time servers (firewall/UDP 123), a VM with a misbehaving hardware clock, or a bad upstream source. I fix sync (start/repair chrony, allow the traffic, point at reliable sources), let it slew back gently, and re-test auth. I avoid a huge manual date jump on a running DB/cluster, which can cause its own problems. Prevention: monitor clock offset as a first-class metric and ensure every host has working, reachable NTP.

Key Interview Signal: Connects skew to Kerberos/TLS/JWT failures, checks chrony sync/offset, and treats clock offset as a monitored metric.

Q93. /var/log filling up disk — log rotation broken

Scenario: /var/log is consuming the disk; a single service log is enormous and rotation isn't kicking in.

Engineer-style Answer: I reclaim safely then fix rotation. I identify the huge file (du -sh /var/log/* | sort -rh), and instead of rm (which won't free space if the service holds it open), I truncate (: > /var/log/huge.log) or, better, trigger proper rotation. Then I diagnose why logrotate failed: run logrotate -d /etc/logrotate.conf (debug/dry-run) to see errors — common causes are a missing/incorrect config for that app, wrong permissions, the logrotate cron/timer not running, or the app not reopening its file after rotation (needs postrotate signal or copytruncate). journald's own size needs SystemMaxUse caps. Prevention: correct logrotate config with size caps + retention, ship logs off-box, and alert on disk at 80% and on abnormal log growth rate so it's caught before 100%.

Key Interview Signal: Truncates rather than rm's an open log, debugs logrotate with -d, and fixes the reopen/copytruncate gap.

Q94. SSH key auth stopped working after server update

Scenario: After a routine update/patch, key-based SSH to a host stops working and you're forced onto a fallback. You must find what the update changed.

Engineer-style Answer: "What changed" is obvious here — the update — so I focus there. ssh -v shows where auth fails (offered key rejected vs no key tried). Top suspects after an update: a newer OpenSSH that disabled legacy key types (e.g. ssh-rsa/SHA-1) or older algorithms, a changed sshd_config (PubkeyAuthentication, AuthorizedKeysFile, or the config reset by the package), or /home/.ssh/authorized_keys permissions now enforced more strictly (sshd refuses too-open perms). I check journalctl -u sshd on the server for the exact rejection reason. Fixes: rotate to a modern key type (ed25519), correct permissions (700 on .ssh, 600 on authorized_keys), or restore the intended sshd_config. Prevention: manage sshd_config as code so updates don't silently reset it, test SSH after patching in staging, and standardize on ed25519 keys.

Key Interview Signal: Treats the update as the change, reads sshd's rejection reason, and knows deprecated-algorithm/permission causes.

Q95. Filesystem went read-only — diagnose and recover

Scenario: An app suddenly can't write; mount shows the root filesystem remounted read-only in the middle of the day.

Engineer-style Answer: A filesystem flipping to read-only is the kernel protecting data after detecting I/O errors or corruption — it's a symptom of an underlying problem, so I don't just remount rw and move on. I check dmesg/journalctl -k for the trigger: disk I/O errors, SATA/NVMe resets, or filesystem inconsistency (errors=remount-ro is the default behavior). If it's hardware (SMART errors, failing disk), remounting rw risks more corruption — I'd get data off and replace the disk. If it's a fixable inconsistency, I plan an fsck (root usually needs a reboot into rescue or single-user, since you can't fsck a mounted root). I communicate that the box is degraded. Prevention: SMART monitoring to catch failing disks early, filesystem-error alerting, and RAID/replication so one bad disk isn't an outage.

Key Interview Signal: Reads it as protective behavior, checks dmesg for hardware vs corruption, and avoids blindly remounting rw over a failing disk.

Q96. Network interface flapping — up and down repeatedly

Scenario: Monitoring shows an interface bouncing up/down, causing intermittent connectivity and dropped connections.

Engineer-style Answer: Flapping has a few distinct causes and I gather evidence before swapping hardware. dmesg/journalctl -k and ethtool <iface> show link resets, speed/duplex renegotiation, and error counters; ip -s link shows errors/drops. Common causes: a bad cable/SFP or port (physical), a duplex/speed mismatch with the switch, a driver/firmware bug, or a flaky NIC. On the switch side, spanning-tree or a port-security event can bounce the link. I check counters over time to distinguish physical (CRC/errors climbing) from config (clean but renegotiating). I coordinate with network engineers since it's often the switch/cable side. Short term, if it's bonded I can drop the bad member; if not, I may fail the workload to another host. Prevention: NIC bonding across two switches, interface error alerting, and replacing the confirmed-bad component.

Key Interview Signal: Uses ethtool/dmesg/counters to separate physical vs config vs switch causes before replacing anything, and leverages bonding.

Q97. Fork bomb executed — system unresponsive

Scenario: A host becomes unresponsive — can't even run new commands — after a user (or malicious script) triggered rapid process creation.

Engineer-style Answer: A fork bomb exhausts the PID/process table so you can't even launch new commands — the classic "can't run anything" symptom. If I have a shell already open, I can sometimes use builtins (no fork) like kill from within the shell or a preloaded loop to kill the offending process group. If nothing runs, I use an out-of-band console. The durable defense is limits set before it happens: per-user process caps via /etc/security/limits.conf (nproc) or systemd TasksMax/user slices, and cgroup pids controller — these cap how many processes a user/service can create so a fork bomb hits a ceiling instead of the whole host. I also review how untrusted code ran as that user. Prevention: nproc/TasksMax limits per user and service, and don't let untrusted workloads run unconstrained.

Key Interview Signal: Understands PID exhaustion, uses no-fork builtins/console to recover, and prevents it with nproc/TasksMax/cgroup pids limits.

Q98. Kernel panic — check and recover

Scenario: A server crashed hard with a kernel panic. It's back after a reboot, but you must find the cause and prevent a repeat.

Engineer-style Answer: A panic means the kernel hit an unrecoverable state and halted. Because RAM is gone after reboot, the evidence I need must have been captured before — I check for a crash dump (kdump//var/crash) and the console/serial logs, which often show the panic stack. Without kdump I'm limited to prior journalctl -b -1, hardware event logs (IPMI/BMC), and the pattern of recurrence. Common causes: faulty RAM (run memtest), a bad driver or a recent kernel update, or hardware (CPU/disk). I correlate with "what changed" — a recent kernel/driver update is the first suspect, and I can boot the previous kernel to test. If hardware, I run diagnostics and consider draining the node. Prevention: enable kdump so the next panic is diagnosable, keep a known-good kernel, and monitor hardware health (ECC/IPMI).

Key Interview Signal: Knows evidence must be pre-captured (kdump/serial), correlates with recent kernel changes, and enables kdump for next time.

Q99. Server compromised — incident response and forensics

Scenario: You find signs a production server is compromised — unexpected outbound traffic and unknown processes. Leadership needs a response.

Engineer-style Answer: I follow incident response, not instinct. First, contain — isolate the host from the network (or restrict to forensic access) to stop lateral movement and data exfil, but I preserve evidence rather than rebooting or "cleaning" it, because a reboot destroys memory and running-process evidence. I preserve: snapshot the disk/memory, and collect volatile data (running processes, network connections ss -tanp, logins last/auth.log, open files, cron/systemd for persistence) before anything changes. I investigate scope: what was accessed, credentials that may be stolen, blast radius. I communicate through the proper channels (security team, incident commander) and assume the attacker may be watching. Recover from known-good — rebuild from a trusted image rather than trusting a compromised host, and rotate all credentials that touched it. Prevention: least privilege, patching, EDR/auditd, network segmentation, and immutable infrastructure so rebuilds are trivial.

Key Interview Signal: Runs contain → preserve evidence → investigate → rebuild-from-clean → rotate creds, and never trusts/cleans a compromised host in place.

Q100. Server suddenly very slow — systematic load diagnosis

Scenario: A server that was fine is now sluggish across the board. No single alert; users just report everything is slow.

Engineer-style Answer: I resist guessing and run a quick, structured triage (the USE/Brendan-Gregg style). Start with uptime/top for load average and whether the bottleneck is CPU. Then partition the resource: is it CPU (top), memory/swap (free, vmstat — swapping crushes performance), disk I/O (iostat -x, high await/%util), or network? vmstat 1 in one glance shows run queue, swap in/out, and I/O wait. High load with idle CPU means I/O or lock contention, not compute. I check "what changed" — a deploy, a cron batch job, a traffic spike, a failing disk. I form one hypothesis, confirm it with a targeted metric, then act, communicating findings as I narrow it. Prevention: baseline dashboards (CPU/mem/IO/net) and alerting on saturation so "everything's slow" becomes a specific, pre-identified signal next time.

Key Interview Signal: Uses a systematic resource-partition triage (CPU/mem/IO/net via vmstat/iostat), correlates with recent changes, and forms one testable hypothesis.

Q101. High load average but CPU idle — I/O wait diagnosis

Scenario: Load average is 20 on an 8-core box, yet top shows CPU mostly idle. Something is very wrong but it isn't compute.

Engineer-style Answer: Load average counts processes that are runnable and those in uninterruptible sleep (D state) — usually blocked on I/O. So high load with idle CPU is the textbook signature of an I/O bottleneck, not a CPU problem, and I won't fix it by adding cores. I confirm with vmstat 1 (high wa/I/O-wait, high b blocked count) and iostat -x 1 (which device has high await and %util). Then ps -eo state,pid,cmd | grep '^D' shows which processes are stuck, pointing at the culprit (a slow disk, a saturated NFS/network mount, or a runaway query). The fix targets the real cause: faster/less-contended storage, fixing the I/O-heavy workload, or repairing the flaky mount. Prevention: monitor I/O-wait and disk latency, not just CPU, so this is caught as a storage signal rather than a mysterious "high load."

Key Interview Signal: Knows load includes D-state I/O waiters, confirms with vmstat/iostat, and finds the blocked processes instead of blaming CPU.

Q102. Can't delete file even as root — immutable flag

Scenario: Even root gets "Operation not permitted" trying to delete or modify a file. Permissions look normal.

Engineer-style Answer: When root can't touch a file and it's not a permissions issue, I suspect an extended attribute — most likely the immutable flag. lsattr file shows it (an i in the attributes); it's set with chattr +i and blocks deletion, modification, rename, and linking regardless of ownership — deliberately, often to protect config or as a tamper/persistence trick by an attacker. I remove it with chattr -i file, then delete/modify. If I didn't set it, I ask why it's there — on a possibly-compromised host an unexpected immutable file (or immutable malware) is a red flag worth investigating rather than just clearing. Other causes: the file is on a read-only mount, or SELinux. Prevention: know lsattr/chattr exist, and treat unexpected immutable flags as a signal to look closer, not just an annoyance.

Key Interview Signal: Reaches for lsattr/chattr -i, and treats an unexplained immutable flag as a possible tampering signal.

Q103. Application can't bind to port 80 — port conflict, capabilities

Scenario: A service fails to start with "permission denied" or "address in use" trying to bind port 80.

Engineer-style Answer: Two different errors, two diagnoses. "Address already in use" means something else holds the port — ss -tulpn | grep :80 shows the PID/process, and I decide whether to stop it or move one service; a lingering process in TIME_WAIT or a duplicate instance is common. "Permission denied" means the app isn't root and ports below 1024 are privileged — the wrong fix is running it as root. The right fix is granting just CAP_NET_BIND_SERVICE (via setcap or systemd AmbientCapabilities) so an unprivileged process can bind low ports, or fronting it with a reverse proxy that owns 80 while the app listens high. I confirm which error it actually is before acting. Prevention: run services non-root with the minimal capability, and manage port ownership explicitly so conflicts don't recur.

Key Interview Signal: Distinguishes "in use" (find PID with ss) from "permission denied" (grant CAP_NET_BIND_SERVICE, not root).

Q104. Script works on one server, fails on another — shell differences

Scenario: The same script succeeds on an older RHEL box but fails on a new Ubuntu one, or vice versa.

Engineer-style Answer: Same script, different host means the environment differs, and I enumerate the usual axes rather than guess. Shell: /bin/sh is bash on some distros, dash on Debian/Ubuntu — bash-only syntax ([[ ]], arrays) breaks under dash, so I check the shebang matches the features used. Tool versions/flags: GNU vs BSD coreutils differ (e.g. sed -i syntax, date options), and a newer/older utility may not support a flag. PATH and installed packages differ. Locale (LC_ALL) can change sorting/number parsing. I reproduce with bash -x on the failing host to see the exact command and value that breaks. Fixes: pin the interpreter, avoid distro-specific flags or detect them, and declare dependencies. Prevention: run shellcheck, test on all target platforms in CI, and prefer portable POSIX or containerize the script so the environment is identical everywhere.

Key Interview Signal: Enumerates shell/coreutils/locale/PATH differences, reproduces with bash -x, and containerizes/CI-tests for consistency.

Q105. High TIME_WAIT connections — socket exhaustion

Scenario: A busy service starts failing to open new outbound connections; ss shows tens of thousands of sockets in TIME_WAIT.

Engineer-style Answer: I first establish whether TIME_WAIT is actually the problem or just visible. TIME_WAIT is normal on the side that closes the connection — it's a 2×MSL wait to handle stray packets — and by itself it's harmless. The real failure is usually ephemeral port exhaustion: the client side runs out of source ports because it opens a huge number of short-lived connections. The right fix attacks the churn: enable HTTP keep-alive / connection pooling so the service reuses connections instead of opening a new one per request — that collapses the TIME_WAIT count at the source. If needed I widen the ephemeral port range and enable tcp_tw_reuse (safe for outbound), but I avoid the old tcp_tw_recycle (removed/harmful with NAT). I measure ss -s before/after. Prevention: connection reuse in the client, and monitor ephemeral port usage so exhaustion is caught early.

Key Interview Signal: Recognizes TIME_WAIT as a symptom of connection churn/port exhaustion, fixes with keep-alive/pooling, and avoids tcp_tw_recycle.

Q106. SSH connects but drops after authentication

Scenario: SSH authenticates successfully but the session dies immediately or hangs right after login. Auth clearly works, so it's something post-login.

Engineer-style Answer: Because auth succeeds, I look at what happens after the shell is handed over. Top causes: a full disk or full /home so the login can't write (.bash_history, temp files) — I check df -h; a broken login shell or a fatal error in .bashrc/.profile that exits immediately — I test with ssh user@host 'echo ok' (bypasses the interactive rc) or set the shell to /bin/sh to compare; the user's shell set to /sbin/nologin; or PAM/quota/limits killing the session. Hangs (vs instant drops) often point at a stuck rc script, DNS reverse-lookup delays in sshd, or a MOTD/pam_exec hanging. ssh -v and the server's journalctl -u sshd show the exact point of failure. Prevention: keep login scripts safe and non-fatal, monitor disk/quota, and validate a minimal login path so a bad .bashrc can't lock users out.

Key Interview Signal: Focuses post-auth (disk/quota, broken rc/shell), bypasses interactive rc to isolate, and reads sshd logs.

Q107. Swap thrashing — system crawling despite available memory

Scenario: The whole system is crawling; vmstat shows heavy swap in/out, even though some memory appears free.

Engineer-style Answer: Thrashing — constant swap-in/out — means the working set doesn't fit in RAM, so the kernel spends its time shuffling pages to slow disk instead of doing work; that's why it crawls even with a little "free" memory (which may be non-reclaimable or fragmented). I confirm with vmstat 1 (si/so columns busy) and find the memory hog (top sorted by RES, or per-process from /proc/*/status). The immediate relief is reducing memory pressure: stop/restart the offending process, or cap it. Tuning vm.swappiness down helps the system avoid swapping cache prematurely, but it doesn't fix a genuine shortage. The real fixes are right-sizing RAM, fixing a leak, or limiting per-service memory via cgroups so one process can't push the whole box into swap. Prevention: alert on swap-in/out rate (not just swap used) and on MemAvailable, and set memory limits per service.

Key Interview Signal: Identifies working-set-exceeds-RAM thrashing via si/so, relieves pressure at the source, and fixes with sizing/limits rather than just swappiness.

Q108. Accidentally ran rm -rf on wrong directory — recovery

Scenario: Someone (maybe you) ran rm -rf on the wrong path and deleted important data. Panic is setting in.

Engineer-style Answer: Calm and act fast to preserve recoverability. The single most important move is to stop writing to that filesystem immediately — remount it read-only or take the app offline — because deleted data survives only until its blocks are overwritten; continued activity destroys the chance of recovery. Then I recover from the reliable source: a backup or snapshot (this is exactly why we have them) — restore is the clean path. If there's no backup and the data is critical, forensic tools (extundelete, photorec, or filesystem-specific undelete) may recover unoverwritten blocks, best done on a disk image, not the live disk. I communicate the impact and blast radius honestly. Prevention is the real lesson: tested backups/snapshots, rm guards (aliases, --preserve-root, avoiding rm -rf $VAR/ where an unset var expands to /), least-privilege, and running destructive commands with confirmation or dry-run in automation.

Key Interview Signal: Stops writes first to preserve data, restores from backup/snapshot, and hardens against recurrence (guards, unset-var traps, backups).

Q109. Accidentally ran chmod -R 777 / — recovery

Scenario: A wildcard/variable mistake ran chmod -R 777 /, wrecking system permissions. Services start failing (sshd may refuse keys, sudo breaks).

Engineer-style Answer: This is serious because many services refuse to work with overly-permissive files (sshd rejects world-writable keys, su/sudo break), so the box is now degraded and I may be about to lose access. First, if I still have a session, I keep it open — I might not be able to log back in. There's no undo for the old modes, so recovery means restoring correct permissions, not reversing chmod: the reliable path is reinstalling packages to reset their shipped permissions (rpm --setperms/--restore on RHEL, or reapplying dpkg/--configure), or restoring from a backup/snapshot/golden image. For a critical prod host, rebuilding from a known-good image is often faster and safer than hand-fixing thousands of files. I communicate that the host is compromised in integrity. Prevention: never run recursive chmod from / or with unvalidated variables, use --preserve-root, test destructive commands, and keep immutable/rebuildable infrastructure so recovery is "replace the instance."

Key Interview Signal: Knows services break under 777, restores perms via package tooling/backup/rebuild rather than "undoing," and preserves the live session.

Q110. Write a script: disk monitoring with alerting

Scenario: You need a lightweight disk-usage monitor that alerts before disks fill, deployable on hosts without a full monitoring stack.

Engineer-style Answer: I'd write it defensively: thresholds configurable, both space and inode checks (since inode exhaustion fills "empty" disks), quiet on healthy, alert once per breach (avoid alert spam), and safe under set -euo pipefail. It should exit non-zero on breach so it composes with cron/monitoring, and log clearly. In production I'd prefer a real agent (node_exporter + Prometheus alerts), but a script like this is a fine gap-filler and a good demonstration of clean scripting.

#!/usr/bin/env bash
set -euo pipefail

THRESHOLD="${THRESHOLD:-85}"        # percent
INODE_THRESHOLD="${INODE_THRESHOLD:-85}"
ALERT_CMD="${ALERT_CMD:-logger -t disk-monitor}"  # replace with mail/slack webhook

check() {
    local kind="$1" flag="$2" limit="$3"
    # -P for portable output, skip pseudo filesystems
    df -P "$flag" -x tmpfs -x devtmpfs 2>/dev/null | awk 'NR>1' | while read -r fs blocks used avail pct mount; do
        local val="${pct%\%}"
        if (( val >= limit )); then
            local msg="ALERT: ${kind} on ${mount} (${fs}) at ${pct} (threshold ${limit}%)"
            echo "$msg"
            $ALERT_CMD "$msg"
        fi
    done
}

breach=0
check "disk space" ""   "$THRESHOLD"    || breach=1
check "inodes"     "-i" "$INODE_THRESHOLD" || breach=1
exit "$breach"

Key Interview Signal: Checks inodes as well as space, alerts idempotently, exits non-zero for composability, and notes a real agent is preferable at scale.

Q111. Parse 50GB log file — top 10 IPs and attack patterns

Scenario: During a suspected attack you must extract the top source IPs and suspicious patterns from a 50GB access log, on a box you can't overload.

Engineer-style Answer: The constraint is size — I never load 50GB into memory or an editor; I stream. A single awk pass aggregates counts without buffering the file, which is far more memory-efficient than sort | uniq -c on the whole thing (that would sort 50GB). For top IPs, awk tallies into an associative array and prints at the end; for attack patterns I filter for signals (4xx/5xx bursts, suspicious paths, unusual user-agents) in the same or a second pass. If it's compressed I stream with zcat. If even one pass is too slow, I can split and process in parallel. I capture output to a file for evidence and share findings (e.g. "one IP is 60% of traffic hitting /admin"). Prevention: centralized log analysis / SIEM so I'm querying indexed logs, not scanning raw 50GB files during an incident.

#!/usr/bin/env bash
set -euo pipefail
LOG="${1:?usage: $0 <access_log>}"

reader() { case "$LOG" in *.gz) zcat "$LOG";; *) cat "$LOG";; esac; }

echo "== Top 10 source IPs =="
reader | awk '{c[$1]++} END{for(ip in c) print c[ip], ip}' \
       | sort -rn | head -10

echo "== Top 10 IPs causing 4xx/5xx =="
reader | awk '($9 ~ /^[45]/){c[$1]++} END{for(ip in c) print c[ip], ip}' \
       | sort -rn | head -10

echo "== Most-hit URLs =="
reader | awk '{c[$7]++} END{for(u in c) print c[u], u}' \
       | sort -rn | head -10

Key Interview Signal: Streams instead of loading 50GB, aggregates in a single awk pass (not a full sort), handles compression, and points to a SIEM as the durable fix.

Q112. Write a deployment script with rollback capability

Scenario: You need a deploy script that can roll back automatically if the new release fails a health check, so a bad deploy self-heals.

Engineer-style Answer: The design principles matter more than the syntax. I make the release switchable, not in-place — deploy into a new versioned directory and flip a current symlink, so rollback is just re-pointing the symlink (atomic, instant, no partial state). I keep the previous release around. I add a health check after the switch and, on failure, automatically revert. I use set -euo pipefail, a trap that rolls back on any error, a lock (flock) to prevent concurrent deploys, and clear logging. In real infrastructure I'd lean on the platform (Kubernetes rolling updates + readiness probes, or blue/green in the load balancer) rather than a bespoke script — but demonstrating the pattern shows I understand atomic release + health-gated rollback.

#!/usr/bin/env bash
set -euo pipefail

APP_DIR="/opt/app"
RELEASES="$APP_DIR/releases"
CURRENT="$APP_DIR/current"
ARTIFACT="${1:?usage: $0 <artifact.tar.gz>}"
NEW="$RELEASES/$(date +%Y%m%d%H%M%S)"
PREV="$(readlink -f "$CURRENT" 2>/dev/null || true)"

rollback() {
    echo "Deploy failed — rolling back"
    [[ -n "$PREV" ]] && ln -sfn "$PREV" "$CURRENT" && reload
    exit 1
}
trap rollback ERR

reload() { systemctl reload-or-restart app.service; }

health_check() {
    for i in {1..10}; do
        curl -fsS http://localhost:8080/health >/dev/null 2>&1 && return 0
        sleep 3
    done
    return 1
}

flock -n /var/lock/deploy.lock bash -c '
    mkdir -p "'"$NEW"'"
    tar -xzf "'"$ARTIFACT"'" -C "'"$NEW"'"
    ln -sfn "'"$NEW"'" "'"$CURRENT"'"
' || { echo "another deploy in progress"; exit 1; }

reload
if health_check; then
    echo "Deploy OK: $NEW"
else
    rollback
fi

Key Interview Signal: Uses atomic symlink releases + health-gated automatic rollback + locking, and notes the platform (k8s/blue-green) is the production-grade version.

End of guide — 112 questions covering Linux fundamentals, shell scripting, advanced systems, and real-world incident response.

Linux Interview Guide

Table of Contents

Linux Fundamentals

Shell Scripting

Advanced Linux & Systems

Incident & Troubleshooting Scenarios

Linux Fundamentals

Q1. Linux boot process — BIOS/UEFI → bootloader → kernel → systemd

Q2. Linux filesystem hierarchy — /, /etc, /var, /opt, /tmp, /proc, /sys

Q3. File permissions — read, write, execute, chmod, chown, chgrp

Q4. Special permissions — SUID, SGID, sticky bit, umask

Q5. File types — regular, directory, symlink, hard link, device, socket, pipe

Q6. Users and groups — /etc/passwd, /etc/shadow, useradd, usermod

Q7. Process management — ps, top, htop, kill, nice, renice, nohup

Q8. Process states — running, sleeping, stopped, zombie, orphan

Q9. Package management — apt, yum/dnf, rpm, dpkg, snap

Q10. systemd — services, units, journalctl, systemctl

Q11. Cron jobs — scheduling, crontab syntax, /etc/cron.d, at

Q12. Disk management — df, du, lsblk, fdisk, mount, fstab

Q13. Filesystem types — ext4, xfs, btrfs, tmpfs, NFS

Q14. I/O redirection — stdin, stdout, stderr, >, >>, 2>&1, pipes

Q15. Text processing — cat, head, tail, grep, wc, sort, uniq, cut

Q16. Find and locate — find by name, size, time, type, -exec

Q17. Archiving and compression — tar, gzip, bzip2, xz, zip

Q18. SSH — key-based auth, ssh-keygen, ssh-agent, config, SCP, rsync

Q19. Environment variables — export, PATH, .bashrc, .bash_profile

Q20. vi/vim basics — modes, navigation, editing, saving

Q21. Linux networking — ip, ifconfig, ping, traceroute, netstat, ss

Q22. DNS tools — nslookup, dig, /etc/resolv.conf, /etc/hosts

Q23. Firewall basics — iptables, ufw, firewalld

Q24. Log files — /var/log/syslog, auth.log, dmesg, journalctl

Q25. Screen and tmux — persistent terminal sessions

Q26. Special files — /dev/null, /dev/zero, /dev/urandom, /dev/stdin

Q27. Signals — SIGTERM, SIGKILL, SIGHUP, SIGUSR1, SIGINT

Q28. Man pages, help, whatis, apropos — navigating documentation

Q29. lsof, fuser, inotifywait — file monitoring and debugging

Q30. tee, watch, diff, comm — everyday utility commands

Shell Scripting

Q31. Shell scripting fundamentals — shebang, variables, quoting, exit codes

Q32. Conditionals — if/elif/else, test, [[ ]], comparisons

Q33. Loops — for, while, until, break, continue

Q34. Functions — declaration, arguments, return, local vars, getopts

Q35. Arrays and associative arrays in Bash

Q36. String manipulation — substring, replacement, length, splitting

Q37. jq for JSON processing — parsing API output and configs

Q38. cURL deep dive — headers, auth, debugging API calls

Q39. Command substitution, process substitution, subshells

Q40. grep, sed, awk — the text processing power trio

Q41. Regular expressions — basic vs extended, classes, quantifiers

Q42. xargs and parallel execution — processing argument lists

Q43. ACLs — setfacl, getfacl, when basic permissions aren't enough

Q44. trap and signal handling — cleanup on exit/error

Q45. Error handling — set -euo pipefail, error trapping, exit codes

Q46. Here documents, here strings, template generation

Q47. File descriptors, ulimits — too many open files, production tuning

Q48. fstab and mount options — noexec, nosuid, nodev, defaults

Q49. File locking and concurrent execution — flock

Q50. Process management from scripts — background, wait, job control

Q51. Bash debugging — set -x, bash -x, PS4, shellcheck

Q52. Backup strategies — rsync, dd, incremental tar, snapshots

Advanced Linux & Systems

Q53. Systemd targets (run levels) — multi-user, rescue, emergency

Q54. Network bonding/teaming — HA networking

Q55. Bash vs sh vs zsh vs dash — shell compatibility

Q56. LVM — physical volumes, volume groups, logical volumes

Q57. RAID levels — 0, 1, 5, 6, 10 — when to use which

Q58. Swap space — sizing, swapon/swapoff, swappiness tuning

Q59. Linux kernel — modules, sysctl, /proc tuning, kernel parameters

Q60. Linux capabilities — fine-grained privileges replacing root

Q61. Namespaces and cgroups — the foundation of containers

Q62. SELinux and AppArmor — mandatory access control

Q63. Linux networking deep dive — routing, iptables chains, NAT, bridging

Q64. TCP/IP stack tuning — socket buffers, conntrack, TIME_WAIT

Q65. Storage performance — I/O schedulers, fio, disk latency

Q66. Memory management — OOM killer, page cache, hugepages

Q67. Linux hardening — CIS benchmarks, automated scanning, lynis

Q68. /proc and /sys — runtime kernel inspection and tuning

Q69. Disk encryption — LUKS setup and key management

Q70. Systemd advanced — custom units, dependencies, resource limits

Q71. Infrastructure automation with shell scripts