Why this list

A server is slow, someone is waiting, and you have just logged in. It is tempting to start guessing. What I do instead is run a fixed set of commands that, in about a minute, tells me whether the problem is CPU, memory, disk or network, and where to look next.

This is the checklist I use, written for the junior and medior sysadmins I work with: what each command is for and the one or two things I look at in its output. The links at the end are where I would go next.

None of the commands change anything on the server. They are safe to run on production. The 1 means a new sample every second; stop them with Ctrl-C after a few samples.

uptime
dmesg -T | tail
vmstat 1
mpstat -P ALL 1
pidstat 1
iostat -xz 1
df -hT
df -hiT
free -m
sar -n DEV 1
sar -n TCP,ETCP 1
top

Make sure all the commands are installed. mpstat, pidstat, iostat and sar come from sysstat, uptime, vmstat, free and top from procps, and dmesg from util-linux. df is part of coreutils. The last three are on almost every box already, but installing them does no harm. dstat, used for the alias at the end, is its own package.

On Debian and Ubuntu:

apt install sysstat util-linux procps coreutils dstat

On RHEL, Rocky, Alma and Fedora (use yum on older releases; the procps package is called procps-ng there):

dnf install sysstat util-linux procps-ng coreutils pcp-system-tools

On RHEL and its clones dstat is provided by pcp-system-tools.

1. uptime

 10:42:01 up 37 days, 3:12, 2 users, load average: 8.12, 5.40, 2.31

The three numbers are the load average over 1, 5 and 15 minutes. Read them as a trend: here the load is climbing (8.12 over the last minute versus 2.31 over fifteen), so the problem is recent and getting worse. If the 1-minute number is much lower than the 15-minute one, whatever it was may already be over. On Linux, load counts processes waiting for CPU and processes stuck in uninterruptible disk I/O, so a high load does not always mean a busy CPU. The next commands tell you which one it is.

2. dmesg -T | tail

The kernel log shows what the kernel already knows: out-of-memory kills, disk errors, network drops, hardware faults. Look for lines like Out of memory: Killed process or TCP: drop open request. -T prints readable timestamps. Quite often the answer is already here and you can stop.

3. vmstat 1

One line per second for the whole system. The columns I read first:

• r: processes running or waiting for CPU. If it is higher than your CPU count, the CPUs are saturated.
• free: free memory in KB.
• si and so: memory swapped in and out. Anything non-zero and steady means the machine is short of memory.
• us sy id wa st: CPU time in user, kernel, idle, I/O wait and stolen. High wa points at the disk. High st on a VM means the hypervisor is taking your CPU away.

The first line shows averages since boot for most columns. Ignore it.

4. mpstat -P ALL 1

The same CPU split, but per core. What I look for is one core at 100% while the others sit idle: a single-threaded process (or a badly balanced interrupt) that adding cores will not fix.

5. pidstat 1

Like top, but it prints a rolling table instead of redrawing the screen, so you can copy it into a ticket. It shows which process is using the CPU and whether it is user (%usr) or system (%system) time. This is where the problem gets a process name.

6. iostat -xz 1

Per-disk statistics. For disks this is the one I would keep if I could only have one:

• r/s, w/s, rkB/s, wkB/s: the workload you are asking of the disk.
• r_await and w_await (a single await on older sysstat versions): average time in milliseconds for a read or write request, queueing included. This is what your application actually feels. Much above what the device normally does means trouble.
• aqu-sz (called avgqu-sz on older versions): requests queued. Above 1 suggests saturation.
• %util: how busy the device is. Near 100% is a problem on a spinning disk. On NVMe and arrays that serve many requests in parallel it is misleading, so trust the await columns more.

-z hides idle devices.

7. df -hT

Disk space per filesystem. -h prints human-readable sizes and -T adds the filesystem type. Look at Use%: a filesystem at or near 100% is the classic cause of applications failing to write, and of databases and logs suddenly stopping. The type column helps you tell real disks from tmpfs and overlay mounts, which fill up in different ways. If df says a filesystem is full but du cannot find the files, a deleted file is probably still held open by a running process.

8. df -hiT

The same listing, but for inodes (-i). Every file uses one inode, and a filesystem has a fixed number of them. When IUse% reaches 100% you get No space left on device errors even though df -hT shows plenty of free space. This usually means millions of tiny files: mail queues, session files, caches or a runaway log directory.

9. free -m

Memory in MiB. Do not panic at a small free column: Linux uses spare RAM for cache on purpose. The column that matters is available, the kernel’s estimate of what can be handed to applications without swapping. Small available plus swap in use is real memory pressure.

10. sar -n DEV 1

Network throughput per interface: rxkB/s and txkB/s. Compare them with what the link can do (a 1 Gbit/s link tops out around 120000 kB/s). If you are close to that, the network is the bottleneck.

11. sar -n TCP,ETCP 1

A few TCP counters:

• active/s: connections this host opened per second (outgoing).
• passive/s: connections this host accepted per second (incoming). A rough measure of the load on a server.
• retrans/s: retransmitted segments per second. Anything consistently above zero hints at a flaky network or an overloaded peer.

12. top

Last, because by now you know what you are looking for. It confirms the picture and catches anything the earlier commands could not show, such as a runaway process that started a second ago. Press 1 for per-core CPU, M to sort by memory, P to sort by CPU.

What to do with the results

After the twelve commands you should be able to say one sentence, something like “this process is eating all the CPU”, “writes on sdb take 200 ms” or “it is swapping”. Sometimes the sentence is “the system looks fine”, which usually means it is the application. Either way, write it in the ticket before you go further. From there you pick a deeper tool for that one resource.

The dstat alias

Between incidents I like to watch the whole machine at once, in one scrolling table. dstat does that. This is the alias I keep in my .bashrc:

alias dstats='dstat -tcmndrylpg'

Each letter turns on a group of columns:

Run dstats and a new line appears every second. When the machine slows down you can see which group changed at that moment. It is also handy to leave running in a second terminal while you go through the list above.

Closing

Run these twelve, in this order, often enough that you no longer need this page. In my experience they are enough to explain most “the server is slow” tickets before anyone opens a dashboard. When they are not, the references below are where I would look next.

References

• Linux Performance
• 19 Linux performance monitoring utilities
• Diagnostic tools for Linux and UNIX
• Linux system info commands
• Troubleshooting hardware problems on Linux