Troubleshooting performance issues in Linux

Server Slowdown:
Performance problems are caused by bottlenecks in one or more hardware subsystems, depending on the profile of resource usage on your system. 

Some elements to consider (in roughly sorted order):

Buggy software
Disk usage
Memory usage
CPU cycles
Network bandwidth

Now, let’s look at the three biggest causes of server slowdown: CPU, RAM, and disk I/O. CPU usage can cause overall slowness on the host, and difficulty completing tasks in a timely fashion. Some tools I use when looking at CPU are top and sar.

SAR Command:

For historical CPU performance data I rely on the sar command, which is provided by the sysstat package. On most server versions of Linux, sysstat is installed by default, but if it’s not, you can add it with your distro’s package manager. The sar utility collects system data every 10 minutes via a cron job located in /etc/cron.d/sysstat (CentOS 7.6). Here’s how to check all of the "Big 3" using sar."sar -A" shows a full report.

To check RAM performance, I use sar command, which give you that day’s memory usage:

$sar -r (starting at midnight)

The main thing to look for in RAM usage is %memused and %commit. A quick word about the %commit field: 

This field can show above 100% since the Linux kernel routinely overcommits RAM. If %commit is consistently over 100%, this result could be an indicator that the system needs more RAM.

The command sar -u gives you info about all CPUs on the system, starting at midnight:

$sar -u (starting at midnight)

As with top, the main things to check here are %user, %system, %iowait, and %idle. This information can tell you how far back the server has been having issues.

For disk I/O performance, I use sar -d, which gives you the disk I/O output using just the device name. 
To get the name of the devices, use sar -dP:

$sar -d
$sar -dP

For this output, looking at %util and %await will give you a good overall picture of disk I/O on the system. The %util field is pretty self-explanatory: It’s the utilization of that device. The await field contains the amount of time the I/O spends in the scheduler.

If any of these commands show a problem, you can go back to see when the server issues started by using:

$sar {-u, -r, -d, -dP} -f /var/log/sa/sa<XX> (where XX is the day of the month you wish to look for).

TOP Command:

The top utility gives you a real-time look at what’s going on with the server. By default, when top starts, it shows activity for all CPUs:

Some things to look for in this view would be the load average (displayed on the right side of the top row), and the value of the following for each CPU:

Command syntax

$ top -c or top

us: This percentage represents the amount of CPU consumed by user processes.

sy: This percentage represents the amount of CPU consumed by system processes.

id: This percentage represents how idle each CPU is.

Each of these three values can give you a fairly good, real-time idea of whether CPUs are bound by user processes or system processes.

Virtual Memory:Report virtual memory statistics

Virtual memory statistics reporter, also known as vmstat, is a Linux command-line tool that reports various bits of system information. Things like memory, paging, processes, IO, CPU, and disk scheduling are all included in the array of information provided.

Basic vmstat Output

The basic output of the vmstat command displays system information in six sections.

1. procs – Process Statistics
r – Active process count.
b – Sleeping process count.

2. memory – Memory statistics
swpd – Total virtual memory. The swap space is initially unoccupied. However, the kernel starts using swap space as the system’s physical memory reaches its limit.
free – Total free memory.
buff – Total memory temporarily used as a data buffer.
cache – Total cache memory.

3. swap – Swap space Statistics
si – The rate of swapping-in memory from disk.
so – The rate of swapping-out memory to disk.

4. io – Input/Output Statistics
bi – Blocks received from a block device per second.
bo – Blocks sent to a block device per second.

5. system – Scheduling statistics
in – The number of system interrupts.
cs – The number of context switches per second.

6. cpu – CPU Statistics
us – The percentage of CPU time spent on non-kernel processes.
sy – The percentage of CPU time spent on kernel processes.
id – The percentage of idle CPU.
wa – The percentage of CPU time spent waiting for Input/Output.
st – The percentage of CPU time stolen by a virtual machine.

Command syntax

The syntax for the vmstat command is rather simple:

$ vmstat [options][delay [count]]

Options to know

The -a option will give us the active and inactive memory of the system:

$vmstat -a

The -f option will give us the number of forks since boot:

$vmstat -f

The -s option displays various memory statistics as well as CPU and IO event counters:

$vmstat -s

The -d option gives you read/write stats for various disks:

$vmstat -d

The -t option gives us timestamp information with every update, a seen here:

$vmstat -t

Using a Time Interval

We can have vmstat provide regular updates to these figures by using a delay value. The delay value is provided in seconds. 

To have the statistics updated every five seconds, we’d use the following command:

$vmstat 5

Using a Count Value

Using too low a delay value will put additional strain on your system. If you need to have rapid updates to try to diagnose a problem, it is recommended that you use a count value as well as a delay value.

The count value tells vmstat how many updates to perform before it exits and returns you to the command prompt. If you do not provide a count value, vmstat will run until it is stopped by Ctrl+C.

To have vmstat provide an update every five seconds—but only for four updates—use the following command:

$vmstat 5 4

Changing the Units

You can choose to have the memory and swap statistics displayed in kilobytes or megabytes using the -S (unit-character) option. This must be followed by one of k , K , m,  or M. These represent:

k:1000 bytes

K: 1024 bytes

m: 1000000 bytes

M: 1048576 bytes

To have the statistics updated every 10 seconds with the memory and swap statistics displayed in megabytes, use the following command:

$vmstat 10 -S M