Chapter 7. System Tuning

Table of Contents

Performance Tuning
A machine is a finite resource
System Model - Sleep Queue
Scheduling, Priority Calculation and the nice value.
The algorithm
Scheduling code - From the process table perspective
More detail on scheduling
Performance Criteria
Limiting the memory usage
Times
Top (Some extracts are from the man pages)
Sar (Some extracts are from the man pages)
Vmstat (Some extracts are from the man pages)
Iostat (Some extracts are from the man pages)
ps (Some extracts are from the man pages)

Performance Tuning

We are going to speed quite a bit of time talking about various subjects that will give you a better understanding of performance tuning.

There are two things that you are going to have to become au-fait with and these are:

  1. Monitoring the system over time and making changes one step at a time - do not rush ahead and change a whole set of allocations and resources without checking how everything affects everything else.

  2. Getting to know how your system works from the hardware, low-level to the memory management and process scheduling aspects.

Truly tuning your machine performance can take months of analysing data and making sure that you make the most of what you have. Checking the system load at low times and at peak time, when developing or when data capturing, it all makes a difference as to how your system can perform.

Some of this will have to be done before you start purchasing your machine and some of it will continue for many months after you have recompiled your kernel for the first time.

So, in order to understand all the statistics we are going to have to analyse, let's get some more back ground of the important aspects of this system.

A machine is a finite resource

If you look at the machine in front of you, it is a finite resource at this time; it consists of a set of interdependent resources or components that have to be shared between users and processes.

Now Linux has a couple of problems to be aware of especially on a busy machine:

  1. There are never enough of each type of resource

  2. Each process will need a different resource mix, and different and at the same times

  3. We will always have badly behaved processes where the program is written to hog the CPU, fill up memory, forget to de-allocate memory after reserving some etcetera.

You cannot take out your current disk and plug in a faster one for a couple of minutes whilst there is a crisis. So really tuning would then be taking the resources that you have and allocating them by balancing conflicts and establishing compromises.

Some Examples:

  1. If a process or processes are waiting for memory, and memory is full at that time, then the system will start to page memory our to the swap space. Now the system is waiting for time on the disk, waiting for time on the CPU and waiting for memory. In this case if it is happening all the time I would suggest that you put in more memory.

  2. A dumb serial card interrupts the CPU with every character whereas an intelligent serial card interrupts when the buffer is full (8+ characters).

System Model - Sleep Queue

Figure 7.1. Let us look again at the sleep queue

Let us look again at the sleep queue

If processes are waiting for disk and you have a slow disk, this will eventually cause a disk bottleneck, where there may be nothing on the run queue and the processes are waiting for disk.

Processes on the run queue only need one more resource before they can run and that is the CPU.

All processes perform IO and may have to wait for their request to be satisfied by the hardware. If a process is reading a file for example login reads /etc/passwd as a user logs in, the kernel will be making requests to the disk (to exec(), the open() etcetera). Disks have built in latencies due to rotational delay, movement of the heads and so on.

While the process is waiting for the disk request to complete it cannot use the CPU, the process is then put onto the sleep queue where it will wait for IO, wait for disk. Once the disk is freed up, the process is again placed into the run-queue.

A substantial amount of input to a process can be interactive such as from a keyboard or mouse. Systems with large numbers of interactive jobs will have most of their processes in wait for IO state. Processes can be held up when outputting to the slower devices such as terminals and printers.

When the hardware devices send an interrupt to the CPU, the clock, or a device that is now freed-up, the process running on the CPU is suspended while the interrupt service routine is executed (process to task_sched not to run queue). The clock interrupt routine is also responsible for gathering statistics dealing with process scheduling, updating page management information etcetera. If a system spends a lot of time dealing with these interrupt service calls it is said to be interrupt-bound.

A good rule of thumb is to make sure that there is enough memory, and that you allocate enough swap space on your disk (at least the same amount of memory again), even though using the swap space as additional memory will take more time than being in RAM at least the processes can still run and do not have to stop altogether.