Swapping Swapping describes what happens when an entire runnable process is moved into swap. If the computer doesn’t have enough physical memory to store a process that isn’t being run at that particular microsecond, the system can move the entire process to swap. Then, the next time the CPU runs that process, the process’s memory is moved from swap into physical memory, and some other process is probably consigned to swap. The problem with swapping is that disk usage goes through the roof and performance drops drastically. Since requests take longer to handle, there are more requests of the system at any one time. And logging in to check the problem only makes the situation worse, because logging runs an extra system process. This performance hit is sometimes called the death spiral. Memory shortages will hurt system performance more than anything else. If you’re frequently swapping, you must get more memory or resign yourself to lousy[6] performance. Note Every system has bottlenecks, or places where performance is limited. If you eliminate one bottleneck, performance will increase until another bottleneck is hit. The system will work at the fastest speed allowed by the slowest component in the system, also called bounds. For example, a Web server is frequently network-bound because the slowest part of the system is the Internet connection. If you upgrade the Internet connection, the system will hand out Web pages as fast as either its CPU or disk allows. Are You Swapping or Paging? FreeBSD includes several programs for examining system performance. Among those are vmstat(8), iostat(8), and systat(1). We’ll discuss vmstat because I find it to be the most helpful. Iostat is similar to vmstat, and systat provides similar information in a more graphic format. Using Vmstat Vmstat(8) shows virtual memory statistics at the current time. While its output takes some getting used to, it is very good at showing large amounts of data in a very small space. Type vmstat at the command prompt, and follow along. ………………………………………………………………………………………. # vmstat procs memory page disks faults cpu r b w avm fre flt re pi po fr sr ad4 da0 in sy cs us sy id 0 0 0 7096 479140 21 0 0 0 9 0 0 0 331 102 437 0 1 99 # ………………………………………………………………………………………. The display is divided into six sections: process (procs), memory, paging (page), disks, faults, and cpu. We’ll look at each then quickly and then dive into detail on the bits that are most important for investigating your performance issues. 415
Note: If you are looking for good and high quality web space to host and run your application check Lunarwebhost PHP Web Hosting services
To renice every process owned by a user, use the -u flag. For example, to make my processes more important than anyone else’s, I could enter this command: ………………………………………………………………………………………. # renice -5 -u mwlucas 1000: old priority 0, new priority -5 # ………………………………………………………………………………………. The 1000 is my user ID number on this system. Again, presumably I have a very good reason for doing this besides a need for personal power.[5] NoteRenicing, rescheduling, and process management don’t create additional CPU time, they simply rearrange the CPU time you do have. If you cannot reschedule processes, and you cannot satisfactorily renice things to tune the way the system behaves, you really do need faster or additional hardware. Some systems have an extra motherboard slot for an additional CPU, which is a quick and inexpensive way to boost performance when the system is CPU-bound. If you have multiple CPUs, definitely take a look at the discussion of SMP in Chapter 11. [2]Some users actually try to use up system resources by starting programs. This is called a forkbomb. These users are like script kiddies, but not as educated. [3]Sluggy Freelance (http://www.sluggy.com/) and Help Desk (http://www.ubersoft.net/), if anyone cares. [4]This might be one of the few circumstances where common sense won out in naming UNIX commands. [5]Being a selfish person doesn’t qualify as a good reason. Or so I’ve been told. When Swap Goes Bad I said earlier that using swap space isn’t bad in and of itself because swap space is used as virtual memory. (In other words, memory space on the hard drive is being used in the same way as RAM.) Swap space is much slower than chip memory, but it does work in a pinch, and many programs don’t need to have everything in RAM in order for them to run. If programs spend 80 percent of their time in 20 percent of their code, then 80 percent of their bulk can be put into swap space without seriously impacting performance. Many sysadmins use the term swapping generically, lumping two different activities (paging and swapping) together without understanding the crucial difference between them. Paging When you read about virtual memory, you’ll see references to pages. A page is simply a section of memory, 4KB on x86 hardware under FreeBSD. (Different platforms have different page sizes.) Data moves between real and virtual memory in units of pages. Paging happens when a portion of a running program is moved onto swap. This process can actually improve performance on a heavily loaded system because unused bits can be stored on disk until they’re needed. 414
Note: If you are looking for good and high quality web space to host and run your application check Lunarwebhost PHP Web Hosting services
If you know that your system is running low on CPU capacity, you can choose to start a command with nice(1) to assign the command a priority. Specify the desired niceness level by putting a single dash in front of the command. For example, to start a make buildworld at nice 15, you would run this command: ………………………………………………………………………………………. # cd /usr/src # nice -15 make buildworld ………………………………………………………………………………………. Only root can assign a negative niceness to a program. To run a program with negative niceness, use a double dash (nice –5). For example, if you have a critical kernel patch that must be applied as soon as possible, and you want the compile to finish as quickly as possible, use a negative niceness like so: ………………………………………………………………………………………. # cd /sys/i386/compile/MYKERNEL # nice –20 make depend && nice –20 make all install ………………………………………………………………………………………. Usually, you won’t have the luxury of telling a command to start off nicely, but will instead need to change a process’s niceness on the fly (generally, when you find out that it’s soaking up all your CPU). You can do so with renice(8), which will reprioritize by process ID or owner. To change the niceness of a process ID, you run renice with the new niceness and the process ID. For example, one of my systems has a FreeBSD CVSup mirror. If I find that the mirror is taking up so much CPU time that it’s getting in the way of things I have to do, I can change its niceness to 20. The maximum niceness we can use is 20, which basically tells the system to run this command only if nothing else at all wants CPU time. To renice a running process, I first need to know its process ID. I know the process is named cvsupd because I’ve looked at this system’s top output over the last several months. I then look at all the processes running on the system, and pull out the one for cvsupd with the following command: ………………………………………………………………………………………. # ps -ax | grep cvsupd 322 ?? Is 0:00.01 /usr/local/sbin/cvsupd -C 5 -b /test2 -s sup # ………………………………………………………………………………………. The first column in the preceding ps output is the process ID, PID 322. Now to renice it, I would enter the following: ………………………………………………………………………………………. # renice 20 322 322: old priority 0, new priority 20 # ………………………………………………………………………………………. Boom! The cvsupd daemon will now only run when nothing else requiring system time is running. This will greatly annoy users of the service, of course, but I presumably have a good reason for doing so. (Since this is a private mirror, not a public one, I feel no particular need to be kind to my users.) 413
Note: If you are looking for good and high quality web space to host and run your application check Lunarwebhost Cheap Web Hosting services
display still shows that I’m using that 100MB of swap, while I have over 200MB of memory free. Using swap space is not a bad thing, especially since a program will typically spend 80 percent of its time running 20 percent of its code. Since much of the rest of that time spent running is startup, shutdown, and error code, you can safely let those bits go to swap space and have minimal impact. So don’t worry if you find that you’re using a bit of swap space on occasion. But, if you’re constantly using swap, you probably need more memory. CPU Usage A processor can do only so many things a second, and if you want to do more than your CPU can handle, the requests will start to queue up. You’ll develop a processor backlog, and the system will slow down. That’s CPU usage in a nutshell. If top shows your CPU hovering around 100 percent all the time, you must take action. While new hardware is certainly an option, you do have other choices. For one, investigate the processes running on your system to see whether they’re all necessary. Did some junior administrator install the SETI@Home client (/usr/ports/astro/setiathome) to hunt for aliens with spare CPU cycles? Are there things running that were important at one time, but are now unnecessary? Find and kill those unnecessary processes and make sure that they won’t start the next time the system boots. Once that’s done, evaluate your system performance again. If you still have a problem, try rescheduling or reprioritizing. Rescheduling Rescheduling is easier than reprioritizing, and it is a relatively simple way to balance system processes so that they don’t load up on CPU time. As discussed in Chapter 9, you can use cron(1) to schedule system tasks for various times, but users can use it too. If you have users who are running massive compile jobs or doing huge database queries, you might consider using cron to schedule them to run at night. Frequently, jobs such as the monthly billing database search can be run between 6 PM and 6 AM, and nobody will care. Similarly, you could schedule your make buildworld && make buildkernel to start at 1 AM. Reprioritizing with Niceness If rescheduling won’t work, you’re left with reprioritizing, which is a bit trickier. When reprioritizing, you tell UNIX to change the importance of a given process. For example, if you want a software install to run, but only when nothing more important is running, you reprioritize it with “niceness,” which is simply a relative measure of how much CPU time a process demands. The nicer a process is, the less CPU time it demands. The default niceness is 0, but niceness runs from 20 (very nice) to -20 (not nice at all). (This might seem backwards; you could argue that a higher number should mean a higher priority. That leads to a language problem, though, as calling this factor “crankiness” or “greed” didn’t seem like a good idea at the time.)[4] In the top display seen earlier (in the “Using Top” section) you saw a PRI column for process priority. FreeBSD calculates a process priority by combining a variety of factors, including niceness, and runs high-priority processes first whenever possible. Niceness affects priority, but you cannot directly edit priority. 412
Note: If you are looking for good and high quality web space to host and run your application check Lunarwebhost PHP Web Hosting services
Time The TIME column gives the total length of time that the process has been running. CPU Usage The WCPU column gives a weighted CPU usage that shows the percentage of CPU time that the process is using, as adjusted for the process’s priority and niceness. The CPU column shows what percentage of CPU time the program is actually using. Command Name Finally, in the COMMAND column, we have the program name. Memory Usage If your system is running slowly, check its memory and CPU usage first. While they’re no more likely to be running amok than any other part of the system, they’re the easiest to measure. Let’s discuss memory first. FreeBSD errs on the side of caching recently accessed data because a surprising amount of information is read from disk time and time again. If this information can be cached in physical memory, it can be accessed very quickly. If the system needs more memory, it dumps the oldest cached chunks in favor of new data. For example, the example top output we’re discussing is from my laptop, which is using a lot of buffer and inactive memory. Part of that is due to my Web browser. I started Mozilla when I booted the system yesterday morning so I could check my morning comics.[3] For a couple of moments, the disk light stayed solidly lit while the system read the program off the disk. I then shut the browser down so I could do some work. Since this Web browser was accessed, it sat in the system buffer cache. When I started the browser again this morning, it only had to be called out of cache rather than from disk, so it started much more quickly. Had I started some other large process, it would have dumped that Web browser from the cache to read in more data. If your system is operating well, you will have at least a few megs of free memory. If you have more than a few megs free, your system is not being used to nearly its full potential. In the example earlier, I could get rid of 128MB of RAM and not affect system performance much at all. If you have a good chunk of memory in cache or buffer, you don’t have a memory shortage. You might make good use of more memory, but it isn’t strictly necessary. Similarly, if you have a lot of free memory, you probably don’t have a memory shortage. If active and wired memory is consuming most of your available memory, more RAM wouldn’t hurt. When you’re out of free space, and have little or no memory in cache or buffer, you should investigate your memory use further. You may well have a memory shortage. Take a look at the Using Vmstat section later in the chapter to check. Swap Space Usage Virtual memory, or swap helps cover brief RAM shortages. For example, if you’re untarring a huge file, you might easily fill up all your physical memory and have to start using virtual memory. It’s not worth buying more RAM for this occasional use when swap suffices. Like memory cache, swap caches data that it has handled recently, and once you’ve touched swap, it never returns to being free. For example, I have a server that has been up for 772 days at this writing. At one point, I used about a hundred megs of swap to handle a massive compile. My top 411
Note: If you are looking for good and high quality web space to host and run your application check Lunarwebhost PHP Web Hosting services
Memory Then we have the Mem line, representing actual physical RAM ({). Unlike Windows, which simply divides memory into “used” and “unused” categories, FreeBSD uses memory in several different ways. Active memory is the total amount of memory in use at the moment for running user programs and their data. When a program ends, the program information is put into inactive memory and the data pulled from the disk is put in the cache memory. Similarly, the Buf entry shows the size of the memory buffer. The memory buffer contains data recently called from disk. Free memory is unused. Wired memory is memory used for in-kernel data structures, as well as for particular system calls that must have a particular piece of memory immediately available. Wired memory is never swapped out. Swap Then we have the Swap line, (|), which simply represents the total swap available and how much is in use. Swapping is when the system uses the disk drive as additional memory. We’ll look at swap in more detail later in the chapter. Process List Finally, we have a list of the processes on the system and their basic characteristics (}). The table format is designed to present as much information as possible in as little space as possible. Every process is on its own line. Let’s look at each column in the following sections. PID First we have the process ID number, or PID. Every process running on the system has a unique PID. When you issue kill commands, you use the PID to identify the process you want to affect. Username Next is the username of the person running the process. If multiple processes consume large chunks of CPU or memory, and they are all owned by the same user ID, you know who to talk to. Priority and Nice The PRI (priority) and NICE columns are interrelated, and indicate how much precedence the system gives these processes. We’ll talk about priority and niceness a little later in the chapter. Size Size is the amount of memory that the system has set aside for this process. Resident Memory The RES column shows how much of the program is actually in memory, or resident, at the moment. A program might have a huge amount of memory reserved for it, but only be using a small fraction of it. State The STATE column shows what the process is doing at the moment. Processes can be in a variety of states at any given time: waiting for input, sleeping until something wakes them, actively running, and so on. 410
Note: If you are looking for good and high quality web space to host and run your application check Lunarwebhost Cheap Web Hosting services
Top lists three load averages. The first (0.14 in our example) is the load average over the last minute, the second (0.08) is for the last 5 minutes, while the last (0.07) is for the previous 15 minutes. If your 15-minute load average is high, but the 1-minute load is low, you had a major spike in activity that has passed. How well did your system hold up? On the other hand, if the 15-minute value is low, but the 1-minute average is high, something happened within the last 60 seconds and may still be going on now. If all of the load averages are high, the condition has persisted for the whole 15 minutes. Uptime The last entry on the first line is the uptime (x), or how long the system has been running. The system in our example has been up for one hour and six minutes, and the current time is 08:12:26. I’ll leave it up to you to figure out when the system booted. Process Counts On the second line you’ll find information about processes that are currently running on the system (y). Running processes are actually doing work; they’re answering user requests, handling mail, or whatever else is going on. Sleeping processes are waiting for input from one source or another, and are just fine. Processes in other states are usually waiting for a resource to become available, or are hung in some way. Large numbers of nonsleeping, nonrunnable processes can be a hint of trouble. Investigate further to find out which processes those are. Process Types The CPU states line (z) indicates what percentage of available time the CPU spends handling different types of processes and other duties. It shows five different process types: user, nice, system, interrupt, and idle. User processes are average everyday programs; they could be daemons run by root, commands run by regular users, or whatever. If it shows up on the list of system processes (that is, on ps -ax), it’s a user process. Nice processes are those whose priority has been deliberately manipulated by the user. We’ll look at this in some detail in “Reprioritizing with Niceness.” System processes are in the kernel, and they include things such as virtual memory handlers, running networking, writing to the disk, and so on. The interrupt category shows how much time the system spends handling interrupt requests (IRQs). Lastly, the idle process shows how much time the system spends doing absolutely nothing. If your system CPU regularly has a very low idle time, you might want to start thinking about rescheduling some jobs or getting a faster processor. Note When you’re working on a multi-CPU system, keep in mind that top displays the average usage among all the processors. You might have one processor completely tied up compiling something, but if the other processor is idle, top will show only 50 percent usage. 409
Note: If you are looking for good and high quality web space to host and run your application check Lunarwebhost PHP Web Hosting services
Using Top To read a top display, you must understand a great deal about how the system works, so we’ll spend a good chunk of time on this. To run top, just type top. To display kernel processes as well as user programs, use top -S. You’ll see a display much like the following, and it will refresh every few seconds. ………………………………………………………………………………………. top -S v last pid: 436; w load averages: 0.14, 0.08, 0.07 x up 0+01:06:16 08:12:26 y 46 processes: 3 running, 43 sleeping z CPU states: 1.2% user, 0.0% nice, 0.8% system, 0.0% interrupt, 98.1% idle { Mem: 70M Active, 102M Inact, 26M Wired, 6016K Cache, 41M Buf, 107M Free | Swap: 200M Total, 200M Free } PID USERNAME PRI NICE SIZE RES STATE TIME WCPU CPU COMMAND 287 mwlucas 2 5 2892K 2136K select 0:13 0.10% 0.10% xsysinfo 378 mwlucas 2 0 101M 64920K RUN 0:08 0.10% 0.10% soffice.bin 376 mwlucas 2 0 35372K 32736K RUN 0:13 0.05% 0.05% mozilla-bin 274 mwlucas 2 0 28208K 26304K select 1:01 0.00% 0.00% XFree86 170 root 2 0 912K 508K select 0:08 0.00% 0.00% moused 277 mwlucas 2 0 3888K 3116K select 0:03 0.00% 0.00% wmaker 5 root 18 0 0K 0K syncer 0:00 0.00% 0.00% syncer 430 mwlucas 28 0 1912K 1160K RUN 0:00 0.00% 0.00% top 399 mwlucas 2 0 4500K 4000K select 0:00 0.00% 0.00% Eterm … ………………………………………………………………………………………. Very tightly packed, isn’t it? Top tries to cram as much data as possible into a standard 80-character by 25-character terminal window. The display updates every two seconds, so you have a fairly accurate, close to real-time, view of your system. We’ll take this a piece at a time and explain what every entry means. PID Values Every process on a UNIX machine has a unique process ID or PID. Whenever a new process is started, it is assigned a PID one greater than the previous process. The last pid value is the last process ID used in the system. In the previous example, the last pid is 436 (v). The next process to be created will be 437, then 438, and so on. You can watch this increment to see if an abnormal number of processes is being created. Hopefully, you’ve looked at your system to see how quickly this number rises when things are running well. If the last pid value keeps climbing rapidly, programs are being started and stopped very quickly. This might indicate some daemon that keeps crashing, or a user trying to start too many programs.[2] Load Average The load average (w) is a somewhat vague number that’s intended to give a rough impression of the amount of load on the system.[2] The load average equals the average number of processes waiting for CPU time, plus the average number of jobs that are waiting for access to the disk. An acceptable load average depends on the system; if the numbers are abnormally high, you should investigate. Many 486s feel bogged down at a load average of 3, while some modern systems feel snappy at a load average of 10. 408
Note: If you are looking for good and high quality web space to host and run your application check Lunarwebhost PHP Web Hosting services
FreeBSD changes continually, and later systems might have new tuning and performance features. Take a look at tuning(7) on your system to find any new performance tips. We’ll cover tuning information that is useful on any FreeBSD (and almost any UNIX) system. Note One word you’re going to keep stumbling across in this chapter is “abnormal.” As the systems administrator, you’re supposed to know what is normal for your system. It’s somewhat like art: You might not be able to define normal, but you need to recognize it when you see it. It’s a good idea for you to use these tools to check your systems regularly when they’re behaving correctly, so you will have a good idea of what is out of whack when the system slows down. We’ll also look at some long-term monitoring tools, so you can gauge system performance over months or years. [1]Technically, network bandwidth is part of input/output. However, it’s special enough that we’ll treat it separately. Disk Input/Output We looked at disk operations in some detail in Chapter 16. When it comes to performance, disk speed is usually a big bottleneck. If programs are waiting for disk activity to complete before proceeding, they will slow your system down. (This is commonly called “blocking on disk,” meaning that the disk is blocking program activity.) The only real solution for this is to use a faster disk or a RAID array, or to split your disk activity between two disks. How do you know if your disk is actually blocking program activity? We’ll look at that in “Using Vmstat,” later in the chapter. Network Bandwidth If your system performance slowdown is due to network problems, you need more bandwidth. In short: You can only push as much bandwidth as you have. If your T1 is full, you need more bandwidth. If your system cannot fill the existing bandwidth, use the tools discussed in Chapter 5 to increase system capacity. To check for this problem, begin by monitoring how much bandwidth your system is using. Chapter 15 discusses how to generate long-term graphs of band-width usage. We also discussed networking in Chapter 5. Consult netstat -m, and increase your kernel’s NMBCLUSTERS, as described in Chapter 4. That’s really all there is to it. Other system conditions are more complicated. CPU and Memory The top(1) tool is a good place to start if you’re examining a system that seems to be running slowly. It provides a good overview of system status, but it only shows information about the CPU and memory usage; input/output and band-width are not touched. 407
Note: If you are looking for good and high quality web space to host and run your application check Lunarwebhost PHP Web Hosting services
Chapter 18: System Performance Overview It’s slow. That’s one of the most dreaded phrases a systems administrator can hear. The user doesn’t know why the system is slow it just “feels” that way. Usually there’s no test case, no set of reproducible steps, and nothing particularly wrong. These two words can cause the systems administrator hours of work, digging through the system trying to figure out what’s going on. One phrase is still more dreadful, especially after you’ve invested those hours debugging the problem: “It’s still slow.” For an inexperienced systems administrator, slow systems are easy to accelerate: Buy faster hardware. This generally fixes speed problems. It also costs a lot of money and simply conceals whatever’s wrong, without really using the equipment you have. FreeBSD includes many tools designed to help you examine system performance, and to give you the information you need to actually find out what’s slowing things down. That will tell you what you need to do to fix the problem. You might very well need faster hardware, but you can quite possibly shift the load around within a system and improve overall performance. The first step is to understand what your problem really is. Computer Resources Speed problems are generally caused by running more on a computer than the computer can handle. That seems obvious, but think about it a moment. What does that mean? A computer has four basic resources: disk input/output,[1] the network band-width, memory, and CPU. If any one of these is filled to capacity, the others cannot be used to their maximum effect. For example, your CPU might very well be waiting for a disk to deliver data or for memory to finish paging. A faster CPU won’t increase system performance in this case. Simply upgrading hardware when the system slows down does fix speed problems, but not in the way that you might think. If you have a program that fills up the system memory, buying a new system with a faster CPU will probably fix the problem. A new system probably has more memory than the old one, after all! By identifying what the system is running short on, and addressing only that need, you can stretch your existing hardware much further. After all, why buy a whole new system when a couple hundred dollars of memory will fix the problem? (Of course, if your goal is to rotate this “slow” system into place as your new desktop, that’s another matter.) Perhaps you can reschedule work; one common cause of system slowdowns is running multiple large programs simultaneously. For example, I once scheduled a massive database log rotation that moved and compressed gigabytes of files at the same time as the system’s automated daily checks. Since the job required shutting down the main database, and hence created system downtime, speed was crucial. Performance on both processes slowed unbearably. Rescheduling the log job greatly reduced downtime. We’re going to examine several FreeBSD tools for examining what a system is doing. Armed with that information, we’ll consider how to fix some performance issues. We have separate tools to examine each of the potential bottlenecks. 406
Note: If you are looking for good and high quality web space to host and run your application check Lunarwebhost PHP Web Hosting services