IRQTUNE -- A Linux x86 IRQ Priority Optimizer
IRQTUNE -- A Linux x86 IRQ Priority Optimizer
Copyright 1996 by Craig Estey.
This FAQ is revision 0.2
Last updated: Mon Aug 26 04:53:01 PDT 1996
See the Changes section at the bottom of this document.
irqtune changes the IRQ priority of devices to allow devices that require
high priority and fast service (e.g. serial ports, modems) to have it.
With
irqtune,
a 3X speedup of serial/modem throughput is possible.
Where do I get irqtune?
irqtune is free software under the terms and conditions of the GNU Public
License. See the file COPYING, included in the distribution, for details.
How do I know if I need irqtune?
You are running Linux on an x86 PC, other architectures to be implemented
later--Sorry.
You probably need
irqtune,
if you are experiencing any of the following:
- SLIP/PPP transfers seem slow. For example, using a 28.8 (or better) modem,
the effective throughput is approximately 700 bytes/second instead of the
expected 2500 bytes/second.
- A running SLIP/PPP dies at random times.
- Serial connections are slow or drop data.
- Netscape hangs mysteriously or stalls when trying to access a web page.
- Equivalent serial/PPP programs under Windoze run much faster than under Linux.
- Disk accesses seem to interfere with SLIP/PPP.
- Interrupt handlers for specialized, time critical devices don't get control
when they need to.
When the PC boots Linux, the timer is
given, by default, the highest IRQ priority in the system (it's IRQ 0 and thus,
priority 0). On a standard configuration, the serial ports are priority
11 and 12!!! This means that 10 other devices have higher priority.
Q: So what does IRQ priority do?
When multiple devices are in contention to interrupt the CPU, their priority
decides which interrupts will occur in what order.
Q: When does this contention occur?
After an arbitrary period of having interrupts disabled (e.g after a cli),
at the point where they're reenabled (sti). This can happen in several
places:
- In an ISR that runs with interrupts locked, it happens in the epilog, just
before attempting to execute the bottom-half.
- The bottom-half itself may do a lock and unlock.
- When a task that enters the kernel to do a system call, the system call
handler may lock and unlock interrupts briefly.
- Almost anywhere in the kernel are brief periods where it does a cli to
lock interrupts and then an sti to unlock them again.
Q: If there are multiple interrupts now pending,which one gets the service,
the serial or some other?
In the default configuration, the serial ISR will usually lose as it's
priority 11.
irqtune gives priority 0 to the whatever device you specify. If we
specify
a serial device, irqtune guarantees that the serial ISR gets control
whenever a contention occurs.
Q: Why does the serial device merit such special treatment?
Serial devices are somewhat unique. Even though they have one of
the slowest data rates (relative to a disk), they are the largest consumer
of interrupts and are extremely sensitive to interrupt latency (the time
from when a device yanks the IRQ line until its ISR is executed).
Q: Could you give a concrete example of this?
- For a modem running at 33.6, we can have a peak maximum (with
compression) of 6700 bytes/second. The serial driver programs the silo to
interrupt after 8 bytes, leaving a latency window of 8 bytes. This means
that when the serial port yanks its IRQ line, it can still absorb 8 more
characters before its buffer will overflow and data will be dropped
- In terms of time, this means that the maximum that the serial ISR may be
delayed is
(8 / 6700) seconds or 1194 microseconds. It also means that the serial ISR
will require 838 interrupts/second.
- Currently there are 52 devices that install their ISR with the SA_INTERRUPT
option.
This means that they wish to run with interrupts disabled.
I have not looked at all the devices, but just
assume for the moment that all will run with interrupts locked for an
arbitrary period.
- Assume we've got 4 of them pending (remember,
at higher priority). Assume that they will lock interrupts individually
for 320, 300, 190, and 500 microseconds, respectively. Assume that the
serial ISR also wants an interrupt.
- This means that the serial ISR will have to wait for (320 + 300 + 190 + 500)
or 1310us
for these other higher priority ISR's to run to completion. This is greater
than the maximum of 1194 us.
- With internal
modems, we probably just see start/stop behavior causing a slowdown.
With an external modem, data will be dropped. This will cause PPP to see
a CRC error and request transmission. Remember, the entire packet must be
retransmitted, which means we just wasted 296-1500 bytes (depending on the
selected MRU).
With an MRU of 1500 bytes, we just
wasted 23.4% of the bandwidth in a given second--all for the loss of a single
byte!
- Worse yet, consider a 115 Kbps ISDN card that is
masquerading as a serial device (Yes, I know it should be a DMA device).
In this case, the maximum delay the serial ISR can wait is 695 us. And that's
just one ISDN channel. Try adding the other and, well, golly :-).
Q: In this example, how would boosting serial IRQ priority help?
- If the serial IRQ had priority 0, it would get in before the others. It
could also re-interrupt after any individual lockout window. Thus, the
maximum
it would be forced to wait would be largest individual time, not the
summation of all of these times. In this example, this means 500 us.
which is now within the 1194 us. maximum for the serial ISR.
- It has been my experience that the serial device must always win these
battles for contention. When serial devices don't get what they need,
when they need it, they slow horribly or drop data outright.
Not really. In actual practice, most devices don't even notice the difference.
Most other
devices (e.g. disks, tape, ethernet) are DMA devices. The DMA does
most of the work, thus greatly reducing their need for interrupts. If the
device allows a request queue, it may function autonomously on several
requests, producing only one interrupt for the entire batch.
Furthermore, serial interrupt services are, themselves, very fast. They
slam their data as quickly as possible and get out ASAP. No fancy
calculations, just the minimum, mindless data transfer. Almost everything
else is handled later, in the bottom-half with interrupts enabled. In
fact, a serial ISR may have to re-interrupt it's own bottom-half several
times.
Those devices that
do experience some slight slowdown are more likely to have long interrupt
disable windows themselves. Having several smaller cli/sti windows is
much better than one large cli/sti window--It's just harder to program.
Q: But suppose I really want both fast serial and fast disk?
Ultimately, it's a bit of a compromise. Which is better:
- Reliable serial and slightly slower disk.
- Slightly faster disk and unreliable serial/modem support.
When paying an ISP for Internet access in $$$/hour, it's an easy decision :-).
No. It's actually an old idea. I've been doing device drivers since 1977
and Unix kernel work since 1981. I've personally written 8 serial drivers
have used this many times commercially. Giving the serial device the
highest priority is actually standard practice in many systems.
With a 4Mhz CPU, these problems used to occur at 1200 baud :-)
How do I install irqtune?
How do I use irqtune? Don't I have to rebuild my kernel?
No, you do not have to rebuild your kernel. irqtune uses insmod and
rmmod to dynamically load and unload a kernel module. But you are correct
in sensing that irqtune is a kernel patch.
Q: Ok, if it's a kernel patch, why not just issue a kernel patch like
everybody else does (e.g. diff -u output)?
irqtune will work even if you don't have the kernel source loaded. It uses
insmod to load the patch, invoke it, and then unload it. The IRQ priority
changes will last so long as the kernel is booted.
Q: How do I invoke it?
irqtune takes two arguments optional arguments:
The only caveat is that you must specify the full pathname, evenif irqtune is placed in a directory that is in $PATH. This is required
because irqtune uses argv[0] to locate its irqtune_mod.o file.
The default is 3 14 which will work for many standard configurations.
More on this later.
Q: Could I do this from my /etc/rc.d/rc.local file?
Yes. Just add a /sbin/irqtune line to this file and you're
in business.
You may also issue another irqtune command at any time.
Q: But aren't kernel patches dependent on the particular revision of the
kernel? What if my kernel revision is different from the kernel revision
that you built it on?
irqtune is 99.44% kernel revision independent. It is built
using ELF binaries, so your insmod must understand them. Also, make sure
that you have the correct insmod for your kernel. But that's
really about it.
Q: But what if I don't have ELF binary support, how can I still do things?
Well, I'd recommend that you upgrade your kernel as ELF binaries are cool :-)
But if you insist, you'll just have to recompile irqtune. Just be sure
that /usr/src/linux/include is installed. The exact procedure for building
a.out binaries can vary with compiler revision, so be sure to check
your documentation on this (You may need to add a parameter or two).
That should do the trick. But, if it doesn't, look at revisionnumbers on insmod, gcc, ld, and the kernel for incompatibilities.
Upgrade as necessary.
irqtune defaults for a standard IRQ configuration. It assumes that the
highest priority device should be on IRQ 3. This is normally the first serial
port on standard configurations, which is what you want.
Q: How do I determine what my IRQ configuration is?
Just type cat /proc/interrupts and you'll get something like:
0: 8578913 timer
1: 109547 keyboard
2: 0 + cascade
3: 86470 + serial
4: 197648 + serial
13: 1 math error
14: 93123 + Ux4F
Note that
/proc/interrupts
only reports on active devices. So to scope out
the serial IRQ's ideally you'd have X Windows up with your serial mouse and
be connected via PPP to the net.
Q: OK, I've got the output from /proc/interrupts, what do I do with it?
The leftmost number is the IRQ number. The rightmost column is the internal
device name (not to be confused with /dev names). In the above case, the two
serial ports are on IRQ 3 and IRQ 4. Just use the lower number, in this
case 3:
This sets IRQ 3 to the highest priority.
In fact, before we invoked irqtune, the IRQ number was also its priority:
IRQ PRIOR
0 0
1 1
2 2
3 3
4 4
5 5
6 6
7 7
After this command, the IRQ priorities are now:
IRQ PRIOR
0 5
1 6
2 7
3 0
4 1
5 2
6 3
7 4
Q: BTW, What's the cascade device I saw in the output of /proc/interrupts?
Glad you asked. There are actually two interrupt controllers, a master
and a slave. The slave is cascaded to the master via its IRQ 2.
The master controls IRQ's 0-7 and the slave controls IRQ's 8-15.
You actually may select two high IRQ priorities, one for the master and one
for the slave. irqtune defaults the slave to IRQ 14, which is
normally the disk controller.
Although the normal notation is to refer to IRQ's as 0-15, it may be easier to
understand if we refer to the master IRQ's as M0-M7 and the slave IRQ's
as S0-S7.
Q: But I've also got an Ethernet controller on IRQ 12. What about that?
In other words, your configuration might look something like this:
0: 8578913 timer
1: 109547 keyboard
2: 0 + cascade
3: 86470 + serial
4: 197648 + serial
12: 17968 + eth
13: 1 math error
14: 93123 + Ux4F
In this case, you might want to use:
because you want your ethernet card to have a higher priority than the diskcontroller. Actually if you did have this configuration, setting 3 14
(the default) would make the ethernet card, the lowest priority device in the
system.
In our new notation IRQ 12 is S4, and the resulting priority would be:
IRQ M/S PRIOR
0 M0 5
1 M1 6
2 M2 7
3 M3 0
4 M4 1
5 M5 2
6 M6 3
7 M7 4
8 S0 12
9 S1 13
10 S2 14
11 S3 15
12 S4 8
13 S5 9
14 S6 10
15 S7 11
Q: Suppose I also had a serial multiplexer card on IRQ 11?
Once again, your configuration might look something like this:
0: 8578913 timer
1: 109547 keyboard
2: 0 + cascade
3: 86470 + serial
4: 197648 + serial
11: 197648 + sermux
12: 17968 + eth
13: 1 math error
14: 93123 + Ux4F
This configuration is a bit tricky because now we've got a serial device on
the slave controller. It would be much better to put all serial cards on
the master controller. Things would stay much simpler.
In this case you would want to use:
The resulting priorities would be more complex and would result in somethinglike:
IRQ M/S PRIOR
0 M0 13
1 M1 14
2 M2 0
3 M3 8
4 M4 9
5 M5 10
6 M6 11
7 M7 12
8 S0 5
9 S1 6
10 S2 7
11 S3 0
12 S4 1
13 S5 2
14 S6 3
15 S7 4
The reason things would be better if all serial devices were on the master
is that now you have serial devices at priorities 0, 8, and 9.
Q: So what's wrong with that?
Well, we boosted the priority of the serial multiplexer at the expense of the
regular serial ports. The only way to allow all serial ports equally high
priority is to group them on consecutive IRQ's and set the high priority for
the lowest of those IRQ's.
How can I tell if irqtune actually did anything for me?
Well, first off, if PPP/SLIP was dying mysteriously, it will probably be more
reliable.
Secondly, run without it and get a feel for the transfer rate:
- Hit many favorite web sites and note the transfer rates in
bytes/second. Make life easy. Netscape is at least one browser
that reports transfer rates in bytes/second in the status line.
- FTP reports the transfer time of a file in bytes/second. Download (or upload)
a few files (300K or greater to smooth out the benchmark) and note the
transfer rates.
- Try several things of varying duration, different times of day, different
sites to accomodate variations in network loading. Don't stop until
there is an average set of numbers that are more or less repeatable.
Repeat this using irqtune and note the transfer times again.
NOTE: IRQTUNE just won't quit--if you want to test in the original mode
again, reboot the system first.
It's a matter of probability. Performance measurement is as much art as
science.
- First, verify that there are no more obvious problems like bad config, bad
cabling, etc.
- We're much more likely to see improvement on a DX2/66 than a Pentium/166.
With the 166, we may be overpowering the problem. We'll still have a
problem, we just won't notice it as much because the Pentium has the
extra speed to burn.
A badly tuned Ferrari may still outperform a well-tuned VW :-)
- System loading is very light. The problems that irqtune will fix are
more likely to happen when more devices and more work are added.
- What problem was occurring before? Which of the symptoms listed earlier is
happening? If performance was 2500
bytes/second before using irqtune, we're less likely to notice the
smaller jump to, say, 2800.
- Verify that irqtune got the correct IRQ numbers for the specific
configuration.
For example, suppose our primary serial port is on IRQ 3, but we gave
irqtune the value 4. We just made the serial device on IRQ 3 into
priority 14, not priority 0.
- We may have a rogue interrupt service that disables interrupts for
something outrageous, say 2 ms. This is singularly longer than the 1194 us. in
the earlier example. irqtune will still help, but the real solution here
is to reduce interrupt lockout times in the other device below the 1194 us.
threshold.
Q: What about disabling Van Jacobsen header compression?
This reduces the amount of bottom-half processing the system has to do at
the expense of larger packets being sent.
Q: What about using ``hdparm -u'' to set the interrupt-unmask flag in hard
disk driver?
It suffers from the same problem as the bottom-half routine.
The disk controller typically uses
IRQ 14 and 15. While the slave interrupt controller would
probably allow preemption,
the master (on IRQ 2) would not because the priority of all slave
devices is higher than the serial IRQ priority.
Q: What about adjusting the MRU/MTU numbers?
This will have less of an effect now. In fact, we normally reduced the MRU
to a minimum (296) to reduce the bottom-half processing and <flip-buffer>
latency at the expense of
adding extra overhead bytes due to the reduced packet size. We may now
actually be able to increase the MRU to regain the efficiency.
Beware: Do this slowly as the optimal may not be 1500. The flip buffer
in the serial/tty drivers is only 512 bytes.
Q: What about going to newer kernel revisions?
Although irqtune will work surprisingly well with just about any kernel
revision, the low level IRQ handlers and device drivers have been vastly
improved in the 2.0.X
kernels. This will only improve irqtune's effect. In fact, 2.0.X and
irqtune actually complement one another.
Didn't you give another explanation before, involving bottom-half routines?
Yes.
- My error was that I had looked at all the existing IRQ code in Linux,
when I was first
investigating the serial problem. I spaced out, misread the code, and
concluded the EOI was done in the epilog. I also was looking at the slow
interrupt prolog/epilog.
- Anyway, I then did measurements and verified that
the serial ISR never got control at an effective rate higher than 1000
bytes/sec.
- It was clear that something was holding up the serial ISR.
- This pointed to a priority problem.
- I coded up irqtune and tried it.
- irqtune worked. I went from 700 bytes/second to 2500 bytes/second.
- I was so happy surfing at 3X my former speed, that I didn't bother to try
to track down the culprit exactly as I had used this priority fix before.
- After running it for about a month, I decided to package it up so others
could use it. In my haste to get it out, I wrote the explanation too quickly.
I also wanted to get some feedback from various configurations and change
things accordingly.
It was a case of round up the usual suspects.
- In fact, now I'll be adding kernel tracing that will give some real world
numbers to back things up. When I have completed this analysis, I'll add them
to this FAQ.
Many thanks to those that pointed this out.
FAQ 0.2 Changes:
- Major rewrite and expansion of the problem explanation section
- More thorough explanation of how and why irqtune works
- Explanation of why serial devices must be highest priority
- Impact on other devices
- Cleaner and better installation instructions
- Better benchmarking section
- Problem resolution section
- Explanation of my prior misread on the EOI thing