1. Linux interrupt basics
1.1 What is an interrupt?
An interrupt is a signal from hardware or software that indicates an event that requires immediate attention from the operating system. When the hardware device requires CPU processing, it sends an interrupt request (IRQ), which pauses the current task, saves the status, and executes the interrupt service routine (ISR) associated with the interrupt.
1.2 Types of interrupts
- Hardware interrupt: Generated by hardware devices (such as keyboard, mouse, network card, etc.)
- Software interruption: Generated by software instructions (such as system calls)
- abnormal: An exception detected by the CPU (such as a zero-dividing error)
1.3 Why do monitoring interrupts need to be done?
- Identify hardware problems
- Diagnostic performance bottlenecks
- Balancing IRQ load
- Optimize system response time
2. Command tool to view interrupt information
2.1 /proc/interrupts - The gold standard for interrupt statistics
/proc/interruptsIt is the most direct and comprehensive way to view system interrupt information. This virtual file records the number of interrupts processed by each CPU core, and is classified by IRQ number.
cat /proc/interrupts
Output example:
CPU0 CPU1 0: 45 0 IO-APIC-edge timer 1: 3 0 IO-APIC-edge i8042 8: 1 0 IO-APIC-edge rtc0 9: 0 0 IO-APIC-fasteoi acpi 12: 4 0 IO-APIC-edge i8042 16: 1157526 0 IO-APIC-fasteoi ehci_hcd:usb1, ath9k 17: 0 2013453 IO-APIC-fasteoi ahci[0000:00:1f.2]
Field explanation：
- Column 1: IRQ number
- Subsequent column: Interrupt counts processed by each CPU core
- Penultimate column: interrupt type (such as IO-APIC-edge)
- Last column: Device name or driver information
Advanced Tips：
- Dynamic monitoring interrupt changes:
watch -n 1 'cat /proc/interrupts' - Focus only on specific interrupts:
grep eth0 /proc/interrupts
2.2 /proc/softirqs - View software interrupts
Software interrupt (softirq) is a mechanism used for delay processing in the Linux kernel, which is often used for network and block device operations.
cat /proc/softirqs
Output example:
CPU0 CPU1
HI: 1 0
TIMER: 1234567 1234568
NET_TX: 567 890
NET_RX: 4567890 3456789
BLOCK: 0 0
IRQ_POLL: 0 0
TASKLET: 1234 567
SCHED: 1234567 1234567
HRTIMER: 0 0
RCU: 4567890 3456789
Common softirq types：
- NET_RX: Network reception interrupt
- NET_TX: Network sending interrupt
- TIMER: Timer interrupt
- SCHED: Scheduled related interrupts
2.3 mpstat - Multi-CPU Statistics Tool
mpstat is part of the sysstat package and can provide interrupt-related CPU statistics.
mpstat -P ALL 1 5
Output%irqThe column shows the percentage of time each CPU handles hardware interrupts,%softThe column shows the percentage of time when the software interrupt is processed.
2.4 vmstat - System activity monitoring
vmstat 1 5
OutputinThe column shows the number of interrupts per second (including clock interrupts),csThe column shows the number of context switches.
2.5 dstat - All-round system resource statistics tool
dstat -cip --top-int
--top-intOptions can display the most active interrupt source.
2.6 irqbalance - Interrupt load balancing tool
For multi-core systems, the irqbalance service can automatically balance interrupt load:
systemctl status irqbalance
Check the CPU affinity of the current interrupt (which CPU handles which interrupt):
cat /proc/irq/*/smp_affinity
3. Advanced interrupt analysis and optimization
3.1 Interrupt affinity setting
In a multi-core system, you can specify which CPU cores handle specific interrupts by setting interrupt affinity:
- Check out the current affinity:
cat /proc/irq/[IRQ]/smp_affinity
- Set affinity (e.g. bind IRQ 16 to CPU 0 and 1):
echo 3 > /proc/irq/16/smp_affinity
(The binary of 3 is 11, indicating CPU 0 and 1)
3.2 Network performance tuning
For high-traffic network servers, network outages can become bottlenecks. Consider:
- Enable RSS (receiver scaling):
ethtool -l eth0
- Adjust the number of queues:
ethtool -L eth0 combined 8
- Enable RPS (receive packet steering):
echo ff > /sys/class/net/eth0/queues/rx-0/rps_cpus
3.3 Identify interruption storm
An interrupt storm is when a device generates too many interruptions, resulting in a degradation in system performance. Diagnostic method:
- Monitor the rate of interrupt growth:
watch -n 1 "cat /proc/interrupts | grep eth0"
- Use the perf tool to analyze:
perf top -e irq:irq_handler_entry
4. Actual case analysis
4.1 Case 1: Network performance issues
symptom: The server responds slowly when it is high in network load, the CPU usage rate is not high but the system load is high.
Diagnostic steps：
- Check
/proc/interruptsFound that a CPU core handled most network interrupts - examine
smp_affinityFound that all network interrupts are bound to the same CPU
Solution：
- Enable irqbalance service
- Or manually set the interrupt affinity of multi-queue network cards
4.2 Case 2: Disk I/O Delay
symptom: Storage server delay increases at high I/O load.
Diagnostic steps：
- Check
/proc/interruptsDisk controller interrupt handling is found uneven -
mpstatShow some CPUs%softVery high value
Solution：
- Adjust the interrupt affinity of block devices
- Increase
vm.dirty_ratioandvm.dirty_background_ratioReduce I/O pressure
5. Automated monitoring scripts
The following scripts can periodically record interrupt distribution:
#!/bin/bash
LOG_FILE="/var/log/"
while true; do
echo "===== $(date) =====" >> $LOG_FILE
cat /proc/interrupts >> $LOG_FILE
echo "" >> $LOG_FILE
sleep 60
done
6. Conclusion
Understanding Linux interrupt mechanism and mastering relevant monitoring tools are the basic skills in system performance analysis and tuning. Through tools such as /proc/interrupts, /proc/softirqs, we can gain an in-depth understanding of the system's interrupt behavior, identify potential performance bottlenecks, and take appropriate optimization measures. In multi-core systems, reasonable interrupt load balancing is crucial to performance.
The above is the detailed content of the complete guide to viewing and analyzing interrupt information in Linux. For more information about viewing and analyzing interrupt information in Linux, please pay attention to my other related articles!