Linux performance debugging.

• top / vmstat / ps aux / mpstat -P all / sar -u / iostat / oprofile / gnome-system-monitor / kde-monitor /proc/*

• top / vmstat -s / ps aur / ipcs / sar -r -B -W / free / oprofile / gnome-system-monitor / kde-monitor /proc/*
• pmap -d <pid> - report memory map of a process
• ps axo pid,comm,min_flt,maj_flt $pid - how many pagefaults has the process? minor=fetch from ram, major=fetch from disc
• consider hugepages if your app can work with that

• top / ps -o pmem / gprof / strace,ltrace / sar / tar
• chrt -p $pid : whats scheduler policy for a process?
• ps axo pid,rtprio,comm,policy
• renice / nice / vmstat / mpstat
• on kernel-bootline the cpu can be freed from usual tasks: isolcpus=1,2,3
• pin processes dedicated to cpus: 'taskset -p 0x00002 123' pins process w/ pid 123 to the second core only

• vmstat
• cat /proc/interrupts
• consider using irq-balancing, its a kernel compile option
• irq-affinity can be configured, binding irqs to cpus. read Documentation/IRQ-affinity.txt for this.
• man ksoftirqd
• blktrace/btrace: who is making i/o on blockdevices?
• seekwatcher: watch your i/o patterns, how many seeks are done etc.

• iostat -x / vmstat -D / sar -DEV # / nfsstat
• drop i/o caches: sync; sysctl -w vm.drop_caches=3
• set different scheduler: echo deadline /sys/block/hda/queue/scheduler
• ionice: get/set program io scheduling class and priority
• measure access: bonnie++, zcav

• see irq-pinning
• investigate module-parameters of network-card-driver
• netstat -s for statistics
• tracepath - simulate mtu path discovery

• ipcs show usage/limits
• ipcrm enforce removal of shared memory segments

• filefrag -v: show fragmentation of a file
• dumpe2fs: fragmentation of a filesystem
• tune2fs: tune reserved percentage on disc, journaling-opts
• mke2fs -b 4096 -J size=10 /dev/sdx: create filesystem with bigger journal
• consider creating journal on external device
• mount filesystem with '-o noatime' - make sure your backup-program doesnt rely on it ;)
• mount -o commit=10: increases journal-commits 5 → 10seconds, but in case of failure you loose now up to 10seconds of data
• cat /proc/locks: watch locks on files