===== What? =====
Linux performance debugging.

=== cpu tools ===
  * top / vmstat / ps aux / mpstat -P all / sar -u / iostat / oprofile / gnome-system-monitor / kde-monitor /proc/*

=== memory tools ===
  * top / vmstat -s / ps aur / ipcs / sar -r -B -W / free / oprofile / gnome-system-monitor / kde-monitor /proc/*
  * pmap -d <pid> - report memory map of a process
  * ps axo pid,comm,min_flt,maj_flt $pid - how many pagefaults has the process? minor=fetch from ram, major=fetch from disc
  * consider hugepages if your app can work with that

=== process tools ===
  * top / ps -o pmem / gprof / strace,ltrace / sar / tar
  * chrt -p $pid : whats scheduler policy for a process?
  * ps axo pid,rtprio,comm,policy
  * renice / nice / vmstat / mpstat
  * on kernel-bootline the cpu can be freed from usual tasks: isolcpus=1,2,3
  * pin processes dedicated to cpus: 'taskset -p 0x00002 123' pins process w/ pid 123 to the second core only

=== irq handling ===
  * vmstat
  * cat /proc/interrupts
  * consider using irq-balancing, its a kernel compile option
  * irq-affinity can be configured, binding irqs to cpus. read Documentation/IRQ-affinity.txt for this.
  * man ksoftirqd
  * blktrace/btrace: who is making i/o on blockdevices?
  * seekwatcher: watch your i/o patterns, how many seeks are done etc.

=== disk i/o tools ===
  * iostat -x / vmstat -D / sar -DEV # / nfsstat
  * drop i/o caches: sync; sysctl -w vm.drop_caches=3
  * set different scheduler: echo deadline /sys/block/hda/queue/scheduler
  * ionice: get/set program io scheduling class and priority
  * measure access: bonnie++, zcav

=== network ===
  * see irq-pinning
  * investigate module-parameters of network-card-driver
  * netstat -s for statistics
  * tracepath - simulate mtu path discovery

=== ipc ===
  * ipcs show usage/limits
  * ipcrm enforce removal of shared memory segments

=== filesystem layer tools ===
  * filefrag -v: show fragmentation of a file
  * dumpe2fs: fragmentation of a filesystem
  * tune2fs: tune reserved percentage on disc, journaling-opts
  * mke2fs -b 4096 -J size=10 /dev/sdx: create filesystem with bigger journal
  * consider creating journal on external device
  * mount filesystem with '-o noatime' - make sure your backup-program doesnt rely on it ;)
  * mount -o commit=10: increases journal-commits 5 -> 10seconds, but in case of failure you loose now up to 10seconds of data
  * cat /proc/locks: watch locks on files