User Tools


Linux performance debugging.

cpu tools

  • top / vmstat / ps aux / mpstat -P all / sar -u / iostat / oprofile / gnome-system-monitor / kde-monitor /proc/*

memory tools

  • top / vmstat -s / ps aur / ipcs / sar -r -B -W / free / oprofile / gnome-system-monitor / kde-monitor /proc/*
  • pmap -d <pid> - report memory map of a process
  • ps axo pid,comm,min_flt,maj_flt $pid - how many pagefaults has the process? minor=fetch from ram, major=fetch from disc
  • consider hugepages if your app can work with that

process tools

  • top / ps -o pmem / gprof / strace,ltrace / sar / tar
  • chrt -p $pid : whats scheduler policy for a process?
  • ps axo pid,rtprio,comm,policy
  • renice / nice / vmstat / mpstat
  • on kernel-bootline the cpu can be freed from usual tasks: isolcpus=1,2,3
  • pin processes dedicated to cpus: 'taskset -p 0x00002 123' pins process w/ pid 123 to the second core only

irq handling

  • vmstat
  • cat /proc/interrupts
  • consider using irq-balancing, its a kernel compile option
  • irq-affinity can be configured, binding irqs to cpus. read Documentation/IRQ-affinity.txt for this.
  • man ksoftirqd
  • blktrace/btrace: who is making i/o on blockdevices?
  • seekwatcher: watch your i/o patterns, how many seeks are done etc.

disk i/o tools

  • iostat -x / vmstat -D / sar -DEV # / nfsstat
  • drop i/o caches: sync; sysctl -w vm.drop_caches=3
  • set different scheduler: echo deadline /sys/block/hda/queue/scheduler
  • ionice: get/set program io scheduling class and priority
  • measure access: bonnie++, zcav


  • see irq-pinning
  • investigate module-parameters of network-card-driver
  • netstat -s for statistics
  • tracepath - simulate mtu path discovery


  • ipcs show usage/limits
  • ipcrm enforce removal of shared memory segments

filesystem layer tools

  • filefrag -v: show fragmentation of a file
  • dumpe2fs: fragmentation of a filesystem
  • tune2fs: tune reserved percentage on disc, journaling-opts
  • mke2fs -b 4096 -J size=10 /dev/sdx: create filesystem with bigger journal
  • consider creating journal on external device
  • mount filesystem with '-o noatime' - make sure your backup-program doesnt rely on it ;)
  • mount -o commit=10: increases journal-commits 5 → 10seconds, but in case of failure you loose now up to 10seconds of data
  • cat /proc/locks: watch locks on files