Linux performance debugging.

• http://www.redhat.com/promo/summit/2010/presentations/summit/decoding-the-code/thurs/dshak-310-performance/larry_shak_perf_summit2010_v4.pdf
• http://www.redhat.com/f/pdf/summit/dshakshober_1010_part_1_performance_analysis.pdf
• http://www.redhat.com/promo/summit/2008/downloads/pdf/Wednesday_1015am_John_Shakshober_and_Larry_Woodman_Decoding_the_Code.pdf

• top / vmstat / ps aux / mpstat -P all / sar -u / iostat / oprofile / gnome-system-monitor / kde-monitor /proc/*

• http://virtualthreads.blogspot.com/2006/02/understanding-memory-usage-on-linux.html - explains a bit on shared library memory
• http://www.berthels.co.uk/exmap/ - utility to see how much memory is shared and among which processes
• top / vmstat -s / ps aur / ipcs / sar -r -B -W / free / oprofile / gnome-system-monitor / kde-monitor /proc/*
• pmap -d <pid> - report memory map of a process
• ps axo pid,comm,min_flt,maj_flt $pid - how many pagefaults has the process? minor=fetch from ram, major=fetch from disc
• consider hugepages if your app can work with that

• top / ps -o pmem / gprof / strace,ltrace / sar / tar
• chrt -p $pid : whats scheduler policy for a process?
• ps axo pid,rtprio,comm,policy
• renice / nice / vmstat / mpstat
• on kernel-bootline the cpu can be freed from usual tasks: isolcpus=1,2,3
• pin processes dedicated to cpus: 'taskset -p 0x00002 123' pins process w/ pid 123 to the second core only

• vmstat
• cat /proc/interrupts
• consider using irq-balancing, its a kernel compile option
• irq-affinity can be configured, binding irqs to cpus. read Documentation/IRQ-affinity.txt for this.
• man ksoftirqd

• iostat -x / vmstat -D / sar -DEV # / nfsstat
• drop i/o caches: sync; sysctl -w vm.drop_caches=3
• set different scheduler: echo deadline /sys/block/hda/queue/scheduler
• ionice: get/set program io scheduling class and priority
• measure access: bonnie++, zcav

• see irq-pinning
• investigate module-parameters of network-card-driver
• netstat -s for statistics
• tracepath - simulate mtu path discovery

• ipcs show usage/limits
• ipcrm enforce removal of shared memory segments

• filefrag -v: show fragmentation of a file
• dumpe2fs: fragmentation of a filesystem
• tune2fs: tune reserved percentage on disc, journaling-opts
• mke2fs -b 4096 -J size=10 /dev/sdx: create filesystem with bigger journal
• consider creating journal on external device
• mount filesystem with '-o noatime' - make sure your backup-program doesnt rely on it ;)
• mount -o commit=10: increases journal-commits 5 → 10seconds, but in case of failure you loose now up to 10seconds of data
• cat /proc/locks: watch locks on files