User Tools

Site Tools


software:performance_co-pilot

Differences

This shows you the differences between two versions of the page.

Link to this comparison view

Both sides previous revision Previous revision
Next revision
Previous revision
software:performance_co-pilot [2019/12/05 02:45]
chris [Compiling on Debian Buster]
software:performance_co-pilot [2020/08/27 08:54]
chris removed
Line 30: Line 30:
 # make # make
 # make install # make install
 +</code>
 +
 +===== Compiling on RHEL8 =====
 +  * install RHEL8.2
 +<code>
 +yum -y install git flex make tar \
 +  bison perl-ExtUtils-MakeMaker.noarch gcc-c++ \
 +  python3-devel rpm-build initscripts ncurses-devel \
 +  readline-devel zlib-devel
 + 
 +# libuv-devel needs to be installed, so pmproxy can
 +# communicate with grafana.
 +  
 +# ensure pcp user/group exist.
 +# 'make install' relies on this
 +useradd pcp
 +
 +mkdir /pcpserv && cd /pcpserv
 +git clone https://github.com/performancecopilot/pcp
 +cd pcp
 +./Makepkgs
 +cd ./pcp-5*/build/rpm
 +
 +<install rpms>
 +
 +systemctl start pmcd
 +systemctl status pmcd
 +pminfo 
 </code> </code>
  
Line 56: Line 84:
 pminfo  pminfo 
 </code> </code>
 +
  
 ===== Compiling on Debian Buster ===== ===== Compiling on Debian Buster =====
Line 65: Line 94:
   libdbd-mysql-perl python3-psycopg2 dh-python libcairo2-dev \   libdbd-mysql-perl python3-psycopg2 dh-python libcairo2-dev \
   libextutils-autoinstall-perl libxml-tokeparser-perl libjson-perl \   libextutils-autoinstall-perl libxml-tokeparser-perl libjson-perl \
-  libnet-snmp-perl libnss3-tools libuv1-dev+  libnet-snmp-perl libnss3-tools libuv1-dev libsystemd-dev
  
 # ensure pcp user/group exist. # ensure pcp user/group exist.
Line 76: Line 105:
 ./Makepkgs ./Makepkgs
 cd build/deb cd build/deb
 +
 +# making a copy of modified configfiles..
 +cp /etc/pcp/netcheck/netcheck.conf /tmp
 +cp /etc/pcp/bind2/bind2.conf /tmp
 +rm -rf /tmp/config.d
 +cp -r /var/lib/pcp/pmdas/openmetrics/config.d/ /tmp/
  
 # remove/install # remove/install
 apt-get remove libpcp-gui2 libpcp-import1 libpcp-pmda-perl \ apt-get remove libpcp-gui2 libpcp-import1 libpcp-pmda-perl \
   libpcp-mmv1 libpcp-pmda3 libpcp-trace2 libpcp-web1 libpcp3 \   libpcp-mmv1 libpcp-pmda3 libpcp-trace2 libpcp-web1 libpcp3 \
-  pcp pcp-conf pcp-webapi python3-pcp+  pcp pcp-conf python3-pcp
 dpkg -i pcp_5.*_amd64.deb libpcp-gui2_5.*_amd64.deb \ dpkg -i pcp_5.*_amd64.deb libpcp-gui2_5.*_amd64.deb \
   libpcp-import1_5.*_amd64.deb libpcp-mmv1_5.*_amd64.deb \   libpcp-import1_5.*_amd64.deb libpcp-mmv1_5.*_amd64.deb \
Line 90: Line 125:
 systemctl start pmcd systemctl start pmcd
 systemctl status pmcd systemctl status pmcd
 +
 +# Recently, on pcp 5.2.0, pmlogger is not coming up properly.
 +# '/etc/init.d/pmlogger start-systemd' executed by systemd
 +# is failing.  Starting once directly fixes this.
 +systemctl stop pmlogger
 +/etc/init.d/pmlogger start
 +/etc/init.d/pmlogger stop
 +systemctl start pmlogger
 +
 +# restore old configs..
 +cd /var/lib/pcp/pmdas/postfix/ && ./Install
 +cp /tmp/netcheck.conf /etc/pcp/netcheck/netcheck.conf
 +cd /var/lib/pcp/pmdas/netcheck/ && ./Install
 +cp /tmp/bind2.conf /etc/pcp/bind2/
 +cd /var/lib/pcp/pmdas/bind2/ && ./Install
 +
 +rm -f /var/lib/pcp/pmdas/openmetrics/config.d/*
 +cp /tmp/config.d/* /var/lib/pcp/pmdas/openmetrics/config.d/
 +cd /var/lib/pcp/pmdas/openmetrics/ && ./Install
 +
  
 # in my setup: # in my setup:
 # systemctl restart pmcd pmlogger pmproxy grafana-server redis-server # systemctl restart pmcd pmlogger pmproxy grafana-server redis-server
-pminfo +systemctl enable --now pmproxy¬†
 +pminfo bind2 netcheck openmetrics
 </code> </code>
  
-===== running the qa suite =====+===== Running the qa suite =====
 <code> <code>
 # preparing for QA tests # preparing for QA tests
Line 118: Line 174:
 </code> </code>
  
-===== own branches =====+===== Own branches =====
 <code> <code>
 # checkout own branch # checkout own branch
Line 131: Line 187:
 # pylint --rcfile=/home/chris/Downloads/.pylintrc \ # pylint --rcfile=/home/chris/Downloads/.pylintrc \
     --output-format=colorized pmdalmsensors.python     --output-format=colorized pmdalmsensors.python
 +</code>
 +
 +===== Openmetrics example =====
 +How to get simple own textfiles into metrics?
 +<code>
 +dnf -y install pcp-pmda-openmetrics
 +cd /var/lib/pcp/pmdas/openmetrics && ./Install
 +
 +### example 1, via URL
 +echo 'file:///tmp/vars' >config.d/myvars.url
 +cat >/tmp/vars<<EOT
 +# HELP my simple test value
 +# Type testval gauge
 +var 12.3
 +EOT
 +
 +### example 2, via executable
 +cat >config.d/system.sh<<EOT
 +#! /bin/sh
 +          awk '{
 +              print("# HELP loadavg local load average")
 +              print("# Type loadavg gauge")
 +              printf("loadavg {interval=\"1-minute\"} %.2f\n", $1)
 +              printf("loadavg {interval=\"5-minute\"} %.2f\n", $2)
 +              printf("loadavg {interval=\"15-minute\"} %.2f\n", $3)
 +          }' /proc/loadavg
 +EOT
 +chmod +x config.d/system.sh
 +
 +### example 3, via executable
 +cat >config.d/system2.sh<<EOT
 +#!/bin/sh
 +cat /proc/meminfo |sed -e 's,:,,' -e 's, kB,,' -e 's, ,"},' -e 's,^,var {interval=",'
 +EOT
 +
 +# as last part in /var/lib/pcp/config/pmlogger/config.default
 +log advisory on every 1 second {
 + openmetrics
 +}
 +systemctl restart pmlogger
 +pminfo openmetrics
 +pmrep openmetrics
 +</code>
 +  * 'man pmdaopenmetrics'
 +  * https://github.com/performancecopilot/pcp/issues/799
 +  * https://prometheus.io/docs/instrumenting/exposition_formats/
 +
 +===== stripping down pmlogger metrics =====
 +You might just archive metrics with pmlogger for creating graphs with grafana, and not want to use them for debugging. In that case, you could
 +
 +  * initially run pmlogger with the metrics suggested by pcp-zeroconf
 +  * then create dashboard which show everything you need. While doing that, you might add metrics to the pmlogger config, so they get archived.
 +  * once you are happy, you can use the dashboard json-code to understand which metrics you use, and create a new config which only contains _these_ metrics.
 +
 +I used this workflow:
 +  * 'sudo dnf -y install xclip'
 +  * in your dashboard, go on the top to 'dashboard settings'
 +  * click 'json model', right click the json code, click 'select all' and 'copy'
 +<code>
 +# then, in a terminal use this to store the dashboard json
 +xclip -o -selection clipboard >grafana.json
 +
 +# lets now extract the sources
 +grep expr grafana.json | sed -e 's,.*": ",\t,' -e 's,".*,,' -e 's,{.*,,'
 +
 +Some values from there should be collapsed, for example
 +  lmsensors.acpitz_acpi_0.temp1
 + lmsensors.coretemp_isa_0000.core_0
 + lmsensors.coretemp_isa_0000.core_1
 +into
 +        lmsensors
 +..which will capture all the deeper metrics.
 +
 +# Then use the metrics in file
 +# /var/lib/pcp/config/pmlogger/config.default . For example:
 +######
 +log advisory on default {
 + lmsensors
 + kernel.all.load
 + mem.util.cached
 + mem.physmem
 + mem.util.free
 + kernel.percpu.cpu.user
 + kernel.percpu.cpu.sys
 + kernel.all.pswitch
 + kernel.all.runnable
 + mem.util.used
 + mem.util.cached
 + mem.util.free
 + mem.vmstat.pgfault
 + mem.vmstat.pgmajfault
 + network.interface.in.bytes
 + network.interface.out.bytes
 + network.interface.in.drops
 + network.interface.out.drops
 + network.interface.in.packets
 + network.interface.in.packets
 + network.tcp.timeouts
 + network.tcp.listenoverflows
 + network.tcp.listendrops
 + network.tcp.retranssegs
 + network.tcp.fastretrans
 + network.tcp.slowstartretrans
 + network.tcp.synretrans
 + disk.dev.read_rawactive
 + disk.dev.write_rawactive
 + disk.dev.read
 + disk.dev.write
 + disk.dev.read_bytes
 + disk.dev.write_bytes
 + disk.dev.avactive
 +}
 +
 +[access]
 +disallow .* : all;
 +disallow :* : all;
 +allow local:* : enquire;
 +###
 +
 +systemctl restart pmlogger
 +</code>
 +
 +===== Issues =====
 +I see this on Debian buster with PCP 5.1.0, a crashing pmda-openmetrics:
 +<code>
 +Log for pmdaopenmetrics on fluxcoil.net started Wed Apr  1 14:56:14 2020
 +
 +[Wed Apr  1 14:56:14] pmdaopenmetrics(15368) Critical: __pmdaSetupPDU: PMDA pmdaopenmetrics send creds: Broken pipe
 +[Wed Apr  1 14:56:14] pmdaopenmetrics(15368) Info: Initializing ... currently in notready state.
 +[Wed Apr  1 14:56:14] pmdaopenmetrics(15368) Info: Config change detected, traversed 3 config entries in 0.0004s, rescanning ...
 +[Wed Apr  1 14:56:14] pmdaopenmetrics(15368) Info: Found source netstats cluster 3
 +[Wed Apr  1 14:56:14] pmdaopenmetrics(15368) Info: Found source ttrss_stats5 cluster 8
 +[Wed Apr  1 14:56:14] pmdaopenmetrics(15368) Info: Found source webacccess cluster 2
 +[Wed Apr  1 14:56:14] pmdaopenmetrics(15368) Info: Ready to process requests
 +[Wed Apr  1 14:56:14] pmdaopenmetrics(15368) Critical: __pmdaSetupPDU: PMDA pmdaopenmetrics send creds: Broken pipe
 +[Wed Apr  1 14:56:14] pmdaopenmetrics(15368) Error: PMDA Initialisation Failed
 +
 +Log finished Wed Apr  1 14:56:14 2020
 +</code>
 +
 +===== Hints =====
 +  * [[https://hub.github.com/#developer|Will use 'hub' for next pull request]]
 +  * pmproxy debugging
 +<code>
 +# Grafana shows only 1 week old data, but archive files with 2 weeks old data exist
 +
 +# looking at grafana query:
 +http://127.0.0.1:3000/api/datasources/proxy/11/series/values?series=ed1fee80843
 +4b8c265752bd81ab18b4d8a45802c&start=1588916484&finish=1588916666&interval=0.01
 +
 +# performing query manually
 +curl 'http://localhost:44322/series/values?series=ed1fee808434b8c265752bd81ab18b4d8a45802c&start=1588916484&finish=1588916666&interval=0.01'
 +[{"series":"ed1fee808434b8c265752bd81ab18b4d8a45802c","instance":"a2b4c26ae379f30881bfe0c82f6c30cc10d0934f","timestamp":1588916524035.758,"value":"4.284000e+00"}[..]
 +
 +=> I always get "timestamp":1587801603901.84, and no newer timestamps.
 +Also querying for start=1000000000 does not change that
 +
 +=> Verify with pmrep on commandline if older data is available:
 +pmrep -a 20200415.0 kernel.all.load -p -s 1000000000 \
 +  -f '%Y-%m-%d %H:%M:%S'|head
 +                     k.a.load  k.a.load  k.a.load
 +                     1 minute  5 minute  15 minut
 +                                                 
 +2020-04-15 00:10:07       N/      N/      N/A
 +2020-04-15 00:10:08       N/      N/      N/A
 +2020-04-15 00:10:09       N/      N/      N/A
 +2020-04-15 00:10:10       N/      N/      N/A
 +2020-04-15 00:10:11     0.070     0.140     0.160
 +2020-04-15 00:10:12     0.070     0.140     0.160
 +2020-04-15 00:10:13     0.070     0.140     0.160
 +=> Data is there..
 </code> </code>