Docker-stats.sh troubleshooting

From RZR, 2 Years ago, written in Plain Text.
URL https://p.libren.ms/view/65368c30
Embed
Download Paste or View Raw
  1. librenms@LibreNMS:~$ ./poller.php -h Smokeping -d -v -m applications
  2. LibreNMS Poller
  3. SQL[select `migration` from `migrations` order by `id` desc limit 1 [] 0.27ms]
  4.  
  5. SQL[select count(*) as aggregate from `migrations` limit 1 [] 0.25ms]
  6.  
  7. SQL[select version() [] 0.24ms]
  8.  
  9. ===================================
  10. Version info:
  11. Commit SHA: a2ad39a38694b9233792d3c3f76e9804e2c278d9
  12. Commit Date: 1641934858
  13. DB Schema: 2021_11_29_165436_improve_ports_search_index (229)
  14. PHP: 7.4.25
  15. MySQL: 10.5.12-MariaDB-0+deb11u1
  16. RRDTool: 1.7.2
  17. SNMP: 5.9
  18. ==================================DEBUG!
  19. Updating os_def.cache  
  20. Override poller modules: applications  
  21. Starting polling run:
  22.  
  23. SQL[SELECT * FROM `devices` WHERE `disabled` = 0 AND `hostname` = 'Smokeping' ORDER BY `device_id` ASC [] 0.53ms]
  24.  
  25. SQL[select * from `devices` where `device_id` = ? limit 1 [11] 0.29ms]
  26.  
  27. SQL[select * from `devices_attribs` where `devices_attribs`.`device_id` = ? and `devices_attribs`.`device_id` is not null [11] 0.23ms]
  28.  
  29. Attempting to initialize OS: proxmox  
  30. Attempting to initialize Group OS: unix  
  31. OS initialized: LibreNMS\OS\Shared\Unix  
  32. Hostname:    Smokeping
  33. Device ID:   11
  34. OS:          proxmox
  35. Assigned IP: 10.146.68.4
  36.  (unix)
  37.  
  38. [FPING] '/usr/bin/fping' '-e' '-q' '-c' '2' '-p' '200' '-t' '200' '-O' '0' '10.146.68.4'
  39.  
  40. response: xmt/rcv/%loss = 2/2/0%, min/avg/max = 0.021/0.037/0.053  
  41. SNMP['/usr/bin/snmpget' '-M' '/opt/librenms/mibs' '-v3' '-l' 'authPriv' '-n' "" '-x' 'AES' '-X' '$Q0GLnG1DhzzNgc6C$sKGJwu' '-a' 'SHA' '-A' '$efjyExR81kNw!PBi#6AbRAx' '-u' 'LibreNMS' '-OQXUte' 'udp:10.146.68.4:161' 'SNMPv2-MIB::sysObjectID.0']  
  42. SNMPv2-MIB::sysObjectID.0 = SNMPv2-SMI::enterprises.8072.3.2.10
  43.  
  44.  
  45. SQL[insert into `device_perf` (`min`, `max`, `avg`, `xmt`, `rcv`, `loss`, `device_id`, `timestamp`) values (?, ?, ?, ?, ?, ?, ?, ?) [0.021,0.053,0.037,2,2,0,11,"2022-01-13 12:20:46"] 0.42ms]
  46.  
  47. SQL[update `devices` set `last_ping_timetaken` = ? where `device_id` = ? [0.037,11] 0.25ms]
  48.  
  49. RRD[last Smokeping/ping-perf.rrd  --daemon unix:/var/run/rrdcached.sock]  
  50. RRDtool Output: 1642069200
  51. OK u:0.00 s:0.00 r:0.23
  52. RRD[update Smokeping/ping-perf.rrd N:0.037 --daemon unix:/var/run/rrdcached.sock]  
  53. RRDtool Output: SQL[select * from `device_outages` where `device_outages`.`device_id` = ? and `device_outages`.`device_id` is not null and `up_again` is null order by `going_down` desc limit 1 [11] 0.29ms]
  54.  
  55. Modules status: Global+  
  56. OS  
  57. Device  
  58.  
  59. #### Load poller module core ####
  60. SNMP['/usr/bin/snmpget' '-M' '/opt/librenms/mibs' '-v3' '-l' 'authPriv' '-n' "" '-x' 'AES' '-X' '$Q0GLnG1DhzzNgc6C$sKGJwu' '-a' 'SHA' '-A' '$efjyExR81kNw!PBi#6AbRAx' '-u' 'LibreNMS' '-OQXUte' '-On' 'udp:10.146.68.4:161' 'SNMPv2-MIB::sysDescr.0' 'SNMPv2-MIB::sysObjectID.0' 'SNMPv2-MIB::sysUpTime.0' 'SNMPv2-MIB::sysName.0']  
  61. .1.3.6.1.2.1.1.1.0 = Linux Smokeping 5.13.19-2-pve #1 SMP PVE 5.13.19-4 (Mon, 29 Nov 2021 12:10:09 +0100) x86_64
  62. .1.3.6.1.2.1.1.2.0 = .1.3.6.1.4.1.8072.3.2.10
  63. .1.3.6.1.2.1.1.3.0 = 146434
  64. .1.3.6.1.2.1.1.5.0 = Smokeping
  65.  
  66.  
  67. SNMP['/usr/bin/snmpget' '-M' '/opt/librenms/mibs' '-v3' '-l' 'authPriv' '-n' "" '-x' 'AES' '-X' '$Q0GLnG1DhzzNgc6C$sKGJwu' '-a' 'SHA' '-A' '$efjyExR81kNw!PBi#6AbRAx' '-u' 'LibreNMS' '-OQXUte' 'udp:10.146.68.4:161' 'SNMP-FRAMEWORK-MIB::snmpEngineTime.0' 'HOST-RESOURCES-MIB::hrSystemUptime.0']  
  68. SNMP-FRAMEWORK-MIB::snmpEngineTime.0 = 1464
  69. HOST-RESOURCES-MIB::hrSystemUptime.0 = 423918
  70.  
  71.  
  72. Uptime seconds: 4239
  73.  
  74. RRD[last Smokeping/uptime.rrd  --daemon unix:/var/run/rrdcached.sock]  
  75. RRDtool Output: 1642069200
  76. OK u:0.00 s:0.00 r:0.27
  77. RRD[update Smokeping/uptime.rrd N:4239 --daemon unix:/var/run/rrdcached.sock]  
  78. RRDtool Output: OK u:0.00 s:0.00 r:0.23
  79. Uptime:  1 hour 10 minutes 39 seconds
  80. SQL[update `devices` set `uptime` = ? where `device_id` = ? [4239,11] 0.29ms]
  81.  
  82.  
  83. >> Runtime for poller module 'core': 0.0461 seconds with 56976 bytes
  84. >> SNMP: [2/0.04s] MySQL: [1/0.00s] RRD: [2/0.00s]  
  85. #### Unload poller module core ####
  86.  
  87. RRD[last Smokeping/poller-perf-core.rrd  --daemon unix:/var/run/rrdcached.sock]  
  88. RRDtool Output: 1642069200
  89. OK u:0.00 s:0.00 r:0.28
  90. RRD[update Smokeping/poller-perf-core.rrd N:0.046087980270386 --daemon unix:/var/run/rrdcached.sock]  
  91. RRDtool Output: OK u:0.00 s:0.00 r:0.27
  92. Modules status: Global+  
  93. OS  
  94. Device  
  95.  
  96. #### Load poller module applications ####
  97. SQL[SELECT * FROM `applications` WHERE `device_id`  = ? [11] 0.23ms]
  98.  
  99. Application: docker, app_id=23SNMP['/usr/bin/snmpget' '-v3' '-l' 'authPriv' '-n' "" '-a' 'SHA' '-A' '$efjyExR81kNw!PBi#6AbRAx' '-u' 'LibreNMS' '-x' 'AES' '-X' '$Q0GLnG1DhzzNgc6C$sKGJwu' '-Oqv' '-m' 'NET-SNMP-EXTEND-MIB' '-M' '/opt/librenms/mibs' 'udp:10.146.68.4:161' 'nsExtendOutputFull.6.100.111.99.107.101.114']
  100. {"version":"1","data":[{"container":"smokeping","pids":23,"memory":{"used":"107.3MiB","limit":"512MiB","perc":"20.96%"},"cpu":"0.04%"}],"error":"0","errorString":""}  
  101.  
  102. RRD[last Smokeping/app-docker-23-smokeping.rrd  --daemon unix:/var/run/rrdcached.sock]  
  103. RRDtool Output: 1642069200
  104. OK u:0.00 s:0.00 r:1.90
  105. RRD[update Smokeping/app-docker-23-smokeping.rrd N:0.04:23:536870912:112512204:20.96 --daemon unix:/var/run/rrdcached.sock]  
  106. RRDtool Output: OK u:0.00 s:0.00 r:0.27
  107. SQL[UPDATE `applications` set `app_state`=?,`app_status`=?,`timestamp`=NOW() WHERE `app_id` = ? ["OK","",23] 0.31ms]
  108.  
  109. SQL[SELECT * FROM `application_metrics` WHERE app_id=? [23] 0.32ms]
  110.  
  111. : SQL[UPDATE `application_metrics` set `value`=?,`value_prev`=? WHERE app_id=? && metric=? [0.04,0,23,"smokeping_cpu_usage"] 0.24ms]
  112.  
  113. USQL[UPDATE `application_metrics` set `value`=?,`value_prev`=? WHERE app_id=? && metric=? [23,22,23,"smokeping_pids"] 0.25ms]
  114.  
  115. U.SQL[UPDATE `application_metrics` set `value`=?,`value_prev`=? WHERE app_id=? && metric=? [112512204,112197632,23,"smokeping_mem_used"] 0.14ms]
  116.  
  117. USQL[UPDATE `application_metrics` set `value`=?,`value_prev`=? WHERE app_id=? && metric=? [20.96,20.9,23,"smokeping_mem_perc"] 0.19ms]
  118.  
  119. U
  120. Application: os-updates, app_id=22SNMP['/usr/bin/snmpget' '-v3' '-l' 'authPriv' '-n' "" '-a' 'SHA' '-A' '$efjyExR81kNw!PBi#6AbRAx' '-u' 'LibreNMS' '-x' 'AES' '-X' '$Q0GLnG1DhzzNgc6C$sKGJwu' '-Oqv' '-m' 'NET-SNMP-EXTEND-MIB' '-M' '/opt/librenms/mibs' 'udp:10.146.68.4:161' '.1.3.6.1.4.1.8072.1.3.2.4.1.2.8.111.115.117.112.100.97.116.101.1']
  121. 0  
  122.  
  123. RRD[last Smokeping/app-os-updates-22.rrd  --daemon unix:/var/run/rrdcached.sock]  
  124. RRDtool Output: 1642069200
  125. OK u:0.00 s:0.00 r:2.91
  126. RRD[update Smokeping/app-os-updates-22.rrd N:0 --daemon unix:/var/run/rrdcached.sock]  
  127. RRDtool Output: OK u:0.00 s:0.00 r:1.90
  128. SQL[UPDATE `applications` set `app_state`=?,`app_status`=?,`timestamp`=NOW() WHERE `app_id` = ? ["OK","0",22] 0.33ms]
  129.  
  130. SQL[SELECT * FROM `application_metrics` WHERE app_id=? [22] 0.31ms]
  131.  
  132. : .
  133.  
  134.  
  135. >> Runtime for poller module 'applications': 2.6328 seconds with 9960 bytes
  136. >> SNMP: [2/2.63s] MySQL: [9/0.02s] RRD: [6/0.00s]  
  137. #### Unload poller module applications ####
  138.  
  139. RRD[last Smokeping/poller-perf-applications.rrd  --daemon unix:/var/run/rrdcached.sock]  
  140. RRDtool Output: 1642069200
  141. OK u:0.00 s:0.00 r:2.91
  142. RRD[update Smokeping/poller-perf-applications.rrd N:2.6328229904175 --daemon unix:/var/run/rrdcached.sock]  
  143. RRDtool Output: OK u:0.00 s:0.00 r:2.90
  144. RRD[last Smokeping/poller-perf.rrd  --daemon unix:/var/run/rrdcached.sock]  
  145. RRDtool Output: 1642069200
  146. OK u:0.00 s:0.00 r:2.91
  147. RRD[update Smokeping/poller-perf.rrd N:2.915 --daemon unix:/var/run/rrdcached.sock]  
  148. RRDtool Output: OK u:0.00 s:0.00 r:2.91
  149.  
  150. Polled in 2.915 seconds
  151. ### Start Device Groups ###
  152. SQL[select * from `device_groups` [] 0.15ms]
  153.  
  154. SQL[select * from `device_group_device` where `device_group_device`.`device_id` = ? [11] 0.18ms]
  155.  
  156. Groups Added:  
  157. Groups Removed:  
  158. ### End Device Groups, runtime: 0.0027s ###
  159.  
  160. #### Start Alerts ####
  161. SQL[select `device_groups`.*, `device_group_device`.`device_id` as `pivot_device_id`, `device_group_device`.`device_group_id` as `pivot_device_group_id` from `device_groups` inner join `device_group_device` on `device_groups`.`id` = `device_group_device`.`device_group_id` where `device_group_device`.`device_id` = ? [11] 0.25ms]
  162.  
  163. SQL[select * from `locations` where `locations`.`id` = ? limit 1 [3] 0.21ms]
  164.  
  165. SQL[select exists(select * from `alert_schedule` where (`start` &lt;= ? and `end` >= ? and (`recurring` = ? or (`recurring` = ? and ((time(`start`) < time xss=removed> ?) or (time(`start`) > time(`end`) and (time(`end`) &lt;= ? or time(`start`) > ?))) and (`recurring_day` like ? or `recurring_day` is null)))) and (exists (select * from `devices` inner join `alert_schedulables` on `devices`.`device_id` = `alert_schedulables`.`alert_schedulable_id` where `alert_schedule`.`schedule_id` = `alert_schedulables`.`schedule_id` and `alert_schedulables`.`alert_schedulable_type` = ? and `alert_schedulables`.`alert_schedulable_id` = ?) or exists (select * from `locations` inner join `alert_schedulables` on `locations`.`id` = `alert_schedulables`.`alert_schedulable_id` where `alert_schedule`.`schedule_id` = `alert_schedulables`.`schedule_id` and `alert_schedulables`.`alert_schedulable_type` = ? and `alert_schedulables`.`alert_schedulable_id` = ?))) as `exists` ["2022-01-13T10:20:49.445397Z","2022-01-13T10:20:49.445397Z",0,1,"10:20:49","10:20:49","10:20:49","10:20:49","%","device",11,"location",3] 0.52ms]
  166.  
  167. SQL[select * from `devices` where `devices`.`device_id` = ? limit 1 [11] 0.33ms]
  168.  
  169. SQL[SELECT DISTINCT a.* FROM alert_rules a
  170.         LEFT JOIN alert_device_map d ON a.id=d.rule_id AND (a.invert_map = 0 OR a.invert_map = 1 AND d.device_id = ?)
  171.         LEFT JOIN alert_group_map g ON a.id=g.rule_id AND (a.invert_map = 0 OR a.invert_map = 1 AND g.group_id IN (SELECT DISTINCT device_group_id FROM device_group_device WHERE device_id = ?))
  172.         LEFT JOIN alert_location_map l ON a.id=l.rule_id AND (a.invert_map = 0 OR a.invert_map = 1 AND l.location_id IN (SELECT DISTINCT location_id FROM devices WHERE device_id = ?))
  173.         LEFT JOIN device_group_device dg ON g.group_id=dg.device_group_id AND dg.device_id = ?
  174.         WHERE a.disabled = 0 AND (
  175.             (d.device_id IS NULL AND g.group_id IS NULL)
  176.             OR (a.invert_map = 0 AND (d.device_id=? OR dg.device_id=?))
  177.             OR (a.invert_map = 1  AND (d.device_id != ? OR d.device_id IS NULL) AND (dg.device_id != ? OR dg.device_id IS NULL))
  178.         ) [11,11,11,11,11,11,11,11] 0.71ms]
  179.  
  180. #### End Alerts ####
  181. ./poller.php Smokeping 2022-01-13 12:20:49 - 1 devices polled in 3.084 secs  
  182.  
  183. SNMP [5/2.69s]: Snmpget[5/2.69s]  
  184. SQL [29/0.10s]: Select[20/0.07s] Insert[1/0.00s] Update[8/0.02s]  
  185. RRD [14/0.00s]: Other[7/0.00s] Update[7/0.00s]  
  186. librenms@LibreNMS:~$
  187. ________________________________________________________________________________
  188.  
  189. rrdtool graph /tmp/aptyyyTKhyRX6x77  --alt-autoscale-max --rigid -E --start 1641982500 --end 1642069134 --width 2559 --height 738 -c BACK\#EEEEEE00 -c SHADEA\#EEEEEE00 -c SHADEB\#EEEEEE00 -c CANVAS\#FFFFFF00 -c GRID\#292929 -c MGRID\#2f343e -c FRAME\#5e5e5e -c ARROW\#5e5e5e -R normal -c FONT\#bfc0c0 --font LEGEND:8:DejaVuSansMono --font AXIS:7:DejaVuSansMono --font-render-mode normal COMMENT:'Memory \(%\)                         Now         Min         Max        Avg\\l' COMMENT:'\\l' DEF:mem_perc1=Smokeping/app-docker-23-smokeping.rrd:mem_perc:AVERAGE  DEF:mem_perc1min=Smokeping/app-docker-23-smokeping.rrd:mem_perc:MIN  DEF:mem_perc1max=Smokeping/app-docker-23-smokeping.rrd:mem_perc:MAX  CDEF:mem_perc_cdef1=mem_perc1,100,/ CDEF:mem_perc_cdef1min=mem_perc1min,100,/ CDEF:mem_perc_cdef1max=mem_perc1max,100,/ LINE2:mem_perc_cdef1\#CC7CCC:'smokeping                 ' GPRINT:mem_perc_cdef1:LAST:%8.0lf%s GPRINT:mem_perc_cdef1min:MIN:%8.0lf%s GPRINT:mem_perc_cdef1max:MAX:%8.0lf%s GPRINT:mem_perc_cdef1:AVERAGE:'%8.0lf%s\\n' COMMENT:'\\n' --daemon unix:/var/run/rrdcached.sock
  190.  
  191. _________________________________________________________________________________
  192.  
  193. librenms@LibreNMS:~/rrd$ rrdtool graph /tmp/aptyyyTKhyRX6x77  --alt-autoscale-max --rigid -E --start 1641982500 --end 1642069134 --width 2559 --height 738 -c BACK\#EEEEEE00 -c SHADEA\#EEEEEE00 -c SHADEB\#EEEEEE00 -c CANVAS\#FFFFFF00 -c GRID\#292929 -c MGRID\#2f343e -c FRAME\#5e5e5e -c ARROW\#5e5e5e -R normal -c FONT\#bfc0c0 --font LEGEND:8:DejaVuSansMono --font AXIS:7:DejaVuSansMono --font-render-mode normal COMMENT:'Memory \(%\)                         Now         Min         Max        Avg\\l' COMMENT:'\\l' DEF:mem_perc1=Smokeping/app-docker-23-smokeping.rrd:mem_perc:AVERAGE  DEF:mem_perc1min=Smokeping/app-docker-23-smokeping.rrd:mem_perc:MIN  DEF:mem_perc1max=Smokeping/app-docker-23-smokeping.rrd:mem_perc:MAX  CDEF:mem_perc_cdef1=mem_perc1,100,/ CDEF:mem_perc_cdef1min=mem_perc1min,100,/ CDEF:mem_perc_cdef1max=mem_perc1max,100,/ LINE2:mem_perc_cdef1\#CC7CCC:'smokeping                 ' GPRINT:mem_perc_cdef1:LAST:%8.0lf%s GPRINT:mem_perc_cdef1min:MIN:%8.0lf%s GPRINT:mem_perc_cdef1max:MAX:%8.0lf%s GPRINT:mem_perc_cdef1:AVERAGE:'%8.0lf%s\\n' COMMENT:'\\n' --daemon unix:/var/run/rrdcached.sock                                                                                                                                                                          
  194. 2640x835
  195.  


Reply to "Docker-stats.sh troubleshooting"