Netdata: Difference between revisions

From Freephile Wiki
Browse history interactively
← Older edit
VisualWikitext
m not 'stats'; data
Expand Ansible section
 
(16 intermediate revisions by 2 users not shown)
Line 1: Line 1:
Netdata will be part of the QualityBox dashboard.
Netdata is one of the [https://discourse.equality-tech.com/t/dashboards-in-qualitybox/107 QualityBox dashboards].


See http://wiki.freephile.org:20000/
<!-- These services/entry points are disabled in production
<!-- netdata.conf -->
 
See [{{SERVER}}:20000/ this website Live]
 
[{{SERVER}}:20000/netdata.conf Configuration]
 
netdata.conf -->


== System Locations ==
== System Locations ==
Depending on how you install netdata, it will be distributed in the normal system locations such as
<pre>
<pre>
   - the daemon    at /usr/sbin/netdata
   - the daemon    at /usr/sbin/netdata
Line 17: Line 23:
</pre>
</pre>


Or, if you use
<pre>bash <(curl -Ss https://my-netdata.io/kickstart-static64.sh)</pre>
to install, you'll get all of netdata installed into <code>/opt/netdata</code>
== With Ansible ==
There is an [https://learn.netdata.cloud/docs/netdata-agent/installation/ansible Ansible Playbook for deploying Netdata Agent] across multiple nodes. At that page, they also describe how to disable the local dashboard by setting <code>web_mode</code> to none. The security configuration is described at https://learn.netdata.cloud/docs/netdata-agent/configuration/securing-agents/


== Host Modifications ==
== Host Modifications ==


A Netdata role is available in [https://github.com/enterprisemediawiki/meza/blob/32.x/src/roles/netdata/tasks/main.yml the 32.x branch of Meza]
Otherwise, you have to make room in HAProxy for netdata:
=== HAProxy ===
=== HAProxy ===
<source lang="python">
<syntaxhighlight lang="python">
frontend netdata  
frontend netdata  
         bind *:20000  
         bind *:20000 ssl crt /etc/haproxy/certs/wiki.freephile.org.pem
         mode http  
         mode http  
         default_backend netdata-back  
         default_backend netdata-back  
Line 29: Line 45:
backend netdata-back  
backend netdata-back  
         server nd1 127.0.0.1:19999
         server nd1 127.0.0.1:19999
</source>  
</syntaxhighlight>


=== Kernel ===
=== Kernel ===
Line 57: Line 73:
<code>systemctl start netdata</code>
<code>systemctl start netdata</code>


== Installation extras ==
To reload configuration:
The configuration will send messages to 'root' so be sure to either edit the conf <code>sudo vim /etc/netdata/health_alarm_notify.conf</code>, or set <code>vim /etc/aliases && newaliases</code>
<code>killall -USR2 netdata</code> <ref>https://docs.netdata.cloud/health/quickstart/#reload-health-configuration</ref>


You'll probably receive alarms for 'tcp listen drops'. This is likely bot-related, and there is a good discussion on how to identify the source of the problem and how to mitigate or resolve it https://github.com/firehol/netdata/issues/3234
== Notifications ==
 
The default configuration will send messages to 'root' so be sure to either edit the conf <code>sudo vim /etc/netdata/health_alarm_notify.conf</code>, or set <code>vim /etc/aliases && newaliases</code>
 
=== Turn off alarm ===
 
 
<pre>
    to: silent # silence notification; still see on website
enabled: no    # disable alarm
</pre>
more details in the [https://docs.netdata.cloud/health/tutorials/stop-notifications-alarms/ netdata docs].
 
 
== Issues ==
 
You'll probably receive alarms for 'tcp listen drops'. This is likely bot-related (sending INVALID packets) and NOT due to your application dropping legitimate packets. There is a good discussion on how to identify the source of the problem and how to mitigate or resolve it [https://github.com/firehol/netdata/issues/3234 Issue #3234][https://github.com/firehol/netdata/issues/3826 Issue #3826] TLDR; increase the threshold to 1 (<code>/etc/netdata/health.d/tcp_listen.conf</code>) so you don't get bogus alerts. 
 
Also, you should modify your firewall to drop invalid packets before they're either counted (by netdata) or dropped (by the kernel).
 
<syntaxhighlight lang="bash">
iptables -A INPUT -m conntrack --ctstate INVALID -j DROP
ip6tables -A INPUT -m conntrack --ctstate INVALID -j DROP
iptables -A INPUT -m tcp -p tcp ! --tcp-flags FIN,SYN,RST,ACK SYN -m conntrack --ctstate NEW -j DROP
ip6tables -A INPUT -m tcp -p tcp ! --tcp-flags FIN,SYN,RST,ACK SYN -m conntrack --ctstate NEW -j DROP
</syntaxhighlight>


Following the advice from NASA at https://wiki.earthdata.nasa.gov/display/HDD/SOMAXCONN, I increased my somaxconn kernel parameter to 1024 from 128
Following the advice from NASA at https://wiki.earthdata.nasa.gov/display/HDD/SOMAXCONN, I increased my somaxconn kernel parameter to 1024 from 128
<source lang="bash">
<syntaxhighlight lang="bash">
  cat /proc/sys/net/core/somaxconn
  cat /proc/sys/net/core/somaxconn
  128
  128
  sysctl -w net.core.somaxconn=1024
  sysctl -w net.core.somaxconn=1024
</source>
</syntaxhighlight>


[[File:Tcp state diagram fixed.svg|600px|TCP State diagram]]
[[File:Tcp state diagram fixed.svg|600px|TCP State diagram]]
Line 78: Line 119:
ln -s /root/netdata/netdata-updater.sh /etc/cron.daily/netdata-updater
ln -s /root/netdata/netdata-updater.sh /etc/cron.daily/netdata-updater
</code>
</code>
{{References}}


[[Category:QualityBox]]
[[Category:QualityBox]]
[[Category:Monitoring]]
[[Category:Monitoring]]

Latest revision as of 09:36, 15 September 2025

Netdata is one of the QualityBox dashboards.


System Locations[edit]

Depending on how you install netdata, it will be distributed in the normal system locations such as 

   - the daemon     at /usr/sbin/netdata
   - config files   in /etc/netdata
   - web files      in /usr/share/netdata
   - plugins        in /usr/libexec/netdata
   - cache files    in /var/cache/netdata
   - db files       in /var/lib/netdata
   - log files      in /var/log/netdata
   - pid file       at /var/run/netdata.pid
   - logrotate file at /etc/logrotate.d/netdata

Or, if you use 

bash <(curl -Ss https://my-netdata.io/kickstart-static64.sh)

to install, you'll get all of netdata installed into /opt/netdata

With Ansible[edit]

There is an Ansible Playbook for deploying Netdata Agent across multiple nodes. At that page, they also describe how to disable the local dashboard by setting web_mode to none. The security configuration is described at https://learn.netdata.cloud/docs/netdata-agent/configuration/securing-agents/

Host Modifications[edit]

A Netdata role is available in the 32.x branch of Meza

Otherwise, you have to make room in HAProxy for netdata:

HAProxy[edit]

frontend netdata 
        bind *:20000 ssl crt /etc/haproxy/certs/wiki.freephile.org.pem
        mode http 
        default_backend netdata-back 
 
backend netdata-back 
        server nd1 127.0.0.1:19999

Kernel[edit]

You have kernel memory de-duper (called Kernel Same-page Merging, or KSM) available, but it is not currently enabled.

Memory de-duplication instructions

To enable it run: 

   echo 1 >/sys/kernel/mm/ksm/run
   echo 1000 >/sys/kernel/mm/ksm/sleep_millisecs

If you enable it, you will save 40-60% of netdata memory.


Ports[edit]

netdata by default listens on all IPs on port 19999. We add a rule to firewalld to allow 20000 and then pass that port through to the backend in haproxy config. 

 http://this.machine.ip:20000/ => http://127.0.0.1:19999

Start/Stop[edit]

To stop netdata run: systemctl stop netdata

To start netdata run: systemctl start netdata

To reload configuration: killall -USR2 netdata [1]

Notifications[edit]

The default configuration will send messages to 'root' so be sure to either edit the conf sudo vim /etc/netdata/health_alarm_notify.conf, or set vim /etc/aliases && newaliases

Turn off alarm[edit]

     to: silent # silence notification; still see on website
enabled: no     # disable alarm

more details in the netdata docs.


Issues[edit]

You'll probably receive alarms for 'tcp listen drops'. This is likely bot-related (sending INVALID packets) and NOT due to your application dropping legitimate packets. There is a good discussion on how to identify the source of the problem and how to mitigate or resolve it Issue #3234Issue #3826 TLDR; increase the threshold to 1 (/etc/netdata/health.d/tcp_listen.conf) so you don't get bogus alerts.

Also, you should modify your firewall to drop invalid packets before they're either counted (by netdata) or dropped (by the kernel). 

iptables -A INPUT -m conntrack --ctstate INVALID -j DROP
ip6tables -A INPUT -m conntrack --ctstate INVALID -j DROP
iptables -A INPUT -m tcp -p tcp ! --tcp-flags FIN,SYN,RST,ACK SYN -m conntrack --ctstate NEW -j DROP
ip6tables -A INPUT -m tcp -p tcp ! --tcp-flags FIN,SYN,RST,ACK SYN -m conntrack --ctstate NEW -j DROP

Following the advice from NASA at https://wiki.earthdata.nasa.gov/display/HDD/SOMAXCONN, I increased my somaxconn kernel parameter to 1024 from 128 

 cat /proc/sys/net/core/somaxconn
 128
 sysctl -w net.core.somaxconn=1024

TCP State diagram


Updates[edit]

Netdata will update itself, and puts a script into cron: ln -s /root/netdata/netdata-updater.sh /etc/cron.daily/netdata-updater


References[edit]

  1. ↑ https://docs.netdata.cloud/health/quickstart/#reload-health-configuration

A worker machine in a Kubernetes cluster.

Retrieved from "https://wiki.freephile.org/wiki/index.php?title=Netdata&oldid=12394"
Powered by QualityBox
Powered by Semantic MediaWiki