Netdata

Netdata is one of the QualityBox dashboards.

System Locations

Depending on how you install netdata, it will be distributed in the normal system locations such as

   - the daemon     at /usr/sbin/netdata
   - config files   in /etc/netdata
   - web files      in /usr/share/netdata
   - plugins        in /usr/libexec/netdata
   - cache files    in /var/cache/netdata
   - db files       in /var/lib/netdata
   - log files      in /var/log/netdata
   - pid file       at /var/run/netdata.pid
   - logrotate file at /etc/logrotate.d/netdata

Or, if you use

bash <(curl -Ss https://my-netdata.io/kickstart-static64.sh)

to install, you'll get all of netdata installed into /opt/netdata

With Ansible

There is an Ansible Playbook for deploying Netdata Agent across multiple nodes. At that page, they also describe how to disable the local dashboard by setting web_mode to none. The security configuration is described at https://learn.netdata.cloud/docs/netdata-agent/configuration/securing-agents/

Host Modifications

A Netdata role is available in the 32.x branch of Meza

Otherwise, you have to make room in HAProxy for netdata:

HAProxy

frontend netdata 
        bind *:20000 ssl crt /etc/haproxy/certs/wiki.freephile.org.pem
        mode http 
        default_backend netdata-back 
 
backend netdata-back 
        server nd1 127.0.0.1:19999

Kernel

You have kernel memory de-duper (called Kernel Same-page Merging, or KSM) available, but it is not currently enabled.

Memory de-duplication instructions

To enable it run:

   echo 1 >/sys/kernel/mm/ksm/run
   echo 1000 >/sys/kernel/mm/ksm/sleep_millisecs

If you enable it, you will save 40-60% of netdata memory.

Ports

netdata by default listens on all IPs on port 19999. We add a rule to firewalld to allow 20000 and then pass that port through to the backend in haproxy config.

 http://this.machine.ip:20000/ => http://127.0.0.1:19999

Start/Stop

To stop netdata run: systemctl stop netdata

To start netdata run: systemctl start netdata

To reload configuration: killall -USR2 netdata

Notifications

The default configuration will send messages to 'root' so be sure to either edit the conf sudo vim /etc/netdata/health_alarm_notify.conf, or set vim /etc/aliases && newaliases

Turn off alarm

     to: silent # silence notification; still see on website
enabled: no     # disable alarm

more details in the netdata docs.

Issues

You'll probably receive alarms for 'tcp listen drops'. This is likely bot-related (sending INVALID packets) and NOT due to your application dropping legitimate packets. There is a good discussion on how to identify the source of the problem and how to mitigate or resolve it Issue #3234Issue #3826 TLDR; increase the threshold to 1 (/etc/netdata/health.d/tcp_listen.conf) so you don't get bogus alerts.

Also, you should modify your firewall to drop invalid packets before they're either counted (by netdata) or dropped (by the kernel).

iptables -A INPUT -m conntrack --ctstate INVALID -j DROP
ip6tables -A INPUT -m conntrack --ctstate INVALID -j DROP
iptables -A INPUT -m tcp -p tcp ! --tcp-flags FIN,SYN,RST,ACK SYN -m conntrack --ctstate NEW -j DROP
ip6tables -A INPUT -m tcp -p tcp ! --tcp-flags FIN,SYN,RST,ACK SYN -m conntrack --ctstate NEW -j DROP

Following the advice from NASA at https://wiki.earthdata.nasa.gov/display/HDD/SOMAXCONN, I increased my somaxconn kernel parameter to 1024 from 128

 cat /proc/sys/net/core/somaxconn
 128
 sysctl -w net.core.somaxconn=1024

Updates

Netdata will update itself, and puts a script into cron: ln -s /root/netdata/netdata-updater.sh /etc/cron.daily/netdata-updater

A worker machine in a Kubernetes cluster.