In this tutorial, I will talk about Prometheus and Grafana. You will learn to install both on CentOS/RHEL and understand how to use Prometheus and Grafana

running a lemmy instance has learned me a lot about devops already!

I love playing around with this, the lemmy instance was a great reason. lemmyfly.org doesn’t have a lot of traffic yet, I think it can handle some more. Currently running on 2 vcpu/4gb ram.

After chatting to my devops colleague at work I wanted to set up a Grafana dashboard with metrics on the server. Seems it’s pretty pretty doable!

  • I installed prometheus node_exporter on my lemmy server
  • I created a second server at my VPS. Very cheap, and don’t think I’ll be running the Prometheus/Grafana 24/7 so I’ll just create a snapshot of it, destroy the server and re-create when wanted
  • Created an internal private network so the instances can communicate without exposing ports to the public
  • Installed Prometheus on the new prometheus server
  • Installed Grafana
  • log in to Grafana, admin:admin then change password
  • Go to the Grafana home dashboard, click on + sign and click on Import. In Import via grafana.com, put the dashboard id 1860 (preset for all prometheus node_exporter metrics) and click on Load.
  • Done!

Last 2 hours in the garden with a beer on the side, WFH FTW

  • lckdscl [they/them]@whiskers.bim.boatsEnglish
    3·
    1 year ago

    I fell into the deep end with Grafana + Prometheus, went crazy with all the metrics and dashboard. But I got burnt out before I even began making alerts, so I just went with Netdata at the end.

  • Jason@lemmy.worldEnglish
    2·
    1 year ago

    Uuuh I absolutely love grafana. Worked half a year with it for my job. I am happy it worked out great for you.

    • majorswitcher@lemmyfly.orgOPEnglish
      1·
      1 year ago

      actually I did delete the server (after creating a snapshot of it) a week or so ago. But this morning I wanted to check lemmyfly.org, couldn’t load the page. Checking my Hetzner dashboard I noticed CPU was spiked at 200%?! It did drop again though, but apparently had last for 2-3 minutes. But prometheus was down, so no graphs apart from the hetzner ones. I doesn’t relate to network traffic spikes, so I don’t know what caused it. I’ve started the prometheus server again (that snapshot was really useful :) ) and will leave it on for a couple of months now.

      current system consumption:

      I might need to get an extra volume for storage, Lemmy is starting to eat up the root filesystem… Does anyone know how I re-configure Lemmy to look at a different volume for storage ?

  • kunic@kbin.social
    1·
    1 year ago

    I’ve been using Cockpit to monitor my server, but this is super clean. Will have to try it out!

  • manned_meatball@lemmy.mlEnglish
    0·
    1 year ago

    lol I literally did this yesterday for my job, using dashboard 1860 too. I used a docker compose stack following their guide here.