Grafana Monitoring Tutorial: Build Dashboards That Actually Help

Grafana is the de facto standard for observability dashboards. But here's the truth: most Grafana dashboards are useless. They show 50 panels of data nobody looks at, with colors that mean nothing and thresholds that trigger at the wrong times.

This tutorial covers not just how to use Grafana, but how to build dashboards that actually help you detect problems, debug incidents, and sleep through the night.

What Grafana Does

Grafana is a visualization layer. It doesn't collect metrics — it queries data sources that do:

Prometheus: metrics (time-series data)
Loki: logs
Tempo/Jaeger: traces
MySQL/PostgreSQL: business data
CloudWatch/Azure Monitor/GCP Monitoring: cloud metrics
Elasticsearch: search and log analytics

You connect data sources, write queries, and Grafana renders panels.

Step 1: Installation

Docker (Fastest)

# docker-compose.yml
version: '3.8'

services:
  grafana:
    image: grafana/grafana:latest
    ports:
      - "3000:3000"
    environment:
      - GF_SECURITY_ADMIN_PASSWORD=admin
    volumes:
      - grafana-data:/var/lib/grafana

  prometheus:
    image: prom/prometheus:latest
    ports:
      - "9090:9090"
    volumes:
      - ./prometheus.yml:/etc/prometheus/prometheus.yml

volumes:
  grafana-data:

prometheus.yml:

global:
  scrape_interval: 15s

scrape_configs:
  - job_name: 'prometheus'
    static_configs:
      - targets: ['localhost:9090']

  - job_name: 'node'
    static_configs:
      - targets: ['node-exporter:9100']

docker-compose up -d
# Grafana is now running at http://localhost:3000
# Default login: admin / admin

Linux (Direct Install)

# Ubuntu/Debian
sudo apt-get install -y adduser libfontconfig1 musl
wget https://dl.grafana.com/oss/release/grafana_11.0.0_amd64.deb
sudo dpkg -i grafana_11.0.0_amd64.deb
sudo systemctl enable grafana-server
sudo systemctl start grafana-server

Step 2: Connect Data Sources

Adding Prometheus

Go to Connections → Data Sources → Add data source
Select Prometheus
URL: http://prometheus:9090 (or http://localhost:9090)
Click Save & Test
You should see "Data source is working"

Adding Loki (for logs)

# Add to docker-compose.yml
  loki:
    image: grafana/loki:latest
    ports:
      - "3100:3100"

Add data source → Loki
URL: http://loki:3100
Save & Test

Step 3: Your First Dashboard

Creating a Panel

Click + → New Dashboard
Click + Add visualization
Select your Prometheus data source

Example Query: CPU Usage

# Average CPU usage percentage per instance
100 - (avg by(instance) (rate(node_cpu_seconds_total{mode="idle"}[5m])) * 100)

Example Query: Memory Usage

# Memory used / total * 100
(node_memory_MemTotal_bytes - node_memory_MemAvailable_bytes) 
/ node_memory_MemTotal_bytes * 100

Example Query: HTTP Request Rate

# Requests per second by status code
sum by(status) (rate(http_requests_total[5m]))

Example Query: Error Rate

# 5xx errors as percentage of total requests
sum(rate(http_requests_total{status=~"5.."}[5m])) 
/ sum(rate(http_requests_total[5m])) * 100

Example Query: P99 Latency

# 99th percentile response time
histogram_quantile(0.99, 
  sum by(le) (rate(http_request_duration_seconds_bucket[5m]))
)

Step 4: Panel Types and When to Use Them

| Panel Type | Use For | Example | |-----------|---------|---------| | Time Series | Metrics over time | CPU, memory, latency | | Stat | Single important number | Current error rate, uptime | | Gauge | Value within a range (0-100%) | Disk usage, CPU | | Bar Gauge | Multiple values with ranges | Per-instance CPU | | Table | Tabular data | Slow queries, alert list | | Heatmap | Distribution over time | Latency percentiles | | GeoMap | Geographic data | Request origin map | | Log Viewer | Log lines | Application errors | | Node Graph | Service dependencies | Distributed trace topology |

Time Series is the most common — use it for 80% of your panels. Use Stat for the single most important numbers at the top of the dashboard.

Step 5: Building USEFUL Dashboards

Most dashboards fail because they show everything. Here's a framework for dashboards that work.

The Three-Dashboard Pattern

Dashboard 1: Overview (for everyone)

Is the system up? (stat panel, green/red)
Error rate (stat + trend)
Request rate (time series)
P50/P99 latency (time series)
Active alerts (alert list)

Dashboard 2: Detailed (for engineers)

CPU, memory, disk, network per instance
Request breakdown by endpoint
Error breakdown by type
Database connection pool
Queue depth

Dashboard 3: Business (for stakeholders)

Active users
Signups/day
Revenue metrics
API usage by tier

Dashboard Design Principles

1. Left to right, top to bottom People read dashboards like text. Put the most critical info top-left. Put deep-dive details at the bottom.

2. Less is more A dashboard with 4 clear panels is better than one with 30 confusing ones. If you need more detail, link to a sub-dashboard.

3. Color means something

Green = healthy
Yellow = warning
Red = critical
Blue = informational

Don't use random colors. Don't use 12 colors. Pick a palette and stick to it.

4. Set thresholds Every panel should have thresholds:

CPU < 70%: green
CPU 70-85%: yellow
CPU > 85%: red

5. Use variables Make dashboards reusable with variables:

Variable: $instance (query: label_values(node_cpu_seconds_total, instance))
Variable: $interval (custom: 5m, 15m, 1h, 6h)

Now one dashboard works for all your servers.

Step 6: Alerts That Don't Cry Wolf

Bad alerts are worse than no alerts — you learn to ignore them.

Alerting Rules

Navigate to Alerting → Alert Rules → New alert rule.

Example: High CPU Alert

# Query A
100 - (avg by(instance) (rate(node_cpu_seconds_total{mode="idle"}[5m])) * 100)

# Condition: WHEN last() OF A IS ABOVE 85 FOR 5m

Example: High Error Rate Alert

# Query A
sum(rate(http_requests_total{status=~"5.."}[5m])) 
/ sum(rate(http_requests_total[5m])) * 100

# Condition: WHEN last() OF A IS ABOVE 5 FOR 2m

Alert Labels and Routing

# Labels
severity: warning  # or critical
team: backend      # or frontend, infra
service: api       # service name

# Annotations
summary: "High CPU on {{ $labels.instance }}"
description: "CPU usage is {{ $values.A }}% for the last 5 minutes"

Notification Channels

Configure in Alerting → Contact points:

Slack/Discord: Good for warnings and daily summaries
PagerDuty/OpsGenie: For critical alerts that need immediate response
Email: For non-urgent notifications
Webhook: Custom integrations

Alert Design Principles

Alert on symptoms, not causes — alert on "users seeing errors", not "CPU is high"
Every alert should be actionable — if you can't fix it, don't alert on it
Set appropriate for durations — avoid flapping alerts (use for: 5m or more)
Use severity levels — critical = page someone, warning = Slack notification
Suppress known maintenance — use maintenance windows

Step 7: Log Integration with Loki

Combine metrics and logs in one view:

LogQL query examples:

# All error logs from the API service
{service="api"} |= "ERROR"

| json | level="error"

# Logs with latency > 1s
{service="api"} | json | duration > 1000000000

# Count errors per minute
sum(count_over_time({service="api"} |= "ERROR" [1m]))

Pro tip: In any time series panel, you can split the panel and show logs below the metrics — so when latency spikes, you see the logs from that exact moment.

Step 8: Dashboard as Code

For production setups, define dashboards as JSON and provision them automatically:

# grafana/provisioning/dashboards/dashboards.yml
apiVersion: 1

providers:
  - name: 'Default'
    orgId: 1
    folder: 'Services'
    type: file
    disableDeletion: false
    updateIntervalSeconds: 30
    options:
      path: /var/lib/grafana/dashboards

Then place dashboard JSON files in that directory. Grafana auto-loads them on startup.

Dashboard JSON structure (simplified):

{
  "title": "API Overview",
  "panels": [
    {
      "type": "stat",
      "title": "Error Rate",
      "gridPos": { "h": 4, "w": 6, "x": 0, "y": 0 },
      "datasource": "Prometheus",
      "targets": [
        { "expr": "sum(rate(http_requests_total{status=~\"5..\"}[5m])) / sum(rate(http_requests_total[5m])) * 100" }
      ],
      "fieldConfig": {
        "defaults": {
          "thresholds": {
            "steps": [
              { "color": "green", "value": 0 },
              { "color": "yellow", "value": 1 },
              { "color": "red", "value": 5 }
            ]
          }
        }
      }
    }
  ]
}

Tools like grafonnet (Jsonnet library) and terraform-provider-grafana let you manage dashboards programmatically.

Step 9: Best Practices Checklist

Performance

[ ] Query ranges match scrape intervals (rate(...[5m]) with 15s scrape interval = good)
[ ] No more than 20 panels per dashboard (Grafana slows down with too many)
[ ] Use recording rules for expensive queries
[ ] Set refresh interval to 30s or 1m (not 5s — it overloads your browser and backend)

Organization

[ ] Dashboards organized in folders (by team or service)
[ ] Consistent naming convention ([Service] - [Dashboard Name])
[ ] Variables for instance/environment selection
[ ] Descriptions on non-obvious panels

Security

[ ] Default admin password changed
[ ] LDAP/OAuth integration for teams
[ ] View-only permissions for non-engineers
[ ] Data source credentials in environment variables, not hardcoded

Observability

[ ] Metrics + logs in the same dashboard
[ ] Links between dashboards (drill-down)
[ ] Annotations for deployments (mark when new versions are released)
[ ] Alert rules tested before going live

Common PromQL Patterns Cheat Sheet

| What | Query | |------|-------| | CPU usage % | 100 - avg(rate(node_cpu_seconds_total{mode="idle"}[5m])) * 100 | | Memory used % | (1 - node_memory_MemAvailable_bytes / node_memory_MemTotal_bytes) * 100 | | Disk used % | 1 - node_filesystem_avail_bytes / node_filesystem_size_bytes | | Network throughput | rate(node_network_receive_bytes_total[5m]) | | HTTP request rate | sum(rate(http_requests_total[5m])) by (handler) | | Error rate % | sum(rate(http_requests_total{status=~"5.."}[5m])) / sum(rate(http_requests_total[5m])) * 100 | | P95 latency | histogram_quantile(0.95, sum(rate(http_duration_bucket[5m])) by (le)) | | Pod restarts | increase(kube_pod_container_status_restarts_total[1h]) |

Conclusion

Grafana is powerful, but power without discipline creates noise. The best Grafana dashboards are:

Focused — one purpose per dashboard
Hierarchical — overview → detailed → deep-dive
Actionable — you can debug from the dashboard
Maintained — stale dashboards are deleted

Start with a simple overview dashboard (4 panels), add detail as needed, and always prefer clarity over completeness.