Twelve days of metrics

Graphite for the masses

Created by Jeff Schroeder / @SEJeff / github: SEJeff / Presentation Video

http://envisionlinux.org/graphite-for-the-masses

Who Am I?

Husband of an awesome woman
Co-maintainer of graphite and salt stack
Lover of open source, bacon, and good beer
GNOME SysAdmin team and foundation member
Professional Linux superhero for a Chicago financial firm

What the heck is graphite?

Eyecandy for devops teams
A fantastic visualization tool for time-series metrics
A community of metrics and graph fanatics

What Graphite is NOT

A friggin' unicorn!

(no unicorns were harmed in the creation of this presentation)

Basic Architecture

graphite-web - web application for adhoc graphing and REST API
carbon - twisted daemon listening for time-series data and persisting it
whisper - python database library similar to RRD with a few perks

Graphite Web

A composer for adhoc graphing, a dashboard for ops, and a REST API for developers/external tools.

Composer

Used for creating and manipulating new graphs on the fly

Dashboard

A built-in dashboard creator to add graphs to a big monitoring screen

Dashboards are Useful!

Mailgun monitors their elasticsearch infrastructure with graphite.

JSON API

The REST API allows other external tools to use graphite data for creating the pretty

sumSeries()
movingAverage() - Smooths out graphs
nonNegativeDerivative() - Network counters that might reset to 0
cactiStyle() - Auto-adds min/max/avg
maximumAbove() - great with averages for errors > N

Carbon

The caching, relay, and storage layer. Carbon is what clients actually send metrics to.

Storage Schemas

Carbon uses storage schemas


[apache_busyWorkers]
pattern = ^servers\.www.*\.workers\.busyWorkers$
retentions = 5m:1d,15m:7d,1h:30d,6h:1y

RRA Example

Feeding Carbon

Minimal carbon client

Graphite docs on sending metrics


#!/bin/bash
# Basically the best carbon client in the world
GRAPHITE_SERVER=${GRAPHITE_SERVER:-localhost}
GRAPHITE_PORT=${GRAPHITE_PORT:-2003}

echo "foo.bar.baz $(($RANDOM % 500)).$(($RANDOM % 5)) $(date +%s)" | \
    nc "$GRAPHITE_SERVER" "$GRAPHITE_PORT"

metric.schema.name value_as_float unix_epoch_time

Aggregation

The carbon aggregator buffers metrics and performs down-sampling operations. Examples could be things like summing requests per hour and only firing the counter to the cache once per hour. Many people use one of the various forms of statsd to do this as well.

TL;DNR: Use statsd, it is awesome!

Carbon Relay

Regex-based rules or consistent-hashing.

Example relay-rules.conf


[db_servers]
pattern = ^collectd\..*db.*
servers = 10.1.2.3, 10.1.2.4:2004, myserver.example.com

Whisper

Whisper is a python library created essentially to be a better RRD.

Better than RRD?

The internets agrees!

RRD

In RRD speak, data retention policies are configured via a RRA using reverse polish notation.

Did Someone Say Polish?

Sorry, not the yummy hot dogs us Chicagoans love!

Example RRA

Taken from stack overflow, so it has to be right


RRA:AVERAGE:0.5:1:288  \ # 5 minute avg for 1 day
RRA:AVERAGE:0.5:3:672  \ # 15 minute avg for 1 week
RRA:AVERAGE:0.5:12:744 \ # 1 hour avg for 1 month
RRA:AVERAGE:0.5:72:1480  # 6 hour avg for 1 year

storage schema example

WAT?

Teh Pretty

Enough time blabbing, time for 12 great graphite tools

Graphene

Graphene is built for tv static dashboards

Giraffe

Giraffe is great viewing pre-generated realtime dashboards

cubism.js

Written by square, cubism, helps build compact and realtime dashboards.

graphite.js

If you love jQuery (I do!), graphite.js, makes it pretty trivial to pull graphs into your existing application

Full dashboard applications

For those who don't prefer a roll your own approach

Descartes

Descartes is a full featured graphing dashboard with builtin collaboration and native "composer" functionality.

Gdash

Gdash uses a simple DSL for generating graphs and is easy to get going

Tasseo

Tasseo is an alerting dashboard that can change graph colors based on thresholds.

Tattle

Tattle is a popular alerting dashboard

Orion

Orion full featured dashboard application with granular access control over metrics

Graph Explorer

Graph Explorer replacement for the graphite composer with tagging and other cleverness

Graphsky

Graphsky is more or less a ganglia replacement dashboard built ontop of graphite

Dusk

Dusk a realtime hotspot dashboard