Collecting and graphing metrics via Apache Camel, Statsd, and Graphite

If you're curious about Apache Camel's metrics reporting via Dropwizard, this guide will help you get Graphite running in a Docker container. Then you will run an instrumented Camel application and view its metrics via Graphite's browser UI.

(1) Install Docker

There are many ways to do this depending on your platform. Google it.

(2) Install and run the Docker container for Statsd and Graphite

Refer to https://github.com/hopsoft/docker-graphite-statsd

The instructions infer that you can run the container without specifying port mappings. This is plain wrong. So use the following command to start the container:

docker run -d --name graphite --restart=always -p 80:80 -p 8125:8125/udp hopsoft/graphite-statsd

It is not necessary to enable Graphite's UDP listener. Statsd receives metrics via UDP and forwards them to Graphite over TCP.

(3) On Windows and OSX, determine the IP address of the graphite Docker container

On Linux, you can use localhost.

On Windows and OSX, there are several ways to determine the IP address of the Docker container. The easiest way is to run Kitematic, click on the container named graphite, click on Settings, and then Ports. This document assumes its IP address is 192.168.99.100.

(4) Install Apache Camel

I used Camel version 2.17.2, but the latest version will likely change by the time you read this. Refer to http://camel.apache.org/download.html

(5) Modify Camel's spring-boot-metrics example

Because the Docker container uses Statsd for UDP message collection, we will need to alter Camel's metrics example to send messages to Statsd. We will also add an additional counter to demonstrate a custom metric collected by the route named fastRoute.

(5a) Go to the following directory under the Camel installation

examples/camel-example-spring-boot-metrics

(5b) Overwrite pom.xml

You may need change package versions to the latest (highlighted in yellow).

<?xml version="1.0" encoding="UTF-8"?>
<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/maven-v4_0_0.xsd">

  <modelVersion>4.0.0</modelVersion>

  <parent>
    <groupId>org.apache.camel</groupId>
    <artifactId>examples</artifactId>
    <version>2.17.2</version>
  </parent>

  <artifactId>camel-example-spring-boot-metrics</artifactId>
  <name>Camel :: Example :: Spring Boot Metrics</name>
  <description>An example showing how to work with Camel and Spring Boot and report metrics to statsd</description>

  <properties>
    <spring.boot-version>${spring-boot-version}</spring.boot-version>
  </properties>

  <dependencyManagement>
    <dependencies>
      <dependency>
        <groupId>org.springframework.boot</groupId>
        <artifactId>spring-boot-dependencies</artifactId>
        <version>${spring.boot-version}</version>
        <type>pom</type>
        <scope>import</scope>
      </dependency>
    </dependencies>
  </dependencyManagement>

  <dependencies>
    <dependency>
      <groupId>org.apache.camel</groupId>
      <artifactId>camel-spring-boot</artifactId>
    </dependency>

    <dependency>
      <groupId>org.apache.camel</groupId>
      <artifactId>camel-metrics</artifactId>
    </dependency>

    <!-- web -->
    <dependency>
      <groupId>org.springframework.boot</groupId>
      <artifactId>spring-boot-starter-web</artifactId>
    </dependency>

    <!-- operations -->
    <dependency>
      <groupId>org.jolokia</groupId>
      <artifactId>jolokia-core</artifactId>
    </dependency>

    <dependency>
      <groupId>org.springframework.boot</groupId>
      <artifactId>spring-boot-starter-actuator</artifactId>
    </dependency>

    <!-- metrics -->
    <dependency>
      <groupId>io.dropwizard.metrics</groupId>
      <artifactId>metrics-core</artifactId>
      <version>3.1.2</version>
    </dependency>

    <dependency>
      <groupId>com.basistech</groupId>
      <artifactId>metrics-statsd</artifactId>
      <version>3.0.0</version>
    </dependency>
  </dependencies>

  <build>
    <!-- we do not want version in the JAR name -->
    <finalName>${project.artifactId}</finalName>

    <plugins>
      <plugin>
        <groupId>org.springframework.boot</groupId>
        <artifactId>spring-boot-maven-plugin</artifactId>
        <version>${spring.boot-version}</version>
        <configuration>
          <mainClass>org.apache.camel.example.springboot.metrics.Application</mainClass>
        </configuration>
        <executions>
          <execution>
            <goals>
              <goal>repackage</goal>
            </goals>
          </execution>
        </executions>
      </plugin>
    </plugins>
  </build>
</project>

(5c) Overwrite src/main/java/org/apache/camel/springboot/metrics/Application.java

Specify the IP address of the Docker container (highlighted in yellow).

package org.apache.camel.example.springboot.metrics;

import java.net.InetSocketAddress;
import java.net.InetAddress;
import java.util.concurrent.TimeUnit;

import com.codahale.metrics.MetricFilter;
import com.codahale.metrics.MetricRegistry;
import org.apache.camel.CamelContext;
import org.apache.camel.builder.RouteBuilder;
import org.apache.camel.component.metrics.routepolicy.MetricsRoutePolicyFactory;
import org.apache.camel.spring.boot.CamelContextConfiguration;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.boot.SpringApplication;
import org.springframework.boot.autoconfigure.SpringBootApplication;
import org.springframework.context.annotation.Bean;
import com.basistech.metrics.reporting.Statsd;
import com.basistech.metrics.reporting.StatsdReporter;

/**
 * A simple Spring Boot application, with a couple of timed camel routes
 * configured with camel-metrics. Reports metrics to Statsd via
 * dropwizard-metrics Statsd sender. Has standard spring-actuator endpoints
 * such as /beans, /autoconfig, /metrics
 */
@SpringBootApplication
public class Application {

    private static final Logger LOG = LoggerFactory.getLogger(Application.class);

    @Autowired
    private MetricRegistry metricRegistry;

    /**
     * @param args no command line args required
     */
    public static void main(String[] args) {
        LOG.info(" *** Starting Camel Metrics Example Application ***");
        SpringApplication.run(Application.class, args);
    }

    /**
     * Create reporter bean and tell Spring to call stop() when shutting down.
     * 
     * @return StatsdReporter
     */
    @Bean(destroyMethod = "stop")
    public 
    StatsdReporter statsdReporter() throws Exception {
        final Statsd statsd = new Statsd("192.168.99.100", 8125);

        final StatsdReporter reporter =
            StatsdReporter
            .forRegistry(metricRegistry)
            .convertRatesTo(TimeUnit.SECONDS)
            .convertDurationsTo(TimeUnit.MILLISECONDS)
            .filter(MetricFilter.ALL)
            .build(statsd);

        reporter.start(5, TimeUnit.SECONDS);

        return reporter;
    }

    /**
     * @return timed route that logs output every 6 seconds
     */
    @Bean
    public RouteBuilder slowRoute() {
        return new RouteBuilder() {
            @Override
            public void configure() throws Exception {
                from("timer://foo?period=6000").routeId("slow-route").setBody().constant("Slow hello world!").log("${body}");
            }
        };
    }

    /**
     * @return timed route that logs output every 2 seconds
     */
    @Bean
    public RouteBuilder fastRoute() {
        return new RouteBuilder() {
            @Override
            public void configure() throws Exception {
                from("timer://foo?period=2000").routeId("fast-route").setBody().constant("Fast hello world!").log("${body}")
                 .to("metrics:counter:counters.camel-example-spring-boot-metrics.fast-route.executions");
            }
        };
    }

    @Bean
    CamelContextConfiguration contextConfiguration() {
        return new CamelContextConfiguration() {
            @Override
            public void beforeApplicationStart(CamelContext context) {
                LOG.info("Configuring Camel metrics on all routes");
                MetricsRoutePolicyFactory fac = new MetricsRoutePolicyFactory();
                fac.setMetricsRegistry(metricRegistry);
                context.addRoutePolicyFactory(fac);
            }

            @Override
            public void afterApplicationStart(CamelContext camelContext) {
                // noop
            }
        };
    }
}

(6) Install JDK8

If you need help with that, you probably shouldn't be reading this.

(7) Install Maven 3.3 or later

Refer to https://maven.apache.org/download.cgi

(8) Run the example

mvn spring-boot:run

(9) Finally, view the metrics

Go to http://192.168.99.100/dashboard. Use the IP address of the graphite Docker container.

View the metrics under stats.gauges, stats.timers, and stats.counters