Search This Blog

Loading...

Thursday, July 07, 2016

Collecting and graphing metrics via Apache Camel, Statsd, and Graphite

If you're curious about Apache Camel's metrics reporting via Dropwizard, this guide will help you get Graphite running in a Docker container. Then you will run an instrumented Camel application and view its metrics via Graphite's browser UI.

(1) Install Docker

There are many ways to do this depending on your platform. Google it.

(2) Install and run the Docker container for Statsd and Graphite

Refer to https://github.com/hopsoft/docker-graphite-statsd

The instructions infer that you can run the container without specifying port mappings. This is plain wrong. So use the following command to start the container:

docker run -d --name graphite --restart=always -p 80:80 -p 8125:8125/udp hopsoft/graphite-statsd
It is not necessary to enable Graphite's UDP listener. Statsd receives metrics via UDP and forwards them to Graphite over TCP.
(3) On Windows and OSX, determine the IP address of the graphite Docker container

On Linux, you can use localhost.

On Windows and OSX, there are several ways to determine the IP address of the Docker container. The easiest way is to run Kitematic, click on the container named graphite, click on Settings, and then Ports. This document assumes its IP address is 192.168.99.100.

(4) Install Apache Camel

I used Camel version 2.17.2, but the latest version will likely change by the time you read this. Refer to http://camel.apache.org/download.html

(5) Modify Camel's spring-boot-metrics example

Because the Docker container uses Statsd for UDP message collection, we will need to alter Camel's metrics example to send messages to Statsd. We will also add an additional counter to demonstrate a custom metric collected by the route named fastRoute.

(5a) Go to the following directory under the Camel installation

examples/camel-example-spring-boot-metrics

(5b) Overwrite pom.xml

You may need change package versions to the latest (highlighted in yellow).

<?xml version="1.0" encoding="UTF-8"?>
<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/maven-v4_0_0.xsd">

  <modelVersion>4.0.0</modelVersion>

  <parent>
    <groupId>org.apache.camel</groupId>
    <artifactId>examples</artifactId>
    <version>2.17.2</version>
  </parent>

  <artifactId>camel-example-spring-boot-metrics</artifactId>
  <name>Camel :: Example :: Spring Boot Metrics</name>
  <description>An example showing how to work with Camel and Spring Boot and report metrics to statsd</description>

  <properties>
    <spring.boot-version>${spring-boot-version}</spring.boot-version>
  </properties>

  <dependencyManagement>
    <dependencies>
      <dependency>
        <groupId>org.springframework.boot</groupId>
        <artifactId>spring-boot-dependencies</artifactId>
        <version>${spring.boot-version}</version>
        <type>pom</type>
        <scope>import</scope>
      </dependency>
    </dependencies>
  </dependencyManagement>

  <dependencies>
    <dependency>
      <groupId>org.apache.camel</groupId>
      <artifactId>camel-spring-boot</artifactId>
    </dependency>

    <dependency>
      <groupId>org.apache.camel</groupId>
      <artifactId>camel-metrics</artifactId>
    </dependency>

    <!-- web -->
    <dependency>
      <groupId>org.springframework.boot</groupId>
      <artifactId>spring-boot-starter-web</artifactId>
    </dependency>

    <!-- operations -->
    <dependency>
      <groupId>org.jolokia</groupId>
      <artifactId>jolokia-core</artifactId>
    </dependency>

    <dependency>
      <groupId>org.springframework.boot</groupId>
      <artifactId>spring-boot-starter-actuator</artifactId>
    </dependency>

    <!-- metrics -->
    <dependency>
      <groupId>io.dropwizard.metrics</groupId>
      <artifactId>metrics-core</artifactId>
      <version>3.1.2</version>
    </dependency>

    <dependency>
      <groupId>com.basistech</groupId>
      <artifactId>metrics-statsd</artifactId>
      <version>3.0.0</version>
    </dependency>
  </dependencies>

  <build>
    <!-- we do not want version in the JAR name -->
    <finalName>${project.artifactId}</finalName>

    <plugins>
      <plugin>
        <groupId>org.springframework.boot</groupId>
        <artifactId>spring-boot-maven-plugin</artifactId>
        <version>${spring.boot-version}</version>
        <configuration>
          <mainClass>org.apache.camel.example.springboot.metrics.Application</mainClass>
        </configuration>
        <executions>
          <execution>
            <goals>
              <goal>repackage</goal>
            </goals>
          </execution>
        </executions>
      </plugin>
    </plugins>
  </build>
</project>

(5c) Overwrite src/main/java/org/apache/camel/springboot/metrics/Application.java

Specify the IP address of the Docker container (highlighted in yellow).

package org.apache.camel.example.springboot.metrics;

import java.net.InetSocketAddress;
import java.net.InetAddress;
import java.util.concurrent.TimeUnit;

import com.codahale.metrics.MetricFilter;
import com.codahale.metrics.MetricRegistry;
import org.apache.camel.CamelContext;
import org.apache.camel.builder.RouteBuilder;
import org.apache.camel.component.metrics.routepolicy.MetricsRoutePolicyFactory;
import org.apache.camel.spring.boot.CamelContextConfiguration;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.boot.SpringApplication;
import org.springframework.boot.autoconfigure.SpringBootApplication;
import org.springframework.context.annotation.Bean;
import com.basistech.metrics.reporting.Statsd;
import com.basistech.metrics.reporting.StatsdReporter;

/**
 * A simple Spring Boot application, with a couple of timed camel routes
 * configured with camel-metrics. Reports metrics to Statsd via
 * dropwizard-metrics Statsd sender. Has standard spring-actuator endpoints
 * such as /beans, /autoconfig, /metrics
 */
@SpringBootApplication
public class Application {

    private static final Logger LOG = LoggerFactory.getLogger(Application.class);

    @Autowired
    private MetricRegistry metricRegistry;

    /**
     * @param args no command line args required
     */
    public static void main(String[] args) {
        LOG.info(" *** Starting Camel Metrics Example Application ***");
        SpringApplication.run(Application.class, args);
    }

    /**
     * Create reporter bean and tell Spring to call stop() when shutting down.
     * 
     * @return StatsdReporter
     */
    @Bean(destroyMethod = "stop")
    public 
    StatsdReporter statsdReporter() throws Exception {
        final Statsd statsd = new Statsd("192.168.99.100", 8125);

        final StatsdReporter reporter =
            StatsdReporter
            .forRegistry(metricRegistry)
            .convertRatesTo(TimeUnit.SECONDS)
            .convertDurationsTo(TimeUnit.MILLISECONDS)
            .filter(MetricFilter.ALL)
            .build(statsd);

        reporter.start(5, TimeUnit.SECONDS);

        return reporter;
    }

    /**
     * @return timed route that logs output every 6 seconds
     */
    @Bean
    public RouteBuilder slowRoute() {
        return new RouteBuilder() {
            @Override
            public void configure() throws Exception {
                from("timer://foo?period=6000").routeId("slow-route").setBody().constant("Slow hello world!").log("${body}");
            }
        };
    }

    /**
     * @return timed route that logs output every 2 seconds
     */
    @Bean
    public RouteBuilder fastRoute() {
        return new RouteBuilder() {
            @Override
            public void configure() throws Exception {
                from("timer://foo?period=2000").routeId("fast-route").setBody().constant("Fast hello world!").log("${body}")
                 .to("metrics:counter:counters.camel-example-spring-boot-metrics.fast-route.executions");
            }
        };
    }

    @Bean
    CamelContextConfiguration contextConfiguration() {
        return new CamelContextConfiguration() {
            @Override
            public void beforeApplicationStart(CamelContext context) {
                LOG.info("Configuring Camel metrics on all routes");
                MetricsRoutePolicyFactory fac = new MetricsRoutePolicyFactory();
                fac.setMetricsRegistry(metricRegistry);
                context.addRoutePolicyFactory(fac);
            }

            @Override
            public void afterApplicationStart(CamelContext camelContext) {
                // noop
            }
        };
    }
}

(6) Install JDK8

If you need help with that, you probably shouldn't be reading this.

(7) Install Maven 3.3 or later

Refer to https://maven.apache.org/download.cgi

(8) Run the example

mvn spring-boot:run

(9) Finally, view the metrics

Go to http://192.168.99.100/dashboard. Use the IP address of the graphite Docker container.

View the metrics under stats.gauges, stats.timers, and stats.counters

Wednesday, June 29, 2016

Load testing with Java: Pound on a resource with multiple threads

Load testing involves multiple threads or processes performing operations against a resource. For example, execute a query against a database server via 50 concurrent threads. The tricky part is waiting for the threads to start up before the commands are actually executed, in order to make sure that the resource is truly getting hammered. Here's an example of how to accomplish this in Java. It is possible to refactor the following code into a utility accepts two parameters: the number of threads and a lambda expression which can be passed as a Runnable object.

final int numThreads = 50;
ExecutorService executor = Executors.newFixedThreadPool(numThreads);
CountDownLatch ready = new CountDownLatch(numThreads);
CountDownLatch start = new CountDownLatch(1);
CountDownLatch done = new CountDownLatch(numThreads);
for (int j = 0 ; j < numThreads ; j++) {
   executor.execute(() -> {
ready.countDown();
try {
   start.await();
// do something here ...
} catch (InterruptedException ex) {
   Thread.currentThread().interrupt();
} finally {
   done.countDown();
}
   });
}
// wait until everybody is ready
ready.await();

// start
start.countDown();
// wait until everybody is done
done.await();

// clean up the thread pool
executor.shutdownNow();
executor.awaitTermination();

Tuesday, April 26, 2016

Useful Process-Related Linux Commands

Amount of free RAM: 
free -t | grep "cache:" | awk '{ print $4}'

Top 5 processes consuming most CPU in their lifetimes:
ps aux | sort -rk 3,3 | head -n 5
The output is not the same as 'top'. The 'top' command outputs instantaneous CPU usage.

Monday, December 07, 2015

Running 2560x1080 on LG 29UM57-P on an old MacBook Pro OSX El Capitan


Yes. A mid-2011 MacBook Pro running OSX El Capitan can drive a 2560x1080 monitor via HDMI.

I was not able to get 60 Hz working. I am settling with 53 Hz. I tried many settings before finding something that works. The refresh rate seems fine to me. Maybe the text could be clearer but for a $200 monitor running this resolution, I'll take it. Apple MacBook 2015 retina resolution is 2560 x 1800. Consider buying a better monitor that may literally give you fewer headaches.
  1. Install SwitchResX
  2. Follow the instructions for disabling SIP
  3. After rebooting, run SwitchResX via System Preferences
  4. Select your LG monitor
  5. Click Custom Resolutions
  6. Click the + icon 
  7. Enter these settings (source)
  8. Exit SwitchResX and save
  9. Reboot
  10. Run SwitchResX via System Preferences
  11. Select your LG monitor
  12. Click Current Resolutions
  13. Select the 2560x1080, 53 Hz setting
If you discover better settings, please leave a comment.

Wednesday, August 19, 2015

Logitech Harmony turns your phone into an IoT magic wand for your house


The Logitech Harmony far exceeded my expectations. It's marketed as a "Universal Remote" and, as such, it's one of the best. However, the Harmony is much more, as it works with popular Internet-connected devices.

Product setup is easy via smartphone/tablet. The first step involves get connecting the Harmony hub to wifi. Since the device database is downloaded from the Internet, Harmony works with almost anything that has a remote: cable boxes, TVs, dvd players, stereos, etc.. You may configure up to 8 devices.

Harmony's remote doesn't need line of sight to communicate with its hub. This means you can place the hub in a solid wood cabinet and control it with the remote or your smartphone/tablet. The Harmony hub has a built-in internal IR blaster for old-school devices. Harmony ships with an additional IR blaster with a long wire so it can be placed, for example, outside your cabinet. Additional IR blasters can be purchased separately.

The provided remote is adequate and is just about the right size. Many universal remotes are too big for my taste. The smartphone/tablet app is more functional and can, for example, display "virtual" remotes that have all the buttons from the original remotes.

Harmony is scriptable. The remote provides easy access to six scripts via the top 3 buttons. 6 operations are provided through 3 buttons via short-taps versus long taps. Each script is fully customizable. Harmony writes the initial script for you, which you can modify whenever you wish. For example, the TV script can turn on your stereo, satellite box, TV, and change the TV input. Changing the TV input is something that my previous universal remote never even dreamed of.

The remote's buttons can also be programmed to perform any operation for any registered device. However, since the remote doesn't have any extra buttons, you usually must lose a function to gain a function. The default button layouts are functional and remapping buttons will likely be a rare activity. You can easily control your blu-ray player and your TV volume without changing modes.

I remapped a button to alter my TV's aspect. If I have one complaint about Harmony, it's that I must configure the same button remapping for every script. At least I only need to set this up once and forget about it.

Harmony appears to save your configuration to the cloud in case it loses power for a prolonged period of time. It also upgrades itself.

But there's more. Logitech Harmony turns your smartphone/tablet into a magic wand for your house. Although Harmony is marketed as a universal remote, it's actually an IoT hub that integrates with wifi and bluetooth devices. So not only can Harmony turn on your TV and Wii U (which uses bluetooth), it can also dim the lights and set your thermostat. Peruse the list of devices.

And you can do all this over the Internet. I didn't have to change my wifi router settings. It's magic!

Wednesday, May 20, 2015

Can I not quote YOU? When TERM is not the same as 'TERM'

As one might expect, there are subtle differences when Perl is running on Linux versus Windows (via cygwin) even when Perl is built from the same source.

On Windows (cygwin), the lines marked with ##1## and ##2## are identical:

use warnings;

my $pid = fork;

if ($pid) {
  sleep 10;
  kill TERM => $pid; ##1##
  kill 'TERM' => $pid; ##2##
  waitpid($pid, 0);
}
else {
    sleep 10000;
    exit;
}

On Linux, only the line marked with ##2## works. Even the POD for 'kill' puts quotes around the signal names. This could be due to the modules I'm using (POE and at least 100 more). Nevertheless, let this be today's all too familiar Perl lesson. I'm just telling you what you already know:

NEVER USE BAREWORDS, even if they are uppercase. Quote 'ALL' the damn strings.

Saturday, February 21, 2015

SL4J and parameterized messages

It's common to log messages that contain run-time information. In Java, this is easier if you use SL4J instead of log4j.


Here's a useful snippet from http://javarevisited.blogspot.com/2013/08/why-use-sl4j-over-log4j-for-logging-in.html

This is how you would do in Log4j, but surely this is not fun and reduce readability of code by adding unnecessary boiler-plate code.


if (logger.isDebugEnabled()) {
    logger.debug("Processing trade with id: " + id + " symbol: " + symbol);
}


On the other hand if you use SLF4J, you can get same result in more concise format as shown below:


logger.debug("Processing trade with id: {} and symbol : {} ", id, symbol);


In SLF4J, we don't need String concatenation and don't incur cost of temporary not need String. Instead, we write log message in a template format with placeholder and supply actual value as parameters.



Wednesday, January 07, 2015

Good coffee requires a good grinder

Coffee makes it possible for me to work. Work makes it possible for money to flow through me. Money makes it possible for the organizations I care about to be successful. 

But whether to grind at the store or at home? Do you want to drink your coffee or other customers' coffee? Do you want a consistent grind or do you want to learn the quirks of every grinder in every store you patronize? I buy from three Peet's stores and I ask for a different grind at each one. No more. I'm buying beans from Blue Tribe Coffee who treats Mexican coffee growers decently. And I'm grinding them at home with the best grinder that I can afford: the Baratza Vario 886.

Saturday, September 27, 2014

Toad - Eclipse plugin for Postgres, Oracle, mySQL

In Eclipse, install the Toad Plugin via http://community-downloads.quest.com/toadsoft/toadextensions/eclipse/freeware

To connect to Postgres databases hosted on Heroku, you must use SSL without certificates. To do this, select Custom Connection String and add the following to the JDBC URL:

?ssl=true&sslfactory=org.postgresql.ssl.NonValidatingFactory