Search This Blog

Thursday, November 30, 2017

Silver City New Mexico November 2017


I don't take enough photos to make compelling albums. I used to be into photography with film cameras but now I try to respect people's privacy. I think our society has overdone it since everyone has a camera now. Instead, I take mental pictures by talking to the locals in order to understand what I'm seeing in a historical dimension. It took me my whole life so far to figure out what I wasn't seeing.

The frontier is alive and well in New Mexico. Silver City is where large-scale cattle ranching was attempted start-up style in the 19th century with VC's. All of the operations went bankrupt due to drought. This was the location of the show Bonanza. The place looks exactly the same. Time has stood still.

If you want to buy a ranch, I know a place. It's amazing.

Silver City is probably the most egalitarian city in the US. Although small, the government obviously has the typical jobs such as judge and police chief. They are filled equally between Latinos and "other." This is the result of Silver City's Mexican colonial history but also, surprisingly, silver and copper mining which has allowed the community to survive without giving their livelihoods over to tourism.

Although we west-westerners associate mining with "bad," the folks in Silver City see it as their best means of not just survival but, even today, prosperity that doesn't rely on the whims of rich young people.

Thursday, March 23, 2017

Join my slack team

My Slack team has several technical channels varying from node.js to Java to C++. Pop in, learn, and ask questions.

Get an invite

Wednesday, February 22, 2017

Upgrading brew

brew complaining that it doesn't support a prerelease version of OSX? Here is the only fix. A mere "brew update" will not work.

cd "$(brew --repo)" && git fetch && git reset --hard origin/master && brew update

Wednesday, August 10, 2016

MySQL: This stored procedure inserts 'empty' rows into a specified table

This stored procedure accepts two arguments: a table name and a row count. It inserts the specified number of rows into the specified table. Of course, all columns in the table must either have default values or allow NULL.




DROP PROCEDURE InsertDimension;
DELIMITER $$
CREATE PROCEDURE InsertDimension(IN TableName VARCHAR(50), IN NumRows INT)
    BEGIN
        DECLARE i INT;
        SET i = 1;
        SET @sql_text = concat('INSERT INTO ', TableName, ' VALUES ()' );
        PREPARE stmt FROM @sql_text;
        START TRANSACTION;
        WHILE i <= NumRows DO
            EXECUTE stmt;
            SET i = i + 1;
        END WHILE;
        COMMIT;
        DEALLOCATE PREPARE stmt;
    END$$
DELIMITER ;

Thursday, July 07, 2016

Collecting and graphing metrics via Apache Camel, Statsd, and Graphite

If you're curious about Apache Camel's metrics reporting via Dropwizard, this guide will help you get Graphite running in a Docker container. Then you will run an instrumented Camel application and view its metrics via Graphite's browser UI.

(1) Install Docker

There are many ways to do this depending on your platform. Google it.

(2) Install and run the Docker container for Statsd and Graphite

Refer to https://github.com/hopsoft/docker-graphite-statsd

The instructions infer that you can run the container without specifying port mappings. This is plain wrong. So use the following command to start the container:

docker run -d --name graphite --restart=always -p 80:80 -p 8125:8125/udp hopsoft/graphite-statsd
It is not necessary to enable Graphite's UDP listener. Statsd receives metrics via UDP and forwards them to Graphite over TCP.
(3) On Windows and OSX, determine the IP address of the graphite Docker container

On Linux, you can use localhost.

On Windows and OSX, there are several ways to determine the IP address of the Docker container. The easiest way is to run Kitematic, click on the container named graphite, click on Settings, and then Ports. This document assumes its IP address is 192.168.99.100.

(4) Install Apache Camel

I used Camel version 2.17.2, but the latest version will likely change by the time you read this. Refer to http://camel.apache.org/download.html

(5) Modify Camel's spring-boot-metrics example

Because the Docker container uses Statsd for UDP message collection, we will need to alter Camel's metrics example to send messages to Statsd. We will also add an additional counter to demonstrate a custom metric collected by the route named fastRoute.

(5a) Go to the following directory under the Camel installation

examples/camel-example-spring-boot-metrics

(5b) Overwrite pom.xml

You may need change package versions to the latest (highlighted in yellow).

<?xml version="1.0" encoding="UTF-8"?>
<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/maven-v4_0_0.xsd">

  <modelVersion>4.0.0</modelVersion>

  <parent>
    <groupId>org.apache.camel</groupId>
    <artifactId>examples</artifactId>
    <version>2.17.2</version>
  </parent>

  <artifactId>camel-example-spring-boot-metrics</artifactId>
  <name>Camel :: Example :: Spring Boot Metrics</name>
  <description>An example showing how to work with Camel and Spring Boot and report metrics to statsd</description>

  <properties>
    <spring.boot-version>${spring-boot-version}</spring.boot-version>
  </properties>

  <dependencyManagement>
    <dependencies>
      <dependency>
        <groupId>org.springframework.boot</groupId>
        <artifactId>spring-boot-dependencies</artifactId>
        <version>${spring.boot-version}</version>
        <type>pom</type>
        <scope>import</scope>
      </dependency>
    </dependencies>
  </dependencyManagement>

  <dependencies>
    <dependency>
      <groupId>org.apache.camel</groupId>
      <artifactId>camel-spring-boot</artifactId>
    </dependency>

    <dependency>
      <groupId>org.apache.camel</groupId>
      <artifactId>camel-metrics</artifactId>
    </dependency>

    <!-- web -->
    <dependency>
      <groupId>org.springframework.boot</groupId>
      <artifactId>spring-boot-starter-web</artifactId>
    </dependency>

    <!-- operations -->
    <dependency>
      <groupId>org.jolokia</groupId>
      <artifactId>jolokia-core</artifactId>
    </dependency>

    <dependency>
      <groupId>org.springframework.boot</groupId>
      <artifactId>spring-boot-starter-actuator</artifactId>
    </dependency>

    <!-- metrics -->
    <dependency>
      <groupId>io.dropwizard.metrics</groupId>
      <artifactId>metrics-core</artifactId>
      <version>3.1.2</version>
    </dependency>

    <dependency>
      <groupId>com.basistech</groupId>
      <artifactId>metrics-statsd</artifactId>
      <version>3.0.0</version>
    </dependency>
  </dependencies>

  <build>
    <!-- we do not want version in the JAR name -->
    <finalName>${project.artifactId}</finalName>

    <plugins>
      <plugin>
        <groupId>org.springframework.boot</groupId>
        <artifactId>spring-boot-maven-plugin</artifactId>
        <version>${spring.boot-version}</version>
        <configuration>
          <mainClass>org.apache.camel.example.springboot.metrics.Application</mainClass>
        </configuration>
        <executions>
          <execution>
            <goals>
              <goal>repackage</goal>
            </goals>
          </execution>
        </executions>
      </plugin>
    </plugins>
  </build>
</project>

(5c) Overwrite src/main/java/org/apache/camel/springboot/metrics/Application.java

Specify the IP address of the Docker container (highlighted in yellow).

package org.apache.camel.example.springboot.metrics;

import java.net.InetSocketAddress;
import java.net.InetAddress;
import java.util.concurrent.TimeUnit;

import com.codahale.metrics.MetricFilter;
import com.codahale.metrics.MetricRegistry;
import org.apache.camel.CamelContext;
import org.apache.camel.builder.RouteBuilder;
import org.apache.camel.component.metrics.routepolicy.MetricsRoutePolicyFactory;
import org.apache.camel.spring.boot.CamelContextConfiguration;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.boot.SpringApplication;
import org.springframework.boot.autoconfigure.SpringBootApplication;
import org.springframework.context.annotation.Bean;
import com.basistech.metrics.reporting.Statsd;
import com.basistech.metrics.reporting.StatsdReporter;

/**
 * A simple Spring Boot application, with a couple of timed camel routes
 * configured with camel-metrics. Reports metrics to Statsd via
 * dropwizard-metrics Statsd sender. Has standard spring-actuator endpoints
 * such as /beans, /autoconfig, /metrics
 */
@SpringBootApplication
public class Application {

    private static final Logger LOG = LoggerFactory.getLogger(Application.class);

    @Autowired
    private MetricRegistry metricRegistry;

    /**
     * @param args no command line args required
     */
    public static void main(String[] args) {
        LOG.info(" *** Starting Camel Metrics Example Application ***");
        SpringApplication.run(Application.class, args);
    }

    /**
     * Create reporter bean and tell Spring to call stop() when shutting down.
     * 
     * @return StatsdReporter
     */
    @Bean(destroyMethod = "stop")
    public 
    StatsdReporter statsdReporter() throws Exception {
        final Statsd statsd = new Statsd("192.168.99.100", 8125);

        final StatsdReporter reporter =
            StatsdReporter
            .forRegistry(metricRegistry)
            .convertRatesTo(TimeUnit.SECONDS)
            .convertDurationsTo(TimeUnit.MILLISECONDS)
            .filter(MetricFilter.ALL)
            .build(statsd);

        reporter.start(5, TimeUnit.SECONDS);

        return reporter;
    }

    /**
     * @return timed route that logs output every 6 seconds
     */
    @Bean
    public RouteBuilder slowRoute() {
        return new RouteBuilder() {
            @Override
            public void configure() throws Exception {
                from("timer://foo?period=6000").routeId("slow-route").setBody().constant("Slow hello world!").log("${body}");
            }
        };
    }

    /**
     * @return timed route that logs output every 2 seconds
     */
    @Bean
    public RouteBuilder fastRoute() {
        return new RouteBuilder() {
            @Override
            public void configure() throws Exception {
                from("timer://foo?period=2000").routeId("fast-route").setBody().constant("Fast hello world!").log("${body}")
                 .to("metrics:counter:counters.camel-example-spring-boot-metrics.fast-route.executions");
            }
        };
    }

    @Bean
    CamelContextConfiguration contextConfiguration() {
        return new CamelContextConfiguration() {
            @Override
            public void beforeApplicationStart(CamelContext context) {
                LOG.info("Configuring Camel metrics on all routes");
                MetricsRoutePolicyFactory fac = new MetricsRoutePolicyFactory();
                fac.setMetricsRegistry(metricRegistry);
                context.addRoutePolicyFactory(fac);
            }

            @Override
            public void afterApplicationStart(CamelContext camelContext) {
                // noop
            }
        };
    }
}

(6) Install JDK8

If you need help with that, you probably shouldn't be reading this.

(7) Install Maven 3.3 or later

Refer to https://maven.apache.org/download.cgi

(8) Run the example

mvn spring-boot:run

(9) Finally, view the metrics

Go to http://192.168.99.100/dashboard. Use the IP address of the graphite Docker container.

View the metrics under stats.gauges, stats.timers, and stats.counters

Wednesday, June 29, 2016

Load testing with Java: Pound on a resource with multiple threads

Load testing involves multiple threads or processes performing operations against a resource. For example, execute a query against a database server via 50 concurrent threads. The tricky part is waiting for the threads to start up before the commands are actually executed, in order to make sure that the resource is truly getting hammered. Here's an example of how to accomplish this in Java. It is possible to refactor the following code into a utility accepts two parameters: the number of threads and a lambda expression which can be passed as a Runnable object.

final int numThreads = 50;
ExecutorService executor = Executors.newFixedThreadPool(numThreads);
CountDownLatch ready = new CountDownLatch(numThreads);
CountDownLatch start = new CountDownLatch(1);
CountDownLatch done = new CountDownLatch(numThreads);
for (int j = 0 ; j < numThreads ; j++) {
   executor.execute(() -> {
ready.countDown();
try {
   start.await();
// do something here ...
} catch (InterruptedException ex) {
   Thread.currentThread().interrupt();
} finally {
   done.countDown();
}
   });
}
// wait until everybody is ready
ready.await();

// start
start.countDown();
// wait until everybody is done
done.await();

// clean up the thread pool
executor.shutdownNow();
executor.awaitTermination();

Wednesday, May 18, 2016

Tuesday, April 26, 2016

Useful Process-Related Linux Commands

Amount of free RAM: 
free -t | grep "cache:" | awk '{ print $4}'

Top 5 processes consuming most CPU in their lifetimes:
ps aux | sort -rk 3,3 | head -n 5
The output is not the same as 'top'. The 'top' command outputs instantaneous CPU usage.

Monday, December 07, 2015

Running 2560x1080 on LG 29UM57-P on an old MacBook Pro OSX El Capitan


Yes. A mid-2011 MacBook Pro running OSX El Capitan can drive a 2560x1080 monitor via HDMI.

I was not able to get 60 Hz working. I am settling with 53 Hz. I tried many settings before finding something that works. The refresh rate seems fine to me. Maybe the text could be clearer but for a $200 monitor running this resolution, I'll take it. Apple MacBook 2015 retina resolution is 2560 x 1800. Consider buying a better monitor that may literally give you fewer headaches.
  1. Install SwitchResX
  2. Follow the instructions for disabling SIP
  3. After rebooting, run SwitchResX via System Preferences
  4. Select your LG monitor
  5. Click Custom Resolutions
  6. Click the + icon 
  7. Enter these settings (source)
  8. Exit SwitchResX and save
  9. Reboot
  10. Run SwitchResX via System Preferences
  11. Select your LG monitor
  12. Click Current Resolutions
  13. Select the 2560x1080, 53 Hz setting
  14. Enable SIP
If you discover better settings, please leave a comment.