The Boy Wonders

Accessible communication with Twilio's Programmable Messaging API, JBang, and PicoCLI

Mon, 26 Jul 2021 00:00:00 +0000

In the mid-to-late 1990s when I was in secondary school in Ireland, I chose to participate in the optional one-year Transition Year (TY) school program. TY lets students combine a regular academic school year with opportunities to participate in independent activites, including volunteer engagements.

I worked one month with the Rehab Group, a charity that provides people with a disability or disadvantage educational services and professional training. As I had developed a decent familiarity with personal computers by then, my responsibility was to train basic computer skills.

Those individuals’ disadvantages included prosthetic limbs, speech impediments, and learning difficulties, among others. Many had never used a computer before. For those that struggled to type, I introduced voice recognition software (Dragon Dictate), so they could speak into a microphone to “write” emails to relatives. I showed them how to use Microsoft Word and find information using a web browser.

There was one incident however that stuck with me all these years later. An elderly gentleman entered the training room and sat down at the computer, visibly nervous.

The first thing I did with every person was to ask them turn on the desktop computer via a button on the front of the box. Most pressed the button without issue, but he was extremely hesitant to touch the device.

I demonstrated the various components - the monitor, the keyboard, the mouse. He expressed concern that if he did the wrong thing, would the computer “blow up”? I reassured him that we were safe and that the computer would not physically harm him.

Once Windows had loaded and the desktop was displayed, it was time for the first lesson - opening an application.

“Move the mouse to the Start Menu over here please”, I said.

He glanced at me, nodded, and looked at the mouse. He then picked it up, raising it into the air and held it to the bottom-left corner of the monitor’s screen.

I do not tell this story to mock him. What I realized that day is the interfaces we use with computers should not be assumed to be natural. That the instructional language we use is often abstract and assumes a level of technical familiarity above what people may be comfortable with, or even capable of.

Ever since then, I’ve always been drawn to designs and solutions that leveraged technology in a manner that people like that gentleman at Rehab could avail of.

Reporting disease cases in remote areas of South-East Asia using paper wheels and SMS

In that vein, I loved recently discovering InSTEDD’s Nicolás di Tada’s blog post from 2010, “IT without Software” describing how his team needed to build a system for workers at remote health centers in Thailand and Cambodia to report disease cases data in a semi-structured way.

Most case reports were being communicated by phone calls to the district offices, which aggregated the data by province, losing the fidelity of the original health center’s report.

The team wished to use SMS as the primary communication medium, but there were several challenges identified that needed to be addressed in determining a reporting syntax, including:

Most people do not know how to send SMS.

Some of them do not know how to read an incoming SMS.

Support for Khmer and Thai characters is not common in the handsets and carriers most people use.

Even if there is support for the characters, writing SMS using them is much more difficult than writing in English due to the amount of letters in the alphabet.

The InSTEDD team devised an ingenious solution to this - physical reporting wheels made of basic materials like paper or card-stock:

The wheels were cheap and easy to build but sturdy, with no batteries required and intuitive within minutes. You may have seen children using similar devices, called decoder wheels, to create “secret codes” to exchange with each other.

The reporting wheel the team created enabled health center workers align 3 independently-rotating wheels, each with a 3-digit code for choosing one of an enumerated set values for the respective data point (in this case, the day of the month, the disease, and the number of cases being reported).

Once each value has been chosen and aligned with a indicator, the health worker would have a nine-digit code that codifies the data values and could send that code via SMS to a cell phone number. The service will then reverse engineer that code back into the original data values.

Codification of the nine-digit message

Most interestingly of all, the system also needed to address some major usability aspects:

How could typos or data entry mistakes be identified to prevent misreporting disease case data?
How could this solution scale to different kinds of reports without having to ask the user to identify the type of wheel being used?

While not outlined explicity in the post, the images above provide the answer. For each wheel, the first code value is a prime number and each subsequent value is a multiple of it. The other wheels also started with a prime number - indeed, sequential primes are being used - 23, 29, 31.

This seemed like a fun idea to experiment with to try out Twilio’s SDK and some other libraries that had recently caught my attention.

Scriptable Java command-line tools with JBang and PicoCLI

JBang, by Max Andersen, is one of my favorite open-source projects of recent years. It’s premise is simple - make scripting with Java as fast and as easy as other languages like python or kotlin.

At SnapLogic, it’s been brilliant for my team to use it for rapid exploration of various SDKs and APIs, and to reproduce specific scenarios quickly.

In fact, I believe I first heard about JBang via Twilio’s Developer Evangelist, Matthew Gillard’s Twitch channel.

Simulating the Reporting Wheel

The first action was to build a script to simulate the reporting wheel demonstrated above. I wanted a command-line interface (CLI) approach where the user could provide the data points as arguments or be prompted for each one, validating the input as it went.

JBang’s landing page demo used PicoCLI so that was good enough for me and I was able to quickly define the options the user would provide:

//usr/bin/env jbang "$0" "$@" ; exit $?
//JAVA 11+
//DEPS info.picocli:picocli:4.2.0
//SOURCES Disease.java

import java.util.Locale;
import java.util.Random;
import java.util.concurrent.Callable;

import picocli.CommandLine;
import picocli.CommandLine.Command;

@Command(name = "ReportingWheel", mixinStandardHelpOptions = true, version = "SendSms 0.1",
        description = "An interactive CLI to simulate using a reporting wheel to generate 9-digit" +
                " codes to send via SMS")
class ReportingWheel implements Callable<Integer> {

    @CommandLine.Option(
            names = {"-d", "--day"},
            description = "The @|bold numeric day|@ of the month")
    private Integer dayOfMonth;

    @CommandLine.Option(
            names = {"-di", "--disease"},
            description = "The @|bold disease code|@ you are reporting")
    private String diseaseCode;

    @CommandLine.Option(
            names = {"-c", "--cases"},
            description = "The @|bold number of cases|@ to report for that day")
    private Integer numCases;

    public static void main(String... args) {
        int exitCode = new CommandLine(new ReportingWheel()).execute(args);
        System.exit(exitCode);
    }
    
    ...

JBang will download Java 11 if the user does not have it installed, it will download the PicoCLI dependencies need to compile the code and package it into an executable JAR file.

Codification of the message via Prime numbers

I wanted to simulate using different reporting wheel types targeting the same reporting service number, so I randomized which prime would be used as the intial seed value for the first “wheel” value. The next 2 prime numbers would then be used as the prime seeds for the second and third wheel values respectively.

@Override
    public Integer call() {
        // primes < 32 since there are 1000 possible values for a 3-digit code, but a max "day"
        // value of 31 so 1000/31 = 32.25...
        // There are only 7 diseases and 20 valid "cases" numbers permitted, so we go with 31 as max
        int[] primes = new int[]{2, 3, 5, 7, 11, 13, 17, 19, 23, 29, 31};

        // we'll be randomly selecting 3 sequential primes, so the max 1st index is 3 from the end
        // this is randomized to simulate different wheels being used for different purposes
        // the point is the user doesn't need to do anything other than text the 9-digit number
        // (no secret key etc. needed)
        int random = new Random().nextInt(primes.length - 2);
        int daySeed = primes[random];
        int diseasesSeed = primes[random + 1];
        int casesSeed = primes[random + 2];

        // check if any command-line inputs are valid and if not, ask for them until they are
        // then simulate each data point being selected on a physical reporting wheel and
        // codified to a 3-digit, zero-padded number
        dayOfMonth = validateDayInputAskAgainIfNeeded(dayOfMonth);
        String codifiedDay = String.format(Locale.ROOT, "%03d", (dayOfMonth * daySeed)); // e.g. 003
        printlnAnsi("@|green day=" + dayOfMonth + ", code=" + codifiedDay + "|@" + System.lineSeparator());
        
        ...

This is how I would ask the user for input, re-asking until valid input values had been provided:

    private Integer validateDayInputAskAgainIfNeeded(Integer day) {
        while (day == null || (day < 1 || day > 31)) {
            printlnAnsi("@|red Missing/invalid day provided (1-31 required)|@");
            try {
                day = askForDayOfMonth();
            } catch (Exception e) {
                day = null;
            }
        }
        return day;
    }

    private Integer askForDayOfMonth() {
        String s = System.console().readLine("What day of the month is this report for?: ");
        return Integer.valueOf(s);
    }

Using PicoCLI like this allowed both a direct and interactive choice of user input. For example, consider the direct invocation:

> jbang ReportingWheel.java -d 26 -di m -c 9
day=26, code=182

disease=MALARIA, code=033

cases=9, code=117

Please text this code 182033117 to +14158493243

compared to the interactive approach:

> jbang ReportingWheel.java 
Missing/invalid day provided (1-31 required)
What day of the month is this report for?: 99

Missing/invalid day provided (1-31 required)
What day of the month is this report for?: -4

Missing/invalid day provided (1-31 required)
What day of the month is this report for?: 26
day=26, code=130

Missing/invalid disease code provided
Which disease are you reporting? (use the single-letter code only): 
c: CHOLERA
d: DENGUE
m: MALARIA
j: JP_ENCEPH
t: TYPHOID
h: HEPATITIS
v: COVID19
m
disease=MALARIA, code=021

Missing/invalid cases metric provided (1-20 required)
How many cases are you reporting?: 9
cases=9, code=099

Please text this code 130021099 to +14158493243
THANK YOU FOR YOUR REPORT!

You may notice that the nine-digit codes generated are different (“182033117” vs “130021099”) for the same input values, but that is due to the randomized prime seed selection simulating different wheel types being used (even though the data points being used are the same).

Decoding the message and replying via Twilio-enabled SMS

To decode the message being sent, I needed a few things first:

A phone number to send SMS messages to
A programmable mechanism to receive the message that was sent
A way to reply to the received SMS with the decoded data values

My team had built the Twilio Snap Pack and had reported on the excellent quality of the APIs and SDKs provided.

I purchased an SMS-capable phone number from Twilio for $1 in about ten seconds.

Next, I read up about webhooks that could be used to receive callback requests when messages had been received by the newly-purchased Twilio number. I didn’t really want to set up a server on the public internet for this, but Twilio’s CLI came to the rescue with an excellent developer-friendly feature.

The tool had integrated with ngrok to create a “tunnel” between ngrok and my laptop. Any requests to the assigned ngrok.io endpoint would be “forwarded” to the web service running on my local machine. This allows real local debugging in my IDE rather than relying on webhook-capture sites like RequestBin.com (Pipedream).

> twilio phone-numbers:update "+14158493243" --sms-url="http://localhost:4567/sms"
SID                                 Result   SMS URL                          
PN14dcf0f9df39d2ae7f207084519db4da  Success  https://dba7b30ce2c2.ngrok.io/sms
ngrok is running. Open http://127.0.0.1:4040 to view tunnel activity.
Press CTRL-C to exit.

This updated my Messaging Service’s configuration automatically:

Finally, it was time to build the web service that would receive the webhook request from Twilio containing the codified message. I tweaked an existing Twilio example that used the Spark Java Framework to create an /sms API endpoint that received the POST request:

/**
 * This Spark web server provides an /sms endpoint to receive Twilio webhook callbacks and
 * decode the codified message, replying to the SMS with the decoded data in a text
*/
public class ReportingServer {

    static int[] primes = new int[]{2, 3, 5, 7, 11, 13, 17, 19, 23, 29, 31};

    public static void main(String args[]) {
        get("/", (req, res) -> "Health Check");

        post("/sms", (req, res) -> {
            String responseText = null;
            try {
                String[] webhookParts = req.body().split("&");
                for (String webhookPart : webhookParts) {
                    if (webhookPart.startsWith("Body=")) {
                        String body = webhookPart.split("=")[1];
                        DecodedMessage msg = decodeMessage(body);
                        if (msg != null) {
                            responseText = msg.toString();
                        } else {
                            responseText = "Oops! Your message does not appear to be valid.";
                        }
                        res.type("application/xml");
                        break;
                    }
                }
            } catch (Exception e) {
                // do nothing for now
            }

            if (responseText == null) {
                responseText = "An error was encountered";
            }

            Body body = new Body.Builder(responseText).build();
            Message sms = new Message.Builder().body(body).build();
            MessagingResponse twiml = new MessagingResponse.Builder().message(sms).build();
            return twiml.toXml();
        });
    }
    
    ...

Run the server via JBang like so - it will start listening for requests from the Twilio webhook:

> jbang ReportingServer.java 
[jbang] Building jar...

My approach to reverse engineer the codified message is a little difficult to explain consisely but it follows this logic:

Build a cache of all the prime number seeds by each of the possible day-of-month 3-digit codes

I didn’t want to blindly build an n * n * n hashtable for all possible combinations of 9-digit codes - I figured there probably was a more efficient approach.

For example, for prime number seed 17, the codified values for 1, 2, and 3 would be 017, 034, 051 and so on. My first cache map would then have entries with keys 017, 034, 051, with each mapping to a value of 17.

Therefore, taking the first three digits of the codified message, I could figure out which prime number seed or seeds was potentially used.

I would need to handle collisions - consider the code 006. It could mean the 3rd day of the month when the prime seed was 2, or it could also mean it was the 2nd day of the month when the prime seed was 3.

Knowing which prime seed was used could be determined by examining the remaining digits of the codified message (identifying the diseases and number-of-cases).

Build a cache of all the possible disease and number-of-cases 3-digit codes for each prime number

In the 006 example, we know that the prime number seed used for the day-of-month value was either 2 or 3. And since the code generation logic uses sequential prime numbers for the 3-digit codes for disease and number-of-cases, then the middle 3-digit code must be one of the possible values when the prime number seed is either 3 (the prime after 2) or 5 (the prime after 3).

Similarly, last 3-digits for the number-of-cases value would use prime number seeds 5 or 7.

The other two cache maps would then have keys 3, 5, 7 etc. whose values were lists of codified number multiples e.g.

=> [003, 006, 009, ...]
=> [005, 010, 015, ...]
=> [007, 014, 021, ...]
...
=> [020, 040, 060, ...] (for number-of-cases only)

/**
 * Messages are encoded by first selecting a random prime number < 32 for the day-of-month seed,
 * and then using the next two primes for the disease and number-of-cases seeds. That means that
 * if you can identify the prime number used for the day-of-month 3-digit code, you know what
 * are the possible valid codes for the other values provided too, and the entire message can be
 * both decoded and validated (since a typo in a code provided by a user will result in a value
 * being provided that doesn't adhere to the sequential-prime-seed rule)
 */
static DecodedMessage decodeMessage(String codifiedMessage) {
    // these data structures will be used for efficient lookups to reverse engineer
    // the codified message to the original day-of-month, disease, and number-of-cases data
    // values that the reporting user originally chose.
    // It does this by attempting to figure which prime number was used as the day-of-month seed
    // and then validates the user's message by confirming that the respective 3-digit codes
    // for the disease and number-of-cases are legal values

    // key = 3 digit code for each day of month (1-31), value = map(key=index in primes
    // array, value=multiplier)
    Map<String, List<Integer>> primeIndexesByDayCode = new HashMap<>();
    // key = index in primes array, value = list(valid disease codes for equivalent prime)
    Map<Integer, List<String>> diseaseCodesByPrimeIndex = new HashMap<>();
    // key = index in primes array, value = list(valid cases codes for equivalent prime)
    Map<Integer, List<String>> casesCodesByPrimeIndex = new HashMap<>();

    // for each permissible prime number, populate the above data structures
    for (int primeIndex = 0; primeIndex < primes.length; primeIndex++) {
        // the day codes will only use the 1st to the third-last prime indexes for 9-digit codes
        if (primeIndex < (primes.length - 2)) {
            // for each day of the month, build the lookup cache for primes by legal 3-digit
            // day-of-month codes
            primeIndexesByDayCode = buildPrimeIndexLookupsByDayCode(primeIndexesByDayCode,
                    primeIndex);
        }

        // the disease codes will only use the 2nd to the second-last prime indexes for 9-digits
        if (primeIndex > 0 && (primeIndex < (primes.length - 1))) {
            // for each prime, build a cache of legal 3-digit disease codes
            diseaseCodesByPrimeIndex =
                    buildDiseaseCodeLookupByPrimeIndex(diseaseCodesByPrimeIndex,
                            primeIndex);
        }

        // the cases codes will only use the 3rd to the last prime indexes for 9-digit codes
        if (primeIndex > 1) {
            // for each prime, build a cache of legal 3-digit number-of-cases codes
            casesCodesByPrimeIndex = buildPrimeLookupByCasesCode(casesCodesByPrimeIndex,
                    primeIndex);
        }
    }
    
    ...

The cache-building logic looks like this:

/**
 * For each day of the month, generate a 3-digit-code and note which prime seeds would result in
 * that code being a legal value
 */
static Map<String, List<Integer>> buildPrimeIndexLookupsByDayCode(
        Map<String, List<Integer>> dayCodes, int primeIndex) {
    for (int dayIndex = 1; dayIndex < 32; dayIndex++) {
        String dayCode = String.format(Locale.ROOT, "%03d", (dayIndex * primes[primeIndex]));
        if (dayCodes.containsKey(dayCode)) {
            dayCodes.get(dayCode).add(primeIndex);
        } else {
            List<Integer> primeIndexesForDayCode = new ArrayList<>();
            primeIndexesForDayCode.add(primeIndex);
            dayCodes.put(dayCode, primeIndexesForDayCode);
        }
    }
    return dayCodes;
}

// as above
static Map<Integer, List<String>> buildDiseaseCodeLookupByPrimeIndex(
        Map<Integer, List<String>> diseaseCodesByPrime, int primeIndex) {
    for (Disease dis : Disease.values()) {
        String diseaseCode = String.format(Locale.ROOT, "%03d",
                ((dis.ordinal() + 1) * primes[primeIndex]));

        if (diseaseCodesByPrime.containsKey(primeIndex)) {
            diseaseCodesByPrime.get(primeIndex).add(diseaseCode);
        } else {
            List<String> diseaseCodes = new ArrayList<>();
            diseaseCodes.add(diseaseCode);
            diseaseCodesByPrime.put(primeIndex, diseaseCodes);
        }
    }
    return diseaseCodesByPrime;
}

// and as above also
static Map<Integer, List<String>> buildPrimeLookupByCasesCode(
        Map<Integer, List<String>> casesCodesByPrime, int primeIndex) {
    for (int casesIndex = 1; casesIndex < 21; casesIndex++) {
        String casesCode = String.format(Locale.ROOT, "%03d",
                (casesIndex * primes[primeIndex]));
        if (casesCodesByPrime.containsKey(primeIndex)) {
            casesCodesByPrime.get(primeIndex).add(casesCode);
        } else {
            List<String> casesCodes = new ArrayList<>();
            casesCodes.add(casesCode);
            casesCodesByPrime.put(primeIndex, casesCodes);
        }
    }
    return casesCodesByPrime;
}

Identify the prime seeds used in the original message

Through a process of elimiation, only one combination will be valid. Once that combination is found, we can fully decode the message via the indexes of the matched codes. If no combinations applied, then the code received is invalid (i.e. likely a typo occurred):

// the first 3 digits correspond the codified day of the month
String impliedDayCode = codifiedMessage.substring(0, 3);

// check if the codified message is valid by checking if each of the day-of-month,
// disease, and number-of-cases 3-digit codes are legal; if they all are, then the message
// is valid and we can reverse engineer the user's original selections
if (primeIndexesByDayCode.containsKey(impliedDayCode)) {
    // the primes that are legal for this day-of-month code
    List<Integer> primeIndexesForDayCode = primeIndexesByDayCode.get(impliedDayCode);
    for (Integer primeIndex : primeIndexesForDayCode) {
        // now check if the disease code is legal
        if (diseaseCodesByPrimeIndex.containsKey(primeIndex + 1)) {
            // get the legal disease codes for the next sequential prime
            List<String> diseaseCodeIndexesByCode =
                    diseaseCodesByPrimeIndex.get(primeIndex + 1);
            String impliedDiseaseCode = codifiedMessage.substring(3, 6);
            if (diseaseCodeIndexesByCode.contains(impliedDiseaseCode)) {
                // finally check if the number-of-cases code is legal
                if (casesCodesByPrimeIndex.containsKey(primeIndex + 2)) {
                    List<String> casesCodeIndexesByCode =
                            casesCodesByPrimeIndex.get(primeIndex + 2);
                    String impliedCasesCode = codifiedMessage.substring(6);
                    if (casesCodeIndexesByCode.contains(impliedCasesCode)) {
                        // if we have reached this point, then all 3-digit codes are
                        // legal and therefore the message is legal and valid. We also know
                        // now what prime number seeds were used by the original user's
                        // reporting wheel and therefore can figure out what the original
                        // non-encoded values were for day-of-month, disease, and
                        // number-of-cases
                        int dayOfMonth =
                                Integer.valueOf(impliedDayCode) / primes[primeIndex];
                        Disease disease =
                                Disease.values()[diseaseCodeIndexesByCode.indexOf(impliedDiseaseCode)];
                        int numberOfCases =
                                casesCodeIndexesByCode.indexOf(impliedCasesCode) + 1;
                        return new DecodedMessage(dayOfMonth, disease, numberOfCases);
                    }
                }
            }
        }
    }
}
return null;

Reply via Twilio SMS

Finally, we have what we need to reply to the sender, confirming the original data values that the health worker chose on the reporting wheel and thanking them for their report:

Body body = new Body.Builder(responseText).build();
Message sms = new Message.Builder().body(body).build();
MessagingResponse twiml = new MessagingResponse.Builder().message(sms).build();
return twiml.toXml();

If the codified message received was invalid, we would let them know also so they can check their wheel and resend again:

The code used in this project is available here: https://github.com/robinhowlett/accessible-sms

Epilogue

There wasn’t anything particularly brilliant on my part about this application. Yes, the prime number seed approach can require some focus to follow, but the idea came from the InSTEDD team and I just applied a simplified version of it.

Twilio also did all the heavy lifting with running virtual phone numbers, on-message webhooks, ngrok integration and a well-designed SDK.

I’m sure there is plenty of other material out there showing much more sophisticated demonstrations of communication platforms with richer user interfaces and experiences.

But what is worth remembering is that when communications techology is used in a way that accomodates everybody, even those sometimes forgotten because of resources, language, education, opportunity or accessibility, it can still provide life-changing benefits to real people, even in the farthest reaches of the globe.

Parsing structured data within PDF documents with Apache PDFBox

Fri, 29 Nov 2019 00:00:00 +0000

PDF continues to be a popular document publishing format because users see them as the digital equivalent of paper documents. Unlike websites, often what you see on the PDF will be exactly how it will be printed on a physical page, with the added benefits of easily distributable files and near-ubiquitous support of software able to read this format on almost any standard digital device.

However, when information, especially structured data, is contained within a PDF document and one wishes to extract that content, the format becomes quite difficult for developers to interact with.

In this post, I outline a real-world example of parsing a large PDF file that contains repeated tables of data. I show how the raw text can be extracted and then detail much more low-level control over the text characters positioned within the pages. I also touch on the actual mechanics of working through a problem like this - using tools like Excel to explore and analyze both the nature of the PDF, as well as the vagaries of the data itself.

Breeders’ Cup Betting Challenge

The Breeders’ Cup Betting Challenge (BCBC) is an annual $10,000 buy-in, live-money horse racing handicapping tournament tied to the two-day, 14-race $30 million Breeders’ Cup World Championships event. In 2018, 391 entries competed for the $1 million prize pool. It would also be the first time that the Breeders’ Cup had taken the decision to publish all the players’ tournament wagers placed at the conclusion of the event.

A few days after the competition ended, a 900+ page PDF file was posted to the Breeders’ Cup website containing a breakdown of all of the wagers placed by each player.

Intrigued by this rare example of transparency into how professional and advanced horse racing tournament players approached this format, I decided to see if I could extricate the data within to conduct some analysis for educational and entertainment purposes.

Apache PDFBox

The Apache PDFBox library is an open-source Java tool for interacting with PDF documents.

It allows the “creation of new PDF documents, manipulation of existing documents and the ability to extract content from documents”.

I’ve found that even for PDFs that turn off the ability to copy text from the document, PDFBox can still extract the content.

(1 of 3) Basic: outputting the raw text line-by-line

When attempting to parse a PDF generally you first want to just output the raw text to examine if there are any obvious patterns that can be used.

A File can be read by PDFBox as a PDF document by using PDDocument.load().

Once the file is a PDDocument, PDFTextStripper’s writeText() method can be used to strip just the text (without any of the formatting and such) and write it to a file:

class BCBCParser extends PDFTextStripper {

    @Override
    public void parse(File source) {
        try (PDDocument document = PDDocument.load(source)) {
            // The order of the text tokens in a PDF file may not be in the same as they appear 
            // visually on the screen, so tell PDFBox to sort by text position 
            setSortByPosition(true);

            try (BufferedWriter writer = Files.newBufferedWriter(Paths.get("bcbc.txt"), UTF_8)) {
            	// This will take a PDDocument and write the text of that document to the writer.
                writeText(document, writer);
            }
        } catch (InvalidPasswordException e) {
            e.printStackTrace();
        } catch (IOException e) {
            e.printStackTrace();
        }
        return null;
    }
}

This resulted in output like the following in bcbc.txt:

Grubbs Charles 10603
Churchill Downs
Charles Grubbs
Date: 20181102
Race: 1
Pool Bets Refunds Winnings Runners
EXA $250.00 $0.00 $0.00 W1-6/6
WIN $250.00 $0.00 $0.00 6
WIN $100.00 $0.00 $0.00 6
$600.00 $0.00 $0.00 
Race: 3
Pool Bets Refunds Winnings Runners
WIN $400.00 $0.00 $0.00 B3,10
EXA $200.00 $0.00 $0.00 W1,8/3,10
EXA $250.00 $0.00 $0.00 W1,4-6,8/3,10
WIN $50.00 $0.00 $0.00 B3,10
$900.00 $0.00 $0.00 
Race: 5
Pool Bets Refunds Winnings Runners
WIN $150.00 $0.00 $0.00 B8,11,12
WIN $150.00 $0.00 $0.00 B8,11,12
WIN $150.00 $0.00 $0.00 B8,11,12
WIN $150.00 $0.00 $0.00 B8,11,12
WIN $100.00 $0.00 $0.00 8
$700.00 $0.00 $0.00 
Race: 6
Pool Bets Refunds Winnings Runners
WIN $200.00 $0.00 $0.00 14
TRI $240.00 $0.00 $11,154.00 W6/14/1-14
TRI $60.00 $0.00 $0.00 W6/1-14/14
TRI $60.00 $0.00 $0.00 W6/1-14/14
...
$934.00 $0.00 $4,136.00 
Race: 9
Pool Bets Refunds Winnings Runners
TRI $240.00 $0.00 $0.00 W6,9/2,4-6,9,10,12,13/2,
13
$240.00 $0.00 $0.00 
Race: 10
...
2-Day Totals $53,420.00 $0.00 $132,250.00 
Penalty Amount: $0.00 Final Score: 86,330.00
Engler Monte 900000778
Twinspires
Monte Engler
Date: 20181102
Race: 4
Pool Bets Refunds Winnings Runners
DOUBLE $50.00 $0.00 $0.00 2/4
DOUBLE $40.00 $0.00 $0.00 W3/1,4
$90.00 $0.00 $0.00 
Race: 6
Pool Bets Refunds Winnings Runners
TRIFECTA $360.00 $0.00 $0.00 W6/2,11,12/1-14
TRIFECTA $120.00 $0.00 $0.00 W1,6/1-14/1,6
DOUBLE $150.00 $0.00 $0.00 W1,11,12/1-10

Immediately, it could be seen that there are aspects of the output that could prove fruitful:

there are repeated text values (e.g. “Final Score: “) that could be used to mark where sections began and ended e.g. the player’s name, ID and home track always follow that line.
as it was a two-day event, the exact race that the wagers were for could be figured out by combining for the “Date: YYYYMMDD” and “Race: N” patterns.
some of the data is derived e.g. the bets/refunds/winnings totals-per-day and the Final Score; I generally prefer to parse just the minimum data and calculate those independently.

However, there were some things of concern that were noted:

For wagers with many combinations, the textual representation of the bet often wrapped to the next line - great care would have to be taken to detect and handle that.
Oddly, a mix of bet type keys was being used. For example for Trifecta bets, some players had “TRI”, others had “TRIFECTA”. Some people seems to have made bets that were actually against the rules (yet possibly not detected by the tournament software). Clearly some data sanitation would be required. It also means that you can’t always rely on the consistency of specific “special” values but rather try to be rules- and/or pattern-based instead.

(2 of 3) Basic: parsing the raw text word-by-word

The BCBC table data is simple enough however to figure most of this out with basic rules and some regexes.

The following is the main parser for the BCBC PDF files (the entire project is available on GitHub).

It builds a list of BCBCEntry objects (corresponding to tournament players), each of which contain a list of the parsed bets.

The overridden writeText() method triggers a variety of calls to other methods that can also be overridden to further control the parsing of the PDDocument, including writePage(), writeCharacters(TextPosition text), and writeString(String text, List<TextPosition> textPositions) among others.

The latter is leveraged below to capture specific words that are expected, along with setting various marker booleans that instruct subsequent iterations to extract the desired information. Some pre-processing and sanitation is done by the BCBCEntry and Bet classes (see GitHub for the details):

public class BCBCParser extends PDFTextStripper implements Parser<List<BCBCEntry>, File> {
    static final Logger LOGGER = LoggerFactory.getLogger(BCBCParser.class);

    // regex to match bets e.g. EX, TRI, DD, WIN
    private static final Pattern BET_TYPE = Pattern.compile("^([A-Z-])+$");
    private final BCBCConfig config;
    private List<String> bcbcEntryRelatedText;
    private List<BCBCEntry> bcbcEntries;

    /**
     * Instantiate a new PDFTextStripper object.
     *
     * @throws IOException If there is an error loading the properties.
     */
    public BCBCParser(BCBCConfig config) throws IOException {
        super();
        this.config = config;
        bcbcEntryRelatedText = new ArrayList<>();
        bcbcEntries = new ArrayList<>();

        // see https://pdfbox.apache.org/2.0/getting-started.html
        System.setProperty("sun.java2d.cmm", "sun.java2d.cmm.kcms.KcmsServiceProvider");
    }

    @Override
    public List<BCBCEntry> parse(File bcbcResults) {
        try (PDDocument bcbcResultsPdf = PDDocument.load(bcbcResults)) {
            setSortByPosition(true);

            try (Writer devNullWriter = new OutputStreamWriter(new OutputStream() {
                @Override
                public void write(int b) {
                    // discard everything
                }
            }, UTF_8)) {
                // this will end up calling #writeString() below with the text of each line
                // each line of text can then be examined in the context of the text already
                // parsed, so can figure out if the text is related to the player or the bet etc
                // the line of text itself will be pulled apart to extract the actual bets
                writeText(bcbcResultsPdf, devNullWriter);
                // at this point, all the players' start-end line indexes have been saved
            }
        } catch (InvalidPasswordException e) {
            LOGGER.error("PDF Password incorrect", e);
        } catch (IOException e) {
            LOGGER.error("Error parsing PDF", e);
        }

        bcbcEntries.forEach(bcbcEntry ->
                bcbcEntry.getBets().addAll(createBetsForEntry(config, bcbcEntry)));

        return bcbcEntries;
    }

    // triggered during stripper.writeText() execution above (by the private stripper.writeLine()
    // method) and called for each "word" of a line processed when parsing the PDF (sometimes a word
    // may actually be a single space-separated piece of text)
    @Override
    protected void writeString(String string, List<TextPosition> textPositions) throws IOException {
        // collect all the words relevant for this entry
        bcbcEntryRelatedText.add(string);

        // if "Final Score" was found, then we at the end of content about this player
        if (string.contains("Final Score")) {
            BCBCEntry entry = new BCBCEntry(bcbcEntryRelatedText);
            bcbcEntries.add(entry);

            // reset
            bcbcEntryRelatedText = new ArrayList<>();
        }
    }

    // word-by-word parsing and building the data structure
    private List<Bet> createBetsForEntry(BCBCConfig config, BCBCEntry entry) {
        List<Bet> bets = new ArrayList<>();
        boolean day2Found = false;
        boolean betTypeFound = false;
        boolean activeBetComplete = false;
        boolean penaltyAmountFound = false;
        int race = 0;

        Bet.Builder betBuilder = new Bet.Builder();

        for (String word : entry.getPlayerText()) {
            // if it's Saturday's date, then Friday bets must be done
            if (word.equals("Date: " + config.getDay2Date())) {
                day2Found = true;
                continue; // go to the next word (it should be the race number)
            }

            // track the race these bets were for
            if (word.startsWith("Race: ")) {
                race = Integer.parseInt(word.substring(word.lastIndexOf(' ') + 1));
                continue; // go to the next word (it should be the start of the betBuilder table)
            }

            // update the entry with the total amount of penalties incurred
            if (penaltyAmountFound) {
                entry.setPenalty(word);
                penaltyAmountFound = false; // reset
            }

            // only betBuilder types (Exacta, Trifecta etc.) are in all-caps around the
            // betBuilder data
            // set this market because next word/line (or two, if wrapped) will be related to
            // a betBuilder
            boolean betTypeDetected = BET_TYPE.matcher(word).find();
            // race totals are derived summary data that aggregate the lines above it
            // we don't need to parse it but it is a marker that all bets for this race have
            // been parsed
            boolean raceTotalsSummaryLineDetected = (word.split(" ").length == 3);

            // first check for any bets that have not yet been fully parsed
            if (betTypeFound) {
                boolean completionOfActiveBetDetected =
                        (betTypeDetected || raceTotalsSummaryLineDetected);
                // check if a betBuilder description has wrapped to the next line and update it
                // if so
                if (!completionOfActiveBetDetected) {
                    // because of trailing spaces between the "Bets", "Refunds", "Winnings", and
                    // "Runners" column values, this "word" contains values intended for
                    // multiple columns
                    //
                    // the setter handles splitting this into its respective components as well as
                    // handling line-wraps and inconsistent delimiters
                    betBuilder.compositeBetInfo(word);
                }
                activeBetComplete = true;
            }

            // build and save the bet if it is ready
            if (activeBetComplete) {
                // set the date and race number as they can apply to multiple bets
                betBuilder.date(day2Found ? config.getDay2Date() : config.getDay1Date());
                betBuilder.race(race);

                bets.add(betBuilder.build());

                // start clean for the next betBuilder
                betBuilder = new Bet.Builder();
                betTypeFound = false;
                activeBetComplete = false;
            }

            // set the marker that a betBuilder is active (ready to be parsed)
            if (betTypeDetected) {
                betTypeFound = true;
                betBuilder.type(word);
                // go to the next word (it should be the composite betBuilder information)
            }

            // set a flag that the next word parsed contains the penalty amount for the entry
            if (word.contains("Penalty Amount:")) {
                penaltyAmountFound = true;
            }
        }

        return bets;
    }
}

I was able to validate the parsing logic by picking a player (I chose 7th-placed Allen Harberg as he was the highest placed finisher that incurred a penalty) and calculated their final score just using the individual bets they placed:

    @Test
    public void parse_With2018Results_FinalScoreCalculatedCorrectly() throws Exception {
        File bcbcResults = new File(
                getClass().getClassLoader().getResource("bcbc_2018.pdf").getFile());
        List<BCBCEntry> bcbcEntries = new BCBCParser(new Config2018()).parse(bcbcResults);

        // the Final Score is the starting bankroll of $7,500 plus all winnings minus all bets
        // minus all penalties
        double finalScore = bcbcEntries.stream()
                .filter(bcbcEntry -> bcbcEntry.getUuid().equals("900000129")) // Allen Harberg
                .flatMapToDouble(bcbcEntry -> DoubleStream.of(bcbcEntry.getBets().stream()
                        .collect(summarizingDouble(bet -> (bet.getWinnings() - bet.getBets())))
                        .getSum() - bcbcEntry.getPenalty() + 7500))
                .findAny().getAsDouble();

        Assert.assertThat(finalScore, equalTo(43025d));
    }

What’s more, I was able to reuse this solution again in 2019, and even created a bar chart race visualization that simulated how the leaderboard changed race-by-race using the data extracted with this solution:

A live leaderboard visualization of the @BreedersCup Betting Challenge 2019 (#BCBC) feat. @PatCummingsTIF, @truxtonstables, @stoolpresidente and @DickJerardi among others.

Based on a recent bar chart race demo by @mbostock using @observablehq : https://t.co/PVVPjCpNAo pic.twitter.com/OhlVROozLG
— Robin Howlett (@robinhowlett) November 14, 2019

(3 of 3) Advanced: outputting each character’s metadata

While I was able to certainly use the basic output of the PDF document’s text for parsing, PDFBox enables exposing the metadata of each individual character present within the document, including:

the X-Y coordinates of the character within the page
the font size (even font name)
the height and width of the printed character
the unicode character value

By leveraging the character metadata, much more fine-grained control of the parsing can be conducted. By using the actual measurements of the positioning of the text within the PDF (for instance, inferring columns in a table and grouping the values), we can avoid regex-driven development of trying to parse the purely textual output returned above, which may be very error prone for less predicatably-structured documents.

We saw earlier that TextPosition is part of the writeString() method signature, but we did not use it above. TextPosition represents “a string and a position on the screen of those characters”.

The following variables within TextPosition can be useful for understanding the positioning, size, spacing, and unicode value:

Variable	Description
xDirAdj	The horizontal positioning of the character within a particular page
yDirAdj	The vertical positioning of the character within a particular page
fontSize	The size of the font for this character
height	The visible height of the character
widthDirAdj	The visible width of the character
widthOfSpace	The measurement of the space character that applies to the character
unicode	The unicode value of the character

There are others too like xScale that may be appropriate for your use cases. View the PDFBox documentation for more.

When comparing the various metadata values of each TextPosition instance with another, certain inferences can be made, for example:

If the difference between the yDirAdj values of two characters is greater than the height of the characters (or some other logical value), then the characters are on different lines.
If the difference between the xDirAdj values of two characters is greater than the widthDirAdj value, then some form of whitespace exists between characters within the string/text (some PDFs store space characters, some do not and must be calculated).
fontSize and similar can also be used to identify superscript values.

An effective way to rapidly identify patterns within the text positions of the characters in the PDF is to use a spreadsheet. To do that, we need to create a CSV where each row is character within the PDF and its relevant TextPosition metadata.

public class BCBCParser extends PDFTextStripper {

    public BCBCParser() throws IOException {
        super();
    }

    public List<TextPosition> parse(File bcbcResults) {
        try (PDDocument document = PDDocument.load(bcbcResults)) {
            setSortByPosition(true);

            try (BufferedWriter writer = Files.newBufferedWriter(
                    Paths.get("text-positions.csv"), UTF_8)) {
                // the first row is the header column names
                String[] headers = {"xDirAdj", "yDirAdj", "fontSize", "xScale", "height",
                        "widthOfSpace", "widthDirAdj", "unicode"};
                // use pipe as a delimiter (just as a personal preference)
                writer.write(String.join("|", headers));
                writer.write(System.lineSeparator());
                // this will call #writeString() below with the line text and positions of each char
                writeText(document, writer);
            }
        } catch (InvalidPasswordException e) {
            e.printStackTrace();
        } catch (IOException e) {
            e.printStackTrace();
        }

        return null;
    }

    @Override
    protected void writeWordSeparator() throws IOException {
        // do nothing as we don't need spaces
    }

    @Override
    protected void writeLineSeparator() throws IOException {
        // do nothing as writeString(String, List<TextPosition>) below handles the new lines
    }

    @Override
    protected void writeString(String text, List<TextPosition> textPositions) throws IOException {
        if (!text.isEmpty()) {
            textPositions.forEach(textPosition -> {
                try {
                	// call the parent's writeString(String) to write to the output writer
                    writeString(String.join("|",
                            String.valueOf(textPosition.getXDirAdj()),
                            String.valueOf(textPosition.getYDirAdj()),
                            String.valueOf(textPosition.getFontSize()),
                            String.valueOf(textPosition.getXScale()),
                            String.valueOf(textPosition.getHeight()),
                            String.valueOf(textPosition.getWidthOfSpace()),
                            String.valueOf(textPosition.getWidthDirAdj()),
                            textPosition.getUnicode()));
                    // write a line/row after each character
                    writeString(System.lineSeparator());
                } catch (IOException e) {
                    e.printStackTrace();
                }
            });
        }
    }
}

This outputs to a file called text-positions.csv the following:

xDirAdj|yDirAdj|fontSize|xScale|height|widthOfSpace|widthDirAdj|unicode
0|28.950012|1.0|10.65|7.364475|2.6625|8.2857|G
2857|28.950012|1.0|10.65|7.364475|2.6625|4.7285995|r
0143|28.950012|1.0|10.65|7.364475|2.6625|5.9214|u
9357|28.950012|1.0|10.65|7.364475|2.6625|5.921398|b
857098|28.950012|1.0|10.65|7.364475|2.6625|5.921398|b
778496|28.950012|1.0|10.65|7.364475|2.6625|4.142849|s
0|28.950012|1.0|10.65|7.364475|2.6625|7.6893005|C
6893|28.950012|1.0|10.65|7.364475|2.6625|5.921402|h
6107|28.950012|1.0|10.65|7.364475|2.6625|5.324997|a
9357|28.950012|1.0|10.65|7.364475|2.6625|4.728607|r
6643|28.950012|1.0|10.65|7.364475|2.6625|2.9606934|l
625|28.950012|1.0|10.65|7.364475|2.6625|4.728607|e
3536|28.950012|1.0|10.65|7.364475|2.6625|4.142853|s
3|27.0|1.0|8.95|6.02335|2.2375|4.4749756|1
77496|27.0|1.0|8.95|6.02335|2.2375|4.4749756|0
24994|27.0|1.0|8.95|6.02335|2.2375|4.4749756|6
7249|27.0|1.0|8.95|6.02335|2.2375|4.4749756|0
1999|27.0|1.0|8.95|6.02335|2.2375|4.4749756|3
...

When opening the same file in Excel (and using Data > Text to Columns to instruct that the pipe character is a delimiter, plus adding some basic decimal formatting), we can now look at the patterns of the character metadata, to make some inferences:

Let’s look at the table that corresponds to the bets made for a particular race (see page 2 of the 2018 BCBC results PDF, race 3 of the second day):

This entry is interesting because some of the bet details (under the “Runners” column) have been wrapped to the next line. Also, the “Pool” column is left-aligned, but all other columns are right-aligned.

A little scroll-and-search within Excel finds the character metadata that corresponds:

This may be hard to see but I’ve split the Excel sheet and on top highlighted the “Pool”, “Bets”, and “Refunds” column header characters, and, below, the first row character values that correspond (“WIN”, “$200,00 “, and “$0.00 “ respectively).

This allows us create rules for detecting when text has been wrapped. For example, see how the yDirAdj value of the “0” character (that was wrapped above in the PDF’s “Runners” columns) of the highlighted row is unlike the rows above and below but that it has a high xDirAdj value, indicating it is positioned to the right of the page:

Again, the BCBC PDF was simple enough in its structure to not need this level of control, but I do have experience with far more complex PDF layouts.

Consider the following PDF from Equibase for a horse racing result chart:

In this case, there are a variety of layouts within the document - multi-line text, multiple key-value pairs on the same line, multiple tables whose text and even number of rows and columns are fully dynamic and based on the nature of the content, superscript text, fractions, even embedded images.

To outline just one technique, for grouping related columns of data within the table (which had semi-predicable headers), I built a TreeSet of xDirAdj and column header indices, so that for any character found within the table, I could find the nearest starting header column using the floor() method:

I was able to fully parse this document (and over a million others like it) into a highly-specialized domain-specific data structure that powered a variety of data-centric tools, APIs and SDKs:

Unfortunately, I ran into some issues when distributing this open-source software, but the general procedure I used was the same as what I have been describing in this post.

Between eyeballing the PDF, noting where obvious patterns exist, and potentionally building a data model using the character position metadata, you can start to construct a collection of data structures that capture the particular domain values you wish to extract from the PDF.

Solved: When the Maven Deploy Plugin silently fails to deploy

Wed, 15 May 2019 00:00:00 +0000

At SnapLogic, we recently noticed that a particular build job that was responsible for deploying build artifacts to a Nexus repository via Maven had suddenly stopped, well, deploying. What was odd was that no error of any kind was being communicated, even in DEBUG mode.

Examining the history of the repository, what had changed was the addition of a new module (“slbugs”) to the existing multi-module Maven build. This module’s reposibility was to run some code health checks using Google’s error-prone static analysis tool to catch some programming mistakes that our developers occassionally made that had a negative effect at runtime.

What was different about this module versus the others was that it did not use the root POM as a parent (as it was sufficiently different from the other more product-focused modules). The other modules were also configured to use the deployAtEnd parameter of the parent’s maven-deploy-plugin plugin configuration.

The problem was that each of the product modules would log that they would be deployed at the end of the build, but after the last module ran its deploy phase, nothing would happen - no uploading of artifacts would be attempted, no warnings or debug messages logged to explain the inaction, and the build would just end with a SUCCESS status.

The solution turned out to be related to the wonderful world of Maven classloaders.

When the problem was reported and some obvious things had been tried (changing plugin versions etc.), I started to look into it. The first thing I did was create a standalone test that mirrored the basic structure of the multi-module project and chose what I deemed to be the pertinent configurations from the production POMs.

It looked like this:

Quite standard as you can see. pom-root defines the modules (module-one and module-two), where module-one does not use pom-root as its parent but module-two does. All dependency and plugin versions are identical, but pom-root (and therefore module-two by inheritence) uses the deployAtEnd configuration for the maven-deploy-plugin. For running the deploy phase, I setup a local empty temporary repository via the distributionManagement definition.

So, I ran mvn clean deploy and expected to see the problem reproduced:

> mvn clean deploy
[MVNVM] Using maven: 3.5.2
[INFO] Scanning for projects...
[INFO] ------------------------------------------------------------------------
[INFO] Reactor Build Order:
[INFO] 
[INFO] module-one
[INFO] pom-root
[INFO] module-two
[INFO] 
[INFO] ------------------------------------------------------------------------
[INFO] Building module-one 1.0-SNAPSHOT
[INFO] ------------------------------------------------------------------------
[INFO] 
[INFO] --- maven-clean-plugin:2.5:clean (default-clean) @ module-one ---
[INFO] Deleting /Users/rhowlett/dev/tmp/pom-root/module-one/target
[INFO] 
[INFO] --- maven-resources-plugin:2.6:resources (default-resources) @ module-one ---
[INFO] Using 'UTF-8' encoding to copy filtered resources.
[INFO] skip non existing resourceDirectory /Users/rhowlett/dev/tmp/pom-root/module-one/src/main/resources
[INFO] 
[INFO] --- maven-compiler-plugin:3.1:compile (default-compile) @ module-one ---
[INFO] Changes detected - recompiling the module!
[INFO] Compiling 1 source file to /Users/rhowlett/dev/tmp/pom-root/module-one/target/classes
[INFO] 
[INFO] --- maven-resources-plugin:2.6:testResources (default-testResources) @ module-one ---
[INFO] Using 'UTF-8' encoding to copy filtered resources.
[INFO] skip non existing resourceDirectory /Users/rhowlett/dev/tmp/pom-root/module-one/src/test/resources
[INFO] 
[INFO] --- maven-compiler-plugin:3.1:testCompile (default-testCompile) @ module-one ---
[INFO] Changes detected - recompiling the module!
[INFO] Compiling 1 source file to /Users/rhowlett/dev/tmp/pom-root/module-one/target/test-classes
[INFO] 
[INFO] --- maven-surefire-plugin:2.12.4:test (default-test) @ module-one ---
[INFO] Surefire report directory: /Users/rhowlett/dev/tmp/pom-root/module-one/target/surefire-reports

-------------------------------------------------------
 T E S T S
-------------------------------------------------------
Running com.snaplogic.AppTest
Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.048 sec

Results :

Tests run: 1, Failures: 0, Errors: 0, Skipped: 0

[INFO] 
[INFO] --- maven-jar-plugin:2.4:jar (default-jar) @ module-one ---
[INFO] Building jar: /Users/rhowlett/dev/tmp/pom-root/module-one/target/module-one-1.0-SNAPSHOT.jar
[INFO] 
[INFO] --- maven-install-plugin:2.4:install (default-install) @ module-one ---
[INFO] Installing /Users/rhowlett/dev/tmp/pom-root/module-one/target/module-one-1.0-SNAPSHOT.jar to /Users/rhowlett/.m2/repository/com/snaplogic/module-one/1.0-SNAPSHOT/module-one-1.0-SNAPSHOT.jar
[INFO] Installing /Users/rhowlett/dev/tmp/pom-root/module-one/pom.xml to /Users/rhowlett/.m2/repository/com/snaplogic/module-one/1.0-SNAPSHOT/module-one-1.0-SNAPSHOT.pom
[INFO] 
[INFO] --- maven-deploy-plugin:2.8.2:deploy (default-deploy) @ module-one ---
Downloading from dev: file:///Users/rhowlett/.m2/temp-repository/com/snaplogic/module-one/1.0-SNAPSHOT/maven-metadata.xml
Uploading to dev: file:///Users/rhowlett/.m2/temp-repository/com/snaplogic/module-one/1.0-SNAPSHOT/module-one-1.0-20190515.171609-1.jar
Uploaded to dev: file:///Users/rhowlett/.m2/temp-repository/com/snaplogic/module-one/1.0-SNAPSHOT/module-one-1.0-20190515.171609-1.jar (2.4 kB at 237 kB/s)
Uploading to dev: file:///Users/rhowlett/.m2/temp-repository/com/snaplogic/module-one/1.0-SNAPSHOT/module-one-1.0-20190515.171609-1.pom
Uploaded to dev: file:///Users/rhowlett/.m2/temp-repository/com/snaplogic/module-one/1.0-SNAPSHOT/module-one-1.0-20190515.171609-1.pom (1.4 kB at 702 kB/s)
Downloading from dev: file:///Users/rhowlett/.m2/temp-repository/com/snaplogic/module-one/maven-metadata.xml
Uploading to dev: file:///Users/rhowlett/.m2/temp-repository/com/snaplogic/module-one/1.0-SNAPSHOT/maven-metadata.xml
Uploaded to dev: file:///Users/rhowlett/.m2/temp-repository/com/snaplogic/module-one/1.0-SNAPSHOT/maven-metadata.xml (767 B at 192 kB/s)
Uploading to dev: file:///Users/rhowlett/.m2/temp-repository/com/snaplogic/module-one/maven-metadata.xml
Uploaded to dev: file:///Users/rhowlett/.m2/temp-repository/com/snaplogic/module-one/maven-metadata.xml (281 B at 94 kB/s)
[INFO] 
[INFO] ------------------------------------------------------------------------
[INFO] Building pom-root 1.0-SNAPSHOT
[INFO] ------------------------------------------------------------------------
[INFO] 
[INFO] --- maven-clean-plugin:2.5:clean (default-clean) @ pom-root ---
[INFO] 
[INFO] --- maven-install-plugin:2.4:install (default-install) @ pom-root ---
[INFO] Installing /Users/rhowlett/dev/tmp/pom-root/pom.xml to /Users/rhowlett/.m2/repository/com/snaplogic/pom-root/1.0-SNAPSHOT/pom-root-1.0-SNAPSHOT.pom
[INFO] 
[INFO] --- maven-deploy-plugin:2.8.2:deploy (default-deploy) @ pom-root ---
[INFO] Deploying com.snaplogic:pom-root:1.0-SNAPSHOT at end
[INFO] 
[INFO] ------------------------------------------------------------------------
[INFO] Building module-two 1.0-SNAPSHOT
[INFO] ------------------------------------------------------------------------
[INFO] 
[INFO] --- maven-clean-plugin:2.5:clean (default-clean) @ module-two ---
[INFO] Deleting /Users/rhowlett/dev/tmp/pom-root/module-two/target
[INFO] 
[INFO] --- maven-resources-plugin:2.6:resources (default-resources) @ module-two ---
[INFO] Using 'UTF-8' encoding to copy filtered resources.
[INFO] skip non existing resourceDirectory /Users/rhowlett/dev/tmp/pom-root/module-two/src/main/resources
[INFO] 
[INFO] --- maven-compiler-plugin:3.1:compile (default-compile) @ module-two ---
[INFO] Changes detected - recompiling the module!
[INFO] Compiling 1 source file to /Users/rhowlett/dev/tmp/pom-root/module-two/target/classes
[INFO] 
[INFO] --- maven-resources-plugin:2.6:testResources (default-testResources) @ module-two ---
[INFO] Using 'UTF-8' encoding to copy filtered resources.
[INFO] skip non existing resourceDirectory /Users/rhowlett/dev/tmp/pom-root/module-two/src/test/resources
[INFO] 
[INFO] --- maven-compiler-plugin:3.1:testCompile (default-testCompile) @ module-two ---
[INFO] Changes detected - recompiling the module!
[INFO] Compiling 1 source file to /Users/rhowlett/dev/tmp/pom-root/module-two/target/test-classes
[INFO] 
[INFO] --- maven-surefire-plugin:2.12.4:test (default-test) @ module-two ---
[INFO] Surefire report directory: /Users/rhowlett/dev/tmp/pom-root/module-two/target/surefire-reports

-------------------------------------------------------
 T E S T S
-------------------------------------------------------
Running com.snaplogic.AppTest
Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.043 sec

Results :

Tests run: 1, Failures: 0, Errors: 0, Skipped: 0

[INFO] 
[INFO] --- maven-jar-plugin:2.4:jar (default-jar) @ module-two ---
[INFO] Building jar: /Users/rhowlett/dev/tmp/pom-root/module-two/target/module-two-1.0-SNAPSHOT.jar
[INFO] 
[INFO] --- maven-install-plugin:2.4:install (default-install) @ module-two ---
[INFO] Installing /Users/rhowlett/dev/tmp/pom-root/module-two/target/module-two-1.0-SNAPSHOT.jar to /Users/rhowlett/.m2/repository/com/snaplogic/module-two/1.0-SNAPSHOT/module-two-1.0-SNAPSHOT.jar
[INFO] Installing /Users/rhowlett/dev/tmp/pom-root/module-two/pom.xml to /Users/rhowlett/.m2/repository/com/snaplogic/module-two/1.0-SNAPSHOT/module-two-1.0-SNAPSHOT.pom
[INFO] 
[INFO] --- maven-deploy-plugin:2.8.2:deploy (default-deploy) @ module-two ---
Downloading from dev: file:///Users/rhowlett/.m2/temp-repository/com/snaplogic/pom-root/1.0-SNAPSHOT/maven-metadata.xml
Uploading to dev: file:///Users/rhowlett/.m2/temp-repository/com/snaplogic/pom-root/1.0-SNAPSHOT/pom-root-1.0-20190515.171609-1.pom
Uploaded to dev: file:///Users/rhowlett/.m2/temp-repository/com/snaplogic/pom-root/1.0-SNAPSHOT/pom-root-1.0-20190515.171609-1.pom (1.8 kB at 602 kB/s)
Downloading from dev: file:///Users/rhowlett/.m2/temp-repository/com/snaplogic/pom-root/maven-metadata.xml
Uploading to dev: file:///Users/rhowlett/.m2/temp-repository/com/snaplogic/pom-root/1.0-SNAPSHOT/maven-metadata.xml
Uploaded to dev: file:///Users/rhowlett/.m2/temp-repository/com/snaplogic/pom-root/1.0-SNAPSHOT/maven-metadata.xml (594 B at 119 kB/s)
Uploading to dev: file:///Users/rhowlett/.m2/temp-repository/com/snaplogic/pom-root/maven-metadata.xml
Uploaded to dev: file:///Users/rhowlett/.m2/temp-repository/com/snaplogic/pom-root/maven-metadata.xml (279 B at 70 kB/s)
Downloading from dev: file:///Users/rhowlett/.m2/temp-repository/com/snaplogic/module-two/1.0-SNAPSHOT/maven-metadata.xml
Uploading to dev: file:///Users/rhowlett/.m2/temp-repository/com/snaplogic/module-two/1.0-SNAPSHOT/module-two-1.0-20190515.171610-1.jar
Uploaded to dev: file:///Users/rhowlett/.m2/temp-repository/com/snaplogic/module-two/1.0-SNAPSHOT/module-two-1.0-20190515.171610-1.jar (2.1 kB at 710 kB/s)
Uploading to dev: file:///Users/rhowlett/.m2/temp-repository/com/snaplogic/module-two/1.0-SNAPSHOT/module-two-1.0-20190515.171610-1.pom
Uploaded to dev: file:///Users/rhowlett/.m2/temp-repository/com/snaplogic/module-two/1.0-SNAPSHOT/module-two-1.0-20190515.171610-1.pom (533 B at 178 kB/s)
Downloading from dev: file:///Users/rhowlett/.m2/temp-repository/com/snaplogic/module-two/maven-metadata.xml
Uploading to dev: file:///Users/rhowlett/.m2/temp-repository/com/snaplogic/module-two/1.0-SNAPSHOT/maven-metadata.xml
Uploaded to dev: file:///Users/rhowlett/.m2/temp-repository/com/snaplogic/module-two/1.0-SNAPSHOT/maven-metadata.xml (767 B at 256 kB/s)
Uploading to dev: file:///Users/rhowlett/.m2/temp-repository/com/snaplogic/module-two/maven-metadata.xml
Uploaded to dev: file:///Users/rhowlett/.m2/temp-repository/com/snaplogic/module-two/maven-metadata.xml (281 B at 94 kB/s)
[INFO] ------------------------------------------------------------------------
[INFO] Reactor Summary:
[INFO] 
[INFO] module-one ......................................... SUCCESS [  2.970 s]
[INFO] pom-root ........................................... SUCCESS [  0.131 s]
[INFO] module-two ......................................... SUCCESS [  0.652 s]
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 3.905 s
[INFO] Finished at: 2019-05-15T11:16:10-06:00
[INFO] Final Memory: 19M/309M
[INFO] ------------------------------------------------------------------------

Odd. The deployment happened perfectly. Then why wasn’t it happening with the production build?

I decided to debug. I added the source for version 2.8.2 of the maven-deploy-plugin to my IDE and ran the same command except this time with mvnDebug and a Remote Debug configuration setup. I put a breakpoint in the execute() method of the DeployMojo class and hoped to see what was causing the deployment not to be executed:

/**
 * When building with multiple threads, reaching the last project doesn't have to mean that all projects are ready
 * to be deployed
 */
private static final AtomicInteger readyProjectsCounter = new AtomicInteger();
    
...

public void execute()
        throws MojoExecutionException, MojoFailureException {
    boolean addedDeployRequest = false;
    if (skip) {
        getLog().info("Skipping artifact deployment");
    } else {
        failIfOffline();

        DeployRequest currentExecutionDeployRequest =
                new DeployRequest().setProject(project).setUpdateReleaseInfo(isUpdateReleaseInfo()).setRetryFailedDeploymentCount(getRetryFailedDeploymentCount()).setAltReleaseDeploymentRepository(altReleaseDeploymentRepository).setAltSnapshotDeploymentRepository(altSnapshotDeploymentRepository).setAltDeploymentRepository(altDeploymentRepository);

        if (!deployAtEnd) {
            deployProject(currentExecutionDeployRequest);
        } else {
            deployRequests.add(currentExecutionDeployRequest);
            addedDeployRequest = true;
        }
    }

    boolean projectsReady = readyProjectsCounter.incrementAndGet() == reactorProjects.size();
    if (projectsReady) {
        synchronized (deployRequests) {
            while (!deployRequests.isEmpty()) {
                deployProject(deployRequests.remove(0));
            }
        }
    } else if (addedDeployRequest) {
        getLog().info("Deploying " + project.getGroupId() + ":" + project.getArtifactId() + ":"
                + project.getVersion() + " at end");
    }
}

It looked clear that the projectsReady boolean would be key. Its value was determined by the equality of a static AtomicInteger called readyProjectsCounter and the size of the reactorProjects collection. The only way for the build to log that it was going to deploy at the end and then not deploy the final module, would be if the readyProjectsCounter and reactorProjects size had some kind of off-by-X error. Indeed, the debugger showed that, for the production build, readyProjectsCounter was one less than the collection size. But what would cause that?

I dived into the debug logs of the production build to try to see if I could identify anything of note. After trying quite a few different things, something caught my eye.

When “slbugs” was running its deploy phase with the maven-deploy-plugin, it logged:

[DEBUG] Created new class realm plugin>org.apache.maven.plugins:maven-deploy-plugin:2.8.2
[DEBUG] Importing foreign packages into class realm plugin>org.apache.maven.plugins:maven-deploy-plugin:2.8.2
[DEBUG]   Imported:  < maven.api
[DEBUG] Populating class realm plugin>org.apache.maven.plugins:maven-deploy-plugin:2.8.2
[DEBUG]   Included: org.apache.maven.plugins:maven-deploy-plugin:jar:2.8.2
[DEBUG]   Included: backport-util-concurrent:backport-util-concurrent:jar:3.1
[DEBUG]   Included: org.codehaus.plexus:plexus-interpolation:jar:1.11
[DEBUG]   Included: junit:junit:jar:3.8.1
[DEBUG]   Included: org.codehaus.plexus:plexus-utils:jar:3.0.15
[DEBUG] Configuring mojo org.apache.maven.plugins:maven-deploy-plugin:2.8.2:deploy from plugin realm ClassRealm[plugin>org.apache.maven.plugins:maven-deploy-plugin:2.8.2, parent: sun.misc.Launcher$AppClassLoader@3d4eac69]

When the parent POM was running, this was logged:

[DEBUG] Created new class realm plugin>org.apache.maven.plugins:maven-deploy-plugin:2.8.2-932771130
[DEBUG] Importing foreign packages into class realm plugin>org.apache.maven.plugins:maven-deploy-plugin:2.8.2-932771130
[DEBUG]   Imported:  < maven.api
[DEBUG] Populating class realm plugin>org.apache.maven.plugins:maven-deploy-plugin:2.8.2-932771130
[DEBUG]   Included: org.apache.maven.plugins:maven-deploy-plugin:jar:2.8.2
[DEBUG]   Included: backport-util-concurrent:backport-util-concurrent:jar:3.1
[DEBUG]   Included: org.codehaus.plexus:plexus-interpolation:jar:1.11
[DEBUG]   Included: junit:junit:jar:3.8.1
[DEBUG]   Included: org.codehaus.plexus:plexus-utils:jar:3.0.15
[DEBUG] Configuring mojo org.apache.maven.plugins:maven-deploy-plugin:2.8.2:deploy from plugin realm ClassRealm[plugin>org.apache.maven.plugins:maven-deploy-plugin:2.8.2-932771130, parent: sun.misc.Launcher$AppClassLoader@3d4eac69]

This -932771130 suffix was strange. When I turned on debug logging for pom-root by running mvn -X clean deploy, I saw no such recreation of a “class realm”. In fact, every plugin was “initialized” once.

But in the production build, many plugins (maven-install-plugin, maven-resources-plugin etc.), all seem to have been “recreated” when the build moved on to the root pom (after the initial “slbugs” module had been deployed). What was triggering that? I started to consume some articles from the web, including:

I’ll admit that I was a bit puzzled. Neither build had any custom extensions defined, but I started focusing on whether there was anything non-standard defined in the production build POM, looking at things like <properties>, <build>, or anything around <plugins>.

Sure enough, I noticed the following:

<pluginRepositories>
    <pluginRepository>
        <id>apache.snapshots</id>
        <name>Apache Snapshots</name>
        <url>http://repository.apache.org/content/groups/snapshots-group/</url>
        <releases>
            <enabled>true</enabled>
        </releases>
        <snapshots>
            <enabled>true</enabled>
        </snapshots>
    </pluginRepository>
</pluginRepositories>

This was added many years ago to the build, probably to avail of some SNAPSHOT-versioned plugin. So I decided to replicate this in the standalone test’s pom-root’s pom.xml by adding the following:

<pluginRepositories>
    <pluginRepository>
        <id>temp.repo</id>
        <name>Temp Repo</name>
        <url>file://${user.home}/.m2/temp-repository</url>
    </pluginRepository>
</pluginRepositories>

I re-ran the build and I finally I had reproduced the problem (the non-deployment of pom-root and module-two):

...
[INFO] --- maven-deploy-plugin:2.8.2:deploy (default-deploy) @ module-two ---
[INFO] Deploying com.snaplogic:module-two:1.0-SNAPSHOT at end
[INFO] ------------------------------------------------------------------------
[INFO] Reactor Summary:
[INFO] 
[INFO] module-one ......................................... SUCCESS [  3.477 s]
[INFO] pom-root ........................................... SUCCESS [  0.228 s]
[INFO] module-two ......................................... SUCCESS [  1.446 s]
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 5.308 s
[INFO] Finished at: 2019-05-15T12:10:55-06:00
[INFO] Final Memory: 24M/437M
[INFO] ------------------------------------------------------------------------

Similarly, the debug log shows the two initializations of the maven-deploy-plugin:

[DEBUG] Created new class realm plugin>org.apache.maven.plugins:maven-deploy-plugin:2.8.2
...
[DEBUG] Created new class realm plugin>org.apache.maven.plugins:maven-deploy-plugin:2.8.2-1119294797

So, the addition of the new module (“slbugs”) in our production build (that didn’t define a <pluginRespositories> and didn’t use the root POM as its parent), created a new context that resulted in new classloaders being used when the root POM (and its child modules) executed. This caused the maven-deploy-plugin’s readyProjectsCounter static AtomicInteger to start once again from 0 (instead of 1 as it should have been incremented when the first module was deployed), resulting in the off-by-one error when compared to the count of the reactor module (which wasn’t static), meaning the parent and child modules never meeting the condition that would trigger the deployment.

As for resolving this, I found that if both pom-root and module-one “matched” when it came to defining the <pluginRepositories>, then the deployment would work; meaning:

if neither defined it, the deployment worked
if both defined it, the deployment worked (however the pluginRepository’s id and urls must be the same)

The standalone project I created to demonstrate this is available on GitHub here: https://github.com/robinhowlett/maven-deploy-plugin-gotcha

The master branch represents the non-working deployment, and the fix branch represents the working example (with the <pluginRepositories> removed from the root POM).

Visualizing inefficient multi-ticket horizontal wagering tickets

Sun, 12 Aug 2018 19:44:27 +0000

Between PPs, result charts, and wagering ticket summaries, the way that horse racing information has been presented to regular fans hasn’t changed very much over the years. In this post, I use data visualizations to try to increase comprehension of advanced wagering advice.

It was May of 2017 when, deep into the development of Handycapper, I took a break and browsed Twitter.

I saw a tweet from Darin Zoccali (@atTheTrack7) referencing a column he published at DRF.com titled, “Trust is a must or your handicapping game is a bust” (archive link).

In this column, Darin explained how a well-known wagering professional, @InsideThePylons (ITP), had called out inefficiencies and mistakes in the construction of Pick 4 tickets Darin had posted in late 2016. Darin reported that this advice had helped him wager smarter, increase churn, and win far more when his opinion was correct.

Intrigued, I wanted to find this conversation, and, with a little Twitter Search-fu, I was able to locate the Pick 4 in question.

As I tried to understand the finer points of ticket construction @InsideThePylons was making, I remember making a mental note that this kind of feedback would be a great candidate to potentially benefit from data visualizations to aid understanding, for the average racing fan, of the interaction between Darin and ITP.

TL;DR? Skip directly to the interactive visualization or view a demonstration video.

The Pick 4 Tickets

The exchange started with Darin posting his $0.50 Pick 4 tickets for @TheMeadowlands Pick 4 pool for races 7 through 10:

Put in my pick 4 tickets for @TheMeadowlands. Pretty high on Alexie Mattosie tonight. Using Fool Me Once in a back-up ticket. pic.twitter.com/B6P4AfxVaM
— Darin Zoccali (@atTheTrack7) December 10, 2016

In short, Darin wagered $99 on four Pick 4 tickets.

He called out a strong preference for #8 Alexie Mattosie in Race 9, but used #7 Fool Me Once in the same race in “back-up” tickets.

Judgement from ITP was swift:

I mean how could you possibly ever make a bet again in your life if it came 1/2 to 4 to 7 to 6?
— Inside The Pylons (@InsideThePylons) December 10, 2016

Eyeballing the individual tickets Darin had posted, I could see ITP was correct.

Indeed, using a Sankey diagram (you may have seen one previously from Hello Race Fans’s “Paths to the Kentucky Derby”) to visualize the four individual Pick 4 tickets emphasized the lack of coverage:

But I had a more fundamental question - would an inefficient multi-ticket structure be immediately identifiable with a more advanced, interactive perspective?

Implicit Choices

You played $36 keying Fool out of the $54 keying Alexie and removed the 2 bombs to save the $18 + keyed the bombs on press?
— Inside The Pylons (@InsideThePylons) December 10, 2016

It’s worth explaining @InsideThePylons’s tweet:

“You played $36 keying Fool out of the $54 keying Alexie”

The $36 refers to the $12 ticket (3,4,5,8 / 1,2,4,5,8,9 / 7 / 6) plus the $24 ticket (3,4,5,8 / 1,2,4,5,8,9 / 7 / 1,3).

The $54 ticket (1,2,3,4,5,8 / 1,2,4,5,8,9 / 8 / 1,3,6) differs only to the above tickets by using #1 and #2 in the first leg (Race 7) and replacing #7 Fool Me Once with #8 Alexie Mattosie in the third leg (Race 9).

“removed the 2 bombs to save the $18”

The “2 bombs” refer to #1 (ML 12/1, race odds 23/1) and #2 (ML 12/1, race odds 30/1) in the first leg (Race 7) that were not present in the tickets keying #7 Fool Me Once in the third leg (Race 9).

These odds are not identified in the tickets posted above, but ITP was aware of them.

By not using these longshots, there was a difference of $18 between the $36 ($24 + $12) tickets and the $54 ticket.

“keyed the bombs on press”

However, Darin also added a $9 ticket (1,2,4 / 8,9 / 8 / 1,3,6), which ITP is referring to as the “press”:

How can you like 124 best in 1st leg but eliminate 12 on the ticket with Fool?
— Inside The Pylons (@InsideThePylons) December 10, 2016

What this $9 ticket implies is that, on top of the $54 ticket that singles #8 Alexie Mattosie in the third leg of the Pick 4 (Race 9), there is further investment in both this opinion, and the opinion that #1, #2, and #4 (race odds, 2.20) are Darin’s “most liked” choices in the first leg (Race 7), and similarly #8 (1.10) and #9 (16.90) in the second leg (Race 8).

The “bombs” (#1 and #2 referred to earlier) are present in this ticket, yet absent from the $36 investment in the two tickets that single #7 Fool Me Once in the third leg (Race 9).

Why would you go out of your way to remove "joker" and save $18 but play another ticket pressing joker that you removed?
— Inside The Pylons (@InsideThePylons) December 10, 2016

Looking back, I think Darin himself would acknowledge that the above tickets were not constructed optimally, and his subsequent DRF article details how he changed his approach going forward.

Visualizing Inefficient Tickets

I had already used Sankey diagrams above to display the four Pick 4 tickets, but the traditional implementation of this chart is to highlight single links between coupled nodes (e.g. between #8 in Race 9 and #1 in Race 10) rather than the end-to-end nature of a Pick 4 ticket (where a selection in every leg must win for the ticket to payoff, excluding consolations).

I wanted something that would aggregate all four Pick 4 tickets into a single parallel set visualization, but still respect the combinatorial mathematics in-play with four separate tickets. I also wanted to quickly answer questions like:

How many ticket combinations use #8 Alexie Mattosie versus #7 Fool Me Once in the third leg (Race 9)?
How many ticket combinations use the #9 in the second leg (Race 8) and the #8 in the third leg (Race 9)?
Who is used in more tickets: the #3 in the first leg (Race 7) or the #9 in the second leg (Race 8)?

Fortunately, I discovered a variation of Sankey with end-to-end highlighting, developed by Ming Qin, which visualized the famous “Titanic Survivors” data set, which is commonly used as an exercise in statistical thinking.

This implementation used a flow-based API that enabled end-to-end highlighting of that flow, plus additional features like rich tooltips and draggable nodes, that I could further leverage.

Using a Google Spreadsheet, I manually described the 198 combinations that Darin’s $99 50c Pick 4 tickets covered (though automating this would be straightforward) and then used a formula to construct the flow node objects used by the library:

Then it was just a matter of copy/pasting the nodes and flows into the appropriate asset file and the visualization was ready.

So here is the actual interactive visualization!

Race 7

Race 8

Race 9

Race 10

Click here for a larger standalone version of the visualization

The standalone visualization is open-source and available on GitHub.

Explanation of the Visualization

The above diagram has a few special features that are worth understanding:

Using your cursor, hover over a rectangular node (representing a horse in a race) to see all the combinations that flow through it across all tickets. In the following example, we can see that #9 in the second leg has 39 combinations including it.

By hovering over #7 Fool Me Once, you can clearly see that no combinations that use him in the third leg (Race 9), also use #1 or #2 in the first leg (Race 7) - exactly to ITP’s point!

Hover over the dark-blue links between the nodes - these represent a connection between a selection in each of two legs, as well as visually showing the paths (and their frequency) of any incoming or outgoing combinations.

In the above example, you can see that 12 of the 198 combinations use #1 in the second leg (Race 8) and #7 Fool Me Once in the third leg (Race 9). This can be also useful to find live tickets quickly.

By double-clicking on a highlighted link, you can get a rich tooltip that outlines all of the actual combinations (e.g. viable paths in a ticket) involving that link:

I can envision interesting information being added here; for example, expected payoffs or even advice that the combination is too “chalky”.

Each rectangular node can be moved by dragging them, but I haven’t really found a compelling reason to do that for this particular visualization.

Conclusion

So, does this visualization satisfy the goal of aiding understanding of the structuring issues behind the multiple Pick 4 tickets?

Maybe.

There are several improvements I can think of, primarily about more clearly delineating each ticket’s combinations and perhaps being able to filter certain tickets and/or combinations.

I do think that ADW’s could start thinking about presenting their wagering tickets in a more dynamic and interactive way, or at least allow them to be shared that way.

There may even be a product opportunity here - a “wagering assistant” that reviews placed wagers and highlights inefficient ticket structures.

However, I go back to what I said when I charted HANA’s track ratings against the 2017 racing calendar in a spreadsheet:

Is there any major insight gained by this data visualization? Probably not. But that's not the point. Today, there are a myriad of distribution options to extend both the reach & the form of content. Much of that can come from regular fans. That engagement should be encouraged.
— Robin Howlett (@robinhowlett) August 7, 2018

Building a Google Chrome Extension (Keyboard Shortcuts, Copying to the Clipboard, and Notifications)

Sat, 12 Nov 2016 07:27:43 +0000

I recently had the quite enjoyable and productive experience of writing Pipeline Linker, my first Google Chrome Extension.

As part of my work with SnapLogic, an enterprise integration platform-as-a-service (iPaaS) provider, I often have to navigate to Pipelines (hosted graphical representations of integrations) across multiple environments (both internal and customer-facing), client accounts, and project folders.

In turn, as the manager of my team, I regularly direct team members to Pipelines that require attention through email, Hangouts, JIRA, Zendesk, and Slack.

For whatever reason, our product had not provided an easy way of linking directly to these pipelines (you had to switch to a different tab and perform a search, before right-clicking on a table entry and copying the link).

Over a weekend, I was pleasantly surprised by the ease and speed I was able to learn and implement a Chrome Extension that would address this gap, as well as add some features that I, in particular, value.

How it works

The name of the Chrome Extension is Pipeline Linker, and it makes it easier to share links to Pipelines by copying the direct link of the active Pipeline to your clipboard:

I have open-sourced the extension’s code on GitHub: https://github.com/SnapLogic/pipeline-linker

When installed, this Chrome Extension will initially be grayed out and disabled:

It will only be enabled if it detects the active tab is the SnapLogic Designer (regardless of whether it was an internal or external environment):

When one wishes to share the direct link to the currently active pipeline in Designer, the extension’s icon beside the Chrome address bar (see above) is clicked or the keyboard shortcut is used (default is Ctrl+i. Mac users, it is control+i, not command(⌘)+i:

When triggered, a Chrome Notification indicates that the Pipeline Link was copied to your clipboard:

The notification message contains the name of the Pipeline, and the smaller contextual message below is the location of the pipeline and the Org name in parentheses.

In your clipboard, a link like the following will be present and ready to be shared:

Writing the extension

I had a short list of requirements for this extension:

It needed to retrieve the rendered HTML of the Designer page so it could be parsed for the information required to build the links.
The generated link should be copied to user’s clipboard.
A notficiation should be given to the user when the link has been copied.
The extension could be triggered with a keyboard shortcut.
The extension should only be enabled for the SnapLogic Designer.

The Chrome Extension Developer Documentation is required reading, so I’m assuming the reader has done that.

For the above requirements, 3 files would comprise the entirety of the functionality needed to cover the above:

the background.js script, containing the majority of the JavaScript logic for the extension.
the content.js script, which is injected into HTML document of the user’s current browser tab, and executes the callback function, passing back the page’s rendered HTML. This is how the background.js file receives the HTML to generate the Pipeline link.
the manifest.json file, which specifies the browser permissions required, the background scripts, that it is a page_action (it applies to a specific page rather than some generic behavior), the image files used as icons, and the keyboard shortcut commands.

The Manifest

{
  "name": "SnapLogic Pipeline Linker",
  "description": "Copy the Active Pipeline's Link to the Clipboard",
  "version": "1.0",
  "manifest_version": 2,
  "permissions": [
    "activeTab",
    "notifications",
    "declarativeContent"
  ],
  "background": {
    "scripts": [
      "js/background.js"
    ],
    "persistent": false
  },
  "page_action": {
    "default_icon": {
      "19": "img/icon-19.png",
      "38": "img/icon-38.png"
    },
    "default_title": "Copy the link to the active pipeline"
  },
  "icons": {
    "128": "img/icon-128.png",
    "48": "img/icon-48.png",
    "16": "img/icon-16.png"
  },
  "commands": {
    "_execute_page_action": {
      "description": "Copy the link to the active pipeline",
      "suggested_key": {
        "default": "Ctrl+I",
        "mac": "MacCtrl+I"
      }
    }
  }
}

The above manifest is quite simple - most of the properties are self-explanatory, but I’ll call out the following that are worth further discussion:

permissions includes activeTab (the extension only cares about the tab the user is currently using), notifications (for pop-ups), and declarativeContent (to use the metadata for the current tab - especially the URL - to control whether the extension is enabled or not).

The notifications permission results in the user being prompted with the following reasonable request when installing the extension:

Both activeTab and declarativeContent have the advantage of not generating any additional permission warnings to the user.
background lists the scripts that run when the extension is active. A single script (background.js) is listed.

The "persistent": false identifies this as an Event Page - a more performant and efficient solution that allows Chrome to load and unload the extension’s scripts automatically, based on the activity of the current page.
commands enables keyboard shortcuts to be defined and configured. When the user performs the shorcut (either using the default, suggested keys or after customising it themselves in their Chrome settings), the reserved _execute_page_action event is triggered.

This event is special in that it doesn’t need to be handled - the extension plugin framework takes care of the execution itself.

Enabling the Extension only for particular pages

Page actions “represent actions that can be taken on the current page, but that aren’t applicable to all pages.” Therefore the pageAction API was used to add listeners etc.

The declarativeContent permission mentioned earlier allows use of the chrome.declarativeContent API. This API (available since Chrome 33) allows the extension “to take actions depending on the content of a page, without requiring permission to read the page’s content.”

// only enable the extension on SnapLogic Designer
chrome.runtime.onInstalled.addListener(function () {
    chrome.declarativeContent.onPageChanged.removeRules(undefined, function () {
        chrome.declarativeContent.onPageChanged.addRules([{
            conditions: [
                new chrome.declarativeContent.PageStateMatcher({
                    pageUrl: {hostSuffix: "elastic.snaplogic.com", pathEquals: "/sl/designer.html"},
                })
            ],
            actions: [new chrome.declarativeContent.ShowPageAction()]
        }]);
    });
});

The chrome.declarativeContent.PageStateMatcher allows definiting the hostSuffix and path that, when matched, enables the extension. When the condition does not match, the extension icon shows us as grayed out and nothing happens when it is clicked:

Parsing the HTML

The background.js script isn’t able to access the HTML of the active tab. Instead, another script (content.js) is injected into the current page which then communicates with background.js by passing messages:

// when the PipelineLinker extension is triggered
chrome.pageAction.onClicked.addListener(function (browserTab) {
    // execute the script that gets injected into page of the current tag
    chrome.tabs.executeScript(null, {file: "js/content.js"}, function () {
        // send a message to content script
        chrome.tabs.sendMessage(browserTab.id, "Background page started.", function (response) {
            // receive the HTML from the tab's page and convert it to a DOM Document
            var doc = htmlToDocument(response);
			...
        });
    });
});

function htmlToDocument(str) {
    // HTML5 <template> allows any element underneath it
    var template = document.createElement("template");
    if (template.content) {
        template.innerHTML = str;
        return template.content;
    }
}

To receive the HTML of the target page the extension wishes to interact with, the above code runs as part of the background.js script. It adds a listener to the page action that, when triggered, sends a message (using Chrome’s Message Passing API) to the content.js script.

The content.js script is extremely simple:

chrome.runtime.onMessage.addListener(function (msg, _, sendResponse) {
    sendResponse(document.all[0].outerHTML);
});

On receiving the message sent by the background.js, it executes the callback function, passing it the entire HTML (as a String) of the active tab.

The anonymous callback function that was passed in background.js then converts that HTML to a Document by using HTML5’s <template> element, which “allows any other element type as a child”.

The rest of the function is then just using selectors to navigate to the portion of the HTML of interest to the extension to build and copy the link, and the send the browser notification.

Copying to the Clipboard

This was surprisingly difficult to identify with Google searches. I tried a few different methods that did not work - it must have been something particular with how background.js executes in the context of the Chrome Extension architecture vs the browser. Anyway, I finally found a function that worked well:

function copyToClipboard(pipelineLink) {
    const input = document.createElement("input");
    input.style.position = "fixed";
    input.style.opacity = 0;
    input.value = pipelineLink;
    document.body.appendChild(input);
    input.select();
    document.execCommand("Copy");
    document.body.removeChild(input);
};

Triggering a Notification

Again, very straightforward thanks to the chrome.notifications API:

function createNotification(pipelineName, pipelineLocation) {
    var opt = {
        type: "basic",
        title: "Pipeline Link Copied",
        message: pipelineName,
        contextMessage: pipelineLocation,
        iconUrl: "img/icon-80.png"
    };
    chrome.notifications.create(null, opt, function (notificationId) {
        timer = setTimeout(function () {
            chrome.notifications.clear(notificationId);
        }, 3000);
    });
}

Being able to control the duration of the popup was a nice extra.

Keyboard Shortcuts

Defining a keyboard shortcut is done entirely within the manifest.json file:

{
  ...
  "commands": {
    "_execute_page_action": {
      "description": "Copy the link to the active pipeline",
      "suggested_key": {
        "default": "Ctrl+I",
        "mac": "MacCtrl+I"
      }
    }
  }
 }

When the user performs the shorcut, the reserved _execute_page_action event is triggered. The suggested_key lists the default and/or OS-specific commands that are bound to the extension out-of-the-box. However, the user can change these to whatever they wish with a link at the bottom their `chrome://extensions page:

There were two “gotchas” worth calling out; on Mac, I couldn’t get a command with two or more combinations of Control/Shift/Command keys plus a letter (as per this SO answer) - hence the default of Ctrl+i. Other responses on that thread appear to have a different experience.

The other thing to watch out for is when OS-global shortcuts override your selected shortcut combination. My Evernote app had shortcuts defined that were interfering, sometimes silently.

Final Thoughts

Publishing the extension to the Chrome Web Store was straightforward. A fair number of logo sizes were recommended. My graphical skills are minimal but I was able to knock something out easily enough.

Overall, creating a Chrome Extension is a concise but pleasant development experience and an extremely effective way of distributing “polyfill”-style features for existing web applications.

Everything you ever wanted to know about SSL (but were afraid to ask)

Tue, 05 Jan 2016 23:17:53 +0000

Or perhaps more accurately, “practical things I’ve learned about SSL”. This post (and the companion Spring Boot application) will demonstrate using SSL certificates to validate and authenticate connections to secure endpoints over HTTPS for some common use cases (web servers, browser authentication, unit and integration testing). It shows how to configure Apache HTTP server for two-way SSL, unit testing SSL authentication with Apache’s HttpClient and HttpServer (Java), and integration testing a REST API within a Spring Boot application running on an embedded Tomcat container.

There are lots of ways for a client to authenticate itself against a server, including basic authentication, form-based authentication, and OAuth.

To prevent exposing user credentials over the wire, the client communicates with the server over HTTPS, and the server’s identify is confirmed by validating its SSL certificate. The server doesn’t necessarily care who the client is, just as long as they have the correct credentials.

An even higher level of security can be gained with using SSL certificates for both the client and the server.

Two-way SSL authentication (also known as “mutual authentication”, and “TLS/SSL with client certificates”) refers to two parties authenticating each other through verifying provided digital certificates, so that both parties are assured of the other’s identity.

Terminology
Authentication with SSL

One-way SSL (server -> client)
Two-way SSL (server <-> client)
File Formats for Certs and Keys
Tools
PKI and the SSL Certificate Chain ("the Chain of Trust")

Create a local SSL server with Apache

Configuring Apache: Creating a Site
Configuring SSL

Unit Testing SSL Authentication with Apache’s HttpClient and HttpServer

Java KeyStores (JKS)
Creating KeyStores and TrustStores with Keytool
One-Way SSL
Two-Way SSL (Client Certificates)

Two-Way SSL Authentication with Spring Boot, embedded Tomcat and RestTemplate

Integration Testing SSL Authentication with Spring’s TestRestTemplate

Two-Way SSL with SnapLogic's REST Snap

Terminology

TLS vs SSL

TLS is the successor to SSL. It is a protocol that ensures privacy between communicating applications. Unless otherwise stated, in this document consider TLS and SSL as interchangable.

Certificate (cert)

The public half of a public/private key pair with some additional metadata about who issued it etc. It may be freely given to anyone.

Private Key

A private key can verify that its corresponding certificate/public key was used to encrypt data. It is never given out publicly.

Certificate Authority (CA)

A company that issues digital certificates. For SSL/TLS certificates, there are a small number of providers (e.g. Symantec/Versign/Thawte, Comodo, GoDaddy, LetsEncrypt) whose certificates are included by most browsers and Operating Systems. They serve the purpose of a “trusted third party”.

Certificate Signing Request (CSR)

A file generated with a private key. A CSR can be sent to a CA to request to be signed. The CA uses its private key to digitally sign the CSR and create a signed cert. Browsers can then use the CA’s cert to validate the new cert has been approved by the CA.

X.509

A specification governing the format and usage of certificates.

Authentication with SSL

SSL is the standard security technology for establishing an encrypted link between a web server and a browser. Normally when a browser (the client) establishes an SSL connection to a secure web site, only the server certificate is checked. The browser either relies on itself or the operating system providing a list of certs that have been designated as root certificates and to be trusted as CAs.

One-way SSL authentication (server -> client)

Client and server use 9 handshake messages to establish the encrypted channel prior to message exchanging:

Client sends ClientHello message proposing SSL options.
Server responds with ServerHello message selecting the SSL options.
Server sends Certificate message, which contains the server’s certificate.
Server concludes its part of the negotiation with ServerHelloDone message.
Client sends session key information (encrypted with server’s public key) in ClientKeyExchange message.
Client sends ChangeCipherSpec message to activate the negotiated options for all future messages it will send.
Client sends Finished message to let the server check the newly activated options.
Server sends ChangeCipherSpec message to activate the negotiated options for all future messages it will send.
Server sends Finished message to let the client check the newly activated options.

Two-way SSL authentication (server <-> client)

Client and server use 12 handshake messages to establish the encrypted channel prior to message exchanging:

Client sends ClientHello message proposing SSL options.
Server responds with ServerHello message selecting the SSL options.
Server sends Certificate message, which contains the server’s certificate.
Server requests client’s certificate in CertificateRequest message, so that the connection can be mutually authenticated.
Server concludes its part of the negotiation with ServerHelloDone message.
Client responds with Certificate message, which contains the client’s certificate.
Client sends session key information (encrypted with server’s public key) in ClientKeyExchange message.
Client sends a CertificateVerify message to let the server know it owns the sent certificate.
Client sends ChangeCipherSpec message to activate the negotiated options for all future messages it will send.
Client sends Finished message to let the server check the newly activated options.
Server sends ChangeCipherSpec message to activate the negotiated options for all future messages it will send.
Server sends Finished message to let the client check the newly activated options.

File Formats for Certs and Keys

Privacy-Enhanced Mail (PEM)

PEM is just Distinguished Encoding Rules (DER) that has been Base64 encoded. Used for keys and certificates.

PKCS12

PKCS12 is a password-protected format that can contain multiple certificates and keys.

Java KeyStore (JKS)

Java version of PKCS12 and also password protected. Entries in a JKS file must have an “alias” that is unique. If an alias is not specified, “mykey” is used by default. It’s like a database for certs and keys.

Tools

OpenSSL

An open source toolkit implementing the SSL (v2/v3) and TLS (v1) protocols, as well as a full-strength general purpose cryptography library.

Keytool

Manages a Java KeyStore of cryptographic keys, X.509 certificate chains, and trusted certificates. Ships with the JDK.

XCA

A graphical tool to create and manage certificates.

PKI and the SSL Certificate Chain (“the Chain of Trust”)

All SSL/TLS connections rely on a chain of trust called the SSL Certificate Chain. Part of PKI (Public Key Infrastructure), this chain of trust is established by certificate authorities (CAs) who serve as trust anchors that verify the validity of the systems being communicated with. Each client (browser, OS, etc.) ships with a list of trusted CAs.

CA-signed Certificates

In the above example, the wildcard certificate for “*.elastic.snaplogic.com” has been issued by the “Go Daddy Secure Certificate Authority - G2” intermediate CA, which in turn was issued by the “Go Daddy Root Certificate Authority - G2” root CA.

Many organizations will create their own internal, self-signed root CA to be used to sign certificates for PKI use within that organization. Then, if each system trusts that CA, the certificates that are issued and signed by that CA will be trusted too.

To trust a system that presents a the above certificate at a particular domain (e.g. https://elastic.snaplogic.com), the client system must trust both the intermediate CA and the root CA (the public certs of those CAs must exist in the client system’s trust/CA store), as well as verifying the chain is valid (signatures match, domain names match, and other requirements of the X.509 standard).

Once a client trusts the intermediate and root CAs, all valid certificates signed by those CAs will be trusted by the client.

If only particular certificates signed by a trusted CA should be trusted, then either limiting the certificates in the CA store, or checking for certain certificate fingerprints, etc. should be considered instead.

Self-signed Certificates

Self-signed certificates have a chain length of 1 - they are not signed by a CA but by the certificate creator itself. All root certificates are self-signed (a chain has to start somewhere).

For example, to create a self-signed certificate (plus private key) for localhost, the following OpenSSL command may be used:

| root@SL-MBP-RHOWLETT.local:/private/etc/apache2/ssl 
| => openssl req -x509 -nodes -days 365 -newkey rsa:2048 -keyout localhost_self-signed.key -out localhost_self-signed.pem
Generating a 2048 bit RSA private key
................................................+++
..............................+++
writing new private key to 'localhost_self-signed.key'
-----
You are about to be asked to enter information that will be incorporated
into your certificate request.
What you are about to enter is what is called a Distinguished Name or a DN.
There are quite a few fields but you can leave some blank
For some fields there will be a default value,
If you enter '.', the field will be left blank.
-----
Country Name (2 letter code) [AU]:US
State or Province Name (full name) [Some-State]:Colorado
Locality Name (eg, city) []:Boulder
Organization Name (eg, company) [Internet Widgits Pty Ltd]:SnapLogic
Organizational Unit Name (eg, section) []:SnapTeam
Common Name (e.g. server FQDN or YOUR name) []:localhost
Email Address []:

The "Common Name" must match the domain that will be presenting the certificate e.g. localhost

To create a .p12 file (that can be imported and trusted within OS X’s Keychain application for example):

| root@SL-MBP-RHOWLETT.local:/private/etc/apache2/ssl 
| => openssl pkcs12 -export -in /etc/apache2/ssl/localhost_self-signed.pem -inkey /etc/apache2/ssl/localhost_self-signed.key -name "SelfSignedServer" -out localhost_self-signed.p12
Enter Export Password:
Verifying - Enter Export Password:

When you want to specifically control a small number of certificates to use within an internal network you control, self-signed certificates can be very useful.

Create a local SSL server with Apache

Modified from: https://gist.github.com/jonathantneal/774e4b0b3d4d739cbc53

Configuring Apache

Switch to root:

sudo su

Within Terminal, start Apache:

apachectl start

In a web browser, visit http://localhost. You should see a message stating that It works!

Configuring Apache for HTTP: Setting up a port 80 Virtual Host

Within Terminal, edit the Apache Configuration:

vi /etc/apache2/httpd.conf

Enable SSL by uncommenting line 143:

LoadModule ssl_module libexec/apache2/mod_ssl.so

Within your editor, replace line 212 to suppress messages about the server’s fully qualified domain name:

ServerName localhost

Next, uncomment line 160 and line 499 to enable Virtual Hosts.

LoadModule vhost_alias_module libexec/apache2/mod_vhost_alias.so

Include /private/etc/apache2/extra/httpd-vhosts.conf

Uncomment line 518 to include httpd-ssl.conf (to listen on port 443):

Include /private/etc/apache2/extra/httpd-ssl.conf

Optionally, uncomment line 169 to enable PHP.

LoadModule php5_module libexec/apache2/libphp5.so

Within Terminal, edit the Virtual Hosts

vi /etc/apache2/extra/httpd-vhosts.conf

Within your editor, replace the entire contents of this file with the following, replacing rhowlett with your user name.

<VirtualHost *:80>
    ServerName localhost
    DocumentRoot "/Users/rhowlett/Sites/localhost"

    <Directory "/Users/rhowlett/Sites/localhost">
        Options Indexes FollowSymLinks
        AllowOverride All
        Order allow,deny
        Allow from all
        Require all granted
    </Directory>
</VirtualHost>

Within Terminal, restart Apache:

apachectl restart

Configuring Apache: Creating a Site

Within Terminal, Create a Sites directory, which will be the parent directory of many individual Site subdirectories:

mkdir ~/Sites

Next, create a localhost subdirectory within Sites, which will be our first site:

mkdir ~/Sites/localhost

Finally, create an HTML document within localhost:

echo "<h1>localhost works</h1>" > ~/Sites/localhost/index.html

Now, in a web browser, visit http://localhost. You should see a message stating that localhost works.

Configuring SSL

Note: I used snaplogic for all passwords below

Modified from: http://www.stefanocapitanio.com/configuring-two-way-authentication-ssl-with-apache/

Within Terminal, create a SSL directory:

mkdir /etc/apache2/ssl

cd /etc/apache2/ssl
mkdir certs private

Create the CA cert

Create a database to keep track of each certificate signed:

echo '100001' > serial
touch certindex.txt

Make a custom config file for openssl to use:

vi openssl.cnf

#
# OpenSSL configuration file.
#

# Establish working directory.

dir = .

[ ca ]
default_ca = CA_default

[ CA_default ]
serial = $dir/serial
database = $dir/certindex.txt
new_certs_dir = $dir/certs
certificate = $dir/cacert.pem
private_key = $dir/private/cakey.pem
default_days = 365
default_md = sha512
preserve = no
email_in_dn = no
nameopt = default_ca
certopt = default_ca
policy = policy_match

[ policy_match ]
countryName = match
stateOrProvinceName = match
organizationName = match
organizationalUnitName = optional
commonName = supplied
emailAddress = optional

[ req ]
default_bits = 2048 # Size of keys
default_keyfile = key.pem # name of generated keys
default_md = sha512 # message digest algorithm
string_mask = nombstr # permitted characters
distinguished_name = req_distinguished_name
req_extensions = v3_req

[ req_distinguished_name ]
countryName = Country Name (2 letter code)
countryName_default = US
countryName_min = 2
countryName_max = 2

stateOrProvinceName = State or Province Name (full name)
stateOrProvinceName_default = Colorado

localityName = Locality Name (eg, city)
localityName_default = Boulder

0.organizationName = Organization Name (eg, company)
0.organizationName_default = SnapLogic

# we can do this but it is not needed normally :-)
#1.organizationName = Second Organization Name (eg, company)
#1.organizationName_default = World Wide Web Pty Ltd

organizationalUnitName = Organizational Unit Name (eg, section)
organizationalUnitName_default = SnapTeam

commonName = Common Name (eg, YOUR name)
commonName_max = 64
commonName_default = localhost 

emailAddress = Email Address
emailAddress_max = 64

# SET-ex3 = SET extension number 3

[ req_attributes ]
challengePassword = A challenge password
challengePassword_min = 4
challengePassword_max = 20

unstructuredName = An optional company name

[ req_distinguished_name ]
countryName = Country Name (2 letter code)
countryName_default = US 
countryName_min = 2
countryName_max = 2

stateOrProvinceName = State or Province Name (full name)
stateOrProvinceName_default = Colorado

localityName = Locality Name (eg, city)

0.organizationName = Organization Name (eg, company)
0.organizationName_default = SnapLogic

# we can do this but it is not needed normally :-)
#1.organizationName = Second Organization Name (eg, company)
#1.organizationName_default = World Wide Web Pty Ltd

organizationalUnitName = Organizational Unit Name (eg, section)
organizationalUnitName_default = SnapTeam

commonName = Common Name (eg, YOUR name)
commonName_max = 64
commonName_default = localhost

emailAddress = Email Address
emailAddress_max = 64

# SET-ex3 = SET extension number 3

[ req_attributes ]
challengePassword = A challenge password
challengePassword_min = 4
challengePassword_max = 20

unstructuredName = An optional company name
[ v3_ca ]
basicConstraints = CA:TRUE
subjectKeyIdentifier = hash
authorityKeyIdentifier = keyid:always,issuer:always

[ v3_req ]
basicConstraints = CA:FALSE
subjectKeyIdentifier = hash

Note that I’ve set default_md to sha512 so that modern browsers won’t complain about Weak Signature Algorithms.

Create a CA by creating a root certificate. This will create the private key (private/cakey.pem) and the public key (cacert.pem, a.k.a. the certificate) of the root CA. Use the default localhost for the common name:

| root@SL-MBP-RHOWLETT.local:/private/etc/apache2/ssl 
| => openssl req -new -x509 -extensions v3_ca -keyout private/cakey.pem -out cacert.pem -days 365 -config ./openssl.cnf
Generating a 2048 bit RSA private key
................................+++
.............................+++
writing new private key to 'private/cakey.pem'
Enter PEM pass phrase:
Verifying - Enter PEM pass phrase:
phrase is too short, needs to be at least 4 chars
Enter PEM pass phrase:
Verifying - Enter PEM pass phrase:
-----
You are about to be asked to enter information that will be incorporated
into your certificate request.
What you are about to enter is what is called a Distinguished Name or a DN.
There are quite a few fields but you can leave some blank
For some fields there will be a default value,
If you enter '.', the field will be left blank.
-----
Country Name (2 letter code) [US]:
State or Province Name (full name) [Colorado]:
Locality Name (eg, city) [Milan]:Boulder
Organization Name (eg, company) [Organization default]:SnapLogic
Organizational Unit Name (eg, section) [SnapLogic]:SnapTeam 
Common Name (eg, YOUR name) [localhost]:
Email Address []:

Create the Server cert

Create a key and signing request for the server. This will create the CSR for the server (server-req.pem) and the server’s private key (private/server-key.pem). Use the default localhost for the common name:

| root@SL-MBP-RHOWLETT.local:/etc/apache2/ssl 
| => openssl req -new -nodes -out server-req.pem -keyout private/server-key.pem -days 365 -config openssl.cnf 
Generating a 2048 bit RSA private key
......+++
..................................+++
writing new private key to 'private/server-key.pem'
-----
You are about to be asked to enter information that will be incorporated
into your certificate request.
What you are about to enter is what is called a Distinguished Name or a DN.
There are quite a few fields but you can leave some blank
For some fields there will be a default value,
If you enter '.', the field will be left blank.
-----
Country Name (2 letter code) [US]:
State or Province Name (full name) [Colorado]:
Locality Name (eg, city) [Boulder]:
Organization Name (eg, company) [SnapLogic]:
Organizational Unit Name (eg, section) [SnapTeam]:
Common Name (eg, YOUR name) [localhost]:
Email Address []:

Have the CA sign the server’s CSR. This will create the server’s public certificate (server-cert.pem):

| root@SL-MBP-RHOWLETT.local:/etc/apache2/ssl 
| => openssl ca -out server-cert.pem -days 365 -config openssl.cnf -infiles server-req.pem 
Using configuration from openssl.cnf
Enter pass phrase for ./private/cakey.pem:
Check that the request matches the signature
Signature ok
The Subject's Distinguished Name is as follows
countryName           :PRINTABLE:'US'
stateOrProvinceName   :PRINTABLE:'Colorado'
localityName          :PRINTABLE:'Boulder'
organizationName      :PRINTABLE:'SnapLogic'
organizationalUnitName:PRINTABLE:'SnapTeam'
commonName            :PRINTABLE:'localhost'
Certificate is to be certified until Oct  4 16:30:23 2016 GMT (365 days)
Sign the certificate? [y/n]:y


1 out of 1 certificate requests certified, commit? [y/n]y
Write out database with 1 new entries
Data Base Updated

Create the Client cert

Each client will create a key and signing request. We will just create one for now. You must use a different common name than the server/CA - here I’m using client. This will create the client’s CSR (client-req.pem) and the client’s private key (private/client-key.pem):

| root@SL-MBP-RHOWLETT.local:/etc/apache2/ssl 
| => openssl req -new -nodes -out client-req.pem -keyout private/client-key.pem -days 365 -config openssl.cnf 
Generating a 2048 bit RSA private key
....................................................+++
...........+++
writing new private key to 'private/client-key.pem'
-----
You are about to be asked to enter information that will be incorporated
into your certificate request.
What you are about to enter is what is called a Distinguished Name or a DN.
There are quite a few fields but you can leave some blank
For some fields there will be a default value,
If you enter '.', the field will be left blank.
-----
Country Name (2 letter code) [US]:
State or Province Name (full name) [Colorado]:
Locality Name (eg, city) [Boulder]:
Organization Name (eg, company) [SnapLogic]:
Organizational Unit Name (eg, section) [SnapTeam]:
Common Name (eg, YOUR name) [localhost]:client
Email Address []:

Have the CA sign the client’s CSR. This will create the client’s public certificate (client-cert.pem):

| root@SL-MBP-RHOWLETT.local:/etc/apache2/ssl 
| => openssl ca -out client-cert.pem -days 365 -config openssl.cnf -infiles client-req.pem 
Using configuration from openssl.cnf
Enter pass phrase for ./private/cakey.pem:
Check that the request matches the signature
Signature ok
The Subject's Distinguished Name is as follows
countryName           :PRINTABLE:'US'
stateOrProvinceName   :PRINTABLE:'Colorado'
localityName          :PRINTABLE:'Boulder'
organizationName      :PRINTABLE:'SnapLogic'
organizationalUnitName:PRINTABLE:'SnapTeam'
commonName            :PRINTABLE:'client'
Certificate is to be certified until Oct  4 16:40:01 2016 GMT (365 days)
Sign the certificate? [y/n]:y

1 out of 1 certificate requests certified, commit? [y/n]y
Write out database with 1 new entries
Data Base Updated

Finally, create the PKCS12 file using the client’s private key, the client’s public cert and the CA cert. This will create the (Mac-friendly) PKCS12 file (client-cert.p12):

| root@SL-MBP-RHOWLETT.local:/etc/apache2/ssl 
| => openssl pkcs12 -export -in client-cert.pem -inkey private/client-key.pem -certfile cacert.pem -name "Client" -out client-cert.p12 
Enter Export Password:
Verifying - Enter Export Password:

Configuring Apache for HTTPS and one-way SSL auth

As root:

vi /etc/apache2/extra/httpd-vhosts.conf

Add a Virtual Host for port 443 and enable SSL:

<VirtualHost *:80>
    ServerName localhost
    DocumentRoot "/Users/rhowlett/Sites/localhost"

    <Directory "/Users/rhowlett/Sites/localhost">
        Options Indexes FollowSymLinks
        AllowOverride All
        Order allow,deny
        Allow from all
        Require all granted
    </Directory>
</VirtualHost>

<VirtualHost *:443>
    ServerName localhost
    DocumentRoot "/Users/rhowlett/Sites/localhost"

    SSLCipherSuite HIGH:MEDIUM:!aNULL:!MD5
    SSLEngine on
    SSLCertificateFile /etc/apache2/ssl/server-cert.pem
    SSLCertificateKeyFile /etc/apache2/ssl/private/server-key.pem

    <Directory "/Users/rhowlett/Sites/localhost">
        Options Indexes FollowSymLinks
        AllowOverride All
        Order allow,deny
        Allow from all
        Require all granted
    </Directory>
</VirtualHost>

Restart Apache:

apachectl restart

(Mac) Install the CA cert into Keychain Access

Open /etc/apache2/ssl in Finder:

Open the CA cert (cacert.pem) by double-clicking it to install it to Keychain Access:

Mark it as trusted:

Open your browser to https://localhost and you should see a successful secure connection:

Configuring Apache for two-way SSL auth

As root:

vi /etc/apache2/extra/httpd-vhosts.conf

Add the SSLVerifyClient, SSLCertificateFile, and SSLCACertificateFile options:

<VirtualHost *:80>
    ServerName localhost
    DocumentRoot "/Users/rhowlett/Sites/localhost"

    <Directory "/Users/rhowlett/Sites/localhost">
        Options Indexes FollowSymLinks
        AllowOverride All
        Order allow,deny
        Allow from all
        Require all granted
    </Directory>
</VirtualHost>

<VirtualHost *:443>
    ServerName localhost
    DocumentRoot "/Users/rhowlett/Sites/localhost"

    SSLCipherSuite HIGH:MEDIUM:!aNULL:!MD5
    SSLEngine on
    SSLCertificateFile /etc/apache2/ssl/server-cert.pem
    SSLCertificateKeyFile /etc/apache2/ssl/private/server-key.pem

    SSLVerifyClient require
    SSLVerifyDepth 10
    SSLCACertificateFile /etc/apache2/ssl/cacert.pem

    <Directory "/Users/rhowlett/Sites/localhost">
        Options Indexes FollowSymLinks
        AllowOverride All
        Order allow,deny
        Allow from all
        Require all granted
    </Directory>
</VirtualHost>

Restart Apache:

apachectl restart

OpenSSL can now confirm that the two-way SSL handshake can be successfully completed:

| rhowlett@SL-MBP-RHOWLETT.local:~/Downloads 
| => openssl s_client -connect localhost:443 -tls1 -cert /etc/apache2/ssl/client-cert.pem -key /etc/apache2/ssl/private/client-key.pem
CONNECTED(00000003)
depth=1 /C=US/ST=Colorado/L=Boulder/O=SnapLogic/OU=SnapTeam/CN=localhost
verify error:num=19:self signed certificate in certificate chain
verify return:0
---
Certificate chain
 0 s:/C=US/ST=Colorado/O=SnapLogic/OU=SnapTeam/CN=localhost
   i:/C=US/ST=Colorado/L=Boulder/O=SnapLogic/OU=SnapTeam/CN=localhost
 1 s:/C=US/ST=Colorado/L=Boulder/O=SnapLogic/OU=SnapTeam/CN=localhost
   i:/C=US/ST=Colorado/L=Boulder/O=SnapLogic/OU=SnapTeam/CN=localhost
---
Server certificate
-----BEGIN CERTIFICATE-----
MIIDPjCCAiYCAxAAATANBgkqhkiG9w0BAQ0FADBtMQswCQYDVQQGEwJVUzERMA8G
A1UECBMIQ29sb3JhZG8xEDAOBgNVBAcTB0JvdWxkZXIxEjAQBgNVBAoTCVNuYXBM
b2dpYzERMA8GA1UECxMIU25hcFRlYW0xEjAQBgNVBAMTCWxvY2FsaG9zdDAeFw0x
NTEwMDYxOTE1MTNaFw0xNjEwMDUxOTE1MTNaMFsxCzAJBgNVBAYTAlVTMREwDwYD
VQQIEwhDb2xvcmFkbzESMBAGA1UEChMJU25hcExvZ2ljMREwDwYDVQQLEwhTbmFw
VGVhbTESMBAGA1UEAxMJbG9jYWxob3N0MIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8A
MIIBCgKCAQEAyvia0x0Nd4tYyvoXEYtI3s/eLIQ3wFsOJIibNy70PLhp35gScQ69
MiIrVDYqIydVbInzyY5kuhttrUIrHCIiDwa5OqEiExJ+ollY9icnrMLrEXJqvv5C
/fduS5byC6StNg7xHQkYlYLUYMw8QQyCZFQVGXlxZeG6i086ffMYduFimkBAkNj5
/LkIwrOELpGnNcrOJxQEnLi8vmRI3oiCrgVc0ugrFBnoj3Tf6y3lx23fYgLbqf9c
bRCS6V3eppa/x9sezv9KQ+pDYly0bwKIcvJ9xLp7qPiO+smGGvS97Ec4NAif8y6v
pU92cPH32cv1p0AIDF0+GMOgVyAYZgSKQwIDAQABMA0GCSqGSIb3DQEBDQUAA4IB
AQBedsAvkB1yNLE2GCJWWQ19qEKOIBYCRQc2z29PgF/LAz5GVOIw/ZiN37C2vTob
jk1NnqfOx5aipQ5Pe5D2yfbarDl0kaqRn9MhBySi+oi3AgUZ5yL0x/nGF9O8jszJ
OM1FUC6qXKic5pR0qTrdXigONlKb0Au+l3z5dFMiqnNmDrNlI8kW1OXrwy/jvyPv
H1bHWAKYFTvHi2v7A0B96V1VvFBLbuQztckPQ3VpFDOwWhWLr2D90vxFd1Ea0SCi
3bysz4ax9XP0bmXJY+968nV31qQJMkk5/3rE5PWVZibsniccfdujgSQYl+yNA3sB
F5h6mCR6pAONZFo6+U3zARSb
-----END CERTIFICATE-----
subject=/C=US/ST=Colorado/O=SnapLogic/OU=SnapTeam/CN=localhost
issuer=/C=US/ST=Colorado/L=Boulder/O=SnapLogic/OU=SnapTeam/CN=localhost
---
Acceptable client certificate CA names
/C=US/ST=Colorado/L=Boulder/O=SnapLogic/OU=SnapTeam/CN=localhost
---
SSL handshake has read 4004 bytes and written 1539 bytes
---
New, TLSv1/SSLv3, Cipher is DHE-RSA-AES256-SHA
Server public key is 2048 bit
Secure Renegotiation IS supported
Compression: NONE
Expansion: NONE
SSL-Session:
    Protocol  : TLSv1
    Cipher    : DHE-RSA-AES256-SHA
    Session-ID: A1D9CE5273963BCF70503B499D7714ECE2B628CEE59CE554615743ACEEA8E281
    Session-ID-ctx: 
    Master-Key: 0920CEE1491E9A116B2DF959430890D449D49DA990A178C0AC980DD5AF359B7E1CDD4B2D8237C8F81BAE186BC06E7BB0
    Key-Arg   : None
    TLS session ticket:
    0000 - 5c 8f 01 d5 5d c1 62 d5-65 d7 8f 05 5f 47 d2 82   \...].b.e..._G..
    0010 - f0 fd 2c 88 be 58 25 6c-9e 9a 1e 78 6a b4 66 c4   ..,..X%l...xj.f.
    0020 - 0d 3d 31 04 97 a2 5f e7-6f 3c 9f c9 b1 44 6a ab   .=1..._.o<...Dj.
    0030 - 84 89 76 e4 63 9b 81 b7-c3 28 e0 95 c6 c3 f5 89   ..v.c....(......
    0040 - d5 f9 7f da df fb 12 f7-de 2a ec e8 c2 01 59 07   .........*....Y.
    0050 - 9e ad 91 56 91 34 88 73-66 d1 ea c1 72 dc 56 ee   ...V.4.sf...r.V.
    0060 - ee 61 fe 5e 38 f0 aa d6-3a 7d ad ef e6 be 2a 15   .a.^8...:}....*.
    0070 - dc cc 9f 04 5e e8 f9 2b-07 21 6b 0f da 9f 08 2e   ....^..+.!k.....
    0080 - 88 af 96 41 98 f3 ff 8a-01 66 1a 1d 61 47 1b e5   ...A.....f..aG..
    0090 - ec ab b7 af 79 aa 7d 25-ca e0 fa f4 2b 2e 9a dd   ....y.}%....+...
    00a0 - 95 0c 4b 35 d8 96 8b f0-1e 20 c1 c3 47 fc 65 ed   ..K5..... ..G.e.
    00b0 - 21 e4 50 59 1e 33 6a 5c-c6 27 f1 65 be 5b 0f 35   !.PY.3j\.'.e.[.5
    00c0 - 1d ba ac bf f5 9c d9 b7-32 87 11 ae b7 87 9b 52   ........2......R
    00d0 - bb 00 6b 66 af e2 94 45-e3 8f fb e0 b4 c6 d7 5a   ..kf...E.......Z
    00e0 - f8 1d 7a af e3 ee bb 6b-93 ff 46 af ed 86 bc f8   ..z....k..F.....
    00f0 - 6d e2 c9 60 eb 61 8e b9-7e bd 4d bb 1e 01 95 d2   m..`.a..~.M.....
    0100 - f5 d5 ee 82 10 4a 1d 23-9e 94 d7 0b 46 e4 d4 32   .....J.#....F..2
    0110 - 11 92 76 4a 94 9e a3 61-21 9b 4c 49 6c df 7b 18   ..vJ...a!.LIl.{.
    0120 - b7 49 66 bd 48 0d eb 9a-ad e9 32 c7 b9 6d 70 1a   .If.H.....2..mp.
    0130 - c7 a1 25 21 b4 f1 03 5b-80 83 e9 da 8d 56 f1 d9   ..%!...[.....V..
    0140 - 8b c5 32 b7 3a 67 5b 9c-51 84 a0 09 04 4f 48 60   ..2.:g[.Q....OH`
    0150 - 27 c0 fe 1c 45 7a 3b b2-22 8d ed 65 72 23 8a bf   '...Ez;."..er#..
    0160 - e3 09 eb 78 98 ec 08 06-9d 37 02 1a 4b ae cd 3a   ...x.....7..K..:
    0170 - 9c a4 bd 5d 47 5e d3 d7-7b 89 7b 97 78 a6 4c 10   ...]G^..{.{.x.L.
    0180 - bf 3e ed 1f f4 fe e5 97-90 ee 31 58 5f ff c6 c2   .>........1X_...
    0190 - 61 b7 df 0a f5 27 c6 a8-ac 61 a3 d0 1e 3a 6a 42   a....'...a...:jB
    01a0 - a9 18 b4 fb 4b 25 87 62-97 26 48 35 d0 16 d1 06   ....K%.b.&H5....
    01b0 - 9d 82 b5 e2 7b f2 24 c5-83 a1 4b fe 8d 38 ae 30   ....{.$...K..8.0
    01c0 - 8e eb e1 ac 8b 48 fa 27-b0 e1 ce b3 17 62 69 f0   .....H.'.....bi.
    01d0 - 30 17 ae 31 9d bf 77 64-66 5b 13 8e a2 63 2e 58   0..1..wdf[...c.X
    01e0 - 02 10 26 e1 3b 0d 55 fc-3d 0f d5 08 2d 1e 28 0a   ..&.;.U.=...-.(.
    01f0 - c2 fd a2 f3 2a 40 25 ed-2b 06 2c 92 c3 78 a3 b3   ....*@%.+.,..x..
    0200 - 35 bc d9 6c 57 97 ca 93-0f f3 b8 e4 60 d8 99 b4   5..lW.......`...
    0210 - b8 ba ae b7 47 4a 59 84-5b f9 5e b2 11 44 42 bd   ....GJY.[.^..DB.
    0220 - e8 46 3d 1d 09 70 72 f6-23 df 89 f8 f7 b7 84 d2   .F=..pr.#.......
    0230 - 7d 42 0e 5d d7 76 c2 da-0b 61 f9 48 3c c9 5f ba   }B.].v...a.H<._.
    0240 - ab be 5f 82 2b 03 07 f1-83 12 69 ee 56 b5 7e 06   .._.+.....i.V.~.
    0250 - 03 d7 8e b3 70 7c 93 75-3d cd e0 a1 1b 8a 14 ef   ....p|.u=.......
    0260 - 91 c6 74 14 1e 16 4c 46-07 c5 62 04 70 a7 fd 5d   ..t...LF..b.p..]
    0270 - e5 67 d8 bf 43 bb 5e f3-7c 37 db 1a 66 cb ad 7d   .g..C.^.|7..f..}
    0280 - cc 30 e4 9b 35 30 b5 6c-d0 4b ba b2 8b 01 71 0e   .0..50.l.K....q.
    0290 - 0a af ec 4e 6a 1a f8 6f-b7 5e 2b b9 e9 ec b6 b6   ...Nj..o.^+.....
    02a0 - 38 1c 70 5c 86 bf ae a4-e6 41 9d c9 9f 40 e4 a0   8.p\.....A...@..
    02b0 - 4b 0d 3d ab 01 90 da 55-cb b8 c8 e6 94 8d 76 35   K.=....U......v5
    02c0 - 94 b5 e2 1a 7c 69 5c b3-ee 08 8b bd 3f 97 c4 31   ....|i\.....?..1
    02d0 - 72 8a 30 a8 c6 3e 74 74-dc 47 c1 d0 ce bd 0b 19   r.0..>tt.G......
    02e0 - f4 93 55 8c 1f 02 b3 6e-f3 4d 44 f1 cc f0 ef 2d   ..U....n.MD....-
    02f0 - 4d 16 92 a3 15 fe 69 db-cc b1 b5 6b d0 4a 49 fc   M.....i....k.JI.
    0300 - 67 9e 0c 47 96 08 0e f2-b2 5c 06 24 45 f3 6a 7d   g..G.....\.$E.j}
    0310 - 6e 1b 2b 9a 68 23 11 3a-43 79 8c 77 9e 98 be 38   n.+.h#.:Cy.w...8
    0320 - 9a 0e e1 a5 17 bd 0f 7b-e0 ac ca 94 ac 48 68 5c   .......{.....Hh\
    0330 - f1 2b 98 b5 8d 36 b6 4f-aa 6f e7 d4 4d a3 f0 4c   .+...6.O.o..M..L
    0340 - cb 09 92 91 01 b9 c2 f1-49 24 64 d3 14 2f a3 5f   ........I$d../._
    0350 - 74 6f c0 54 16 73 c8 40-33 bc 7e e9 3b d8 d5 7c   to.T.s.@3.~.;..|
    0360 - 78 49 5c 80 83 88 4e 4b-46 f2 7a 6b 62 c4 ca 42   xI\...NKF.zkb..B
    0370 - 18 b6 22 40 77 fc 26 0e-28 50 89 7a 14 49 ba b0   .."@w.&.(P.z.I..
    0380 - 2c d7 26 7a 30 f9 9b 90-ba 9a 1f 3b 80 1b 0b 25   ,.&z0......;...%
    0390 - f0 e7 83 83 55 1f 1e f0-71 5b 64 a4 1e 76 91 bb   ....U...q[d..v..
    03a0 - d9 19 f5 2d 2e 54 d7 3a-93 95 29 ae 44 09 e6 cd   ...-.T.:..).D...
    03b0 - ec 79 8d b6 3c 09 d5 05-8d fc 2b 79 88 37 25 92   .y..<.....+y.7%.
    03c0 - 73 ae e6 8a d6 0c 1a eb-7b b9 08 44 4e 81 67 36   s.......{..DN.g6
    03d0 - a6 3a 57 43 d0 ed dc 3e-bb 0f 87 02 f5 fe 80 bb   .:WC...>........
    03e0 - 28 17 6e 7e ad c4 d9 4c-0a 53 fa 41 d2 d2 7c 76   (.n~...L.S.A..|v
    03f0 - a4 95 10 26 1d 5b 7d 19-23 dd 28 a0 48 c1 96 d9   ...&.[}.#.(.H...

    Start Time: 1444189099
    Timeout   : 7200 (sec)
    Verify return code: 0 (ok)
---
closed

(Mac) Install the client PKCS12 file into Keychain Access

Open /etc/apache2/ssl in Finder and double-click the client PKCS12 file (client-cert.p12) to install it to Keychain Access:

Open https://localhost in your browser again and select the client cert when prompted:

You should then once again see a successful secure connection:

cURL will also work:

| => curl -v --cert /etc/apache2/ssl/client-cert.p12:snaplogic https://localhost
* Rebuilt URL to: https://localhost/
*   Trying ::1...
* Connected to localhost (::1) port 443 (#0)
* WARNING: SSL: Certificate type not set, assuming PKCS#12 format.
* Client certificate: client

| => curl -v --cert /etc/apache2/ssl/client-cert.p12:snaplogic https://localhost
* Rebuilt URL to: https://localhost/
*   Trying ::1...
* Connected to localhost (::1) port 443 (#0)
* WARNING: SSL: Certificate type not set, assuming PKCS#12 format.
* Client certificate: client
* TLS 1.0 connection using TLS_DHE_RSA_WITH_AES_256_CBC_SHA
* Server certificate: localhost
* Server certificate: localhost
> GET / HTTP/1.1
> Host: localhost
> User-Agent: curl/7.43.0
> Accept: */*
> 
< HTTP/1.1 200 OK
< Date: Tue, 05 Jan 2016 23:16:29 GMT
< Server: Apache/2.4.16 (Unix) PHP/5.5.29 OpenSSL/0.9.8zg
< Last-Modified: Fri, 02 Oct 2015 18:34:41 GMT
< ETag: "19-521236aaf5240"
< Accept-Ranges: bytes
< Content-Length: 25
< Content-Type: text/html
< 
<h1>localhost works</h1>
* Connection #0 to host localhost left intact

Unit Testing SSL Authentication with Apache’s HttpClient and HttpServer

Apache’s HttpComponents provides HttpClient, “an efficient, up-to-date, and feature-rich package implementing the client side of the most recent HTTP standards and recommendations.”

It also provides HttpCore, which includes an embedded HttpServer, which can be used for unit testing.

Generate a PKCS12 (.p12) file from the public server-cert.pem, the private server-key.pem, and the CA cert cacert.pem created above to be used by the local test HttpServer instance:

| rhowlett@SL-MBP-RHOWLETT.local:~/dev/robinhowlett/github/everything-ssl/src/main/resources/ssl 
| => openssl pkcs12 -export -in /etc/apache2/ssl/server-cert.pem -inkey /etc/apache2/ssl/private/server-key.pem -certfile /etc/apache2/ssl/cacert.pem -name "Server" -out server-cert.p12
Enter Export Password:
Verifying - Enter Export Password:

If you see unable to write 'random state', run sudo rm ~/.rnd and try again

Java KeyStores (JKS)

Java has its own version of PKCS12 called Java KeyStore (JKS). It is also password protected. Entries in a JKS file must have an “alias” that is unique. If an alias is not specified, “mykey” is used by default. It’s like a database for certs and keys.

For both the “KeyStore” and “TrustStore” fields in the REST SSL Account settings, we are going to use JKS files. The difference between them is for terminology reasons: KeyStores provide credentials, TrustStores verify credentials.

Clients will use certificates stored in their TrustStores to verify identities of servers. They will present certificates stored in their KeyStores to servers requiring them.

The JDK ships with a tool called Keytool. It manages a JKS of cryptographic keys, X.509 certificate chains, and trusted certificates.

Creating KeyStores and TrustStores with Keytool

Create the Server’s KeyStore from the PKCS12 file:

| rhowlett@SL-MBP-RHOWLETT.local:~/dev/robinhowlett/github/everything-ssl/src/main/resources/ssl 
| => keytool -importkeystore -deststorepass snaplogic -destkeypass snaplogic -destkeystore server_keystore.jks -srckeystore server-cert.p12 -srcstoretype PKCS12 -srcstorepass snaplogic -alias server

View the server keystore to confirm it now contains the server’s cert:

| rhowlett@SL-MBP-RHOWLETT.local:~/dev/robinhowlett/github/everything-ssl/src/main/resources/ssl 
| => keytool -list -v -keystore server_keystore.jks 
Enter keystore password:  

Keystore type: JKS
Keystore provider: SUN

Your keystore contains 1 entry

Alias name: server
Creation date: Jan 4, 2016
Entry type: PrivateKeyEntry
Certificate chain length: 2
Certificate[1]:
Owner: CN=localhost, OU=SnapTeam, O=SnapLogic, ST=Colorado, C=US
Issuer: CN=localhost, OU=SnapTeam, O=SnapLogic, L=Boulder, ST=Colorado, C=US
Serial number: 100001
Valid from: Tue Oct 06 13:15:13 MDT 2015 until: Wed Oct 05 13:15:13 MDT 2016
Certificate fingerprints:
	 MD5:  62:83:6B:84:1B:CB:DE:26:CA:E0:9D:E8:04:84:B6:C1
	 SHA1: AD:D4:27:FF:9A:68:77:25:95:C3:A2:BE:F6:22:AD:82:5C:2B:AF:EB
	 SHA256: 8D:8D:EA:E5:7C:7A:E9:42:C9:9E:71:2A:76:C7:BE:BE:34:CC:4A:CC:83:ED:FE:C8:8E:C6:06:D2:D8:89:59:4A
	 Signature algorithm name: SHA512withRSA
	 Version: 1
Certificate[2]:
Owner: CN=localhost, OU=SnapTeam, O=SnapLogic, L=Boulder, ST=Colorado, C=US
Issuer: CN=localhost, OU=SnapTeam, O=SnapLogic, L=Boulder, ST=Colorado, C=US
Serial number: e4e00ed07233a969
Valid from: Tue Oct 06 13:14:51 MDT 2015 until: Wed Oct 05 13:14:51 MDT 2016
Certificate fingerprints:
	 MD5:  F3:5E:28:E4:28:47:F2:EC:82:E2:BD:16:31:DC:90:02
	 SHA1: 6F:0F:49:BA:A9:30:01:E9:4C:60:B3:A1:85:7D:BB:C6:79:1F:41:7B
	 SHA256: A7:9D:25:E4:A6:34:8A:A3:5B:9A:CD:F3:62:D0:D8:2F:6A:A0:71:6A:6D:19:F3:04:A1:FD:BC:FB:21:40:DE:A1
	 Signature algorithm name: SHA512withRSA
	 Version: 3

Extensions: 

#1: ObjectId: 2.5.29.35 Criticality=false
AuthorityKeyIdentifier [
KeyIdentifier [
0000: 03 09 12 6E 8B DD 7A 80   FB F5 21 AB 75 D9 B8 49  ...n..z...!.u..I
0010: 79 5B 61 1F                                        y[a.
]
[CN=localhost, OU=SnapTeam, O=SnapLogic, L=Boulder, ST=Colorado, C=US]
SerialNumber: [    e4e00ed0 7233a969]
]

#2: ObjectId: 2.5.29.19 Criticality=false
BasicConstraints:[
  CA:true
  PathLen:2147483647
]

#3: ObjectId: 2.5.29.14 Criticality=false
SubjectKeyIdentifier [
KeyIdentifier [
0000: 03 09 12 6E 8B DD 7A 80   FB F5 21 AB 75 D9 B8 49  ...n..z...!.u..I
0010: 79 5B 61 1F                                        y[a.
]
]



*******************************************
*******************************************

Create the client’s truststore and import the server’s public certificate:

| rhowlett@SL-MBP-RHOWLETT.local:~/dev/robinhowlett/github/everything-ssl/src/main/resources/ssl 
| => keytool -import -v -trustcacerts -keystore client_truststore.jks -storepass snaplogic -alias server -file /etc/apache2/ssl/server-cert.pem
Owner: CN=localhost, OU=SnapTeam, O=SnapLogic, ST=Colorado, C=US
Issuer: CN=localhost, OU=SnapTeam, O=SnapLogic, L=Boulder, ST=Colorado, C=US
Serial number: 100001
Valid from: Tue Oct 06 13:15:13 MDT 2015 until: Wed Oct 05 13:15:13 MDT 2016
Certificate fingerprints:
	 MD5:  62:83:6B:84:1B:CB:DE:26:CA:E0:9D:E8:04:84:B6:C1
	 SHA1: AD:D4:27:FF:9A:68:77:25:95:C3:A2:BE:F6:22:AD:82:5C:2B:AF:EB
	 SHA256: 8D:8D:EA:E5:7C:7A:E9:42:C9:9E:71:2A:76:C7:BE:BE:34:CC:4A:CC:83:ED:FE:C8:8E:C6:06:D2:D8:89:59:4A
	 Signature algorithm name: SHA512withRSA
	 Version: 1
Trust this certificate? [no]:  yes
Certificate was added to keystore
[Storing client_truststore.jks]

View the client’s truststore to confirm it contains the server’s cert:

| rhowlett@SL-MBP-RHOWLETT.local:~/dev/robinhowlett/github/everything-ssl/src/main/resources/ssl 
| => keytool -list -v -keystore client_truststore.jks 
Enter keystore password:  

Keystore type: JKS
Keystore provider: SUN

Your keystore contains 1 entry

Alias name: server
Creation date: Jan 4, 2016
Entry type: trustedCertEntry

Owner: CN=localhost, OU=SnapTeam, O=SnapLogic, ST=Colorado, C=US
Issuer: CN=localhost, OU=SnapTeam, O=SnapLogic, L=Boulder, ST=Colorado, C=US
Serial number: 100001
Valid from: Tue Oct 06 13:15:13 MDT 2015 until: Wed Oct 05 13:15:13 MDT 2016
Certificate fingerprints:
	 MD5:  62:83:6B:84:1B:CB:DE:26:CA:E0:9D:E8:04:84:B6:C1
	 SHA1: AD:D4:27:FF:9A:68:77:25:95:C3:A2:BE:F6:22:AD:82:5C:2B:AF:EB
	 SHA256: 8D:8D:EA:E5:7C:7A:E9:42:C9:9E:71:2A:76:C7:BE:BE:34:CC:4A:CC:83:ED:FE:C8:8E:C6:06:D2:D8:89:59:4A
	 Signature algorithm name: SHA512withRSA
	 Version: 1


*******************************************
*******************************************

One-Way SSL

At this point we have enough to demonstrate one-way SSL with the local test HttpServer instance. The createLocalTestServer method instantiates an embedded HttpServer instance with an (optional) sslContext (null meaning HTTP-only) and a boolean “forceSSLAuth” indicating if client certificates are required or not:

protected HttpServer createLocalTestServer(SSLContext sslContext, boolean forceSSLAuth)
        throws UnknownHostException {
    final HttpServer server = ServerBootstrap.bootstrap()
            .setLocalAddress(Inet4Address.getByName("localhost"))
            .setSslContext(sslContext)
            .setSslSetupHandler(socket -> socket.setNeedClientAuth(forceSSLAuth))
            .registerHandler("*",
                    (request, response, context) -> response.setStatusCode(HttpStatus.SC_OK))
            .create();

    return server;
}

The getStore method loads the JKS files from the classpath, and the getKeyManagers and getTrustManagers methods turn that store into the respective Key- or TrustManager arrays that are used to initialize an SSLContext:

private static final String JAVA_KEYSTORE = "jks";

/**
 * KeyStores provide credentials, TrustStores verify credentials.
 *
 * Server KeyStores stores the server's private keys, and certificates for corresponding public
 * keys. Used here for HTTPS connections over localhost.
 *
 * Client TrustStores store servers' certificates.
 */
protected KeyStore getStore(final String storeFileName, final char[] password) throws
        KeyStoreException, IOException, CertificateException, NoSuchAlgorithmException {
    final KeyStore store = KeyStore.getInstance(JAVA_KEYSTORE);
    URL url = getClass().getClassLoader().getResource(storeFileName);
    InputStream inputStream = url.openStream();
    try {
        store.load(inputStream, password);
    } finally {
        inputStream.close();
    }

    return store;
}

/**
 * KeyManagers decide which authentication credentials (e.g. certs) should be sent to the remote
 * host for authentication during the SSL handshake.
 *
 * Server KeyManagers use their private keys during the key exchange algorithm and send
 * certificates corresponding to their public keys to the clients. The certificate comes from
 * the KeyStore.
 */
protected KeyManager[] getKeyManagers(KeyStore store, final char[] password) throws
        NoSuchAlgorithmException, UnrecoverableKeyException, KeyStoreException {
    KeyManagerFactory keyManagerFactory = KeyManagerFactory.getInstance(
            KeyManagerFactory.getDefaultAlgorithm());
    keyManagerFactory.init(store, password);

    return keyManagerFactory.getKeyManagers();
}

/**
 * TrustManagers determine if the remote connection should be trusted or not.
 *
 * Clients will use certificates stored in their TrustStores to verify identities of servers.
 * Servers will use certificates stored in their TrustStores to verify identities of clients.
 */
protected TrustManager[] getTrustManagers(KeyStore store) throws NoSuchAlgorithmException,
        KeyStoreException {
    TrustManagerFactory trustManagerFactory =
            TrustManagerFactory.getInstance(TrustManagerFactory.getDefaultAlgorithm());
    trustManagerFactory.init(store);

    return trustManagerFactory.getTrustManagers();
}

The SSLContext is created and initialized like so:

/*
Create an SSLContext for the server using the server's JKS. This instructs the server to
present its certificate when clients connect over HTTPS.
 */
protected SSLContext createServerSSLContext(final String storeFileName, final char[]
        password) throws CertificateException, NoSuchAlgorithmException, KeyStoreException,
        IOException, UnrecoverableKeyException, KeyManagementException {
    KeyStore serverKeyStore = getStore(storeFileName, password);
    KeyManager[] serverKeyManagers = getKeyManagers(serverKeyStore, password);
    TrustManager[] serverTrustManagers = getTrustManagers(serverKeyStore);

    SSLContext sslContext = SSLContexts.custom().useProtocol("TLS").build();
    sslContext.init(serverKeyManagers, serverTrustManagers, new SecureRandom());

    return sslContext;
}

The following unit test shows making a HTTPS request to the local test HttpServer instance and validating the server’s public certificate with the client’s truststore:

private static final boolean ONE_WAY_SSL = false; // no client certificates

private static final char[] KEYPASS_AND_STOREPASS_VALUE = "snaplogic".toCharArray();
private static final String SERVER_KEYSTORE = "ssl/server_keystore.jks";
private static final String CLIENT_TRUSTSTORE = "ssl/client_truststore.jks";
   
private CloseableHttpClient httpclient;

@Before
public void setUp() throws Exception {
    httpclient = HttpClients.createDefault();
}

@Test
public void httpsRequest_With1WaySSLAndValidatingCertsAndClientTrustStore_Returns200OK()
        throws Exception {
    SSLContext serverSSLContext =
            createServerSSLContext(SERVER_KEYSTORE, KEYPASS_AND_STOREPASS_VALUE);

    final HttpServer server = createLocalTestServer(serverSSLContext, ONE_WAY_SSL);
    server.start();

    String baseUrl = getBaseUrl(server);

    // The server certificate was imported into the client's TrustStore (using keytool -import)
    KeyStore clientTrustStore = getStore(CLIENT_TRUSTSTORE, KEYPASS_AND_STOREPASS_VALUE);

    SSLContext sslContext =
            new SSLContextBuilder().loadTrustMaterial(
                    clientTrustStore, new TrustSelfSignedStrategy()).build();

    httpclient = HttpClients.custom().setSSLContext(sslContext).build();

    /*
    The HTTP client will now validate the server's presented certificate using its TrustStore.
     Since the cert was imported to the client's TrustStore explicitly (see above), the
     certificate will validate and the request will succeed
     */
    try {
        HttpResponse httpResponse = httpclient.execute(
                new HttpGet("https://" + baseUrl + "/echo/this"));

        assertThat(httpResponse.getStatusLine().getStatusCode(), equalTo(200));
    } finally {
        server.stop();
    }
}

protected String getBaseUrl(HttpServer server) {
    return server.getInetAddress().getHostName() + ":" + server.getLocalPort();
}

The above unit test is included in the everything-ssl GitHub project, along with the following (which are useful to see the behavior when the SSL handshake fails, when server certificate validation is bypassed, malformed contexts etc.)

execute_WithNoScheme_ThrowsClientProtocolExceptionInvalidHostname
httpRequest_Returns200OK
httpsRequest_WithNoSSLContext_ThrowsSSLExceptionPlaintextConnection
httpsRequest_With1WaySSLAndValidatingCertsButNoClientTrustStore_ThrowsSSLException
httpsRequest_With1WaySSLAndTrustingAllCertsButNoClientTrustStore_Returns200OK
httpsRequest_With1WaySSLAndValidatingCertsAndClientTrustStore_Returns200OK

Two-Way SSL (Client Certificates)

To configure two-way SSL we have to create the server’s truststore and create the client’s keystore.

Since the client’s certificate is signed by a CA we created ourselves, import the CA cert into the server truststore:

| rhowlett@SL-MBP-RHOWLETT.local:~/dev/robinhowlett/github/everything-ssl/src/main/resources/ssl 
| => keytool -import -v -trustcacerts -keystore server_truststore.jks -storepass snaplogic -file /etc/apache2/ssl/cacert.pem -alias cacert
Owner: CN=localhost, OU=SnapTeam, O=SnapLogic, L=Boulder, ST=Colorado, C=US
Issuer: CN=localhost, OU=SnapTeam, O=SnapLogic, L=Boulder, ST=Colorado, C=US
Serial number: e4e00ed07233a969
Valid from: Tue Oct 06 15:14:51 EDT 2015 until: Wed Oct 05 15:14:51 EDT 2016
Certificate fingerprints:
	 MD5:  F3:5E:28:E4:28:47:F2:EC:82:E2:BD:16:31:DC:90:02
	 SHA1: 6F:0F:49:BA:A9:30:01:E9:4C:60:B3:A1:85:7D:BB:C6:79:1F:41:7B
	 SHA256: A7:9D:25:E4:A6:34:8A:A3:5B:9A:CD:F3:62:D0:D8:2F:6A:A0:71:6A:6D:19:F3:04:A1:FD:BC:FB:21:40:DE:A1
	 Signature algorithm name: SHA512withRSA
	 Version: 3

Extensions: 

#1: ObjectId: 2.5.29.35 Criticality=false
AuthorityKeyIdentifier [
KeyIdentifier [
0000: 03 09 12 6E 8B DD 7A 80   FB F5 21 AB 75 D9 B8 49  ...n..z...!.u..I
0010: 79 5B 61 1F                                        y[a.
]
[CN=localhost, OU=SnapTeam, O=SnapLogic, L=Boulder, ST=Colorado, C=US]
SerialNumber: [    e4e00ed0 7233a969]
]

#2: ObjectId: 2.5.29.19 Criticality=false
BasicConstraints:[
  CA:true
  PathLen:2147483647
]

#3: ObjectId: 2.5.29.14 Criticality=false
SubjectKeyIdentifier [
KeyIdentifier [
0000: 03 09 12 6E 8B DD 7A 80   FB F5 21 AB 75 D9 B8 49  ...n..z...!.u..I
0010: 79 5B 61 1F                                        y[a.
]
]

Trust this certificate? [no]:  yes
Certificate was added to keystore
[Storing server_truststore.jks]

Viewing the server truststore will show the CA’s certificate:

| rhowlett@SL-MBP-RHOWLETT.local:~/dev/robinhowlett/github/everything-ssl/src/main/resources/ssl 
| => keytool -list -v -keystore server_truststore.jks 
Enter keystore password:  

Keystore type: JKS
Keystore provider: SUN

Your keystore contains 1 entry

Alias name: cacert
Creation date: Jan 5, 2016
Entry type: trustedCertEntry

Owner: CN=localhost, OU=SnapTeam, O=SnapLogic, L=Boulder, ST=Colorado, C=US
Issuer: CN=localhost, OU=SnapTeam, O=SnapLogic, L=Boulder, ST=Colorado, C=US
Serial number: e4e00ed07233a969
Valid from: Tue Oct 06 15:14:51 EDT 2015 until: Wed Oct 05 15:14:51 EDT 2016
Certificate fingerprints:
	 MD5:  F3:5E:28:E4:28:47:F2:EC:82:E2:BD:16:31:DC:90:02
	 SHA1: 6F:0F:49:BA:A9:30:01:E9:4C:60:B3:A1:85:7D:BB:C6:79:1F:41:7B
	 SHA256: A7:9D:25:E4:A6:34:8A:A3:5B:9A:CD:F3:62:D0:D8:2F:6A:A0:71:6A:6D:19:F3:04:A1:FD:BC:FB:21:40:DE:A1
	 Signature algorithm name: SHA512withRSA
	 Version: 3

Extensions: 

#1: ObjectId: 2.5.29.35 Criticality=false
AuthorityKeyIdentifier [
KeyIdentifier [
0000: 03 09 12 6E 8B DD 7A 80   FB F5 21 AB 75 D9 B8 49  ...n..z...!.u..I
0010: 79 5B 61 1F                                        y[a.
]
[CN=localhost, OU=SnapTeam, O=SnapLogic, L=Boulder, ST=Colorado, C=US]
SerialNumber: [    e4e00ed0 7233a969]
]

#2: ObjectId: 2.5.29.19 Criticality=false
BasicConstraints:[
  CA:true
  PathLen:2147483647
]

#3: ObjectId: 2.5.29.14 Criticality=false
SubjectKeyIdentifier [
KeyIdentifier [
0000: 03 09 12 6E 8B DD 7A 80   FB F5 21 AB 75 D9 B8 49  ...n..z...!.u..I
0010: 79 5B 61 1F                                        y[a.
]
]



*******************************************
*******************************************

Finally, the client keystore stores the client certificate that will presented to the server for SSL authentication. Import the cert from client’s PKCS12 file (created above):

| rhowlett@SL-MBP-RHOWLETT.local:~/dev/robinhowlett/github/everything-ssl/src/main/resources/ssl 
| => keytool -importkeystore -srckeystore /etc/apache2/ssl/client-cert.p12 -srcstoretype pkcs12 -destkeystore client_keystore.jks -deststoretype jks -deststorepass snaplogic
Enter source keystore password:  
Entry for alias client successfully imported.
Import command completed:  1 entries successfully imported, 0 entries failed or cancelled

Viewing the created client_keystore.jks file will show the client entry in the keystore:

| rhowlett@SL-MBP-RHOWLETT.local:~/dev/robinhowlett/github/everything-ssl/src/main/resources/ssl 
| => keytool -list -v -keystore client_keystore.jks 
Enter keystore password:  

Keystore type: JKS
Keystore provider: SUN

Your keystore contains 1 entry

Alias name: client
Creation date: Jan 4, 2016
Entry type: PrivateKeyEntry
Certificate chain length: 2
Certificate[1]:
Owner: CN=client, OU=SnapTeam, O=SnapLogic, ST=Colorado, C=US
Issuer: CN=localhost, OU=SnapTeam, O=SnapLogic, L=Boulder, ST=Colorado, C=US
Serial number: 100002
Valid from: Tue Oct 06 13:15:41 MDT 2015 until: Wed Oct 05 13:15:41 MDT 2016
Certificate fingerprints:
	 MD5:  F1:EF:60:64:48:DC:9B:C1:92:37:61:90:ED:48:01:1C
	 SHA1: C5:4B:1C:EF:85:C1:8C:5A:AA:74:54:49:F0:B5:97:F1:EC:34:49:6F
	 SHA256: B0:00:E4:C1:AE:03:92:95:9C:A2:BB:DB:13:3A:B6:38:BE:B4:BF:04:D0:72:41:6D:62:A6:93:D0:46:7E:3C:97
	 Signature algorithm name: SHA512withRSA
	 Version: 1
Certificate[2]:
Owner: CN=localhost, OU=SnapTeam, O=SnapLogic, L=Boulder, ST=Colorado, C=US
Issuer: CN=localhost, OU=SnapTeam, O=SnapLogic, L=Boulder, ST=Colorado, C=US
Serial number: e4e00ed07233a969
Valid from: Tue Oct 06 13:14:51 MDT 2015 until: Wed Oct 05 13:14:51 MDT 2016
Certificate fingerprints:
	 MD5:  F3:5E:28:E4:28:47:F2:EC:82:E2:BD:16:31:DC:90:02
	 SHA1: 6F:0F:49:BA:A9:30:01:E9:4C:60:B3:A1:85:7D:BB:C6:79:1F:41:7B
	 SHA256: A7:9D:25:E4:A6:34:8A:A3:5B:9A:CD:F3:62:D0:D8:2F:6A:A0:71:6A:6D:19:F3:04:A1:FD:BC:FB:21:40:DE:A1
	 Signature algorithm name: SHA512withRSA
	 Version: 3

Extensions: 

#1: ObjectId: 2.5.29.35 Criticality=false
AuthorityKeyIdentifier [
KeyIdentifier [
0000: 03 09 12 6E 8B DD 7A 80   FB F5 21 AB 75 D9 B8 49  ...n..z...!.u..I
0010: 79 5B 61 1F                                        y[a.
]
[CN=localhost, OU=SnapTeam, O=SnapLogic, L=Boulder, ST=Colorado, C=US]
SerialNumber: [    e4e00ed0 7233a969]
]

#2: ObjectId: 2.5.29.19 Criticality=false
BasicConstraints:[
  CA:true
  PathLen:2147483647
]

#3: ObjectId: 2.5.29.14 Criticality=false
SubjectKeyIdentifier [
KeyIdentifier [
0000: 03 09 12 6E 8B DD 7A 80   FB F5 21 AB 75 D9 B8 49  ...n..z...!.u..I
0010: 79 5B 61 1F                                        y[a.
]
]



*******************************************
*******************************************

So, in summary, the server will present the certificate in its keystore to the client. The client will use its truststore to validate the server’s certificate. The client will present its certificate in its keystore to the server, and the server will validate the client certificate’s chain using the CA certificate in the server’s truststore.

The HttpServer instance can now be created with the forceSSLAuth parameter set to true (see the TWO_WAY_SSL boolean) which will require client certificates. The client’s SSLContext now has both the client truststore and keystore loaded:

private static final boolean TWO_WAY_SSL = true; // client certificates mandatory

@Test
public void httpsRequest_With2WaySSLAndHasValidKeyStoreAndTrustStore_Returns200OK()
        throws Exception {
    SSLContext serverSSLContext =
            createServerSSLContext(SERVER_KEYSTORE, KEYPASS_AND_STOREPASS_VALUE);

    final HttpServer server = createLocalTestServer(serverSSLContext, TWO_WAY_SSL);
    server.start();

    String baseUrl = getBaseUrl(server);

    KeyStore clientTrustStore = getStore(CLIENT_TRUSTSTORE, KEYPASS_AND_STOREPASS_VALUE);
    KeyStore clientKeyStore = getStore(CLIENT_KEYSTORE, KEYPASS_AND_STOREPASS_VALUE);

    SSLContext sslContext =
            new SSLContextBuilder()
                    .loadTrustMaterial(clientTrustStore, new TrustSelfSignedStrategy())
                    .loadKeyMaterial(clientKeyStore, KEYPASS_AND_STOREPASS_VALUE)
                    .build();

    httpclient = HttpClients.custom().setSSLContext(sslContext).build();

    try {
        CloseableHttpResponse httpResponse = httpclient.execute(
                new HttpGet("https://" + baseUrl + "/echo/this"));

        assertThat(httpResponse.getStatusLine().getStatusCode(), equalTo(200));
    } finally {
        server.stop();
    }
}

Once again, the above unit test is included in the everything-ssl GitHub project, along with the following:

httpsRequest_With2WaySSLAndUnknownClientCert_ThrowsSSLExceptionBadCertificate
httpsRequest_With2WaySSLButNoClientKeyStore_ThrowsSSLExceptionBadCertificate

Two-Way SSL Authentication with Spring Boot, embedded Tomcat and RestTemplate

Spring Boot “takes an opinionated view of building production-ready Spring (Java) applications”. Spring Boot’s Starter POMs and auto-configuration make it quite easy to get going:

``` xml pom.xml <?xml version=”1.0” encoding=”UTF-8”?>

4.0.0 org.springframework.boot spring-boot-starter-parent 1.3.1.RELEASE com.robinhowlett everything-ssl 0.0.1-SNAPSHOT jar everything-ssl Spring Boot and SSL UTF-8 1.8

2.4

org.springframework.boot spring-boot-starter-security org.springframework.boot spring-boot-starter-web org.apache.httpcomponents httpclient commons-io commons-io ${commons-io-version} org.springframework.boot spring-boot-starter-test test org.springframework.boot spring-boot-maven-plugin org.apache.maven.plugins maven-failsafe-plugin 2.19 integration-test verify

The Spring Boot documentation [describes the properties required to configure SSL](https://docs.spring.io/spring-boot/docs/current/reference/html/howto-embedded-servlet-containers.html#howto-configure-ssl). I wanted however to support both HTTP and HTTPS for testing purposes, so an explicit SSL Connector for the embedded Tomcat container needed to be created:

``` java Config.java

/**
 * Configure embedded Tomcat and SSL connectors
 */
@Configuration
public class Config {

    @Autowired
    private Environment env;

    // Embedded Tomcat with HTTP and HTTPS support
    @Bean
    public EmbeddedServletContainerFactory servletContainer() {
        TomcatEmbeddedServletContainerFactory tomcat = new
                TomcatEmbeddedServletContainerFactory();
        tomcat.addAdditionalTomcatConnectors(createSSLConnector());
        return tomcat;
    }

    // Creates an SSL connector, sets two-way SSL, key- and trust stores, passwords, ports etc.
    protected Connector createSSLConnector() {
        Connector connector = new Connector(Http11Protocol.class.getCanonicalName());
        Http11Protocol protocol = (Http11Protocol) connector.getProtocolHandler();

        File keyStore = null;
        File trustStore = null;

        try {
            keyStore = getKeyStoreFile();
        } catch (IOException e) {
            throw new IllegalStateException("Cannot access keyStore: [" + keyStore + "] or " +
                    "trustStore: [" + trustStore + "]", e);
        }

        trustStore = keyStore;

        connector.setPort(env.getRequiredProperty("ssl.port", Integer.class));
        connector.setScheme(env.getRequiredProperty("ssl.scheme"));
        connector.setSecure(env.getRequiredProperty("ssl.secure", Boolean.class));

        protocol.setClientAuth(env.getRequiredProperty("ssl.client-auth"));
        protocol.setSSLEnabled(env.getRequiredProperty("ssl.enabled", Boolean.class));

        protocol.setKeyPass(env.getRequiredProperty("ssl.key-password"));
        protocol.setKeystoreFile(keyStore.getAbsolutePath());
        protocol.setKeystorePass(env.getRequiredProperty("ssl.store-password"));
        protocol.setTruststoreFile(trustStore.getAbsolutePath());
        protocol.setTruststorePass(env.getRequiredProperty("ssl.store-password"));
        protocol.setCiphers(env.getRequiredProperty("ssl.ciphers"));

        return connector;
    }

    // support loading the JKS from the classpath (to get around Tomcat limitation)
    private File getKeyStoreFile() throws IOException {
        ClassPathResource resource = new ClassPathResource(env.getRequiredProperty("ssl.store"));

        // Tomcat won't allow reading File from classpath so read as InputStream into temp File
        File jks = File.createTempFile("server_keystore", ".jks");
        InputStream inputStream = resource.getInputStream();
        try {
            FileUtils.copyInputStreamToFile(inputStream, jks);
        } finally {
            IOUtils.closeQuietly(inputStream);
        }

        return jks;
    }
}

The application.properties file then defines the required SSL properties:

ssl.port=8443
ssl.scheme=https
ssl.secure=true
ssl.client-auth=true
ssl.enabled=true
ssl.key-password=snaplogic
ssl.store=ssl/server_keystore.jks
ssl.store-password=snaplogic
ssl.ciphers=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_DHE_RSA_WITH_AES_128_GCM_SHA256,TLS_DHE_DSS_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_SHA256,TLS_ECDHE_ECDSA_WITH_AES_128_SHA256,TLS_ECDHE_RSA_WITH_AES_128_SHA,TLS_ECDHE_ECDSA_WITH_AES_128_SHA,TLS_ECDHE_RSA_WITH_AES_256_SHA384,TLS_ECDHE_ECDSA_WITH_AES_256_SHA384,TLS_ECDHE_RSA_WITH_AES_256_SHA,TLS_ECDHE_ECDSA_WITH_AES_256_SHA,TLS_DHE_RSA_WITH_AES_128_SHA256,TLS_DHE_RSA_WITH_AES_128_SHA,TLS_DHE_DSS_WITH_AES_128_SHA256,TLS_DHE_RSA_WITH_AES_256_SHA256,TLS_DHE_DSS_WITH_AES_256_SHA,TLS_DHE_RSA_WITH_AES_256_SHA

The ssl.client-auth=true property enforces two-way SSL.

We are re-using the JKS files created earlier (above)

A simple REST interface is defined to return JSON representations of Greeting instances:

``` java GreetingController.java

/**

Just says hello */ @RestController public class GreetingController {

private static final String template = “Hello, %s!”;

@RequestMapping(“/greeting”) public Greeting greet( @RequestParam(value = “name”, required = false, defaultValue = “World!”) String name) { return new Greeting(String.format(template, name)); }

}

	
Spring Security auto-configuration will enable basic authentication on the REST endpoint by default, so let's switch it off:

``` java HttpSecurityConfig.java

/**
 * Disable default-enabled basic auth
 */
@EnableWebSecurity
public class HttpSecurityConfig extends WebSecurityConfigurerAdapter {

    @Override
    protected void configure(HttpSecurity http) throws Exception {
        http.httpBasic().disable();
    }

}

Integration Testing SSL Authentication with Spring’s TestRestTemplate

Spring Boot really is great - it was quite straightforward to write an integration test to demonstrate two-way SSL authentication with the application running on the embedded Tomcat container.

First I created an integration-test.properties file to set the HTTPS port to be different than then main application (the HTTP port will be set to a random open port by an annotation on the test itself). The only contents of this file is the ssl.port property. All the other properties will come from the application.properties file detailed above:

ssl.port=54321

The integration test class has annotations that instruct it to run using the SpringJUnit4ClassRunner, the scan for @Configuration classes at the base package of the EverythingSSLApplication class, to choose a random available HTTP port with the @WebIntegrationTest annotation (which itself is a combination of the @IntegrationTest and @WebAppConfiguration annotations), and finally the integration-test.properties file is denoted as a @TestPropertySource.

``` java ITEverythingSSL.java

/**

Integration test that uses the embedded Tomcat instance configured by this Spring Boot app */ @RunWith(SpringJUnit4ClassRunner.class) @SpringApplicationConfiguration(classes = EverythingSSLApplication.class) @WebIntegrationTest(randomPort = true) @TestPropertySource(locations = “classpath:integration-test.properties”) public class ITEverythingSSL {

public static final String CLIENT_TRUSTSTORE = “ssl/client_truststore.jks”; public static final String CLIENT_KEYSTORE = “ssl/client_keystore.jks”;

@Rule public ExpectedException thrown = ExpectedException.none();

@Value(“${local.server.port}”) private int port = 0;

@Value(“${ssl.port}”) private int sslPort = 0;

@Value(“${ssl.store-password}”) private String storePassword;

… ```

The first test confirms that plain HTTP is supported:

@Test
public void rest_OverPlainHttp_GetsExpectedResponse() throws Exception {
    Greeting expected = new Greeting("Hello, Robin!");

    RestTemplate template = new TestRestTemplate();

    ResponseEntity<Greeting> responseEntity =
            template.getForEntity("http://localhost:" + port + "/greeting?name={name}",
                    Greeting.class, "Robin");

    assertThat(responseEntity.getBody().getContent(), equalTo(expected.getContent()));
}

TestRestTemplate is “a convenience subclass of Spring’s RestTemplate that is useful in integration tests”

The getRestTemplateForHTTPS method creates a TestRestTemplate instance with an SSLContext set to support the client’s keystore and truststore:

private RestTemplate getRestTemplateForHTTPS(SSLContext sslContext) {
    SSLConnectionSocketFactory connectionFactory = new SSLConnectionSocketFactory(sslContext,
            new DefaultHostnameVerifier());

    CloseableHttpClient closeableHttpClient =
            HttpClientBuilder.create().setSSLSocketFactory(connectionFactory).build();

    RestTemplate template = new TestRestTemplate();
    HttpComponentsClientHttpRequestFactory httpRequestFactory =
            (HttpComponentsClientHttpRequestFactory) template.getRequestFactory();
    httpRequestFactory.setHttpClient(closeableHttpClient);
    return template;
}

The test that then demonstrates two-way SSL is quite simple:

@Test
public void rest_WithTwoWaySSL_AuthenticatesAndGetsExpectedResponse() throws Exception {
    Greeting expected = new Greeting("Hello, Robin!");

    SSLContext sslContext = SSLContexts.custom()
            .loadKeyMaterial(
                    getStore(CLIENT_KEYSTORE, storePassword.toCharArray()),
                    storePassword.toCharArray())
            .loadTrustMaterial(
                    getStore(CLIENT_TRUSTSTORE, storePassword.toCharArray()),
                    new TrustSelfSignedStrategy())
            .useProtocol("TLS").build();

    RestTemplate template = getRestTemplateForHTTPS(sslContext);

    ResponseEntity<Greeting> responseEntity =
            template.getForEntity("https://localhost:" + sslPort + "/greeting?name={name}",
                    Greeting.class, "Robin");

    assertThat(responseEntity.getBody().getContent(), equalTo(expected.getContent()));
}

The following integration tests have also been included in the project:

rest_WithMissingClientCert_ThrowsSSLHandshakeExceptionBadCertificate
rest_WithUntrustedServerCert_ThrowsSSLHandshakeExceptionUnableFindValidCertPath

The companion Spring Boot application is available on GitHub

Two-Way SSL with SnapLogic’s REST Snap

Naturally, SnapLogic’s REST Snap makes this all very easy:

Thank Yous

Thank you to Ed Heneghan for correcting my initial mistakes.

Farewell SportsLabs, Hello SnapLogic

Fri, 07 Aug 2015 04:13:17 +0000

After close to 5 years with Silver Chalice/SportsLabs, today, Friday August 7th 2015, will be my last day with the company. On Monday, I join SnapLogic, an enterprise integration platform-as-a-service (iPaaS) provider that I believe has an amazing future.

It has been a tremendous, life-changing journey.

Silver Chalice

I joined Silver Chalice shortly after marrying my wife and moving to America from Ireland. I had completed a small social app for the 2010 Breeders’ Cup and was finishing up my responsibilities as a server-side engineer for IBM, when I started looking for startups in the Denver/Boulder area that I could be a part of. I knew Boulder, with its Techstars incubator, would be a great place to find fast-growing companies.

Towards the end of 2010, I stumbled across a job posting for a web developer position at new media company that was being spun off from the Chicago White Sox.

Ignoring the fact my baseball knowledge at that point in time amounted to “it’s kind of like Rounders”, and that I wasn’t a web developer, I was intrigued by a major sports franchise looking to invest in opportunities in the digital and new media space. So I applied. And heard nothing back.

I followed up a while later and got their attention this time. Silver Chalice was really only a handful of people at that point, split between Chicago and Boulder. They had outsourced any previous client work and had no engineers on staff. I met the hiring manager and, over a burger, he described the need for a system that could power market-leading news, weather and sports apps. How it did that, he had no idea. We fleshed out, in detail, all the ways we didn’t know what to build, but we did realize that it definitely wasn’t a web developer they were looking for. So I took a leap of faith, joined and set to work on building a Platform and rapidly hiring a team - neither of which I’d ever done before but had always wanted to do.

Within a couple of weeks I was on a plane to the headquarters of a major media company with a slide deck and somewhat vague idea of how we (the engineering department of 1) were, in the next 4 months, going to build apps and mobile websites for 31 TV stations, and a platform to power them all, for the 1-million-plus active users they had with their legacy provider’s systems. I was grilled pretty hard by their CIO but I discovered I had a knack for spinning a good tale and making a sale - we won the deal.

What followed in the next few months was a madcap scramble to hire great engineers, build a cloud-hosted, scalable content delivery architecture, an integration engine to ingest the myriad of content needed for a modern news/weather/sports app, and to build the apps and websites themselves. As a fan of Eric Ries, I was a proponent of API-driven development and continuous deployment, and we quickly built a system that, driven by auto-scaling, workers scheduled and consumed jobs from AWS SQS queues, used the Apache Camel integration framework to source and combine the information from dozens of data sources across many protocols, and wrote “API response” data to AWS S3 and Cloudfront for durable, reliable content delivery, even under bursty, high-traffic scenarios like major breaking news. We got the job done and the apps and sites launched on time and the client experienced a significant increase in active users, and being #1 in many of their markets.

It wasn’t all plain sailing. My very first paycheck bounced - you can imagine how that made me feel about joining a startup! Luckily it turned out to be the bank’s error. I was working very long hours at this stage, trying to keep up with building the platform, interviewing for all new employees, identifying market opportunities and pitching new business, and setting out a strategic vision for our technology. And we were still working by the skin of our teeth - on launch weekend, a not-insignificant portion of the platform was running on my laptop.

SportsLabs

We took this platform into sports by powering CBS Interactive’s college sports apps. That’s when the company saw a major opportunity to do what Major League Baseball had done with MLB Advanced Media (BAM) but for college sports - create a “roll-up” platform that could serve the digital needs of top-tier collegiate athletic institutions and organizations, as well as independent sports media companies, sports media rights holders, and global media powerhouses. The Boulder arm of Silver Chalice was rebranded as SportsLabs and we said goodbye to our news media friends and focused exclusively on the college sports vertical.

Today, SportsLabs is a major player in the space, streaming live video of the ACC Basketball Championships, building Notre Dame’s WatchND media properties, powering the location-based services, social streams and live stats for 30+ IMG Gameday iOS and Android native apps, winning premium clients like the University of Kentucky and Kansas State University, and signing one of the largest sports audio streaming deals with Samsung.

From the 5-person Boulder office I joined in 2011, SportsLabs has now 50+ employees in Boulder and Silver Chalice must have 150+ in Chicago. We moved into new, much larger offices right on beautiful Pearl St. and have built market-leading products.

As with any fast-paced environment, it was difficult to just keep up. We naturally had to rewrite the platform for the API to be flexible enough to support all these products - so we brought in the Spring stack including Spring MVC, Spring Data etc., plus MongoDB, Postgres, and an RoR/AngularJS stack to power our CMS and websites. After working seemingly non-stop I definitely experienced burn-out a few times. Sports, like most live entertainment experiences, is skewed to take place at nights or weekends - especially the highest profile events. The stress of a busy November Saturday evening with hundreds of events being live scored and streamed to the general public on a national stage meant that I rarely got a break from even thinking about the platform and our products.

We added more and more data providers and the integration challenge began to be significant. Trying to balance developing and coding the platform, plus managing the platform team, was tricky. Plus, as my future CEO Gaurav Dhillon so wisely put it, the importance of timing is “vital”. While we have definitely seen the market move to “cutting the cord” and consuming content online (OTT solutions like HBO Now and SlingTV are super promising), sports on TV is still a great experience and, in many ways, the reason why TV, especially cable, has been able to fight off these challenges from far more flexible online properties. ESPN especially shapes the college football broadcast world, both in terms of distribution and money. And the broadcast rights for the big games are everything. Yahoo live streaming an NFL game this year is a gamechanger - with the worst part of it being it’s a Bills-Jaguars game.

It really does feel that sports broadcasting online is going to go through some major shifts in the next few years - it’s just a little behind where I’d thought it would be now back in 2011. Either way, SportsLabs has marvelous opportunities ahead of it, working with world-famous clients in one of the most interesting media/technology spaces out there. They’ll do great things.

SnapLogic

In my time with Silver Chalice/SportsLabs I experienced a few themes that I think are going to be incredibly commonplace for businesses in the near future.

Firstly, Silver Chalice was a new company and to get up and running quickly, we used SaaS heavily - both to build our products and to simply operate ourselves. From the usual suspects like Google Apps, Dropbox, Paylocity, and Concur, to the plethora of services we use in Engineering including Splunk, New Relic, PagerDuty, GitHub, Google Analytics, DFP, Atlassian JIRA Studio, and of course Amazon Web Services. While some of these services talk to one or two of the others, there wasn’t a way to easily move information between all of them.

So much communication was lost between wiki pages defining product requirements, ticket systems detailing bugs, email and chat messages clarifying queries, continuous integration systems outlining test failures, logs, monitoring systems highlighting anomalies etc.! In fact, in my opinion, this largely technical limitation of sharing basic information across systems was costing us millions in time and effort wasted. And we as platform engineers were exacerbating the problem by trying to make data sources talk to each other in non-consistent ways.

Secondly, with Camel we started with an ESB-like solution for integrating data and, while it worked quite well, we got sucked into a fairly tightly coupled architecture, even though it supported using a variety of ways to decouple parts of the architecture. We eventually moved to a JSON-based REST API solution that offered increased flexibility, but we strongly typed far too many of the contracts and integration boundaries. As the number of data sources increased, the complexity of trying to integrate them quickly into the platform, and scale them appropriately, increased also. Not having a GUI like the SnapLogic Designer that described the integrations, meant that a significant portion of developers’ time was spent just understanding how the data would flow through the system. The code-only solution also limited the number of people that were able to help understand and document the flows too. Maintaining the flow descriptions in wikis etc. was both too time consuming for the (expensive) developers and too static to be relied upon.

Thirdly, we were only scratching the surface with the amount of data becoming available to us. In the past, live sports might have been described as having “a lot of data”, but it pales in comparison to the volume of information now accessible through app analytics of user actions, streaming metadata from live broadcasts, social engagement with players and teams, location-based services at stadiums, and so on. Designing our platform to incorporate social, mobile, analytical, cloud and Internet of Things (IoT) data to further customize, operate, and gain insights into our products would require a major engineering effort.

So when I joined our neighborhood website recently, I was amazed that the first post I saw was from a neighbor, Kevin Hahn, who lived just a couple of doors down, describing opportunities with a company named SnapLogic, who had just opened an office in Boulder. A quick perusal of snaplogic.com piqued my interest further. Watching fascinating interviews with SnapLogic CEO and Informatica co-founder Gaurav Dhillon and reading Darren Cunningham’s excellent blog posts whetted my appetite even more.

After speaking with Kevin, I made it my business to become part of SnapLogic. I can’t wait to start. It’s wonderfully exciting being part of a something that I believe is going to be huge - both in its success and its impact.

Reflections

I read an article where Gaurav said “Leaving [Informatica] was the hardest and best thing I ever did”. I have similar feelings leaving SportsLabs and joining SnapLogic. Naturally, the team I’m leaving behind weighs on me the most.

I can’t thank John Burris enough for the opportunity he gave me, how he looked out for me and my family, and the introduction to this startup world that I don’t want to ever leave. I’m very proud to be part of your team that built this company.

I have a deep gratitude in having worked beside Nancy Beaton and Chris Jackson. Chris has an amazing talent for making a connection with clients. Nancy is one of the most impressive people I have ever had the pleasure of meeting. I’ll never forget the time we travelled to what we thought was a small, getting-to-know-you meeting with Samsung and instead entered a conference room with about 25 people, and standing room only, to listen to our pitch. I was amazed with the technical acumen displayed as she deftly took the audience through the finer points of streaming live audio, while also being responsible for securing one of the most lucrative contracts our space has ever had - a superstar.

As for other terrific Silver Chalice/SportsLabs employees, both former and present, there are simply too many to for me to remember them all, but I do want to thank Jason Gall, Trey Hicks, Eric Foote, Colin Holm, Tim Gray, Kevin Gorham, Jonathan Kreuger, Arlie Sisson, Peter Laird, Nick Cassidy, Mike Olson, Colin Pilloud, Rebecca Cameron, Ryan Pensy, Dave Wiedenheft, Brian Grayless, Sean Ortiz, and Ryan Karlin for the help and support they gave me over the years. I’m sure I’ve left plenty out that deserve individual recognition too.

I joined Silver Chalice/SportsLabs fresh off the boat (so to speak). I was newly married, living a new country where I didn’t know anyone, and not really sure what the future held. My experience completely opened up a new world for me - professionally and personally. My home is Boulder now, where my children are growing up. I gained confidence to pursue ambitious opportunities and engineer things that were worth something and made an impact in the market. I gained friendships that I hope will last a lifetime.

I’ve enjoyed every moment.

Spring Social Bootstrap: Create REST API SDKs and CLIs that can Record and Replay HTTP requests

Sun, 26 Apr 2015 22:50:33 +0000

I joined SportsLabs (then still under the Silver Chalice brand) way back in 2011 as one of its earliest employees and the first engineer.

We started work on envisioning and building the Advanced Media Platform - a system to ingest, process, transform, distribute, and stream sports, news, social, and media content to create market leading mobile, web, and social products for clients such as Samsung, the University of Notre Dame, the ACC, the College Football Playoff, IMG College, the Mountain West and Campus Insiders, among others.

Since then, SportsLabs has consumed data from dozens of sources including STATS LLC, Twitter, and Ooyala, but also from proprietary systems that were never foreseen as integration points.

Data providers’ APIs use combinations of JSON, XML and/or CSV. Some are spec-compliant, others are not. Some rely heavily on query parameters, while others favor HTTP headers. Some API providers use OAuth 2.0 plus API rate limits, while others have rolled their own security solutions. Some integrations were with partners willing to work with us on evolving their web services. Others were with competitors who were not motivated to make things easy.

This plethora of ways to configure, consume, learn from, and integrate with APIs led us to create Spring Social Bootstrap, a family of projects intended to aid creating and managing API clients for many of the above scenarios.

Spring Social Bootstrap is comprised of the following:

Spring Social Bootstrap SDK: a Spring Social-based framework for rapidly and consistently developing API clients
Bootstrap Shell: a Spring Shell-based framework to aid creating command-line interface (CLI) applications for API clients built on Spring Social Bootstrap SDK
HAR Mar Interceptor: allows capturing HTTP request/response exchanges, persisting them in a HAR or ALF-compatible format, and re-executing those requests at a later time.

![Spring Social Bootstrap SDK](http://i.imgur.com/asX8yGM.jpg) **Spring Social Bootstrap SDK** is a simple [Spring Social](http://projects.spring.io/spring-social)-based framework for rapidly and consistently developing API clients. It combines the strengths of Spring Social, such as [native support for OAuth-based service providers](http://docs.spring.io/spring-social/docs/current/reference/htmlsingle/#connectFramework) (including support for OAuth 1 and OAuth 2), with consistent and well-defined extension points for new API clients. Spring Social, despite its name, has no requirement on integrations being only with social networks. In truth, its primarily a set of interfaces for predictable implementations of API clients - which is our goal too. Spring Social Bootstrap SDK provides CRUD and query capabilities out-of-the-box e.g. ```java testApi.testOperations().create(testBaseApiResource); testApi.testOperations().get(testBaseApiResource.getId()); testApi.testOperations().update(testBaseApiResource); testApi.testOperations().delete(testBaseApiResource.getId()); testApi.testOperations().query(); ``` as well as guidelines on how to create developer-friendly features like query builders e.g. ```java testApi.testOperations().qb().withPaging(1, 25, "name", Sort.Direction.ASC).query(); ``` I'd advise taking a look at the [Spring Social Bootstrap SDK README](https://github.com/robinhowlett/spring-social-bootstrap/blob/master/spring-social-bootstrap-sdk/README.md) to learn more on how to create API clients based on Spring Social Bootstrap SDK.

---

![Bootstrap Shell](http://i.imgur.com/PypWRjx.png) **Bootstrap Shell** is a [Spring Shell](http://docs.spring.io/spring-shell/docs/current/reference/htmlsingle/)-based framework to aid creating command-line interface (CLI) applications for API clients built on [Spring Social Bootstrap SDK](https://github.com/robinhowlett/spring-social-bootstrap/tree/master/spring-social-bootstrap-sdk). When developing API clients, SportsLabs learned that is worth leveraging the clients in data monitoring, QA, testing, and simulation scenarios. As such, we created a small CLI framework to allow highly interactive command-line applications as well as taking advantage of the power of command line tools, such as [pipes](http://man7.org/linux/man-pages/man2/pipe.2.html), [`xargs`](http://unixhelp.ed.ac.uk/CGI/man-cgi?xargs), [`jq`](http://stedolan.github.io/jq/), and even opening the browser to online side-by-side diff tools like [mergely](http://www.mergely.com/editor?lhs=http://mockbin.com/bin/800a818b-5fb6-40d4-a342-75a1fb8599db/view&rhs=http://mockbin.com/bin/3c149e20-bc9c-4c68-8614-048e6023a108/view). We also liked Spring Shell's Spring-based interactive shell, but the XML-only configuration, the `toString()`-based command outputs and lack of menu-style multi-step support required a fork of the project to add the following: * [SHL-106: Java Configuration support](https://github.com/spring-projects/spring-shell/pull/66): Permits configuring a Spring Shell application using Java `@Configuration` classes instead of XML ([JIRA](https://jira.spring.io/browse/SHL-106)) * [SHL-174: Multi-Step Commands](https://github.com/spring-projects/spring-shell/pull/67): Permits commands annotated with `@CliStepIndicator` to perform additional logic during its execution, including accepting user input e.g. pagination instructions ([JIRA](https://jira.spring.io/browse/SHL-174)) * [SHL-175: Multiple Output Formats for Commands](https://github.com/spring-projects/spring-shell/pull/68): Permits command results to be printed to the console with different formats by using Spring Type Converters, denoted by `@CliPrinter` annotations on Command parameters ([JIRA](https://jira.spring.io/browse/SHL-175)) The result was a handy framework for developers and QA engineers to quickly start using our API clients in a variety of situations. _e.g. [Mockbin CLI](https://github.com/robinhowlett/mockbin-cli): an Bootstrap Shell-based CLI application for [Mockbin](http://mockbin.com/) powered by [Spring Social Mockbin](https://github.com/robinhowlett/spring-social-mockbin)_ ![CLI Example](http://i.imgur.com/8Eca4p3.gif) Again, check out the [Bootstrap Shell README](https://github.com/robinhowlett/spring-social-bootstrap/blob/master/bootstrap-shell/README.md) for more information.

---

![HAR Mar Interceptor](http://i.imgur.com/oaqfyPi.png) **HAR Mar Interceptor** allows capturing HTTP request/response exchanges, persisting them in a [HTTP Archive (HAR)](http://www.softwareishard.com/blog/har-12-spec/) or [API Log Format (ALF)](https://github.com/Mashape/api-log-format)-compatible format, and re-executing those requests at a later time. Often one of the challenging things about creating clients for proprietary APIs is the lack of documentation or samples available to learn from. Also, some API providers can be very strict with rate limits and test API endpoints are rarely provided. In Sports, this issue is complicated further by the time sensitive nature of the domain - events rarely last longer than a couple of hours and every so often rare scenarios like weather-related postponements, data-entry errors and [22 inning baseball games](http://www.ncaa.com/news/baseball/article/2014-06-01/tcu-wins-22-innings-second-longest-game-ncaa-tourney-history) create challenges for how well our products work for users. Early on in SportsLabs' life, we created our own system for storing the request history of particular API endpoints, but I discovered the [HTTP Archive (HAR)](http://www.softwareishard.com/blog/har-12-spec/) spec, created by [@JanOdvarko](https://twitter.com/janodvarko), lead of the [Firebug](http://www.getfirebug.com/) project. If you are familar with Firebug you'll know you can see the requests the browser is making for a particular page and the time taken for each response to be received: ![Firebug](http://core0.staticworld.net/images/idge/imported/article/itw/2013/11/22/netpanel-100521990-orig.png) [Mashape](https://www.mashape.com/) [API Log Format (ALF)](https://github.com/Mashape/api-log-format) is based on HAR and is particularly designed for API consumer use cases. HAR Mar Interceptor ships with a customized Spring [`ClientHttpRequestInterceptor`](http://docs.spring.io/spring/docs/current/javadoc-api/org/springframework/http/client/ClientHttpRequestInterceptor.html) for capturing HTTP request/response exchanges and storing them in ALF to a JDBC database: ![HAR Table](http://i.imgur.com/siNRX66.png) The persisted data can be loaded into an `AlfHar` POJO and represented in JSON using [Jackson](http://wiki.fasterxml.com/JacksonHome). This JSON can even be stored in a `.har` file and loaded into HAR-compatible tools like [Charles Web Debugging Proxy](http://www.charlesproxy.com/). It also provides `ReplayAlfHarTemplate` to replay entries in the HAR log at either real-time, fixed or immediate intervals. This allows us to capture what our systems are requesting and/or responding under certain conditions, replay system inputs and/or outputs for debugging, plus design and tweak scenarios based on real data. It also aids us when integrating with new data providers lacking documentation, as we can learn from the archive log and, for those who wish to limit testing efforts against their infrastructure, use services like Mashape's [Mockbin](http://mockbin.com/) to mock their endpoints with observed data - using [Spring Social Mockbin](https://github.com/robinhowlett/spring-social-mockbin), naturally.

Supporting Multi-Step Commands with Spring Shell

Thu, 19 Mar 2015 21:44:25 +0000

Out of the box, Spring Shell supports printing command results to the terminal in a fairly basic way.

Spring Shell also provides the ExecutionProcessor interface, allowing a “command provider to be called in a generic fashion just before, and right after, executing a command”.

The interface defines three lifecycle events that can be intercepted:

before a command has been invoked
after an invocation has been returned
after an exception was thrown

I was interested in hooking into the afterReturningInvocation to provide “step logic” - potentially allowing user or system input to execute additional logic based on the result of the initial command result (and/or each step result) e.g. paging backwards or forwards on the command line through lists of data.

I was able to achieve this and opened a JIRA ticket and the following pull request on Spring Shell’s GitHub repo: SHL-174: Multi-Step Commands #67

When writing command execution results to the terminal, Spring Shell’s AbstractShell examines the instance being returned by the command and logs at INFO level the toString() output of the result object. If the result object is an instance of Iterable, it iterates over the collection and logs the toString() of each entry:

``` java AbstractShell.java protected void handleExecutionResult(Object result) { if (result instanceof Iterable<?>) { for (Object o : (Iterable<?>) result) { logger.info(o.toString()); } } else { logger.info(result.toString()); } }

When planning out my solution, I came up with the following back-of-the-napkin outline:

* a new custom annotation would be created and would be put on the command method. The `ParseResult invocationContext` instance passed into the processor methods could then be checked to see if the method being invoked had that annotation present, denoting a multi-step command.

* an abstract implementation of `ExecutionProcessor` would override the `afterReturningInvocation` method and detect if a multi-step command had been invoked. Command classes requiring multi-step logic would extend this class.

* the step logic would be configurable in that it could:
	* determine if there were more steps to execute, 
	* configure each step in preparation of execution, 
	* execute the step, and 
	* handle each step execution's result (e.g. logging to the shell), if any.

* the solution should also ensure that the command's original/final result was handled correctly.

<p />
---
**`@CliStepIndicator`**

I created a new annotation that denotes a `@CliCommand`-annotated method supports multi-step processing:

``` java CliStepIndicator.java
@Inherited
@Documented
@Retention(RetentionPolicy.RUNTIME)
@Target(ElementType.METHOD)
public @interface CliStepIndicator {

}

AbstractStepExecutionProcessor.java

I created a new abstract class implementing ExecutionProcessor. The afterReturningInvocation method is overridden with the following logic:

checks if this invocation is on a multi-step command
the initial result from the command is handled and this is fact is stored on the shell
while there are more steps
- configure the next step
- execute the next step in the workflow
- handle the step execution result

Example

StepCommand.java is an example command class with a multi-step command (step-test) method annotated with @CliStepIndicator.

Each step increments a int variable (see configureStep), up to 3 times (see hasMoreSteps). Each step execution logs that it is executing (see executeStep), and step results are handled by printing the current int variable value (see handleStepExecutionResult).

The following integration test executes both the step-test multi-step command (to increment the int variable 3 times) and the step-check comand (to confirm the incrementation took place).

The test passes and prints the following output:

I’ll be posting shortly about how I used this feature in a recent CLI project to navigate API pagination, run simulations, diff audit log entries etc.

Adding Java Config support to Spring Shell

Thu, 19 Mar 2015 19:58:12 +0000

Be it DVCS workflows, JSON transformations, or blogging frameworks, I always favor tools that allow me to use the terminal.

I’ve recently started using Spring Shell for rapidly experimenting with consuming data from a series of APIs I had created.

I have become allergic to XML configuration for Spring applications in recent times, so I was disappointed to see a lack of support for Java configuration within Spring Shell.

However, I did find a JIRA issue tracking the feature request. The ticket creator had even submitted a pull request with a potential solution. However, Spring Shell lead Mark Pollack responded that the feature could be provided in a simpler manner and even provided guidance to the solution.

Implementing this solution seemed quite straightforward, so I gave it a go and it turned out well.

I’ve submitted the following pull request to the Spring Shell GitHub repository: SHL-106: Java Configuration support #66

The pull request detailed:

Updated Bootstrap.java to accept basePackages String varargs and to include them in ClassPathBeanDefinitionScanner scan
Updated sample HelloWorld project to demonstrate mixing XML and Java Configuration
Updated docbook-reference-plugin dependency group ID and version (needed to get my gradle build working)

This was my first pull request submission for a Spring project (albeit still awaiting approval). I was very happy to “give a little back” finally!

Custom Jackson Polymorphic Deserialization without Type Metadata

Thu, 19 Mar 2015 18:28:24 +0000

At SportsLabs we regularly encounter proprietary, non-standard APIs and formats. Our job is to integrate with these APIs, normalize them, and distribute the data in web- and mobile-friendly web services.

One of the scenarios we often encounter is a provider supplying multiple resource JSON-based APIs that share a lot of the same data in their responses, but without any particular field dedicated to identying the type of the resource e.g.

{
...
	"common": "a common field within multiple resource responses",
	"one": "one is a field only within this response type"
...
}

and

{
...
	"common": "a common field within multiple resource responses",
	"two": "two is a field only within this response type"
...
}

Instead of mapping 1-to-1 with these APIs, we often try to follow DRY principles and model them as implementations of a common polymorphic abstraction.

When using Jackson for polmorphic deserialization and not being in control of the API response data, the lack of any kind of type identifier requires a different solution.

One of the ways we’ve addressed this problem is to identify fields and properties that are unique to a particular resource API’s response.

We then add this field to a registry of known unique-property-to-type mappings and then, during deserialization, lookup the response’s field names to see if any of them are stored within the registry.

Planning out the solution, there were a couple of things I wanted to incorporate:

the deserializer should be initialized with the abstract class representing the shared response data
a register(String uniqueProperty, Class<? extends T> clazz) method will add the field-name-to-concrete-class mapping to the registry
the custom deserializer would be added to a Jackson SimpleModule which in turn would be registered with the ObjectMapper.

The deserialize method reads each of the fields in the response and looks up the registry to see if it is present. If it finds a match, the mapper.treeToValue method is invoked with the response object and the mapped class returned by the registry. If no match is found an exception is thrown.

For the unit test, I created an inner static abstract class AbstractTestObject (containing shared data) with two concrete implementations (TestObjectOne and TestObjectTwo) that each contain a property unique to that type.

The test also contains a inner class TestObjectDeserializer that extends UniquePropertyPolymorphicDeserializer. The test’s setUp method:

initializes the custom deserializer,
registers the unique-field-name-to-type mappings, and
adds the custom deserializer to a new SimpleModule which is registered with the ObjectMapper in turn.

If others have solved this problem differently, I’d love to hear about it!

Building a Custom Jackson Deserializer

Fri, 02 Jan 2015 04:35:31 +0000

A partner recently provided a useful HTTP-based API for me at short notice.

The API returned simple JSON representations of the current state of various events they had periodically ingested from our REST APIs.

A decision they made was to use "yes"/"no" values in the response, rather than booleans e.g.

{
...
	"exists": "yes",
	"has_ended": "no"
...
}

I had written a Spring Social-based API client to interact with their API but wanted to deserialize their API response representation into a POJO that followed Java convention.

Jackson is a great Java library for processing JSON data format. It’s quite straightforward to use it to solve a problem like the above with a custom JsonDeserializer.

Planning out the solution, there were a couple of things I wanted to incorporate:

the deserializer should be case-insensitive
a null value in the JSON document should be treated as false (due to autoboxing considerations around Java’s primitive (boolean) and Object-based Boolean representations)
an exception should be thrown if something other than "yes" or "no" was provided

Sketching out the JUnit test class, I needed a static inner class to represent the POJO that Jackson would deserialize to, an ObjectMapper to deserialize a sample JSON document, and test cases for the above conditions.

For the field that will use the custom deserializer, the @JsonDeserialize annotation is added, with the using parameter pointing at our implementation class.

For the implementation, we must override the JsonDeserializer.deserialize(JsonParser jp, DeserializationContext ctxt) method.

If Jackson has identified the field value token currently being deserialized as a String, it is case-insensitively checked for either a "yes" or "no" value, throwing a JsonMappingException with an explaination through the DeserializationContext.weirdStringException(String value, Class<?> instClass, String msg) method if an unsupported String value is received instead.

As Java supports both the primitive boolean and the Object Boolean, and can autobox between them, null values will be interpreted as false. To do this, JsonDeserializer.getNullValue() is overridden and returns Boolean.FALSE. Then within the deserialize method, the current token is checked if it is a JsonToken.VALUE_NULL and the getNullValue() method is called if it is.

Finally, an exception is thrown if the token is of any other type.

Spring app migration: from XML to Java-based config

Wed, 13 Feb 2013 10:46:00 +0000

Our team recently built a Spring MVC 3.1 application for a web service API. We had used the traditional XML-based configuration but I wanted to see how easy would it be to migrate a Spring application from an XML-based to a Java annotation-based configuration.

I referenced three great resources for this migration:

@baeldung’s excellent Bootstrapping a web application with Spring 3.1 and Java based Configuration series
@doughaber’s post, Pagination with Spring MVC, Spring Data and Java Config, included some great “gotchas”, especially around web tier configuration
@cbeam and @rstoya05’s presentation, Configuration Enhancements in Spring 3.1, laid out in terrific detail how Spring configuration has evolved and how most XML configurations could be achieved through Java annotations

The web.xml defined the context configuration and dispatch servlet locations:

``` xml web.xml <?xml version=”1.0” encoding=”ISO-8859-1”?>

contextConfigLocation

classpath:META-INF/spring/root-context.xml

...

spring-mvc-dispatcher

org.springframework.web.servlet.DispatcherServlet

contextConfigLocation

classpath:META-INF/spring/servlet-context.xml

spring-mvc-dispatcher

`root-context.xml` was intended to define shared resources visible to all other web components, servlets etc and for cross-cutting concerns like security - for the moment though it was just an empty `<beans>` XML file.

`servlet-context.xml` was the `DispatcherServlet` and a wrapper around three other contexts; `persistence-context.xml` for Spring Data/JPA configuration, `bizniz-context.xml` for business/service configuration and `web-context.xml` for web service configuration:

``` xml servlet-context.xml
<?xml version="1.0" encoding="UTF-8"?>
<beans xmlns="http://www.springframework.org/schema/beans"
	xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
	xsi:schemaLocation="http://www.springframework.org/schema/beans http://www.springframework.org/schema/beans/spring-beans-3.1.xsd">
	
	<import resource="classpath:META-INF/spring/persistence-context.xml" />
	<import resource="classpath:META-INF/spring/bizniz-context.xml" />
	<import resource="classpath:META-INF/spring/web-context.xml" />
</beans>

web-context.xml contained an <mvc:annotation-driven> declaration to enable Spring MVC’s annotation based model. This was the first place to start.

From <mvc:annotation-driven> to @EnableWebMVC

<mvc:annotation-driven> enables the Spring MVC @Controller programming model and adds support for @RequestMapping annotations etc.

We had added to the web service a PageableArgumentResolver for Spring Data paging query parameter support and an instance of MappingJackson2HttpMessageConverter with a custom Jackson ObjectMapper. The custom JacksonObjectMapper just extended ObjectMapper and added a serialization inclusion configuration set to NON_NULL, so empty JSON properties would not be outputted.

``` xml web-context.xml <beans …>

…

<mvc:annotation-driven>
	<mvc:argument-resolvers>
		<bean class="org.springframework.data.web.PageableArgumentResolver" />
	</mvc:argument-resolvers>
	<mvc:message-converters>
		<bean class="org.springframework.http.converter.json.MappingJackson2HttpMessageConverter">
			<property name="objectMapper" ref="jacksonObjectMapper" />
		</bean>
	</mvc:message-converters>
</mvc:annotation-driven>

<bean id="jacksonObjectMapper" class="com.silverchalice.contracts.json.mappers.JacksonObjectMapper" />

…

</beans>

The first thing that needed to be done was to create a new configuration class, `WebConfig`, annotated with `@Configuration` and `@EnableWebMVC`. The new class was added to a new package, `com.silverchalice.api.config`.

It is common in many Spring MVC application configurations to just use the default self-closing `<mvc:annotation-driven />` tag; therefore the new configuration class could be just an empty class annotated with `@EnableWebMVC`.

However, since we had added an argument resolver and message conveter customizations, the config class would need to be able to configure these too. This was acheived by extending `WebMvcConfigurerAdapter`. The needed customizations could then be added by simply overriding the appropriate method:

``` java WebConfig.java
package com.silverchalice.api.config;

...

/**
 * Spring MVC Configuration.
 * 
 * Extends {@link WebMvcConfigurerAdapter}, which provides convenient callbacks that allow us to customize aspects of the Spring Web MVC framework.
 * These callbacks allow us to register custom interceptors, message converters, argument resovlers, a validator, resource handling, and other things.
 * 
 * @author robin
 * @see WebMvcConfigurerAdapter
 */
@Configuration
@EnableWebMvc
public class WebConfig extends WebMvcConfigurerAdapter {

	@Override
	public void addArgumentResolvers(List<HandlerMethodArgumentResolver> argumentResolvers) {
		// equivalent to <mvc:argument-resolvers>
	}
	
	@Override
	public void configureMessageConverters(List<HttpMessageConverter<?>> converters) {
		// equivalent to <mvc:message-converters>
	}
}

To implement the above methods, we needed references to the beans that they used; PageableArgumentResolver, MappingJackson2HttpMessageConverter and JacksonObjectMapper.

This was a simple as adding @Bean annotated methods that returned instances of each type that Spring would then manage. The overridden configuation methods could then just add those instances to their respective collections:

``` java WebConfig.java package com.silverchalice.api.config;

…

@Configuration @EnableWebMvc public class WebConfig extends WebMvcConfigurerAdapter {

@Override
public void addArgumentResolvers(List<HandlerMethodArgumentResolver> argumentResolvers) {
    argumentResolvers.add(resolver());
}

@Override
public void configureMessageConverters(List<HttpMessageConverter<?>> converters) {
	converters.add(converter());
}

@Bean
public ServletWebArgumentResolverAdapter resolver() {
	return new ServletWebArgumentResolverAdapter(pageable());
}

@Bean 
public PageableArgumentResolver pageable() {
	return new PageableArgumentResolver();
}

@Bean
public MappingJackson2HttpMessageConverter converter() {
	MappingJackson2HttpMessageConverter converter = new MappingJackson2HttpMessageConverter();
	converter.setObjectMapper(mapper());
	return converter;
}

/**
 * Provides the Jackson ObjectMapper with custom configuration for our JSON serialization.
 * @return The Jackson object mapper with non-null serialization configured
 */
@Bean
public JacksonObjectMapper mapper() {
	return new JacksonObjectMapper();
} } ```

You may have noticed that the PageableArgumentResolver was wrapped in a ServletWebArgumentResolverAdapter before being added to the argumentResolvers list.

This is because Spring Data’s PageableArgumentResolver interface uses the old ArgumentResolver interface instead of the new (Spring 3.1) HandlerMethodArgumentResolver interface. The XML config handles this behind the scenes but with a Java config it must be done manually. Thanks to @doughaber for this gotcha.

As the Spring MVC Java annotation-based configuration was complete, the references to the <mvc:annotation-driven> and jacksonObjectMapper beans in web-context.xml were removed.

We now had an annotated configuration class that configured the container - but only the Spring MVC configuration. The dispatcher servlet configuration (servlet-config.xml) was still being referenced as the contextConfigLocation under <servlet> and we hadn’t actually replaced the application context (root-context.xml) with an annotated equivalent.

So, a basic @Configuration annotated class AppConfig was created to replace root-context.xml:

``` java AppConfig.java package com.silverchalice.api.config;

import org.springframework.context.annotation.Configuration;

/**

Configuation defining shared resources visible to all other web components,
servlets etc and for cross-cutting concerns like security
@author robin */ @Configuration public class AppConfig {

}

`root-context.xml` was then deleted.

Similarly, `ServletConfig` was created to replace `servlet-context.xml`. However, we had only partially migrated the configuration. We needed to include the new `WebConfig` but also the existing `persistence-context.xml`, `bizniz-context.xml`, and `web-context.xml` XML-based configurations.

This was done using the `@Import` and `@ImportResource` annotations respectively:

``` java ServletConfig.java
package com.silverchalice.api.config;

…
	
/**
 * Configuation intended to aggregate the context for 
 * each architectural layer of the application
 * 
 * @author robin
 */
@Configuration
@Import(WebConfig.class)
@ImportResource({
	"classpath:META-INF/spring/persistence-context.xml",
	"classpath:META-INF/spring/bizniz-context.xml",
	"classpath:META-INF/spring/web-context.xml"
})
public class ServletConfig {

}

The WebConfig configuration class was explicity imported using @Import. Since we didn’t need servlet-context.xml anymore, we used @ImportResource to import in the existing XML-based context configurations and deleted servlet-context.xml.

Next, we needed to modify our web.xml so that the context knows about these Java annotation-based configurations.

The only thing needed here was to add contextClass params for both the application and dispatch servlet contexts referencing AnnotationConfigWebApplicationContext, and modifying the contextConfigLocation params to use the appropriate configuration class:

``` xml web.xml

…

contextClass

org.springframework.web.context.support.AnnotationConfigWebApplicationContext

contextConfigLocation

com.silverchalice.api.config.AppConfig

…

spring-mvc-dispatcher

org.springframework.web.servlet.DispatcherServlet

contextClass

org.springframework.web.context.support.AnnotationConfigWebApplicationContext

contextConfigLocation

com.silverchalice.api.config.ServletConfig

spring-mvc-dispatcher

	
The Spring MVC configuration was now completely Java annotation-based and the rest of the configuration, while still XML-based, was set up for incremental migration to our new way.

After deploying to Tomcat 7, the logs show that the `AnnotationConfigWebApplicationContext` has worked for both the application and dispatcher servlet configurations (which, in turn, loads the web configuration), while still picking up the renaming XML configurations:

``` logtalk
INFO : org.springframework.web.context.ContextLoader - Root WebApplicationContext: initialization started
INFO : org.springframework.web.context.support.AnnotationConfigWebApplicationContext - Refreshing Root WebApplicationContext: startup date [Thu Dec 27 23:34:24 MST 2012]; root of context hierarchy
INFO : org.springframework.web.context.support.AnnotationConfigWebApplicationContext - Successfully resolved class for [com.silverchalice.api.config.AppConfig]

…

INFO : org.springframework.web.servlet.DispatcherServlet - FrameworkServlet 'spring-mvc-dispatcher': initialization started
INFO : org.springframework.web.context.support.AnnotationConfigWebApplicationContext - Refreshing WebApplicationContext for namespace 'spring-mvc-dispatcher-servlet': startup date [Thu Dec 27 23:34:24 MST 2012]; parent: Root WebApplicationContext
INFO : org.springframework.web.context.support.AnnotationConfigWebApplicationContext - Successfully resolved class for [com.silverchalice.api.config.ServletConfig]
INFO : org.springframework.beans.factory.xml.XmlBeanDefinitionReader - Loading XML bean definitions from class path resource [META-INF/spring/persistence-context.xml]
INFO : org.springframework.beans.factory.xml.XmlBeanDefinitionReader - Loading XML bean definitions from class path resource [META-INF/spring/bizniz-context.xml]
INFO : org.springframework.beans.factory.xml.XmlBeanDefinitionReader - Loading XML bean definitions from class path resource [META-INF/spring/web-context.xml]

From <context:component-scan> to @ComponentScan

Next was to convert how packages are scanned to register beans within the application context.

web-context.xml contained the following line:

<context:component-scan base-package="com.silverchalice.api.controller" />

The com.silverchalice.api.controller package contains our Spring MVC components for the presentation layer (the web service interface we are exposing) e.g.

``` java SportsController.java @Controller @RequestMapping(“/sports”) public class SportsController { … }

To replace this component scanning directive with its Java equivalent, the `@ComponentScan` annotation is added to the web configuration class:

``` java WebConfig.java
@ComponentScan(basePackageClasses = { SportsController.class })
@Configuration
@EnableWebMvc
public class WebConfig extends WebMvcConfigurerAdapter {
	...
}

It would have been possible to just add @ComponentScan("com.silverchalice.api.controller") but by using the basePackageClasses specifier with a class reference(s), the packages of each class noted will be scanned, which is both type-safe and adds IDE support for future refactoring.

Indeed, Spring documentation suggests to “consider creating a special no-op marker class or interface in each package that serves no purpose other than being referenced by this attribute”.

<context:component-scan … /> was removed from web-context.xml.

From <beans profile="…"> to @Profile

With Spring 3.1 also came the ability to conditionally include beans under certain environment bean definition profiles.

Swagger is a JSON-based specification for “describing, producing, consuming, and visualizing RESTful web services”. We forked @marty_pitt’s Swagger Spring MVC implementation to provide deployment-tailored Swagger documentation for our web service.

We used this functionality in web-context.xml to declare a DocumentationController bean for each deployment environment:

``` xml web-context.xml …

…

Depending on which profile is active at application context start, a `documentationController` bean will be available for the specified environment.

[@cbeam](http://www.twitter.com/cbeam)'s blog post, [Spring 3.1 M1: Introducting @Profile](http://blog.springsource.org/2011/02/14/spring-3-1-m1-introducing-profile/), recommended *object-oriented configuration* by declaring configuration interfaces and then using `@Autowired` to inject the profile-dependent configuration themselves. I liked that, so we first created an interface for the application environment configuration:

``` java AppEnvConfig.java
/**
 * Environment configuration interface for application
 * 
 * @author robin
 */
public interface AppEnvConfig {

}

Before creating an environment configuration interface for the web layer, I created an interface for the Swagger configuration that would support profiles:

``` java SwaggerConfig.java import com.mangofactory.swagger.springmvc.controller.DocumentationController;

/**

Configuration for Swagger Spring MVC API Documentation support
@author robin */ public interface SwaggerConfig {

DocumentationController apiDocs();

}

The web environment configuration interface then extended both `AppEnvConfig` and `SwaggerConfig`:

``` java WebEnvConfig.java
/**
 * Environment configuration interface for web layer
 * 
 * @author robin
 */
public interface WebEnvConfig extends AppEnvConfig, SwaggerConfig {

}

Each web environment configuration that implemented this interface was then annotated with @Profile whose value specified they profile name:

``` java WebEnvDevConfig.java /**

Environment configuration for the web layer (development profile)
@author robin */ @Configuration @Profile(“dev”) public class WebEnvDevConfig implements WebEnvConfig {

@Override public DocumentationController apiDocs() { DocumentationController docs = new DocumentationController();
```
 docs.setApiVersion("1.0");
 docs.setSwaggerVersion("1.0");
 docs.setBasePath("http://localhost:8080/webservice");
	
 return docs;  }
```

}

	
The `apiDocs` method is then overridden per profile and the `DocumentationController` instance is configured for that environment. Similarly, the production environment configuration (`WebEnvProdConfig`) class would instantiate an instance customized for production URLs.

<p />
---
**From `<context:property-placeholder location="…" />` to `@PropertySource`**

However, the various implementations of `WebEnvConfig` with slightly modified `DocumentationController` instances felt like a ["code smell"](http://www.codinghorror.com/blog/2006/05/code-smells.html).

Instead I preferred to use `.properties` files to detail environment-specific settings, for example, `web-dev.properties`:

``` properties web-dev.properties
apiVersion="1.0"
swaggerVersion="1.0"
basePath="http://localhost:8080/webservice"

To use properties previously with XML-based configuration, a property placeholder definition would be used:

<context:property-placeholder location="web-dev.properties" />

With Spring 3.1, the Environment abstraction allows searching for properties across a hierarchy of property sources.

The @PropertySource annotation will specifiy a location of properties that will be added to the environment. This makes it very easy to both add to and retrieve properties from the Environment.

First, the WebEnvConfig abstract class (that now contains the apiDocs method) was altered to use the @Autowired Environment instance to retrieve the appropriate property value:

``` java WebEnvConfig public abstract class WebEnvConfig implements AppEnvConfig, SwaggerConfig {

@Autowired Environment env;

@Override
public DocumentationController apiDocs() {
	DocumentationController docs = new DocumentationController();
	
	docs.setApiVersion(env.getProperty("apiVersion"));
	docs.setSwaggerVersion(env.getProperty("swaggerVersion"));
	docs.setBasePath(env.getProperty("basePath"));
	
	return docs;
}

}

The, for each `@Profile` annotated concrete web configuration class, the `@PropertySource` annotation's value denotes the location of the `.properties` file to be used:

``` java WebEnvDevConfig
@Configuration
@Profile("dev")
@PropertySource("classpath:web-dev.properties")
public class WebEnvDevConfig extends WebEnvConfig {

}

Therefore, for whatever the active profile was (for instance, if the JVM parameter -Dspring.profiles.active=dev was used), the .properties file for that profile was loaded.

This meant that we could inject in a WebEnvConfig instance into our WebConfig class and expose our Swagger Spring MVC functionality that would be customized for the environment profile it was launched under.

To do this, we needed to import the profile annotated configuration classes, autowire a configuration instance and them expose a single bean with the desired functionality:

``` java WebConfig.java @ComponentScan(basePackageClasses = { SportsController.class }) @Configuration @EnableWebMvc @Import({WebEnvDevConfig.class, WebEnvProdConfig.class}) public class WebConfig extends WebMvcConfigurerAdapter {

// the web @Configuration class injected for this @Profile
@Autowired WebEnvConfig webEnvConfig;

…

/**
 * Swagger Spring MVC API Documentation support
 * 
 * @return swagger documentation controller
 */
@Bean
public DocumentationController documentationController() {
	return webEnvConfig.apiDocs();
}

}

Naturally, this approach could be extended across all application layers.

<p />
---
**From `web.xml` to `WebApplicationInitializer`**

The last piece of the puzzle of converting our Spring MVC app's web layer to a fully Java annoatation-based configuration was the conversion of the deployment descriptor itself.

At this moment, our web service was set up as a Servlet 2.5 application, but, since we were using Apache Tomcat 7 as our servlet container, we could use [Servlet 3.0 and its annotation support](http://explodingjava.blogspot.in/2010/05/servlet-30-annotations.html) instead. This also opened up the opportunity to replace most, if not all, of `web.xml` with a programmatic equivalent written in Java.

In order to configure the `ServletContext` programmatically, we needed to create a class that implemented `WebApplicationInitializer`. Implementations of this interface would be detected automatically and bootstrapped by the Servlet 3.0 container.

``` java WebInit.java
package com.silverchalice.api.config;

import javax.servlet.ServletContext;
import javax.servlet.ServletException;

import org.springframework.web.WebApplicationInitializer;

public class WebInit implements WebApplicationInitializer {

	@Override
	public void onStartup(ServletContext container) throws ServletException {
		// TODO Auto-generated method stub
	}

}

This is simply a matter of replicating our changes to web.xml within the WebInit class.

As we are using @Profile now, we need to tell the container context what profile to use if no active profile has been configured (either as a JVM argument, an environment variable etc.). This was done in web.xml by setting a context init-param:

<!- If no active profile is set via -Dspring.profiles.active or 
some other mechanism then this will be used. -->
<context-param>
        <param-name>spring.profiles.default</param-name>
        <param-value>dev</param-value>
</context-param>

In our new WebInit class, this is very easy:

``` java WebInit.java public class WebInit implements WebApplicationInitializer {

@Override
public void onStartup(ServletContext container) throws ServletException {
	/*
	 * If no active profile is set via -Dspring.profiles.active or 
	 * some other mechanism then this will be used.
	 */
	container.setInitParameter("spring.profiles.default", "dev");
	
	…
	
}

}

	
Next we created the root application context itself, set its display name, registered the root application configuration and added a context loader listener to bootstrap the container - all very standard stuff. We had set this all up earlier in `web.xml`:

``` xml
<?xml version="1.0" encoding="ISO-8859-1"?>
<web-app version="2.5" xmlns="http://java.sun.com/xml/ns/javaee" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://java.sun.com/xml/ns/javaee http://java.sun.com/xml/ns/javaee/web-app_2_5.xsd">
...
	<display-name>Silver Chalice Web API</display-name>
	<context-param>
	        <param-name>contextClass</param-name>
	        <param-value>
				org.springframework.web.context.support.AnnotationConfigWebApplicationContext
	        </param-value>
	</context-param>
	<context-param>
	        <param-name>contextConfigLocation</param-name>
	        <param-value>com.silverchalice.api.config.AppConfig</param-value>
	</context-param>
	<!-- Creates the Spring Container shared by all Servlets and Filters -->
	<listener>
	        <listener-class>org.springframework.web.context.ContextLoaderListener</listener-class>
	</listener>
...
</web-app>

but it was very straightforward to do this through Java:

// Creates the root application context
AnnotationConfigWebApplicationContext appContext = 
		new AnnotationConfigWebApplicationContext();

appContext.setDisplayName("Silver Chalice Web API");

// Registers the application configuration with the root context
appContext.register(AppConfig.class);

// Creates the Spring Container shared by all Servlets and Filters
container.addListener(new ContextLoaderListener(appContext));

Filters were straightforward to convert also:

<filter>
        <filter-name>PerfLoggingFilter</filter-name>
        <filter-class>com.silverchalice.api.filter.PerfLoggingFilter</filter-class>
</filter>
<filter-mapping>
        <filter-name>PerfLoggingFilter</filter-name>
        <url-pattern>/*</url-pattern>
</filter-mapping>

became:

container.addFilter("PerfLoggingFilter", PerfLoggingFilter.class)
			.addMappingForUrlPatterns(null, false, "/*");

Finally the DispatchServlet itself was migrated from:

<servlet>
	<servlet-name>spring-mvc-dispatcher</servlet-name>
	<servlet-class>org.springframework.web.servlet.DispatcherServlet</servlet-class>
	<init-param>
		<param-name>contextClass</param-name>
		<param-value>
			org.springframework.web.context.support.AnnotationConfigWebApplicationContext
		</param-value>
	</init-param>
	<init-param>
		<param-name>contextConfigLocation</param-name>
		<param-value>com.silverchalice.api.config.ServletConfig</param-value>
	</init-param>
	<load-on-startup>1</load-on-startup>
</servlet>
<servlet-mapping>
        <servlet-name>spring-mvc-dispatcher</servlet-name>
        <url-pattern>/</url-pattern>
</servlet-mapping>

to:

// Creates the dispatcher servlet context
AnnotationConfigWebApplicationContext servletContext = 
		new AnnotationConfigWebApplicationContext();

// Registers the servlet configuraton with the dispatcher servlet context
servletContext.register(ServletConfig.class);

// Further configures the servlet context
ServletRegistration.Dynamic dispatcher = 
		container.addServlet("spring-mvc-dispatcher", 
				new DispatcherServlet(servletContext));
dispatcher.setLoadOnStartup(1);
dispatcher.addMapping("/");

web.xml can be deleted now and the application starts just fine:

INFO : org.springframework.web.context.ContextLoader - Root WebApplicationContext: initialization started
INFO : org.springframework.web.context.support.AnnotationConfigWebApplicationContext - Refreshing Silver Chalice Web API: startup date [Tue Jan 01 19:21:33 MST 2013]; root of context hierarchy
INFO : org.springframework.web.context.support.AnnotationConfigWebApplicationContext - Registering annotated classes: [class com.silverchalice.api.config.AppConfig]
…
INFO : org.springframework.web.context.ContextLoader - Root WebApplicationContext: initialization completed in 323 ms
DEBUG: com.silverchalice.api.filter.PerfLoggingFilter - initializing perf logging filter
…
INFO: Initializing Spring FrameworkServlet 'spring-mvc-dispatcher'
INFO : org.springframework.web.servlet.DispatcherServlet - FrameworkServlet 'spring-mvc-dispatcher': initialization started
INFO : org.springframework.web.context.support.AnnotationConfigWebApplicationContext - Refreshing WebApplicationContext for namespace 'spring-mvc-dispatcher-servlet': startup date [Tue Jan 01 19:21:34 MST 2013]; parent: Silver Chalice Web API
INFO : org.springframework.web.context.support.AnnotationConfigWebApplicationContext - Registering annotated classes: [class com.silverchalice.api.config.ServletConfig]
…

And that’s it.

From <aop:aspectj-autoproxy /> to @EnableAspectJAutoProxy

The single remaining item left in web-context.xml was this:

<beans …>
		<!-- Enable AspectJ auto-wiring -->
		<aop:aspectj-autoproxy />
</beans>

We were using AspectJ with Perf4J to profile our web service. We had added the Perf4J @Profiled annotation (not to be confused with the Spring annotation @Profile) to our controllers and had created an aop.xml file configuring our timing aspect.

To remove web-context.xml completely we just needed to add the @EnableAspectJAutoProxy annotation to our WebConfig configuration class:

@ComponentScan(basePackageClasses = { SportsController.class })
@Configuration
@EnableAspectJAutoProxy
@EnableWebMvc
@Import({WebEnvDevConfig.class, WebEnvProdConfig.class})
public class WebConfig extends WebMvcConfigurerAdapter {

	…
	
}

Then the reference to web-context.xml was removed from ServletConfig and the file itself was deleted.