A Blog by Robin Howlett
After close to 5 years with Silver Chalice/SportsLabs, today, Friday August 7th 2015, will be my last day with the company. On Monday, I join SnapLogic, an enterprise integration platform-as-a-service (iPaaS) provider that I believe has an amazing future.
It has been a tremendous, life-changing journey.
I joined SportsLabs (then still under the Silver Chalice brand) way back in 2011 as one of its earliest employees and the first engineer.
We started work on envisioning and building the Advanced Media Platform - a system to ingest, process, transform, distribute, and stream sports, news, social, and media content to create market leading mobile, web, and social products for clients such as Samsung, the University of Notre Dame, the ACC, the College Football Playoff, IMG College, the Mountain West and Campus Insiders, among others.
Since then, SportsLabs has consumed data from dozens of sources including STATS LLC, Twitter, and Ooyala, but also from proprietary systems that were never foreseen as integration points.
Data providers’ APIs use combinations of JSON, XML and/or CSV. Some are spec-compliant, others are not. Some rely heavily on query parameters, while others favor HTTP headers. Some API providers use OAuth 2.0 plus API rate limits, while others have rolled their own security solutions. Some integrations were with partners willing to work with us on evolving their web services. Others were with competitors who were not motivated to make things easy.
This plethora of ways to configure, consume, learn from, and integrate with APIs led us to create Spring Social Bootstrap, a family of projects intended to aid creating and managing API clients for many of the above scenarios.
Spring Social Bootstrap is comprised of the following:
Out of the box, Spring Shell supports printing command results to the terminal in a fairly basic way.
Spring Shell also provides the ExecutionProcessor interface, allowing a “command provider to be called in a generic fashion just before, and right after, executing a command”.
The interface defines three lifecycle events that can be intercepted:
I was interested in hooking into the afterReturningInvocation to provide “step logic” - potentially allowing user or system input to execute additional logic based on the result of the initial command result (and/or each step result) e.g. paging backwards or forwards on the command line through lists of data.
I was able to achieve this and opened a JIRA ticket and the following pull request on Spring Shell’s GitHub repo: SHL-174: Multi-Step Commands #67
Be it DVCS workflows, JSON transformations, or blogging frameworks, I always favor tools that allow me to use the terminal.
I’ve recently started using Spring Shell for rapidly experimenting with consuming data from a series of APIs I had created.
I have become allergic to XML configuration for Spring applications in recent times, so I was disappointed to see a lack of support for Java configuration within Spring Shell.
However, I did find a JIRA issue tracking the feature request. The ticket creator had even submitted a pull request with a potential solution. However, Spring Shell lead Mark Pollack responded that the feature could be provided in a simpler manner and even provided guidance to the solution.
Implementing this solution seemed quite straightforward, so I gave it a go and it turned out well.
I’ve submitted the following pull request to the Spring Shell GitHub repository: SHL-106: Java Configuration support #66
At SportsLabs we regularly encounter proprietary, non-standard APIs and formats. Our job is to integrate with these APIs, normalize them, and distribute the data in web- and mobile-friendly web services.
One of the scenarios we often encounter is a provider supplying multiple resource JSON-based APIs that share a lot of the same data in their responses, but without any particular field dedicated to identying the type of the resource e.g.
{
...
"common": "a common field within multiple resource responses",
"one": "one is a field only within this response type"
...
}
and
{
...
"common": "a common field within multiple resource responses",
"two": "two is a field only within this response type"
...
}
Instead of mapping 1-to-1 with these APIs, we often try to follow DRY principles and model them as implementations of a common polymorphic abstraction.
When using Jackson for polmorphic deserialization and not being in control of the API response data, the lack of any kind of type identifier requires a different solution.
One of the ways we’ve addressed this problem is to identify fields and properties that are unique to a particular resource API’s response.
We then add this field to a registry of known unique-property-to-type mappings and then, during deserialization, lookup the response’s field names to see if any of them are stored within the registry.