Clojure at a Bank – Freeing the Rules

November 29, 2012 § 1 Comment

I’ve written previous posts about a team at an investment bank making the switch from Java to Clojure. In this post I’d like to focus on the business rules being moved in the process.

Rules/Memes

I’ve found that business rules at large institutions tend to resemble viral memes and genes. This is to say that they are regularly duplicated amongst systems, manual or automated, and most will persist long after the people maintaining them move on. The hardy ones manage to jump from dying systems into new ones.

In our case the rules are migrating from a large monolithic Java stack into a new Clojure code base. We wanted to give them a reformed existence that would be more streamlined, free of elaborately crafted OO structures, and where they would not be hemmed in by overly enthusiastic and rigidly defined tests.

Tags rather than Types

One of the first things we did differently was to use tags rather than types for modeling the business domain. In the old world a financial product would immediately be given its own Java class and its similarities with other products expressed through inheritance and interfaces, perhaps with some mixin styled object composition thrown in. This approach has failed our system. Sure that with the right people, time and foresight, most systems in most languages can be beautiful, but in my opinion the generalistic approach of modeling the problem with static types opens the door wide open to epic waste and codebases with limited options.

For our Clojure codebase we instead simply treated products and trades as maps of immutable data and we built up a corresponding set of keywords – ‘tags’ – to describe what we know about them.

Imagine that we’re processing orders for cups of coffee. Without breaking the problem domain down to the nth degree, we can start by introspecting and tagging whatever data comes our way with business significant terms such as :hot, :milky, and :has-froth. Predicates to drive business rules become trivial to write, i.e. (when (superset? tags #{:sprinkles :caffiene :large}) (do-something)). The code matching up data to tags would then become an all-important kernel of the system.

CSS styled JSON building

We used tags for a range of things, one of the prime cases being to build up JSON for sending to a downstream system. We used tags to determine a) JSON schemas and b) JSON values.

Taking values first, imagine that we’ve built up a JSON structure but with the values omitted, i.e:

{:beverage {:type _ :temperature _ }}

A simple DSL for defining values may then look like:

(values/add :type #{:cappucino} "Coffee")

and:

(values/add :temperature #{:cappucino} "Hot")

Here the rules look a lot like CSS. We use path selectors such as :type and :temperature and instead of matching on HTML class attributes we simply match on tags; #{:cappucino}.

The rules for building up the schema could work in a similar way. A :frappuccino may need a different outgoing data structure than for a :cappucino, perhaps to do with some inherent complexity of iced drinks.

What’s covered here is fairly trivial stuff but it’s a starting point for more. You could extend this DSL as to allow multiple value-rules to be defined at once, to add predicates, and to make the values Clojure functions that work off the input data. You do of course need some core boilerplate code for mashing the JSON schema and value rules together but this should be simple to write.

Like CSS there are some pros and cons. Pros are that adding new rules becomes trivial and that the rules themselves are inherently reusable. Cons are of having to manage a large set of flattened rules.

Rules as Data

On our project we’ve got a fight on our hands in that the number of business rules is large whilst having a wide amount of variance across them. Instead of spreading the rules out we have now a few thousand – and growing – number of rules living in a concentrated handful of Clojure namespaces. At first glance this looks like poor design, this is fugly and bad right? What about storing the rules in a DB or reapplying a fair bit of OO styled modeling?

First because all Clojure code is data then we absolutely do have a rule DB, and not just a pile of namespace code editable in Emacs. By thinking of the rules entirely as data we’ve opened options for ourselves.

For example we’ve built a UI that allows non-technical users to directly browse the rules and to walk the hierarchies between them. Users can play around with input data and tags to see what rules are used for a given context. They can view information about why a certain rule is selected above another, the CSS-like specificity behavour. We also show the source code, namespace and line number behind each rule. A team member recently added the ability for devs to click on an html link that opens up Emacs at exactly the right point where the rule-code is.

The rules form an audit trail of how payloads were crafted. Since we also use them for data reconciliation purposes users can now write comments on data mismatches, directly in the context of a rule.

REPL, Builds and Tests

The tags and rules-as-data approach has allowed us to build up a set of tools for experimenting in the REPL. For example it’s common to query an ElasticSearch instance from the REPL to bring back a certain population of test-data and then to throw it at the rule engine to see what happens. We’ve got build-server agents doing a similar thing to alert us when changes in the rules bring about unexpected consequences.

As a more immediate line of defense we’ve got unit tests utilising core.logic to make sure that the rules are sane and that repetition and redundancy are kept to a minimum if not prohibited. We’ve added to the UI the ability to highlight sets of rules that are too close to each other, where a revision of tags might be applied to clean things up.

And if one day we decided that our rules needed to be stored in a completely different way, then we could always write some more Clojure code to read them back in and to spit them out again as something different. The door to moving them into a graph DB, RDF, or into a fact based DB like Datomic is not closed.

Wrap up

We’ve learnt that when you’re working with rules as data then many more possibilities open up as to what you can do with them. Quite the opposite to what you typically see at large institutions where the rules are tucked away like diamonds in the earth inside of type-heavy OO modeled systems. For us Clojure has been a great emancipator of business rules.

§ One Response to Clojure at a Bank – Freeing the Rules

Leave a Reply

Your email address will not be published. Required fields are marked *

You may use these HTML tags and attributes: <a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <strike> <strong>

What’s this?

You are currently reading Clojure at a Bank – Freeing the Rules at Pithering About.

meta