Clojure at a Bank – Moving from Java

October 23, 2012 § 17 Comments

An increasing number of large institutions are now wanting to leverage Clojure. Large institutions are inherently complex and can often be dysfunctional in various areas. They do however attract lots of very good people and good people often bring with them the best tools.

Investment banks in particular have hard problems to solve. The financial instruments themselves have varying degrees of complexity, although whatever complexity they have often pales in comparison against the systems ecosystem that maintains them. I said once to a senior manager at one place that the number of systems I was seeing in various strates reminded me of Darwin’s survival of the fittest, beasts coming in all shapes and sizes. He said I was wrong because in the wild animals actually die rather than live on to relentlessly cause problems.

Over a year and half ago I was a team lead on a medium sized development team running a strategically important middle-office system. We were 100% a Java team and we had some pretty good devs onboard with decent backgrounds. We had plans to split up our Spring/Hibernate 1 million LOC behemoth and were getting in some decent wins to make this happen. Then I persuaded a Swedish colleague I’d worked with before to come work with us. I had the feeling the team needed someone to come in to and give us a kick and to challenge our general direction – he certainly lived up to this. Large scale refactorings I was fairly proud of he denounced as ‘just moving shit around’. If we were serious about changing our system then the only way to do this was to change our course radically. Although he was challenged fairly robustly – to the extent that at times I know he got quite pissed off – a few things we were already conscious of.

Firstly Java encourages lots of modelling. “Let’s sit down with all the power of OO and create of lots of classes that attempt to model the problem at hand. We won’t use UML but instead we’ll evolve it in an agile way.” The model in the investment banking world is actually quite complicated so this OO graph grows quite large. Abstraction is brought in do deal with variance; abstract classes and interfaces proliferate. The type system encourages this. I had used some dynamical languages before and it was quite obvious that we were essentially forcing lots of schema and type definition on to a problem domain that just didn’t want or need it. Sure you can minimize the pain with patterns such as composition over inheritance, better abstractions etc, but we had already admitted to ourselves in some quarters that we were fighting the wrong battle.

And soon it becomes like wading in cement. Our system was predominately conceived around 2007 when it was the heyday of using Test Driven Development (TDD) and to have a heavy reliance on dependency-injection frameworks like Spring. I should state now that I don’t actually dislike Spring with any religious fervour – it just looks redundant in retrospect now that I’m working 95% with FP code. Our app back then also had lots of Mock framework usage – we had both JMock and Mockito flavours. Before our big move to Clojure I’d already decided I just didn’t get on with mock frameworks. They slow down development as you constantly find yourself fighting against lots of incomprehensible tests written dogmatically that rarely test anything useful. I really can’t really think of many situations right now when using Mocking would be a good idea, even in a Java/OO setting.

I had also picked up a consensus view amongst industry peers that the only way to truly break up a big monolithic system like ours was to enforce hardline decoupling around service boundaries. The best way to achieve this is naturally to use different languages. If we’d have stuck with Java we’d never have ‘gone for broke’ to drop a crowbar into our system and to wrestle it apart. Now we’re using Clojure this is essentially what we’re doing, and not in a bad way.

You also need to ask yourself when contemplating a fairly seismic shift of tool usage what about the business? What about the stakeholders? Who is going to pay for this kind of re-engineering and why?

In the end there are a lot of answers to these questions. My answer of choice tends to be waste. Waste on the scale lots of project have should not be tolerated. Waste in terms of time spent wading through cement-styled Java, time spent hunting through layers upon layers of indirection, interfaces, unit/integration/acceptance tests, Spring XML files, just to find the one if-statement that actually has some business significance, Eureka! We had/have waste on a fairly large scale and to start developing our system in a fundamentally different way almost seems the only sensible option especially if you want people to stick around.

Here are some specifics – our new code is going to be an order of magnitude less in volume than the old and this is being conservative. With the new code you can use your REPL to navigate in and to run any little bit of it you like. You can change any code and see the changes immediately in your browser (our app lives inside of web-server instances) or REPL. Instead of modelling the world, every man and his dog, we’re going to instead write code that operates against simple data structures. If then we need to be able to determine what is actually going on – wtf – we’ll add lots of introspective abilities to our code. In short, to replace copious amounts of Java code along with built up DSLs that only devs use, we’re going to build a simple rule engine. And we’re going to provide some visualisation tools so that you can actually see the rules themselves. A rule editor may follow.

Sure we had lots of debates about why we couldn’t do what we need to do in Java, and in the end it became clear that you couldn’t really have the debate on these terms. The decision of switching to Clojure fell in to three natural brackets. 1) What’s good for the long term nature of the project? 2) What’s good for this present team? and 3) What’s good for the individuals involved? 1) FP seemed a better fit for our system – it’s basically functional blocks of code hanging of financial market events. Our system also suits a dynamic language making all the rigid schema definition less of a problem. On a different tack surely the long term view project is best served by using the best tools and being able to attract the best people. 2) The team will be motivated learning something new and we’ll likely get better retention. This is ultimately good for the costs of the business. 3) The majority of the team wanted to try Clojure. As Hakan made the point – we don’t get paid to enjoy ourselves at work, but it doesn’t hurt. It’s an obvious point – a happier team is more productive.

In the end at some point a team just has to take a jump. Certainly there’s been problems and long standing issues in our move to Clojure and there are some significant questions still yet to be resolved. We’re a global team and this brings with it a wealth of complexities. A year ago we were also a bunch of Clojure newbies finding our way and some of our code and decisions reflect this. I’ll write about these in a subsequent post.

  • zero

    Regarding TDD and mocking, you should read the GOOS book, because I think using a functional language won’t save you from writing isolated tests (Unless you want no tests, or test everything endtoend).

    What you wrote about your system, seems to me that it has more general design problems than problems which FP claims to solve. There wasn’t any word about concurrency, state management or side effects. Designing and maintaining large systems in a functional language needs design skills, and I don’t think this industry has too much experience with it.

  • Jon Pither

    Hi zero. I plan to write in a separate post a bit about our testing strategy. We do have a significant amount of testing in place (isolated at the unit level as well as acceptance/integration) but we went about it differently than using plain old fashioned TDD – i.e. we wrote tests where we knew we needed them – more so for regression and pain relief purposes than for needing TDD to help steer design. It’s an interesting topic – when you have REPL you are TDD’ing in a different way as opposed to constructing persistent unit-test classes that could potentially form a layer of cement.

    I’ve read a bit of the GOOS book but I’ve suffered so much around mocks in the past I’m just not sure they justify their existence 99% of the time.

    WRT to the rest of your comment I plan to write more around the ‘immaturity’ phase of what we were doing.

  • Jakub Holý

    Hi Jon,

    Thank you for an inspiring post! If an investment bank could do the move then hopefully more enterprises can follow :-)

    I really like that you have sound justification for making the switch, i.e. because FP and a dynamic language are better match for the domain.

    I would have similar concerns as Zero regarding industry experiences with FP design and maintenance of large systems so I’m really looking forward to your follow-up posts.

    Regarding TDD, I’m a big fan of this technique and I know it has a great value. REPL can replace some throw-away tests but there is still value in good, high-level tests for the purpose of regression and refactoring. It is not easy to write good tests that provide a valuable safety net without cementing the design but it is possible and worth the effort. I’m not fond of mocking and TDD can well work without it (the father of TDD himself doesn’t use mocks). I firmly believe that people must deeply understand the value of testing and TDD to apply it beneficially without producing tests that cause more pain than gain (->

  • Pingback: Most interesting links of October ’12 « The Holy Java()

  • Cabins hocking hills

    Thank you so much for sharing such an informative article that is very well-written too.

  • Pingback: My 2012 in Review « The Holy Java()

  • Pingback: Clojure in Finance | Tales from a Trading Desk()

  • Yujun Liang

    Why not use Scala?

  • John

    John – could you point us to online resources that describe the FP programming way that allow you to reduce your code by an order of magnitude over Java? I’m really interested in expanding my horizons; I’ve been an OO programmer for my entire career and would like to branch into FP.

  • James

    As was said a long time ago by Paul Graham

    The phrase “object-oriented” means a lot of things. Half are obvious, and the other half are mistakes.

  • hnthrowaway

    Paul Graham is a fairly sophomoric commenter on computer languages and speaks from an experience level of effectively zero in context of actually creating, deploying, and maintaining software in the enterprise. (No, a shopping cart web app does not make him a tech guru.)

    • bobfoster

      Just because Paul Graham never wasted his time working on beetle-brained “enterprise” systems “written by large (and frequently changing) teams of mediocre programmers” doesn’t mean he was wrong.

      • SadForTheWorld

        >> written by large (and frequently changing) teams of mediocre programmers” doesn’t mean he was wrong.

        The distribution of skills sucks, yes, but it is reality. For example, I’ve interviewed 30 people in the last month to join our team. Only 4 were able to code simple algorithms during an interview coding exercise. These were people with 5+ yrs experience and degrees in Dallas, TX. For *most* businesses, we have to try to go to market with the programmers we can get…and they’re shitty. The proficient are rare, the experts work in San Fran+NYC. I am skeptical that with the typical mediocre team, FP is going to be the big win. Though I can hope.

  • Torbjörn Josefsson

    Your plan is good – and I can assure you that it will have benefits. We’ve been doing a very similar thing for, oh, more than 10 years, and we seem to be having remarkably few problems :)

    The graphical rule editor is, in my opinion, one huge advantage. Nobody wants to keep track of a 10000 lines logic rule-base which needs to change regularly according to customer needs. The drawings are much better to aid memory and helps navigation Tremendously.

    We genererate Prolog code – mostly run it as a service (but also in test and simulation clients), and it Is a good fit for banking and stuff 😉

    Not sure how much more I can tell you, but none of these things are secrets to my company, as far as I know, and most of what I’ve said is advertised on the website (Emric AB)

    We are also Swedish… hmmm. Maybe it’s something in the water?

  • ozo

    I will be quite interested to know what your present day thoughts are on Clojure, and whether you and your team (if you still reside in the same office) are still flying high with Clojure. Do you still reckon it is a language of the future, or are you considering something else?

What’s this?

You are currently reading Clojure at a Bank – Moving from Java at Pithering About.