An increasing number of large institutions are now wanting to leverage Clojure. Large institutions are inherently complex and can often be dysfunctional in various areas. They do however attract lots of very good people and good people often bring with them the best tools.
Investment banks in particular have hard problems to solve. The financial instruments themselves have varying degrees of complexity, although whatever complexity they have often pales in comparison against the systems ecosystem that maintains them. I said once to a senior manager at one place that the number of systems I was seeing in various strates reminded me of Darwin’s survival of the fittest, beasts coming in all shapes and sizes. He said I was wrong because in the wild animals actually die rather than live on to relentlessly cause problems.
Over a year and half ago I was a team lead on a medium sized development team running a strategically important middle-office system. We were 100% a Java team and we had some pretty good devs onboard with decent backgrounds. We had plans to split up our Spring/Hibernate 1 million LOC behemoth and were getting in some decent wins to make this happen. Then I persuaded a Swedish colleague I’d worked with before to come work with us. I had the feeling the team needed someone to come in to and give us a kick and to challenge our general direction – he certainly lived up to this. Large scale refactorings I was fairly proud of he denounced as ‘just moving shit around’. If we were serious about changing our system then the only way to do this was to change our course radically. Although he was challenged fairly robustly – to the extent that at times I know he got quite pissed off – a few things we were already conscious of.
Firstly Java encourages lots of modelling. “Let’s sit down with all the power of OO and create of lots of classes that attempt to model the problem at hand. We won’t use UML but instead we’ll evolve it in an agile way.” The model in the investment banking world is actually quite complicated so this OO graph grows quite large. Abstraction is brought in do deal with variance; abstract classes and interfaces proliferate. The type system encourages this. I had used some dynamical languages before and it was quite obvious that we were essentially forcing lots of schema and type definition on to a problem domain that just didn’t want or need it. Sure you can minimize the pain with patterns such as composition over inheritance, better abstractions etc, but we had already admitted to ourselves in some quarters that we were fighting the wrong battle.
And soon it becomes like wading in cement. Our system was predominately conceived around 2007 when it was the heyday of using Test Driven Development (TDD) and to have a heavy reliance on dependency-injection frameworks like Spring. I should state now that I don’t actually dislike Spring with any religious fervour – it just looks redundant in retrospect now that I’m working 95% with FP code. Our app back then also had lots of Mock framework usage – we had both JMock and Mockito flavours. Before our big move to Clojure I’d already decided I just didn’t get on with mock frameworks. They slow down development as you constantly find yourself fighting against lots of incomprehensible tests written dogmatically that rarely test anything useful. I really can’t really think of many situations right now when using Mocking would be a good idea, even in a Java/OO setting.
I had also picked up a consensus view amongst industry peers that the only way to truly break up a big monolithic system like ours was to enforce hardline decoupling around service boundaries. The best way to achieve this is naturally to use different languages. If we’d have stuck with Java we’d never have ‘gone for broke’ to drop a crowbar into our system and to wrestle it apart. Now we’re using Clojure this is essentially what we’re doing, and not in a bad way.
You also need to ask yourself when contemplating a fairly seismic shift of tool usage what about the business? What about the stakeholders? Who is going to pay for this kind of re-engineering and why?
In the end there are a lot of answers to these questions. My answer of choice tends to be waste. Waste on the scale lots of project have should not be tolerated. Waste in terms of time spent wading through cement-styled Java, time spent hunting through layers upon layers of indirection, interfaces, unit/integration/acceptance tests, Spring XML files, just to find the one if-statement that actually has some business significance, Eureka! We had/have waste on a fairly large scale and to start developing our system in a fundamentally different way almost seems the only sensible option especially if you want people to stick around.
Here are some specifics – our new code is going to be an order of magnitude less in volume than the old and this is being conservative. With the new code you can use your REPL to navigate in and to run any little bit of it you like. You can change any code and see the changes immediately in your browser (our app lives inside of web-server instances) or REPL. Instead of modelling the world, every man and his dog, we’re going to instead write code that operates against simple data structures. If then we need to be able to determine what is actually going on – wtf – we’ll add lots of introspective abilities to our code. In short, to replace copious amounts of Java code along with built up DSLs that only devs use, we’re going to build a simple rule engine. And we’re going to provide some visualisation tools so that you can actually see the rules themselves. A rule editor may follow.
Sure we had lots of debates about why we couldn’t do what we need to do in Java, and in the end it became clear that you couldn’t really have the debate on these terms. The decision of switching to Clojure fell in to three natural brackets. 1) What’s good for the long term nature of the project? 2) What’s good for this present team? and 3) What’s good for the individuals involved? 1) FP seemed a better fit for our system – it’s basically functional blocks of code hanging of financial market events. Our system also suits a dynamic language making all the rigid schema definition less of a problem. On a different tack surely the long term view project is best served by using the best tools and being able to attract the best people. 2) The team will be motivated learning something new and we’ll likely get better retention. This is ultimately good for the costs of the business. 3) The majority of the team wanted to try Clojure. As Hakan made the point – we don’t get paid to enjoy ourselves at work, but it doesn’t hurt. It’s an obvious point – a happier team is more productive.
In the end at some point a team just has to take a jump. Certainly there’s been problems and long standing issues in our move to Clojure and there are some significant questions still yet to be resolved. We’re a global team and this brings with it a wealth of complexities. A year ago we were also a bunch of Clojure newbies finding our way and some of our code and decisions reflect this. I’ll write about these in a subsequent post.