I was fortunate to have worked on some low latency applications built within the equities teams of some top investment banks.

Also, recently I built an Order Manager application for a fintech and was able to get a performance of <1 millis per order processing. The application had to connect with multiple market makers over FIX and each of them were sending simultaneous orders with different price levels. 

The requirement was also to seamlessly scale out as the number of market makers grew.

Here are the key concepts behind designing and implementing such low latency applications.

Look to Scale up and Scale out

Scaling up (vertical scaling)

Each of the processing nodes should be written with ultra-low latency (<1millis) in mind. In the Java world, one key aspect is to avoid Garbage Collection (GC) cycles and going with zero GC – an approach very common in the equities world.
Additionally, building event-driven modules is a big plus. This means nothing gets blocked. The fast part is separated from the slow part in the application. And for all the slow calls, instead of using blocking I/O calls, use an event-driven model.

Scaling out (horizontal scaling or elasticity)

As the nodes are scaled, more often than not, they will share some state between themselves. This is called Connection Stickiness. For example the same order could have been created by Node1 but executed by Node2. This requires Node1 and Node2 to synchronise order states and this is a blocking operation. People get this wrong at times by choosing a distributed cache like Hazelcast. Distributed caches are a big ‘no’ for ultra low latency systems as it can result in unpredictable slowness while trying to ensure state synchronisation causing losses in peak business hours. The best approach is to use a message bus like Kafka and use its topic partitioning facility for handling connection stickiness.

Tend towards Zero Garbage Collection

Object allocation and reclamation are expensive tasks in Java. Depending on the object type, the creation time could be substantial (Java Date object etc.) and then when it reclaimed, the Garbage Collection can slow down the application threads and at times block trading entirely for millis or seconds potentially causing financial losses.

Here are some of the approaches to reducing Garbage Collection in Java:

Recycling of objects using a Object Pool Manager

 You can implement an Object Pool Manager for a defined set of POJOs. The pool manager could learn how many objects of each POJO were in use on the last day and can create many of these objects in advance when the application loads in the morning. The client code can ask for an object from the pool manager by providing its type, use the object and then call its recycle() function to put the object back in the pool. If the Pool Manager runs out of objects, it can use the Object’s factory to create a new one.

With this approach, since the objects live forever, there is no GC required, hence allowing a very high throughput for the application. Additionally, there is no object creation cost during peak trading hours.

LMAX Disruptor also does this for the message elements in its ring buffer.

Reduce memory footprint

Reuse objects as much as possible and reduce the number of bytes per object. Lean towards using mutable objects with a ThreadLocal instead of going for immutable objects. For example you can define a ReusableString class which is a mutable version of Java’s String class and can use it from an object pool within a ThreadLocal construct.

You can also define your own alternative objects to Java. Like a LeanString class similar to Java’s String class but backed by 1 byte instead of 2. This should be fine if you only care for UTF-8 encoding and will reduce your memory footprint by half. Remember low memory footprint means more optimised code can sit on the L1 cache of the CPU.

Choose the appropriate Garbage Collector

CMS is better than Parallel Mark and Sweep as it blocks application threads (StopTheWorld) for a minimum time. Compare the performance benchmark between CMS and G1 collector and then make a decision. Switch on the GC logs and heap dump options. Choose whether to set Xms the same as Xmx or slightly lower. Please bear in mind that when Xms is equal to Xmx, there is a chance of an OutOfMemoryError as GC doesn’t clean up on time sometimes. It isn’t advisable to tinker with the young and old generation ratios unless you really need it and know what you are doing, as again it can cause OutOfMemoryExceptions.

Also, monitor the GC using VisualVM

Avoid Thread synchronisation in latency-sensitive execution path

In the last decade, engineers have learnt that multi-threading can cause more bad than good at times. Modern machines have 8 to 16 CPU cores. The CPU cores have a cache hierarchy starting with the L1 cache which resides within the CPU core, then the L2 cache which is shared between CPU cores.

When there is an L1 cache miss, the data is looked up in the L2 cache, which costs around 50-60 cycles of CPU processing. Ideally, we want to avoid these L1 cache misses. The way to do this is by CPU pinning. So we segregate the slow steps from fast critical steps, and then we spin through a logic without any wait (busy spin/busy wait). If the thread is spinning within every 50 micros, the optimised jvm code remains within the L1 cache of the same CPU core as there is no thread scheduling. And the processing is ultra fast.
LMAX Disruptor uses this concept and can process up to 6 million^[1] orders per second. It uses something call Single-Writer pattern.

Single-Writer Pattern avoids Race Conditions by design

Go with singler-writer pattern^[2] by design for critical paths of the application, instead of using multiple threads mutating shared data, to avoid race conditions. Race conditions are nasty beasts, and if the software design doesn’t care about thread safety, there will be bugs right, left and centre throughout the year. It’s costly, hard to fix and often involves adding hacks to make it work, rendering the application a maintenance hell.

Build Event-Driven workflows

Event Driven Design is great for building low latency workflows as there are no blocking calls. However one of the major challenges is complexity. When you don’t use event-driven design, then your calls are synchronous in nature, and the calling thread waits for the response and goes into a blocked state. The machine’s underlying architecture around dealing with threads handles this. So when the response comes back, the waiting thread gets notified by the system’s interrupts and will pick up where it left off.

Whereas, when you are designing an event-driven system, your requests to slow parts/external systems are asynchronous in nature i.e. you don’t wait for the result within that thread. But instead you have to save the current request’s state in some in-memory cache and let the thread move on to handle the next event. The challenge is to reconcile the request with the response event received later. This is generally done using some common ID used in the request and response messages and also by building some sort of a state-machine in our application.

Prefer pub/sub over point to point calls

Prefer a message bus like Kafka instead of point to point calls like REST. Kafka has in-built support for event-driven workflows, whereas REST calls are blocking and if there are 100s of simultaneous client calls it could mean that many of those client threads leads to connection breaks with the server. Additionally, the REST client side code will have to decide on the max threads and this requires a lot of testing for peak time load and is still unreliable. 

With a message bus, the client threads can be freed up the moment the message is written to the bus, and the server can handle the events at its own pace.

Avoid Database or other Blocking I/O calls in latency sensitive paths

There shouldn’t be a Database(DB) call in the critical path of the workflow. It normally takes 50 millis to write to a standard DB like Postgres. The critical path should create a deep clone of the object and hand it to the DB thread without involving a thread scheduling (LMAX Disruptor). If the DB writer is trailing more than the defined threshold compared to the incoming messages then halt it. Design your system in a way that it doesn’t rely on DB during its run. All the static and config data should be read in-memory at startup.

Use EhCache

Keep all items in EhCache and only hit the DB for reads when there is a cache-miss. Pre-populate your ehCache objects during startup.

Warm up your application at start-time

During start up, 1000s of dummy messages can be replayed, so that the HotSpot JIT compiler would run and cache the optimized code. Hence improving the first message being played. This was proven by Peter Lawrey (Java low latency expert) in his Jitter application that the first 10000 or so messages were slower and so he recommended such replays to also include sessions/connections ^[3].

Iterative Performance Testing and Benchmarking

The best way to improve performance is by isolating the system under test and iteratively trying different changes and observing its impact. Capturing the stats after each run and finally picking the changes which is most performant. At times isolating the system by mocking external calls can take some effort, but this is worth doing. I was able to reduce latency from 5-6 millis per order processing to under 1 milli sec with this approach.

Collocation

And finally, apart from the code being perfect, network latency does matter. So you need to avoid the network cost and run your processes next to the Data Centre where the exchanges are.

References

https://martinfowler.com/articles/lmax.html

https://stackoverflow.com/questions/36198278/why-does-the-jvm-require-warmup

https://mechanical-sympathy.blogspot.com/2011/09/single-writer-principle.html