For time-sensitive Big Data research, renting processing capacity is often better than buying and maintaining farms of dedicated computers, and it helps organizations achieve tremendous leaps in productivity and ease of use.
Leaders of a High Frequency Trading (HFT) hedge fund retained Princeton Consultants to create a state-of-the-art environment for researching and simulating their alpha and execution ideas. In the typical cycle, proposed research ideas are discussed, approved, coded, and then back-tested against prior market and related data, using different combinations of parameters. In this never-ending quest, HFT funds face several classic challenges:
- The sheer quantity of data. For a single research idea, back-testing may involve reading hundreds of terabytes of data, and require tens of thousands of hours of processing to find and validate the best parameter set.
- Time. The rate of change in the markets continues to accelerate, and research turnaround times that used to be acceptable in weeks must now be done in days, even hours.
- Personnel. Unlike many areas, where vast groups of people can be hired and orchestrated, HFT research requires a uniquely skilled and multifaceted individual, whose time is extremely precious to the overall performance of the fund. Often these researchers are working massive numbers of hours and still find they can only investigate a fraction of their ideas effectively.
- Keeping up with the technology arms race. The drive for ever lower latencies and ever more nimble algorithms has created an almost insatiable need for computation and data storage. Towards this end, it is not unusual for a large financial institution to purchase and maintain expensive, dedicated supercomputers and farms of disk drives for its quantitative research.
Approach & Solution
Princeton Consultants elected to use as a backbone Amazon Elastic Compute Cloud (Amazon EC2) Spot Instances, the real-time market in which users bid for spare Amazon EC2 computing capacity.
In an innovative approach, Princeton Consultants refactored the HFT research away from massive, multi-hour jobs into hundreds of smaller pieces. After evaluating multiple existing multi-processing controller frameworks, Princeton Consultants created its own, OptiSpotter™, www.optispotter.com.
Based on custom optimization algorithms, OptiSpotter™ uses the best mix of servers that is optimal for the circumstance, to complete the current workload in the required time. An OptiSpotter™ user maps which pieces can run concurrently, and which need to be gated. OptiSpotter™ marshals these pieces into thousands of small sub-jobs, each with a specific calculation for a specific region of data.
OptiSpotter™ then places these jobs into SQS queues. The choice of queue depends on the memory and I/O requirements of the sub-job, as well as other factors. For instance, there might exist several possible instance types in different availability-zones that satisfy the sub-job requirements, all at different and fluctuating prices.
Using techniques similar to the ones used for the HFT models themselves, the system monitors the dynamics of the SQS queues, the recent spot price history in several Availability Zones and instance performance. Using custom optimization algorithms, OptiSpotter™ determines the most efficient collection of spot instances to promptly process the queue contents for the least cost. In rare cases, the optimal cluster may include some On Demand instances if current spot prices are highly volatile. Load can be automatically moved to other Regions if cost merits it.
Leveraging the unique advantages of AWS Spot Instances enabled the hedge fund to greatly speed up its research, improving turnaround time by 5x-10x. Research ideas that formerly would have required overnight runs are now available within an hour. This acceleration is a critical competitive advantage in today’s fast-moving market conditions.
By moving the load from On Demand instances to the price-optimized spot cluster, the fund achieved a 90% reduction in the cost of running the system. The savings meant the fund can research ten times as many ideas for the same cost—and provide better and more rigorous algorithms for its investors.
Finally, the relatively low administrative overhead of an Amazon EC2 cluster, combined with automatic monitoring of OptiSpotter™ of Spot Instances, saves valuable time for the hedge fund’s IT personnel. They can focus on the truly proprietary aspects of their trading infrastructure.
For this work, Princeton Consultants was a winner of the first AWS Spotathon, the global contest for applications that leverage AWS EC2 Spot Instances. Matt Wood, Chief Data Scientist of AWS, said: "Princeton Consultants' OptiSpotter is an exciting, innovative application taking advantage of AWS Spot Instances that dramatically improves the speed and cost-effectiveness of big data quantitative research. The judges of our first Spotathon coding challenge were impressed with their creative approach."
Visit www.optispotter.com for more information about our work in this area. We welcome a conversation about your cloud strategy, architecture and design challenges.