QUANTTRADING DATAMANAGEMENT BYTHENUMBERS
“Data is the oxygen of our information society.”
Ana Botín, Executive Chair, Banco Santander*
QUANT TRADING DATA IS DIFFERENT
Quantitative trading data is different. Market data is fast, voluminous and complicated, and the tech landscape continues to evolve, merge, and morph quickly. Regulation reigns.
“Real-time
data is the lifeblood of trading. Without it, traders operate in the dark.”
- Michael Bloomberg¹
Thanks to pioneering work in streaming analytics, time-series data management and, now, generative AI, data management on Wall Street has come a long way from the first digitized trading rooms of the 80s. The challenge, of course, is time. The time between a market micro-movement and the action that must be taken. The time to build new trading strategies that out-alpha your competitors. The time it takes to build systems that implement new ideas yet comply with ever-increasing compliance demands
The other challenge is change There are new types of data to unlock, new opportunities thanks to the rise of of generative AI and vector databases, and new demands to run applications as managed services in the cloud, but safely, securely, and with the same low-latency demands as on-premise, bare-metal performance.
From handling latency to massive volumes and throughput, to time-series comparison on the fly, trading data is just different. But, hey, this is Wall Street – let’s look at how trading data is different by the numbers
QUANTITATIVE TRADING DATA
Quantitative trading data must yield insights across a full range of decision-making latency – from near real-time to strategic, historical insights.
MILLISECONDS MATTER 10
Low-latency decision-making with data is essential for quantitative trading, as it is for any automated environment. On Wall Street, milliseconds matter, but so does the right strategic context to make a good decision, not just a fast decision.
Quant trading demands a data fabric that provides insight into high-frequency data and historical context at the same time
MATTERS
1
BILLION TICKS
PER DAY FOR 2,700 STOCKS IN A REGIONAL MARKET
¹
It’s not just fast data, it’s tick data, and tick data is different. Each change in market price, order, execution and position matters.
Streaming data lies at the heart of quantitative trading systems, so the first job of a data platform is to process fast data in a way that organizes it by time for analysis, query, and replay.
That means three things: high-frequency, low-latency, and time series structure for storage
HIGH-FREQUENCY
High-volume ingestion is the first step to effective tick data management. And it’s not just a billion ticks a day, but financial data bursts in important ways, so ingest rates of 100,000 ticks a second are common
Quant trading management uses in-memory processing, time-series data management, and streaming analytics to ingest tick data at these extreme rates.
LOW LATENCY
Depending on the data and use case, acceptable latencies can range from low single-digit milliseconds to seconds
Low-latency absorption of data is achieved by databases that use in-memory and micro-batching techniques.
A solid quant trading data platform handles streaming data from any financial system, from market data like ICE, Bloomberg, and NYSE to order management systems, execution management systems, and more
HIGH FREQUENCY LOW LATENCY
TIME SERIES
TIME SERIES
Finally, quantitative trading data should be organized in temporal order for aggregation, comparison, query, analysis, and replay For intraday trading decisions, every tick and state change must be stored, indexed, and queryable.
Time-series quant trading databases are structured in a way that stores data in temporal order so queries can be executed quickly for lowlatency decision-making
Quantitative trading data is different. Design systems for high-volume, low-latency, time series data management and you’ll be off to the quantitative trading races.
3 32
PROJECTS PER YEAR
Conventional data technologies aren’t designed for time-series, streaming, or vector data. This mismatch impedes progress and creates a drag on IT productivity, experimentation, and innovation.
In one study, a top-tier bank measured the code required to manage tick and trading data with a conventional store (SQL and NoSQL based) and a store designed for streaming, time-series, and vector data management.
They found the time needed (initial development and ongoing support) to require 5 versus 15 FTEs, and 45 days versus six months. The biggest areas of time and code inefficiency came from:
High-frequency data collection, 1 Aggregation, cleansing, normalizing data, 2. Performance tuning, scale-out 3. AI/ML model operationalization, recoding of Python 4. Cloud integration and deployment 5.
They found these tasks were all significantly faster with a fit-forpurpose data platform with time-series, streaming, and vector support.
The resulting increased cadence in delivery was due to re-use and fit with the data architecture designed for streaming data.
FORCING TRADING DATA INTO A CONVENTIONAL DATABASE IS LIKE POUNDING A SQUARE PEG INTO A ROUND HOLE.
Red rows arenon-value-add tasks like connecting to unstructured and streaming data, error correction, data quality, tuning
Greenrows arevalue-add business logic tasks.
Projectoutputincreased from3to 32projects per year. Sprint time reduced to 1 5 months from 6 months and headcount per project from 15 to 5 FTE
1
LINE OF CODE
As Steve Jobs said, the fastest, most efficient, easiest-to-debug line of code is the line you never write.
Domain specific programming languages are popular for quantitative trading because, for time series data, they’re designed to eliminate looping structures, which makes them up to 100 times more efficient.
MarkPalmer,withMidjourney
"The fastest, most efficient, most errorfree, easiest to maintain code is the code you never write."
LOADING, JOINING,
AND
ANALYZING DATA
WAS NOT ONLY FASTER AND REQUIRED LESS MEMORY, IT COULD ALSO BE DONE WITH FAR LESS CODE.*
¹ NickPsaris,QTips:Fast,ScalableandMaintainableKdb+
Domain specific languages fit the domain in which they’re used. In the case of KX, hundreds of lines of code are often replaced with a single operator.
- SteveJobs
INNOVATIVE TRADING APPS
&
In the capital markets, real-time data isn’t oil, its oxygen. We’ll explore seven innovative case studies and the secrets of their success.
QUANT RESEARCH & DIGITAL TWIN
A real-time view of the market for asif comparisons
ALGORITHMIC TRADING
Trade execution in real time with historical context
REAL-TIME TICKER PLANT
A real-time view of trillions of market data and order events a day
MACHINE LEARNING AS A SERVICE
Self-service access to high-frequency and historical data via Python
EXECUTION ANALYTICS
Analyze trade prices to determine favorable pricing and cost
CONTINUOUS SURVEILLANCE
Continuously anticipate anomalies and proactively maintain compliance
GENERATIVE AI QUANT RESEARCH
Explore edge cases and uncover novel strategies
To explore these, download Seven Innovative Trading Apps and Seven Best Practices You Can Steal eBook
1
MILLISECOND VS. 5 MINUTES
TIMES MORE DATA WITH
Conventional data technologies aren’t designed for streaming data, or will require careful and sophisticated programming to force-fit quantitative data inside. Choose carefully.
Time-series databases designed for streaming data are an essential tool for any quantitative trading desk. They’re designed for streaming data, organizing it by time, and supporting high-speed ingest
CONVENTIONAL DATABASES ARE OF LIMITED USE FOR QUANT TRADING
Data technologies like SQL, Snowflake, or NoSQL databases weren’t built to manage streaming time-series data. While it’s possible to store quant trading data in a store like this, it’s sub-optimal and may lead to systems that are unable to answer trading questions in a reasonable timeframe
This is because conventional databases are designed to manage rows and columns of data, graphs, or purely unstructured data.
CONVENTIONAL DATABASES ARE A BAD FIT FOR QUANT TRADING
Quantitative trading requires data structured in temporal order to provide not only the ingest rates required, but also query performance Because five minutes isn’t fast enough
